Fix hang in wxMSW wxFileDialog when using multi-thread COM

Fall back to old style file dialogs in this case, as IFileDialog just
doesn't seem to support this COM threading model, as hinted at in
SHBrowseForFolder() documentation.

Also document this limitation.

See #23578.

(cherry picked from commit 6c4bee846cf36b74046462705f8415cc1a846767)

docs/changes.txt

@@ -273,6 +273,7 @@ wxMSW:
 - Fix building with LLVM clang (Sergey Khalyutn, Maarten Bent, #23464).
 - Fix discrepancy between GDI and Direct2D in high DPI (Kumazuma, #23486).
 - Fix wxTreeCtrl::ScrollTo() with hidden root item (#23534).
+- Fix hang in wxFileDialog if COINIT_MULTITHREADED was used (#23578).
 
 wxOSX:
 

interface/wx/filedlg.h

@@ -146,6 +146,12 @@ const char wxFileSelectorDefaultWildcardStr[];
     @ref page_samples_dialogs, please check it for more details.
 
 
+    @note New style file dialogs can only be used in wxMSW when the apartment,
+        COM threading model is used. This is the case by default, but if the
+        application initializes COM on its own using multi-threaded model, old
+        style dialogs are used, at least when they must have a parent, as the
+        new style dialog doesn't support this threading model.
+
     @beginStyleTable
     @style{wxFD_DEFAULT_STYLE}
            Equivalent to @c wxFD_OPEN.

src/msw/filedlg.cpp

@@ -1227,12 +1227,39 @@ int wxFileDialog::ShowModal()
     wxWindowDisabler disableOthers(this, parent);
 
     /*
-        We need to use the old style dialog in order to use a hook function
-        which allows us to use custom controls in it but, if possible, we
-        prefer to use the new style one instead.
+        We prefer to use the new style dialog if possible, but have to fall
+        back on the old common dialog in a few cases.
     */
 #if wxUSE_IFILEOPENDIALOG
-    if ( !HasExtraControlCreator() )
+    bool canUseIFileDialog = true;
+
+    /*
+        We need to use the old style dialog in order to use a hook function
+        which allows us to use custom controls in it.
+     */
+    if ( HasExtraControlCreator() )
+        canUseIFileDialog = false;
+
+    /*
+        We also can't use it if we're in a multi-threaded COM apartment _and_
+        have a parent, as IFileDialog::Show() simply hangs in this case, see
+        #23578.
+     */
+    if ( hWndParent )
+    {
+        // Call this function just to check in which apartment we are: it will
+        // return S_OK if COINIT_APARTMENTTHREADED had been used for the first
+        // COM initialization and an error if not.
+        const HRESULT hr = ::CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
+
+        if ( hr == RPC_E_CHANGED_MODE )
+            canUseIFileDialog = false;
+
+        // This just undoes the call above, COM remains initialized.
+        ::CoUninitialize();
+    }
+
+    if ( canUseIFileDialog )
     {
         const int rc = ShowIFileDialog(hWndParent);
         if ( rc != wxID_NONE )