I’ve run into an issue where pyzmq wheels for Windows Python which bundle (C++ library) libzmq apparently work for users with CPython 3.12.3, but crash with CPython 3.12.3 from the Windows Store. This is the same user on the same system, with the same pyzmq wheel. Other users have confirmed, and nobody has been able to confirm encountering the issue without Python from the Windows Store.
My question is really: what could be different about a Windows Store Python from a Python.org installer that would cause crashes in the same C++ library loaded by an Extension? And how should I as a packager test for these differences?
I haven’t been able to find any way to run a Windows VM with Windows Store, so I’ve never seen this error myself. All the cloud things I have access to only have Windows Server, and when I install a local VM, the Store somehow isn’t part of it. I’m on an ARM Mac, so I can’t use the Microsoft official VMs, which only run on x86_64.
The most common difference is that Store apps are automatically opted into safe DLL handling (basically, ignore PATH), while non-Store apps have this configured by the system admin (or parent process).
So if you’re modifying PATH to locate DLLs instead of putting it in the same directory as the extension module loading it (or using os.add_dll_directory, or using a full path with ctypes), it may be finding a different copy of the DLL (e.g. from the System32 directory) that has the same name but is incompatible.
Failing that, the next most likely thing is that the app is writing something to its own AppData directory, then launching a non-Python process that expects to find those (private) files. Store apps keep AppData local to the app, so that it can be safely backed up, restored, and reset, but this does mean that apps outside of Python won’t necessarily see them in the shared AppData directory. This wouldn’t normally crash, unless it happens to find a different file with incorrect info (though normally Python would also find and modify this file rather than using its private area, so we’re looking at precise timing here too).
It’s also possible that this crash is related to a recent change in the C++ runtime that means you really do need to bundle or statically link a copy of the version it was built with. matplotlib did this recently, and unbundled their copy of msvcp140.dll, so maybe if their newer version is being installed (though I’m not 100% sure it’s shipped yet…) then perhaps that’s the cause? Or if theirs is still shipping an old msvcp140.dll and yours is built with a newer compiler but not bundling it (or not loading it first), then it may result in the same issue. This shouldn’t only affect the Store though, it should affect all Windows users (except Conda users, who will get the right bundled files).
I believe it’s possible to install an MSIX directly on Windows Server. You can grab the msix build artifact for 3.12.5 and try double-clicking the python-3.12.5-amd64.msix (or possibly using Add-AppxPackage <path> in Powershell). The Store interface is (was?) some kind of add-on.
Otherwise, yeah, you’re going to be hard pressed to find anyone hosting a client OS. There is an ARM64 version of the OS now that ought to be fine for this kind of testing, but I’m not sure it’s widely available yet.
Thanks so much! That’s all super useful information. I just got a report that pyzmq is bundling a not-the-latest msvcp DLL via delvewheel, so I’m going to try following matplotlib’s example and statically link it and see if that solves it. I’ll feel a lot better if I can ever manage to confirm these fixes myself.
I finally managed to reproduce this, but unfortunately only in an x86_64 VM on ARM via emulation where e.g. launching Powershell takes about a minute or two, so ironing out cases is a slow process.
The bug (EBADFD in epoll_ctl from wepoll) only occurs when all of these are true:
User with non-ascii username
Python 3.12.5 from MS Store
The same wheel does not produce the issue for the same user with:
The same wheel with the same Store Python also does not produce the error with:
User with ascii username
(I’m 100% sure this is a bug in the library, I’m just trying to narrow down the situation where it happens to I can try to fix it. I suspect it has something to do with trying to bind an AF_UNIX socket to a path created by _wtmpnam_s.)
Installing the msix directly is a great suggestion! I was able to download them, but unfortunately not install. It’s unclear if this is something about my VM (e.g. why it lacks the Store in the first place), but it fails with:
error 0x800701C8: While preparing to process the request, the system failed to register the
windows.capability extension due to the following error: The custom capability’s SCCD has an invalid catalog.
Are you installing by double-clicking or using the Powershell Add-AppxPackage command? For a while, there was a bug where only the Powershell command would work for packages with special capabilities (which in this case I believe is the ability to touch arbitrary files). It’s fixed in Win11, but if you’re on Windows Server then I don’t think there’s a fix for it yet, and you need to use the Powershell command.
It’s Windows 11, both double clicking and powershell report the same error. But don’t spend too much energy thinking about it, this is a weird arm vm that I don’t trust to be representative of a typical user system. Thanks for your help! I’ll poke around.
Fascinating. I’ve tracked this down to failing to connect to an AF_UNIX socket in the default $env:TMP directory in AppData after successfully binding a socket there.
Binding an ipc socket works fine
Connect fails with EINVAL
Binding and connecting with the ipc path in other directories also works fine, as long as they are not in ~/AppData.
Setting $env:TMP="$env:USERPROFILE\tmp" also fixes things
It’s still utterly bizarre to me that this requires Windows Store builds + non-ascii username. Both non-ascii user + Python.org and ascii user + Windows Store work without issue. If it affected sockets in AppData for all users on Store builds, the Windows Store difference would make a lot more sense to me.
I was also finally able to get the MS Store working on my ARM VM thanks to this hack, and the behavior is the same.