Specifying supported Windows in Wheels

Packaging
#1

I have two Windows-wheel-related questions:

  • Are there any hints / guides / steps / examples for building a compiled wheel on GitHub actions that is compatible with a given Windows version? I’m specifically struggling with supporting Windows older than 8.1 since that’s what my toolchain happens to target, but I don’t understand how the base target version is specified at all.
  • Is there a manylinux-style mechanism for indicating that a wheel requires a base Windows version, and if not, is this on purpose or just “we haven’t gotten there yet”? I have a feature request for AF_UNIX support which I understand is new in Windows 10, and I don’t know how to go about building appropriate wheels.

Background:

I just published my first release (v21) of pyzmq from GitHub actions using cibuildwheel after migrating away from my own manual setup on appveyor, and immediately faced some compatibility issues. My new build toolchain with VS2017 is targeting Windows 8.1 as the minimum supported version (import on Windows 7 fails with “ImportError: DLL load failed: The specified module could not be found.” due to missing symbols revealed by Dependency Tracker, e.g. API-MS_WIN_CORE_COM_L1-1-0.DLL).

I’m not a Windows build expert, but I cannot find a way to build wheels that work on Windows 7, a la MACOSX_DEPLOYMENT_TARGET. It seems Python’s own builds with apparently the same toolchain do work, so there might be someone here who knows. VS2017 doesn’t appear to be compatible with the 7.1 SDK, so I don’t understand what to do to make things compatible. Do I just need to bundle more of the vc_redist files?

If it is not feasibly fixable (or in general if I had a project with newer Windows required on purpose, such as for AF_UNIX support), how can I communicate that my latest wheel requires Windows 8.1 (or whatever version) and should not be considered for installation on earlier versions?

#2

Unfortunately, we don’t capture that level of detail in wheel tags.

For the most part, Windows has excellent backwards/forwards compatibility, so unless you’re coding directly against the latest Windows APIs, you’ll rarely hit issues.

In general, you want to use the latest compilers and latest Windows SDK, and then define preprocessor variables to target a specific version by hiding newer APIs. Python’s header files specify these as the earliest version supported by that release, so if you include Python.h first, you’ll get the right setting.

Those API-* DLLs are not always reliably handled by Dependency Walker (unless it’s been given an upgrade), as they aren’t all real DLLs. And anything in those DLLs should be new - anything available prior to them being added should be being linked directly. So if your pyd is directly referencing them, it’s because you’ve used APIs that don’t exist (which means your preprocessor variables mentioned above are higher than necessary).

And notice that the missing DLLs in the Dependency Walker screenshots in your thread are referenced from DLLs that do exist, which means they’ll be handled fine and it’s just Dependency Walker missing them.

Beyond the things that CPython itself ensures are installed, such as the UCRT, everything else is basically like a native dependency via apt on Linux - you’ll just have to document that it’s needed and leave it to the system integrators doing the installs to make sure it’s there.

For something like socket options, you’ll be best off offering runtime detection, ultimately. I believe creating a socket with AF_UNIX will fail immediately on older OS, which you can handle as early or as late as you like. That shouldn’t need separate wheels, and things will be much simpler here with runtime conditional code compared to compile/build-time conditional code.

#3

Thank you so much! After a bit of reading, it seems to me that adding this bit makes sense as default behavior (I’m trying it out in my PR):

// default to Python's own target Windows version(s)
// override by setting WINVER, _WIN32_WINNT, (maybe also NTDDI_VERSION?) macros
#ifdef Py_WINVER
  #ifndef WINVER
    #define WINVER Py_WINVER
  #endif
  #ifndef _WIN32_WINNT
    #define _WIN32_WINNT Py_WINVER
  #endif
#endif

I see that pyconfig.h sets these only when building Python itself. The comment (from 2007) in that snippet:

we don’t want to force these values on extensions

suggests that this snippet would be forcing it on Extensions, but it would only be setting the default behavior to match Python instead of the compiler itself. Extensions could still override them all by defining WINVER, _WIN32_WINNT prior to including pyconfig.h due to the ifndefs. Does it make sense to set these as defaults for Extensions as well? I believe that would be more consistent with how the equivalent MACOSX_DEPLOYMENT_TARGET is handled on macOS: default to match Python itself, easily overridden with platform-standard methods. This lets the default be the most compatible option for developers who perhaps don’t know much about Windows compatibility (i.e. me) while allowing standard methods for setting a more recent target for the smaller, presumably more expert, subset who need it.

Thanks for your tips on AF_UNIX, too, I’ll give it a try when I have the chance. I’m still a little unsure about detecting what I’ve pulled in as a dependency and/or required minimum version implicitly, since no change in my code, only the compilers I used, changed not only the target Windows version but also the CRT DLL dependencies.

I really appreciate all your help with Python on Windows! You are an incredible benefit to the community :heart:.

#4

I haven’t checked out your actual build, but I’d honestly be surprised if it changed the target Windows version in any way that isn’t handled properly at runtime (as I said, Dependency Walker hasn’t had an update in a while and isn’t reliable anymore). The slightly different C Runtime dependency is possible, but seems unlikely unless you’re using C++ exceptions.

As for the proposed snippet, yeah, that should work. I think if you drop the first ifdef and put it before the Python.h include it will work as well, because of the order that the preprocessor will do substitutions. But yeah, I think it would probably be okay to set those variables when undefined (though still better for the code using the API to manage its own requirements, or it may get different compiler errors - those variables should only affect what is available at compile time, not what gets linked to during link time).

#5

To be clear, I am only looking into this because changing the compilers meant that my wheels stopped working on many versions of Windows and I started getting bug reports. Changing the compilers from VS2019 to VS2017 fixed import on Server 2012R2, the oldest version of Windows I have access to, but I’m still hearing that it doesn’t work on Windows 7. Compiling with VS2019 also brings in a dependency on vcruntime140_1.dll, which I haven’t bundled yet, but at least I understand how to fix, if not why it changes. Could this be triggered by requesting features loaded by checking MSC_VER?

I’ve tried setting WINVER in a few ways, the latest being with define_macros in compiler configuration:

define_macros.extend([
    ('WINVER', '0x0601'),
    ('_WIN32_WINNT', '0x0601'),
])

however I’m not getting a discernibly different result from them being unset, which makes me think something else might be wrong, or that I’m not setting them correctly. It is in a C++ project

My iterations are pretty slow, though, because I haven’t found a way to programmatically check on the build system whether the issue is resolved. I have to do a full CI build, then manually download and test the artifact on an old VM each time.

#6

They only affect whether certain function definitions are available, not whether you use them. And the linker is only going to link the ones you use, so if your source code is compatible with Win7 already, this shouldn’t matter.

You could try generating the output of “dumpbin /imports my.pyd” during build and comparing it. That will show what it’s trying to load at startup, though not recursively. Dumpbin lives next to the compiler cl.exe, which may not be on PATH by default because of platform options. Transitive dependencies only matter for things you’re distributing yourself though - don’t rely on the output for CRT or system DLLs.