Delocate/auditwheel, but for Windows?`

Has someone already written a tool that does the same job auditwheel does on Linux, and delocate does on MacOS, but for Windows? That is, for each compiled-code extension in a wheel, find all of the DLLs on the system that the extension needs in order to work correctly, filter out the ones that are guaranteed to be provided by Windows itself, and then copy the rest of them into the wheel? I think on Windows there isn’t any tricky object code rewriting to be done, so it shouldn’t be too complicated and basically I’m hoping someone else has already coded it up.

You do need tricky object code rewriting (the windows loader has the exact same funny glitch that the glibc loader does, that necessitates rewriting library names to avoid isolation between wheels), but that code exists. Just no one has packaged it up into a convenient tool. I think adding support to auditwheel itself would be nice, to avoid making the packaging zoo even more confusing.

See:

Is this the kind of thing the PSF and/or some external party could fund? I know debugging DLL issues in Windows wheels is kind of a chore for us, and any help through better tooling would be more than welcome :wink:

cc @brettcannon

Are you debugging using an actual debugger on Windows? Because WinDBG, Visual Studio or even Process Monitor are very good tools for this, and at a simpler level, dumpbin (to list load time dependencies) is helpful too.

By debugging I meant finding out which DLL fails loading when we get an ImportError. The only way I’ve found to do that is to use Process Monitor and search manually through trace entries to find the offending one. It is extremely tedious and, more annoying still, we’d need a CLI tool that works in CI (because situations like “the wheel fails loading on this particular AppVeyor setup but we can’t reproduce locally” have occurred several times already).

(as a last resort, we could use AppVeyor remote desktop and run Process Monitor over a high-latency connection. Let’s just say it’s not the most convenient solution)

If you’ve got a magic dumpbin incantation that finds out DLL dependencies recursively (and which ones are missing) then I’d love to know about it :slight_smile:

2 Likes

I assumed I got looped in for this sentence. For package-related funding the package WG has typically handled that and it requires a proposal to the PSF for a grant.

I’m pretty sure the CLI version would be based on xperf, but I don’t know the incantation to collect the events you want. Running under a native debugger should break at the load, but that’s not trivial to analyze either.

Dumpbin doesn’t attempt to resolve anything, so you really need to debug a running process to take search paths etc into account.

You might want to try:
https://lucasg.github.io/2018/04/29/Dependencies-command-line/

(rewrite of the invaluable Dependency Walker. I have used the Dependencies GUI version a few years ago, but have not tried the CLI)

2 Likes

I’m trying to wrap a poppler utility using pybind11, and getting up to speed on packaging C extensions (that have shared library dependencies) in the process. It was relatively easy to build Linux/macOS wheels compared to Windows ones, but this week I was able to build Windows wheels thanks to all the replies in this post (and all the linked posts)! I’m using cibuildwheel to build the wheels using GitHub Actions, vcpkg to install the external shared libraries, and a wheel_repair.py to copy the shared library DLLs into the wheel. wheel_repair.py follows a scoped down version of the workflow that @njs described in this post:

  • Look for a PYD file in the wheel
  • Get the DLL dependencies (both direct and transitive) for that PYD file using pefile, and look for them in the vcpkg directory
  • Mangle the DLL names using their sha256 hash (using the same function as auditwheel) and copy them into the wheel directory (alongside the PYD file)
  • Modify the PYD import table with the new mangled DLL names using machomachomangler by Nathaniel (and also do this for each DLL which depends on other mangled DLLs)

There are some limitations to this approach, (1) it assumes that there’s just one PYD file in the wheel which might not be the case for complex projects but works for my simple extension, and (2) it doesn’t look for DLLs in system directories (as I wasn’t sure what all directories to look into).

The built Windows wheel worked on my machine but I need to test it on a fresh Windows instance to see if I need to bundle any other DLLs.

I wanted to ask you all if this is an ok approach to bundle DLLs within a Windows wheel, if the wheels would work on Windows versions older than 10, and if there’s anything that I might’ve missed or could improve. @steve.dower I would love to have your advice on this! And if you have some more time, could you please look at the DLL imports (Dependencies tool output as a GitHub gist) for my compiled PYD file and tell me if I need to bundle any other system DLLs apart from freetype, libpng16, jpeg62 (and all of their DLL dependencies that I install using vcpkg)?

I’ve also documented my experience building Windows wheels for others looking to do the same. Here are the blog posts in reverse chronological order:

1 Like

I faced a similar issue so I used your script wheel_repair.py and extended it to add new functionalities:

  • It supports hierarchical folder architecture with multiple .pyd.
  • Similarly to auditwheel, the dll are bundled in package_name.libs directory, alongside the package but not inside. This way, common dependencies to multiple .pyd are shared, not duplicated.

Here is the source.

I have wanted a fully-functioning tool like delocate/auditwheel for Windows for a while, so I have developed one myself. It scans the wheel for DLL dependencies, figures out which dependencies are not provided by the Windows system, copies those DLLs into the wheel, mangles the DLL names, and patches __init__.py so that these DLLs are loaded at runtime.

Source: https://github.com/adang1345/delvewheel
Release: https://pypi.org/project/delvewheel/

6 Likes