Need auditwheel-like utility for other platforms

Title says it all. On Windows for example, your wheel can succeed loading locally even though some dependent DLLs are not bundled, because those wheels exist on your local system.

Typical example is zlib.dll, which can be found on many Windows setups but not all of them.

1 Like

+1 to the overall idea.

Maybe, it might make sense to add this to auditwheel itself – splitting the logic internally based on which platform it sees.

+1 in principle. In practice, I suspect the auditwheel team as it currently stands has enough on their plate just handling Linux, so I expect this will need someone with both the time and the Windows knowledge to step up and offer to help.

I agree that auditwheel would be a good place for this. There’s a lot of logic for scanning and repacking wheel files that could be shared across platforms. And it’s already very challenging for package maintainers to understand all the different platforms well enough to build wheels on all of them. Being able to use the same tool on different platforms will make this a little bit easier.

I don’t have the time to work on this myself, but I did implement the core primitive you need a while ago, and I would be delighted to see it integrated into auditwheel: https://github.com/njsmith/machomachomangler

I think the pieces you need are:

  • Scan the wheel to find PE format files (DLLs and EXEs)
  • For each PE file, extract the list of DLLs that it links to – IIRC machomachomangler can do this, or probably lots of PE libraries can
  • Compare those to some list of which libraries are built into Windows and don’t need vendoring. This will probably need refinement over time, but you can probably get close by looking at a fresh Windows install. Or maybe Steve Dower can help.
  • For libraries which do need vendoring, move them into a directory in the wheel, and mangle their names, like how we do on Linux. AFAIK machomachomangler is the only existing tool for this part.
  • Possibly the most annoying part: windows doesn’t have an equivalent of RPATH. So we need to rewrite the wheel’s top-level __init__.py and inject some code to add our vendored libs dir to the DLL search path. There are a lot of ways to approach this, they’re all kinda gross, but I think we can hold our noses and make it work in practice.

I’m on my phone so can’t easily look up their usernames, but someone should probably CC Matthew Brett and the auditwheel maintainers into this thread.

1 Like

cc @steve.dower @takluyver

Hopefully os.add_dll_directory() is the least gross of these :slight_smile: I’d be just as happy to see it validate and pass/fail rather than trying to fix anything automatically, though.

As with everyone else, +1 in principle, but not committing my own time to it. I could probably help making a list of the intersection of pre-installed DLLs across clean, supported Windows versions though.

IIRC, delocate does something similar for Macs.