Over the past 18 months or so I’ve spent a bunch of time experimenting with cross-compiling Python wheels. One of the outputs of that work is repairwheel, a tool that wraps the “repair” steps from
delvewheel into a single package with no non-python dependencies that can be run cross-platform (at least that’s the goal).
Work items included:
- reimplement bits from
- reimplement bits from the macos tools
codesign (machotools.py, machosign.py)
- various monkeypatches to get
delocate to work in these strange new environments with new tools
delvewheel already works cross-platform, so
repairwheel just wraps it and invokes its repair operation.
repairwheel also normalizes the final output, rewriting the zip file with entries sorted in a stable order and given constant timestamps, with the goal of bitwise-identical outputs regardless of what the build host is.
You can see it used in this example that can cross-compile
psycopg2 and build a wheel with vendored
libpq (warning: this example uses Bazel, and doesn’t work on Windows).
Very nice, thanks for sharing. Any plans to add an importable API? Would be interesting to be able to include this as a final step for the wheel building process in tools like https://pypi.org/p/build.
Thank you for this! I will be interested to try it out at some point.
One awkwardness I had with
delvewheel is the need to strip binaries built with MinGW. I don’t understand enough the reasons for this but is that something that
repairwheel needs as well or is it something that can be “solved” somehow?
No plans, but that seems like a reasonable request and should be pretty straightforward. As a least common denominator tool, the interface is currently pretty minimal (although I could see it growing with more usage/use cases).
I don’t think this will be solved.
repairwheel doesn’t really do anything special with the
delvewheel invocation since it already works cross-platform. The only thing that
repairwheel adds is that wheel file normalization step, but that doesn’t do any stripping.
delvewheel 1.3.2, stripping binaries is not necessary anymore in most situations. Previously, they needed to be stripped because trailing data such as debug symbols in the DLLs made it difficult to name-mangle the DLLs. Since version 1.3.2, I revised the name-mangling technique so that it works with most unstripped DLLs. (https://github.com/adang1345/delvewheel/blob/master/CHANGELOG.md#132-21-february-2023)
I have my setup for python-flint working now so I don’t want to change anything until there is a need but I expect that someone attempting something similar in future would save a chunk of work because of this.
Also if repairwheel can work for all the major OS options and if it was used by cibuildwheel by default (without needing manual Windows configuration) then that would also save a chunk of work.
Nice work @jvolkman! I’ll be following this for my company’s wheel distributions.
I’m biased since it’s my tool, but have you considered integrating
abi3audit or implementing similar checks? It can’t provide any additional fixes on top of the other tools, but it can detect incorrect and/or potentially dangerous wheel builds that the others can’t (namely, when a wheel is tagged as abi3-compatible but actually isn’t).
I haven’t considered it, but it sounds like a great idea. Does
abi3audit work cross-platform today?
I believe so –
abi3audit itself is written in pure Python and I’ve tested it on macOS and Linux, but it may have non-pure dependencies (since I haven’t fully checked).