Since there doesn’t currently seem to be any consensus around it, I thought I would start a topic here to try to address the question: What are the best practices for deterministically normalizing wheel ZIP metadata?
This is particularly in reference to enabling reproducible builds. One major blocker to that right now is wheel ZIP metadata.
I’m hoping that a discussion around this might generate some blanket recommendations/best practices, and then those recommendations could be applied to implement blanket solutions in centralized tools.
python-stripzip does less modifications than strip-nondeterminism. In particular, it doesn’t change order of entries, so if .dist-info files are placed at end of the ZIP – as is best practice – it will leave them so.
Is there any reference for this “best practice”? Does it actually have tangible, measured benefits, or is it just some piece of technical lore?
It’s mentioned in the wheel spec. It has genuine technical benefits - pip, for example, reads wheel metadata using HTTP range requests to only download enough of the file to find the metadata, and putting .dist-info at the end maximises the chance of getting it in the first read. Now that PyPI publishes metadata separately, this is less useful, but it’s not required of all indexes, so it still has the possibility of being beneficial (for example, with pytorch wheels, which are huge and hosted on a separate index).
I should probably reframe that initial section, since the way it’s written now can be misleading. My main takeaways about ordering of ZIP entries are really:
The ordering seems to already be deterministic by default – at least with the handful of experiments I’ve run.
It probably depends on which tool is used for generating the wheels. For example, delvewheel does not seem to make any particular effort to make the order of ZIP entries deterministic, except for ensuring that dist-info files come last.
@pitrou Agreed. And my third takeaway would also be exactly what you said:
It could be good to have a tool that can do deterministic normalization on wheel metadata, and which covers all relevant aspects, as a catch-all – even if some aspects will often be effectively a no-op. A single tool that can be run a single way, no matter the OS or the build backend.
They also point out a couple of still-unresolved limitations:
We don’t preserve file permissions. The exception is the executable bit, which means that builds aren’t reproducible between Unix and Windows if the repository contains a file with the executable bit
Similarly, platform-specific file system features such as symlinks and junctions are treatly differently by Git depending on the platform, and with a different repository checkout on disk, there are different artifacts.
These limitations could maybe be addressed in a similar way to the timestamp workaround I suggested for the sdist-lacks-VCS-metadata problem: __source-date-epoch.txt.
The basic idea is: When creating sdist, extract data from git/VCS metadata and store it as files in the sdist. Then the wheel generator, or post-processor, can read those files and use it to set ZIP metadata.
The __source-date-epoch.txt thing is relevant to wheels, because it is a (kludgy) way of carrying a meaningful timestamp from the VCS, through the sdist, into the wheel ZIP timestamps. (I think many people, understandably, balk at the idea of all ZIP timestamps always being hardcoded to e.g. 1980-01-01.)
As for the executable bit and other properties (symlinks, junctions etc.), I haven’t yet dug deep into the file formats or how Python tooling uses them. But at first glance, ZIP does store executable bits. So that’s ZIP metadata that would need to be normalized.
Reproducible builds generally assume that you’re using a controlled setup, right? Otherwise many things may differ. One platform may use a different compression library version (such as zlib-ng) and therefore wheel compression could produce different outputs. Expecting the same exact build artifacts from Unix and Windows sounds a bit irrealistic.
Without claiming to be an expert… I’d say it’s generally assumed there is some control of the environment, but not total control. For example, reproducible-builds.orgrecommends that a build not be sensitive to username/hostname.
Exactly where the line is drawn, for a particular project/framework, seems to vary. I think the key thing is that the build “inputs” are defined clearly, and that the output is fully determined by those inputs. And it’s conceivable to classify platform type (e.g. "unix" | "windows") as part of the build inputs.
Personally I don’t have any strong opinions on whether platform type should be in or out of scope.
I see it might be a nice property to have, for a pure-Python wheel to come out the same on both e.g. Debian and WSL.[1] But if it makes things too complex, it might not be worth it.
Regarding WSL… I’m not sure whether the uv dev’s original comment was implicitly referring to regular-Windows, WSL, or both. Apparently executable bits with WSL work like Linux sometimes, but like Windows other times: it depends on whether “the file resides on a Windows filesystem and not within the WSL internal filesystem” (source).
I hadn’t considered compression. I guess I have another category to add to my notes. This is still just Linux-only, but auditwheel apparently un-zips and re-zips wheels, so it will normalize that at least somewhat. No idea if auditwheel tries to control for its own underlying compression library.
Of course, I wouldn’t expect compiled binary artifacts to come out the same. That definitely seems unrealistic. ↩︎
