That still means that wheels generated from the same source code end up with different contents based on the operating system on which the packaging is done.
I guess it does, yes. In ~5 years, the question of reproducibility between Posix & Windows platforms hasn’t come up, as far as I can remember.
Do you have a plausible use case that would require the execute bit to be set?
Yes, and it will have different contents depending on the C compiler used, and the dependencies installed. The people who care about
SOURCE_DATE_EPOCH only expect to be able to reproduce the artifacts bit-for-bit when everything else is kept the same.
A wheel could bundle a native executable to do some of its work. That file would need the
Maybe the bit should be preserved only on platform specific wheels then?
I preserved it mostly because version control systems (inc. git) do that, which seemed like a reasonable precedent to follow.
I can come up with scenarios like @dholth describes where having the execute bit set matters, though I doubt they’re common. It’s not necessarily limited to native binary executables - you could bundle e.g. a bash script which is launched as a child process.
It’s harder to come up with a concrete reason why not having it set might matters, so you may argue we should just make everything executable. But that would look a bit strange, because files are generally non-executable by default.
It really just isn’t going to come up. Thing that no one will complain about:
bug: I can’t package reproduceable Linux executables on Windows.
My only personal concern here is that it could be abused for packaging scripts where
console_scripts should have been used. But if the consensus is to preserve the executable bit, I am fine with that.
I did actually package scripts that way in early versions of Flit, before
console_scripts was in a written spec. I switched because scripts didn’t get exe wrappers on Windows, even though PEP 427 recommended that they should.
But if that’s the concern, maybe the wheel spec 2.0 should discourage using the scripts folder, or even outright forbid it. Though I’m sure someone will have a use case that requires it.
FYI I also do basically this in the new (unreleased) version of Hatch except I support a
reproducible option in
pyproject.toml which is enabled by default.
There’s an old Python saying “we are consenting adults”. Everyone is publishing and using wheels because they want to. It is not possible to coerce the publisher, who is so highly motivated to please you already that they gave you quality software for free, by omitting features from the format. Except for post install scripts!
console_scripts are usually the best option, but publishers should use old fashioned scripts, or even bash scripts, when they are needed. If you remove that feature you don’t get more
console_scripts, you only push the publishers that need other kind of scripts to other ecosystems.
First a side comment: versioning shared libraries is certainly not Linux specific. It is a necessary thing in some situations on all OSes. It’s usually done with symlinks. Windows is the exception there, versioning shared libraries is uncommon and can’t be done with symlinks - but it is still possible to version shared libraries on Windows, see for example this description for MKL.
Now the main reason I came to this thread - here is another use case besides shipping shared libraries: support for editable installs when the build system uses out-of-place builds.
Out-of-place builds are the only option in Meson, and also good practice for CMake. For out-of-place builds, you end up with compiled extension modules and generated files in the build directory, and
.py files in the source directory. To put those together into a working editable install, the most straightforward solution is putting symlinks to all files in a wheel. xref Add support for editable wheels · Issue #47 · mesonbuild/meson-python · GitHub for more details.
I have revised my opinion on this matter since I posted that. My view is now:
- There have been sufficient use cases presented that I think symlink support is worth having. A cross-platform use case that hasn’t been mentioned here so far is implementing editable installs via symlinks in the created wheel.
- I remain -1 on using the “native” support for symlinks in the zip format. It seems not to be standardised, and support isn’t universal. Also, I expect that we would want better control of what happens when symlinks aren’t available than just leaving that to a library to deal with.
- We would need some rules around what symlinks are allowed - for example, do we want to allow links to files not bundled in the wheel? That’s necessary for editable wheels, but do we want to somehow limit it to only that use case?
- As I mentioned above, we need to decide what behaviour we want when an installer tries to install a wheel containing symlinks on a platform/filesystem/session without symlink support.
Because of all of these points, I think we should treat symlink support as needing a PEP, and hence a new version of the wheel spec. I will note that the spec doesn’t distinguish between “minor” and “major” specification version changes, so it technically doesn’t matter whether this is Wheel 1.1 or Wheel 2.0.
In particular, I do not think that symlink support qualifies as simply a “clarification” of the wheel spec.
Is anyone interested in writing a PEP / championing this change?
If we are updating the spec, there are two things that I would like the PEP to take a stance on: timestamps and file attributes (whether they should be stored in wheels).
I’d be fine with including those in the same PEP. Or they could be a separate PEP, which would hopefully be small and relatively non-controversial. That’s going to be up to the PEP author(s), ultimately.
I was wrong on this point, by the way. If a consumer sees a wheel with a version it doesn’t recognise, it must warn if it’s a newer minor version, and error if it’s a newer major version. But I don’t know whether symlink support needs to be a minor or major version change based on this requirement, unfortunately
Again, I’ll let the PEP author decide in the first instance, and community consensus dictate the final answer.