Clarifications to the wheel specification

[agronholm (Alex Grönholm)]

@takluyver

Permissions are normalised to either 0o755 or 0o644, i.e. only a single bit is preserved for executable-ness (executability?). This seems to match what popular version control systems store.

That still means that wheels generated from the same source code end up with different contents based on the operating system on which the packaging is done.

[takluyver (Thomas Kluyver)]

I guess it does, yes. In ~5 years, the question of reproducibility between Posix & Windows platforms hasn’t come up, as far as I can remember.

[agronholm (Alex Grönholm)]

Do you have a plausible use case that would require the execute bit to be set?

[dholth (Daniel Holth)]

Yes, and it will have different contents depending on the C compiler used, and the dependencies installed. The people who care about SOURCE_DATE_EPOCH only expect to be able to reproduce the artifacts bit-for-bit when everything else is kept the same.

A wheel could bundle a native executable to do some of its work. That file would need the +x bit.

[agronholm (Alex Grönholm)]

Maybe the bit should be preserved only on platform specific wheels then?

[takluyver (Thomas Kluyver)]

I preserved it mostly because version control systems (inc. git) do that, which seemed like a reasonable precedent to follow.

I can come up with scenarios like @dholth describes where having the execute bit set matters, though I doubt they’re common. It’s not necessarily limited to native binary executables - you could bundle e.g. a bash script which is launched as a child process.

It’s harder to come up with a concrete reason why not having it set might matters, so you may argue we should just make everything executable. But that would look a bit strange, because files are generally non-executable by default.

[dholth (Daniel Holth)]

It really just isn’t going to come up. Thing that no one will complain about: bug: I can’t package reproduceable Linux executables on Windows.

[agronholm (Alex Grönholm)]

@takluyver

It’s harder to come up with a concrete reason why not having it set might matters, so you may argue we should just make everything executable. But that would look a bit strange, because files are generally non-executable by default.

My only personal concern here is that it could be abused for packaging scripts where console_scripts should have been used. But if the consensus is to preserve the executable bit, I am fine with that.

[takluyver (Thomas Kluyver)]

I did actually package scripts that way in early versions of Flit, before console_scripts was in a written spec. I switched because scripts didn’t get exe wrappers on Windows, even though PEP 427 recommended that they should.

But if that’s the concern, maybe the wheel spec 2.0 should discourage using the scripts folder, or even outright forbid it. Though I’m sure someone will have a use case that requires it.

[ofek (Ofek Lev)]

@takluyver

Flit preserves the timestamps from the filesystem in wheels by default, except for generated files which have an arbitrary, fixed timestamp (1st January 2016). But if the SOURCE_DATE_EPOCH environment variable is set, all timestamps are generated from this.

FYI I also do basically this in the new (unreleased) version of Hatch except I support a reproducible option in pyproject.toml which is enabled by default.

[dholth (Daniel Holth)]

There’s an old Python saying “we are consenting adults”. Everyone is publishing and using wheels because they want to. It is not possible to coerce the publisher, who is so highly motivated to please you already that they gave you quality software for free, by omitting features from the format. Except for post install scripts! console_scripts are usually the best option, but publishers should use old fashioned scripts, or even bash scripts, when they are needed. If you remove that feature you don’t get more console_scripts, you only push the publishers that need other kind of scripts to other ecosystems.

[rgommers (Ralf Gommers)]

@pf_moore

The only use case presented so far seems to be Linux specific (it refers to the “Linux Shared Library Versioning Guidelines”).

First a side comment: versioning shared libraries is certainly not Linux specific. It is a necessary thing in some situations on all OSes. It’s usually done with symlinks. Windows is the exception there, versioning shared libraries is uncommon and can’t be done with symlinks - but it is still possible to version shared libraries on Windows, see for example this description for MKL.

Now the main reason I came to this thread - here is another use case besides shipping shared libraries: support for editable installs when the build system uses out-of-place builds.

Out-of-place builds are the only option in Meson, and also good practice for CMake. For out-of-place builds, you end up with compiled extension modules and generated files in the build directory, and .py files in the source directory. To put those together into a working editable install, the most straightforward solution is putting symlinks to all files in a wheel. xref Add support for editable wheels · Issue #47 · mesonbuild/meson-python · GitHub for more details.

[pf_moore (Paul Moore)]

@pf_moore

I don’t personally have an opinion on whether we should add symlink support.

I have revised my opinion on this matter since I posted that. My view is now:

1. There have been sufficient use cases presented that I think symlink support is worth having. A cross-platform use case that hasn’t been mentioned here so far is implementing editable installs via symlinks in the created wheel.
2. I remain -1 on using the “native” support for symlinks in the zip format. It seems not to be standardised, and support isn’t universal. Also, I expect that we would want better control of what happens when symlinks aren’t available than just leaving that to a library to deal with.
3. We would need some rules around what symlinks are allowed - for example, do we want to allow links to files not bundled in the wheel? That’s necessary for editable wheels, but do we want to somehow limit it to only that use case?
4. As I mentioned above, we need to decide what behaviour we want when an installer tries to install a wheel containing symlinks on a platform/filesystem/session without symlink support.

Because of all of these points, I think we should treat symlink support as needing a PEP, and hence a new version of the wheel spec. I will note that the spec doesn’t distinguish between “minor” and “major” specification version changes, so it technically doesn’t matter whether this is Wheel 1.1 or Wheel 2.0.

In particular, I do not think that symlink support qualifies as simply a “clarification” of the wheel spec.

Is anyone interested in writing a PEP / championing this change?

[agronholm (Alex Grönholm)]

If we are updating the spec, there are two things that I would like the PEP to take a stance on: timestamps and file attributes (whether they should be stored in wheels).

[pf_moore (Paul Moore)]

I’d be fine with including those in the same PEP. Or they could be a separate PEP, which would hopefully be small and relatively non-controversial. That’s going to be up to the PEP author(s), ultimately.

[pf_moore (Paul Moore)]

@pf_moore

I will note that the spec doesn’t distinguish between “minor” and “major” specification version changes, so it technically doesn’t matter whether this is Wheel 1.1 or Wheel 2.0.

I was wrong on this point, by the way. If a consumer sees a wheel with a version it doesn’t recognise, it must warn if it’s a newer minor version, and error if it’s a newer major version. But I don’t know whether symlink support needs to be a minor or major version change based on this requirement, unfortunately

[pradyunsg (Pradyun Gedam)]

Major.

Require packing wheels without files in the root directory

Skipping the terminology question… If we want to add functionality to the wheel format, we need a new major version bump. If we want to remove functionality from the wheel format, we need a minor version bump. So far, this is the only proposed change that I’ve seen to remove functionality from the format.

[pf_moore (Paul Moore)]

Again, I’ll let the PEP author decide in the first instance, and community consensus dictate the final answer.