How to reinvent the wheel

Packaging Standards
6

The following is from the perspective of a pip maintainer, rather than as PEP delegate.

I think the first thing we need to look at is how things work today. That imposes some rather strong (and not ideal) constraints on any solution.

  • If pip (and uv) chooses to install a particular version of a package, and the selected wheel is greater than version 1.0, the install fails. The installer doesn’t backtrack, or try other versions or other wheels.
    • One consequence of this is that if a project releases a version 2.0 wheel, the project will not be installable without a maxiumum version constraint (which is generally considered bad practice).
    • Furthermore, any package that depends on the one with a 2.0 wheel will also fail to install. And changing the dependency to add a version cap isn’t possible, without releasing a new (wheel 1.0!) version of the package with updated dependency data.
    • In practical terms, the only solution is likely to be for every user to maintain a constraints file which excludes any package versions which use wheel 2.0, and apply it to every install.
  • Once a build backend starts producing version 2.0 wheels, installers won’t be able to install packages that use that backend from source, unless the backend offers a “use wheel version 1.0” flag and the user manually includes it.

This is the position now, but of course a transition plan could include a process for installers to offer a more graceful transition. For such a plan, we need to decide how long the transition process should last, before it’s acceptable to assume that enough installer users have upgraded to at least a transitional version. Until that point arrives, build backends cannot produce 2.0 wheels and indexes (especially PyPI) cannot host version 2.0 wheels.

Ideally, we’d say that users are told to keep on the latest version of pip, so the transition could be relatively short (and I’d assume uv users are generally aware that they are using a rapidly developing tool, so they would upgrade frequently as well). But the feedback we got on the symlinks-in-wheels thread is that this isn’t the reality. So someone will need to make an assessment of what we should really use as a transition timescale.

The alternative (or maybe something we could do as well) is mandate that installers ignore wheels that they can’t handle[1]. This would reduce the issue, although it would still leave problems with packages that only ship sdists (or ship sdists alongside 2.0 wheels). That could be added to the spec and then, once installers start doing that, the transition becomes a lot simpler. However, in order to do that, it’s important (so that we have acceptable performance) that installers can detect the wheel version without downloading the whole wheel. That means something along the lines of:

  • Expose the wheel version via the index API
  • Extend the wheel API to expose the WHEEL file in the same way that the METADATA file is currently exposed.

Alternatively we could change the filename format to explicitly include the wheel version. This allows projects to publish both version 1 and version 2 wheels simultaneously. But that’s an even bigger change, and it’s not clear to me that we need to go that far at this point.

Again, we’d still need to wait until we judge that (nearly) all users are using an installer that conforms to the new spec before taking the next step, but at least we would then have a framework for making future version changes less painful.

The alternative is to simply publish the wheel 2.0 spec, let tools and indexes implement support for it, and manage the damage by getting users to upgrade as needed. But this feels to me like the sort of “big bang, break the world” approach that the Python 2-to-3 transition took, and I doubt users will have the appetite for anything like this (I know that I don’t, personally :slightly_frowning_face:)

By the way, as a side note, this is all based on the need for a major version update to the wheel format. The rules for adding a new minor version would make transition a lot simpler, but it’s not at all clear to me what exactly would be appropriate in a minor version bump. All the current spec has to say on the matter is

A wheel installer should warn if Wheel-Version is greater than the version it supports, and must fail if Wheel-Version has a greater major version than the version it supports.

Does anyone have a feel for what a minor version change would look like? If we could formally establish what can go in a minor change, it might help us structure future proposals to work as a series of minor changes, rather than always needing a major change.

  1. technically this is a change to the spec, as the current version explicitly states to abort the install ↩︎

4 Likes