I think we agree that installers should say something about upgrading, or link to a webpage with more information, if they encounter an incompatible wheel.
@Liz I am curious about your thoughts on the example I gave. I guess the more general question is “what can we do about wheels with versions incompatible due to the environment resolving before compatible ones”. I expect this to be quite common when using extra indices, which often offer more specific wheels.
I agree that scenario isn’t great. But I also think the scenario where numpy makes a release using 2.0 wheels and tens (hundreds?) of thousands of users have to go out of their way to fix CI pipelines is not great.
So really it becomes a question of which is less bad, or is there an alternative that is better. There’s the option of delaying the uploading of wheels to PyPI, but that greatly slows adoption of new wheels.
Thank you for doing this analysis! I think this definitely paints a more encouraging picture. I am a bit concerned that as far as I know, PyPI still “just works” for older Pythons. I don’t know if this is intentional policy or not, but really the only breakage I can think of is the TLS 1.0/1.1 removal 7 years ago now.
I think if we go down this path we still probably would need to delay uploads 2-3 years, which is longer than I’d prefer. It’s still definitely better than 4-5 years!
Just to be I’m understanding you right: you’re saying that the big compressed wheel is an alternative to the small pure-Python one, with the same package name? At first I thought you meant the big compressed one was a separate package like mypackage-accel or whatever. . . which would be one way to resolve this, right? Also, are you saying that the advanced compression is a CPython build option that not all builds of that Python version would have? Because otherwise just having the right requires-python would work, right? If advanced compression was added in Python 3.123 and the wheel requires it, it should require that as the minimum version, and then you won’t get that wheel if you’re only on 3.122.
Apart from that, one simple answer (maybe ): could the user download the pure-Python wheel manually and install it explicitly? Obviously that’s not a great user experience but it’s still meaningfully different from “installing this package is flat-out impossible”.
But here’s my bigger question: How would changes to the wheel format resolve this issue without changing the wheel extension? As in, imagine that this PEP had been proposed some years ago and the change was already made so now everything was using .wheel, but we never actually got around to making any changes to the format itself. Now we want to make changes to the “new” wheel format, in order to fix the problem you described. How would that be done without again changing the wheel extension?
What still seems to me like the “right” answer for your problem: it could be resolved by having metadata external to the wheel file that resolvers can use. That metadata can have whatever extra info is needed for resolvers to recognize that the big wheel is incompatible with the environment, and they can resolve in accordance with that constraint. The installer should never even know which actualy wheel file is to be downloaded until after the entire resolution process is done with pure metadata.
I’m going to include sdists for one part of this, but it won’t further complicate my already complicated answer.
Ideal version
I think the ideal here requires multiple moving parts, and this is a really good example to figure out if these moving parts can fit into the current extension or if we need to use the one time break to set this up.
require a reference implementation of both packing and spreading a new major wheel version for each new major wheel version.
prior to the release of wheel 2 in preparation, we improve index metadata. This would be optional, but when available, revolvers and installers should prefer to use it.
on a per-version basis, allow projects to specify if falling back to an older wheel format is preferable this only applies within the same release version. The default is no.
on a per-version basis allow projects, specify if falling back to an sdist is allowed for that version. The options are yes, no, and only without wheel match. The default would be only without wheel match.
On the user side, tools should be configurable to override any of those and it is strongly preferable that tools support a persistent configuration of those overrides.
When overriding those impacts the chosen and installed wheel or sdist, it must be logged that an override was part of the calculation as part of install and what change it caused, helping with the visibility for those both normal use and debugging.
On the installer side, if the wheel selected that is supported for the environment markers is not supported by the installer, there is an older format that is compatible with both for the version and the behavior is not to fall back to an older format, the error message should inform the user that an older wheel format matching their environment for the same version was available. In an interactive install setting, it is allowed to directly prompt to use that one, but support for interactive use is not required.
This might also have a reason to intersect with the priorities on configurable index priorities.
In none of these cases do I think falling back to an older version be implicit behavior. reproducibility settings like passing a date or temporarily installing an older version directly should likely be the used in CI cases, with a job that is allowed to fail without blocking CI uncapping the date giving a signal for when things will break, but still running CI for believed good release ranges without hard capping them by version on an individual basis in CI.
I am aware that that requires improving index metadata, specifying additional resolver behavior, specifying additional knobs for project maintainers and users. I don’t think this will be easy, but I don’t think any option that gets us from where we are to having all the bells and whistles will be.
Acceptable version
require a reference implementation of both packing and spreading a new major wheel version for each new major wheel version.
Require all uploads be the same wheel version for a given release even if not all features are used
Prevent uploads of a wheel version that has new mandatory features (major version) for 2 months after the next python version to be released after that wheel version is accepted. This gives time for maintainers preparing for the next python version to not simultaneously be working on the next wheel version, and builds in a more likely natural situation for pip at minimum to support the wheel version before. This should also help with the CI pipeline concern, as there will be a predictable date that projects should ensure their CI tooling needs to be capable of handling a wheel version by.
There’s probably other answers to this that strike good balances of concerns too, the ideal gives more knobs, but it’s more work. The acceptable gives predictable timelines, but restricts options for those uploading, and leaves multiple-index situations not fully addressed.
File extensions can be renamed so ideally the data is in the file itself
Regarding remote URL fetch, you can optimise it by making sure the metadata data for the wheel version is as close to the first few bytes
These installers can read these first few bytes and then abort when it’s downloaded enough to determine the version
Unless you are on dial up the cost is negligible, the server itself could also send your code the mime type in the headers (application/wheel2) which is yet another efficient way to determine it
I think “just works” in this situation means that PyPI continues to provide compatible old wheels to those versions. Actively-maintained packages are going bump their requires-python over time as they drop support for EOL versions, and newer releases won’t be found by those old installers.
The current release of pip supports >=3.8. They support anything that’s not EOL at time of release, so the 25.2 release will be >=3.9[1]. A hypothetical transition to wheel v2 could come with language that’s something like:
With helpful docs/error messages and the right timing, this would mean an older installer will almost never try to install a new wheel. A project that never upgrades their wheel version can continue to support new versions of python without issues.
I think a revised version of this PEP could simply…put these wheels in motion, as it were[3]. Set a real timeline and migration path for PyPI and installers to support a new wheel version, even if that version doesn’t change anything beyond metadata, and we will ensure that future changes can be handled more gracefully. It’s tempting to try to stick some actual improvements into the new version but I think that’ll just slow down the process.
One question I have about this plan, it seems to assume that new wheel versions will be incremental, with each version adding features over the previous version. So, for example, if you want to include symlinks in your wheel, you may have to use a wheel version that also requires zstd compression. Is that correct? Because one of the suggestions in PEP 777 is that we might be able to have wheel “features”, with the format allowing wheels to declare which features an installer needs to support to install this wheel.
I don’t know whether an independent feature model is possible, or even whether it’s useful, but it would be good to be explicit about what capabilities any given proposal allows, given that we’re (to an extent) discussing this in a purely theoretical context, with no actual new features being proposed at this point…
The ideal alternative is simply to not only allow, but encourage (perhaps by making it very easy) maintainers to publish different wheel versions for the same binary distribution.
I don’t think it has to be this way. My understanding of PEP 777’s plan was that the v2 wheel includes feature flags in some form[1], so that future improvements don’t need a version bump.
But that concept isn’t in the current proposal. It could make sense to introduce it, if people think it will be useful–whenever it is introduced will probably require a version bump to tell installers “check the feature flags”. If the goal of this PEP is minimal disruption to give installers forward compatibility, some form should go in now?
If I was asked to do this, I would flatly refuse. This goes against my values on a fundamental level as it is wasteful of community resources, and I am confident that anything added in future wheel versions can already be emulated with clever use of importlib metadata and a pair of packages, one containing data, the other depending on that data-only package and assembling it differently.[1]
For those who don’t view it on a values reason, it also interacts poorly with project file size limits on pypi.
I don’t think CI pipelines should be using open-ended versions and blocking merges/release at the same time. If people want advance warning of things that will break them, they should have CI for both a supported configuration and an open ended one, with only the supported configuration being a hard blocker.
I think this is also solved for the majority of users by just making an implementation in pip being mandatory as part of the wheel version being upgraded.
The first step of any open-ended version CI pipeline would then be to update pip (or other installer), and anyone who doesn’t (with pip at least, and that’s the vast majority of users) is only finding an error in an unsupported configuration, since pip only supports the latest version of pip, and only on supported python versions.
The ability to do that puts a finite limit on how much process people should be dealing with to put a wheel out there, I may prototype a build process that proves this this weekend if I have time, but it would be another “odd” package similarly, yet differently to how nvidia-stub is ↩︎
Who is doing such clever tricks currently, and how is it more convenient/less work than simply shipping two different wheel versions? How would it work for symlink support, or more efficient compression schemes?
Project file size limits can easily be bumped by PyPI admins if need be.
It wouldn’t be less work once, but as a build tool that does it automatically, it would save space rather than increase the amount of space spent on duplicating data, decreasing use of community resources that on the backend with amazon ends up significantly more impactful that just the estimation that was done for “how much network overhead is caused by use of zip” if this means a library should distribute 3 wheels for a single platform by the time we’re done with just the features we know people want.
I don’t know of anyone doing them currently, I intend to prototype it later, but I hope nobody needs to because I don’t believe it is the right long-term answer. I will be disappointed if this outcome is the best option for the concerns I value personally because we couldn’t do something that relies on trusting users to update and communicating with users about updates.
I think any answer here that makes tradeoffs to avoid “just tell users to upgrade pip whenever a situation where pip should be upgraded happens” is making the wrong tradeoff. pip has been designed in a way that upgrading pip should always be possible with minimal friction for supported python versions. The general idea is that all wheel formats should result in a compatible package, it’s just making the distribution format more efficient. There shouldn’t ever be a python package that can’t be packaged in one wheel format but can for another for the same platform, so I should just tell users “upgrade pip”, and pass along the efficiency.
In principle, I agree with this. However, transition processes are important - if a job that has worked reliably for a long time suddenly starts crashing because of changes[1] that the user isn’t even aware of, it’s a bit harsh to describe that crash as “notifying the user that they should upgrade pip”.
for example, a key package publishing a new format wheel ↩︎
right, but that goes back to not having CI jobs that block processes that use open-ended dependencies. There’s myriads of ways that same scenario could break CI without it being from a major wheel version. There’s really no way to know what future change some library might have that breaks you. Even for something with relatively strongly communicated intent of versioning policies, for example, python at the language level broke the cases where int(some_string) could error in a patch version.
If it’s not sufficient to tell people that, then how do we ever add anything?
With all due respect to those who are trying or have tried a cautious approach, we’re years into the wheel being blocked by caution, and I find that this is the only constraint that people have expressed that is holding us here. It’s a load-bearing problem, and I think it’s also the easiest of the problems to tackle.
There are several constraints here, many in direct conflict with each other, but this is the only one that does not have a solution at a format level, and I don’t see any possible way for there to be one. Fundamentally, it isn’t possible to require new features and then not want to require users to update to have the required features; And in direct contention to it, we have issues for end users when silent fallbacks happen without a clear reason.
All of the other problems have a reasonable solution if we can assume the installer is updated regularly.
With that in mind, what can we do to ensure that that is a reasonable enough assumption that people are comfortable with the approach?
I’d be fine with various options that ensure a level of time between “new wheel format is here and supported”, and “new wheel format exists on pypi” as well as various improvements that help ensure we shorten the time from “format exists” to “user’s tools handle the format”
A list of things we could do that focus on that aspect of the problem, some paraphrased from prior discussion:
Provide good guidance in a canonical location for the topic that all tools may link to.
Require a reference implementation of packing and unpacking a format before upgrade.
Require tools inform users of the need to update.
Disallow new formats for a set period of time.
Don’t allow retroactively adding wheels for formats newer than a release date.
Provide an example of CI jobs that can error for new features being unavailable without blocking
Provide an example of CI jobs that include updating the installer.
Set a maximum wheel format version that is tied to the lowest python version a wheel supports
Sorry if my (intended to be amusing) comment suggested that I think what we’ve done in the past was the right idea. I think previous attempts have got bogged down with an excess of caution. But that doesn’t mean we have to go to the other extreme.
I’d like to see proper consideration given to the question of how we avoid simply breaking people’s workflows or CI pipelines without simply saying “well, you shouldn’t have been doing that”. We know people shouldn’t use unpinned dependencies in production jobs. We know people should upgrade their tools. We know people should keep aware of incoming ecosystem changes and test how they will be affected. But people don’t do these things, and telling them that they are bad, and any consequences are their fault, doesn’t help anyone (it doesn’t help them, because they still have a problem, and it doesn’t help us, because we’re trying to improve people’s view of the packaging ecosystem).
So I’ll ask again. How do we mitigate the damage that would be caused by a new format version just triggering an installer error? Can we provide some sort of advance warning from the tools themselves? Can we provide some sort of graceful degradation for people using out of date tools?
Whether you like the approach PEP 777 takes of older tools ignoring new-format wheels or not, it has the advantage of meaning that users who don’t upgrade continue to see the world exactly as it was before the new format was introduced. That has downsides, particularly around user confusion as to why they can’t install the new release of a package that has switched to the new format, or why they are suddenly installing from sdist far more often, but at least it doesn’t break those users.
Maybe there’s a middle ground, giving an error while still allowing users to proceed in the short term. PIp has --use-deprecated and --use-feature. We could allow --use-deprecated=ignore_new_wheel_format for a few releases to offer a transition. But we can’t do that unless the new wheel format has a structure that allows it to be ignored.
I don’t know, should we wait for build tools to do now what they haven’t been doing for years? Especially as it’s not obvious how this would work under the hood, concretely.
The list above provides some options there, the last one tying the maximum wheel version to the minimum python version is the most drastic on the list, but it gives a place people already expect to need to check compatibility.
Possibly, though likely limited to degradation only after an update to be prepared for such a degradation. All wheel formats are going to have to be analogs of each other based on what’s in scope. Even something as drastic as a wheel version that allows referencing a prior release + a patch file can still be reconstituted by applying that, then repacking to an older format. We could theoretically require that a package maintained by PyPA exists that can convert between all wheel formats, but that conversion should only be used as a fallback and say that installers are allowed to augment the package list with the most recent version of this package as needed.
): could the user download the pure-Python wheel manually and install it explicitly? Obviously that’s not a great user experience but it’s still meaningfully different from “installing this package is flat-out impossible”.
