I’m actually really struggling to find a non-loaded way to explain this while reflecting what the reality is going to be. Because “old installers will continue to give the same result” isn’t true either (unless you’ve pinned all your dependencies, or at least everything that might get updated to a version with a new lower-bound pin on a package that only has new-style wheels).
Perhaps “resolvers should take into account the wheel version” is most neutral? It doesn’t really explain much for anyone who isn’t pretty familiar with how the resolve/install process works, and it may not transfer well to different tools that may handle wheels, but as a result it also doesn’t prejudice the argument (at least until the argument becomes 100 straw men fighting, which seems pretty inevitable).
I think the problem is that we haven’t been able to achieve consensus on that. I honestly don’t know without going back over previous discussions who had concerns about that approach - much less what those concerns were (beyond a generic “failing with an error is bad”, which is hard to engage with as it’s so non-specific). But it seems to me that we either (1) propose some wheel enhancement, and have discussion in the context of that - and my feeling is why would we expect the conclusion to change if we haven’t changed anything about the proposal, or (2) do some research to find out what the objections were, and address them directly in this thread, to get consensus on the transition process we want to use in future.
What I don’t want to do is have this discussion fizzle out with a generic “we don’t need to do anything” conclusion, and for us to still not be able to make any changes to the wheel format that need a major version bump.
I agree, the reality is far too nuanced to be easily reducible to a simple phrase. My preference is “installers ignore unrecognised wheel versions” because it focuses on the mechanics (which are simple), rather than the implications (which are not). And it’s easily contrasted with “installers raise an error on unrecognised wheel versions”.
One objection is the user experience: installers download the wheel before they learn the metadata version, so it can take them a while before they error. A concrete improvement there is to make the wheel version available before download[1]. I don’t know how much work this involves, and I’m unclear on whether it must be proposed in a PEP, or could simply be a PR to pypi/warehouse and the like.
Another UX issue is the minimal messaging when users encounter a new wheel version. It might seem obvious that the user should update their installer, but the current error messages don’t say that and the user might assume it’s just broken for them. It would be simple and worthwhile to improve that message.
If those two changes were made, pip and uv could a) rapidly check the wheel versions of requested packages and b) fail with a useful suggestion (“update!”) if they encounter a v2 wheel in the resolved solution. In a couple years, almost everyone would be using the new installers, and I think that makes the user experience for a version bump far more palatable.
maybe this can be accomplished with a range request, without a metadata update? ↩︎
I view the problem of future wheel compatibility as a two stage process:
Stage 1: How to introduce the first breaking change without breaking too much
and
Stage 2: how to make it easy (or at least, as easier) to bump the wheel major version to add independent new features to the wheel format
To me, the question of whether to rename the wheel file extension is pertinent to stage 1. However, I’m going to punt on that because stage 2 is probably more important long term. Defining how to handle upgrading the wheel version is central to making both stages work well.
So let’s focus on just the question of installer behavior for now.
One of the central arguments to having installers raise an error on unrecognized wheel versions is that users should be able to easily upgrade their installer to resolve the error, which minimizes the impact. After a lot of thought on this, I’ve come to the conclusion that either this greatly limits what future wheel versions can do, or is just not going to be the case in the future.
Here are a couple of examples of where a user cannot simply upgrade their installer to avoid an error:
The wheel being installed uses symlinks (and requires them!), yet the filesystem on the user’s machine, e.g. NFS, does not support symlinks.
The wheel being installed uses lzip (for arguments sake), an lzma based compression format, but the user did not build their CPython with lzma.
In both of these cases, the installer can be impacted by features (or lack thereof) in the user’s environment. A user cannot simply upgrade their way out of seeing an error with these wheels.
So how do we resolve this? One proposed workaround would be to enhance repository metadata to indicate how fallbacks should be handled. But this doesn’t work if I’m invoking pip --find-links with a directory containing a mix of wheels. It also doesn’t handle situations where users have multiple indices. How do we handle the instance where there are 2.0 wheels in some 3rd party index, but 1.0 wheels on PyPI?
For what it’s worth, I originally agreed that not aborting was a problem:
But as I’ve thought about various future wheel changes people want, I have come to believe that the wheel compatibility story must be handled by revolvers. We already have a robust mechanism by which tools select wheels that are compatible with your environment, it makes sense to me we should leverage that when talking about selecting wheels compatible with your installation environment, which can be influenced by your runtime environment.
I’ve also seen people suggest that we could define weaker versions of the standards, such that symlinks be an optional feature. But that isn’t realistically workable either in my opinion. If I ship some libraries that I want users to be able to load at runtime but also link to dynamically, I really want both libfoo.so and libfoo.so.1 in my site-packages, and I want to be able to rely on that. If I can’t, I have to include patches to my build process or loading code, which I’d have to do without symlinks anyway, so I win very little. Features that depend on the operating environment should be able to be mandatory, and I think we would limit ourselves otherwise.
I also think we can mitigate concerns about installers ignoring unrecognized/incompatible wheel versions.
Regarding sdists, we can either solve the problem wholesale (e.g. the discussions in Provide a way to signal an sdist isn't meant to be built?) or simply specify in PEP 777 that if incompatible wheels were skipped for a given version, resolvers should skip sdists of the same version. This will force resolvers to backtrack to an older version that should have a compatible wheel, else the installer can indicate that no compatible wheels were found and suggest trying to upgrade as a resolution (or some other resolution dependent on the feature).
I also think with a big loud warning about skipped wheels suggesting people upgrade, people will upgrade when they need to.
I wholeheartedly agree, I don’t think we should expect everyone to be using the same wheel version, or even encourage it. We practically can’t anyway, since there are 12 million and counting files on PyPI today that aren’t going to be upgraded
Why would this wheel format ever be accepted? Maybe you are thinking of this in terms of a future with variant wheels, where the resolver should see that there’s one variant it can’t support, and another it can, but they’re otherwise the same version. In which case, the installers needs to support that feature to deal with this. But I think variant wheels is a separate proposal?
It seems unlikely that there will be a new format that just doesn’t work on a Tier 1 platform, with no fallback.
But there must exist an installer that does have lzma support (vendored into it, as they tend to do), or again that wheel format seems like a nonstarter?
I don’t find these convincing examples for “you can’t upgrade out of it”, but maybe I misunderstand the scenarios.
I think we resolve 1 not by allowing this to fail, but by specifying it with behavior that prefers symlinks but works without (such as a metadata file that describes the symlinks, but just extracting results in duplicated files)
for 2, We can require that installers carry that compression going forward. Then upgrading the installer means the installer will by necessity tell the user if they need to go rebuild python with lzma support or possibly the installer brings it’s own lzma support. There’s already threads open to consider python built without zlib as broken currently, I imagine that we can avoid this by being specific about what is required in each new feature and ensuring it must be available either in python or brought by compliant installers.
Neither of these appear to be cases where upgrading the format isn’t enough, because we can upgrade the format in a way that is aware of platform issues with symlinks or that installers now have a new requirement to be compliant with the next version.
If the behavior of resolvers taking into account wheel version is still users receiving an error that need to update their installer/resolver, that would be okay. I’m not against a faster process here. I’m also okay with a flag that turns this into a logged message rather than an error and attempts to resolve using only wheels the tool already knows about to partially address the CI situation, but I don’t see a way out of this in the current specification that won’t break someone’s CI job that has uncapped dependencies but a pinned (explicitly or implicitly via lack of update) installer version, and I don’t think we can do anything about this that is reasonable other than get the message out to people not to do this and why.
Three concrete objections to the silent fallback:
The already hammered in bit on user experience with sdist fallbacks
The potential that this creates unwitting security or stability issues when someone does pip install -U somepkg and gets an old version that isn’t supported or may actively have flaws that were known and fixed [1]
Users getting unexpected outcomes without visibility will lead to more bug reports for tools, more support tickets for libraries, and more places where people debugging issues get frustrated that not only is the tool not doing what they expect, but they don’t even know why. This might sound like it’s the same user-experience issue, but this is describing what happens on the second-order effects of that. The people offering any level of support for those users also have to deal with this.
they may not have been strictly vulnerabilities, they may not have been known, they may not have been in supported version anymore. Tangential, as part of better messaging in general, we should probably get people more comfortable with yanking though, which is another thing not all packages consistently do the “right thing” on, even some large projects with CVEs acknowledged and fixed versions haven’t always yanked. ↩︎
But this can lead to silent explosion of disk space usage. I doubt people want 3 copies of every library on their system. There are also other use cases of symlinks that I think would be nice to have, such as PEP 660 support, which simply do not work with copies.
I am thinking of using variant wheels in this scenario, the point I am trying to make is that we can do much more with the wheel format if we make wheel compatibility part of resolution. Variants is an important factor in resolution.
I think there are use cases where certain platform-specific features may make sense. The symlink chain for libraries won’t be used by windows libraries, so I expect if there was a PEP that added symlinks it wouldn’t need to impact Windows. Not all features need be mandatory for every wheel.
Regarding my second example:
I think today it would be a non-starter, but I think in a world where users could upload multiple wheels using different compression and the selection is handled as part of resolution, it could exist.
There would be some installer that vendors things, or is free-standing of the environment (e.g. uv), but that doesn’t help someone who is using pip when they get an error at install time. pip doesn’t take binary dependencies as a policy (ref pip/src/pip/_vendor at main · pypa/pip · GitHub) and I doubt a Python implementation would be a workable alternative, both due to complexity and performance. pip has always relied on parts of the standard library. I am of the opinion that if we want advanced compression it must be an optional feature that installers either support or don’t, and that users should upload multiple wheels to handle that compatibility while also providing performance improvements.
I think telling the user they need to rebuild their Python is a non-starter. Users may be stuck with certain builds for security or other reasons, and I think it is a significantly larger lift to ask of users rather than simply upgrading their installer. As I stated above, I also don’t see how pip would vendor liblzma. It would break the assumption that pip is easy to bootstrap.
I expect in both scenarios we will see more errors and more bug reports and support tickets. I don’t think we can authoritatively say one will be more than the other because we don’t have a basis for measuring how confused people will be or how many people will be affected for each scenario.
This is the current status quo if your wheel is relying on each of those paths containing that library. If it isn’t relying on that, I’m not sure why each of those are in the wheel. I don’t see a downside to only distributing one wheel that can gracefully keep the status quo, but save disk space when the user’s filesystem supports it.
This just doesn’t make sense to me. I’ll pick the most efficient compression that I expect my users to have access to and distribute it once rather than intentionally in n-copies, and having an installer that can handle that becomes a requirement to use the library.
sorry, I realized I’ve split this into more messages than intended, last one in response for a while.
This breaks the concept of a wheel for me, and if this becomes the suggested way to do things, I’m probably just going to find other solutions and ignore this.
Wheels are just a way to get a prebuilt binary distribution to a user. I already have a valid reason to have separate wheels for platform compatibility, but I’m not going to upload multiple wheels for the same platform compatibility. I’ve already made it clear that on a personal values basis, I think even 200bytes per wheel would be worthwhile savings, but lets just look at what wheels are here to solve: Wheels are meant to be portable and solve the case where users don’t have build tools installed. They also act as a rather efficient build cache. Installers were meant to be environment agnostic. If a feature breaks the portability goal to the point where multiple wheels are needed for the same library on the same platform with it behaving identically once installed at runtime, I think we’ve done a poor job of it. We can probably specify many things at a level of abstraction that isn’t the wheel format here. We could probably today as a minor version specify a metadata file that installers can optionally use to mark which files should be considered safe to dedupe via symlink.
If we need to more regularly make advanced compression guaranteed available, lets do that. Lets not require 4 uploads for people to use that.
This isn’t the status quo though. Right now the wheel itself copies the library 3 times and increases the wheel size. At $dayjob, we only ship the SONAME copy (libfoo.so.N), and users need to handle linking it.
Of course, we could change this to have symlinks be pollyfilled with copies during installation, but again, that prohibits symlinks being used for PEP 660 editable installs, because if it degrades, the result is a broken editable installation that copies the source rather than symlinks to it.
That’s totally your choice for your packages! But not everyone may want to make that same choice. People who ship large libraries may wish to produce n wheels so that their users can take advantage of advanced compression for faster downloads.
That seems a critical point. So you ship ~broken wheels in that they install via a standard installer broken and a user must hand repair by adding symlinks after?
Well, I suppose you could view them as broken, but I would say they are “efficiently optimized”
On a more serious note, users do not add the symlinks themselves, they modify their build tools (e.g. CMake) to modify how those tools search for the libraries.
Ok, but I think that’s enough to take that example off the table. I think @mikeshardmind is right, a working wheel would include the duplicate copies today. And in the future you’d optimize that waste out with the symlink support.
I agree that platform-specific features make sense, but “with no fallback” is the issue. There will have to be a way to install the package on supported platforms, even when the feature is unavailable. A compliant installer will need to know how do about that. Thus, upgrading the installer will let them install the new wheel.
If your package simply doesn’t support a platform, the installer wouldn’t try to install it in the first place (unless you upload an sdist!).
Maybe there’s a disconnect here in terms of the intended use of feature flags. Is the idea “make a bunch of wheels with different flags” or “your installer reads the flags and installs the best thing it can”? I’m imagining the latter, and a flag like this should have a fallback.
Noticed a few points, but still havent had the time to finish the comparison of hypothetical outcomes I was working on. Discussion here has actually caused some expanding scope.
I don’t think it’s fair to say it isn’t the status quo if you are already having to subvert what the wheel does and ship a wheel that doesn’t work without manual intervention. your status quo is firmly outside of the wheel specification.
edit: sorry, seems multiple people were typing the same point and discourse didn’t load it as I was trying to grab quotes for things I saw.
Reasonable, but if it can’t be vendored easily, this puts a standard library adoption delay in.
Can we get agreement on this point right now as part of this?
Thinking about it more, maybe that’s a little too strong–if you’re uploading separate wheels for separate platforms, it seems possible to upload different wheel features for those platforms. The thing that doesn’t make sense is to allow people to upload a linux-only wheel format for something that purports to support multiple platforms.
I would word it that universal wheels need to only have wheel features compatible with all tier 1 and 2 platforms, and that platform specific wheels must only use wheel feature that always work on that platform.
edit: but symlink support can’t be guaranteed on any platform. This can be a filesystem limitation and will always need a fallback behavior defined.
I view it differently - features that are mandatory must be designed in a way that means they work anywhere.
But this is irrelevant. The premiss of PEP 777 is that currently, a major version bump in the wheel format is impossible. Specifically, the PEP says it would be “catastrophically disruptive”, which is extremely strong language. And I think there’s been enough disagreement in this thread that the PEP needs to justify that statement. Not by appealing to possible new wheel features, but in the abstract - exactly the way the statement is made.
So let’s get down to the minimal possible case. If wheel version 2 was defined as being absolutely identical to wheel version 1, with the only change being to the version number, and if projects started publishing version 2 wheels, then the PEP asserts that this would be “catastrophically disruptive” to the packaging ecosystem. I think you need to prove that statement. Why would this be such a problem? Why isn’t it enough for people to upgrade their copy of pip and/or uv?
To be explicit here, I’m not saying that all possible new features are this simple. We may indeed find that some features need special consideration as far as the transition process goes. But that special consideration should be coupled to the proposal itself, not to the general idea of a major version bump.
