I think the big difference between whlx and variants (or other proposals) is that there shouldn’t be any need to build twice to make both a whl and a whlx. The version in this PEP doesn’t specify any difference to the built package at all–it’s just an organization of the zip file. So a tool that can build a whlx should be able to build a whl in the same step, if side-by-side versions are desired.
2 Likes
Making that work would be a much bigger change to tools like pip that build wheels.
What would (presumably) be possible would be for someone to write a new tool that repacked an old-style wheel (.whl format) to a new-style (.wheel) one. Or vice versa, as long as the .wheel didn’t use any new features not supported in .whl.
I don’t understand why it’s much bigger than supporting the whlx format in the first place, but I’ll take your word for it. edit: maybe I should clarify, I only mean when building a whlx v2.0, i.e. the specification in this PEP with no additional features.
(I early-sent in my other message, here is my actual reply)
My definition intentionally includes the existing wheel forma!
I think the new extension is necessary because people don’t upgrade their installers frequently enough. See my analysis referenced in the PEP that shows a long tail of installers Appendix: Analysis of Installer Usage on PyPI | peps.python.org
If you have a reference to that previous discussion, I’d love a link. I will say that variants work much better with a new wheel format. My colleague and I are working on a variant design that wheel 2.0 will lay the groundwork for.
This PEP proposes that installers emit a warning if a wheel is skipped because of an incompatible version. That’s a great opportunity to suggest users upgrade. If that isn’t sufficient, we could also specify that sdists of the same release as an all-incompatible release be skipped as well.
It would greatly slow adoption if uploading 2.0 wheels would break users. People would wait until most users adopt new installers to upload 2.0 wheels. I think we want to encourage early adoption of new wheel formats. Also, pip’s existing error message about incompatible wheel version mentions nothing about upgrading, so I don’t know if users will actually know how to fix the problem.
See above mentioned analysis, I just don’t think this is realistic. There are always going to be users who are on old Linux distributions who will be using older Pythons and versions of pip. I don’t want to wait 4 years to adopt a new wheel version just to maintain the existing file extension.
1 Like
Yes, I think this should be explicitly possible, at least in the .whlx → .whl direction. The new format I am drafting has some information not present in .whl such as top level packages, so I’m not sure the .whl → .whlx direction can be done without providing additional data.
One thing I do not want to do in wheel 2.0 (and leave for a future revision) is to change the installation procedure beyond unpacking a zip file. The format the metadata/compatibility tags is stored may be slightly different, but still compatible with importlib.metadata.
1 Like
Upgrading the installer is going to be the only way for users to adopt. So your choices really do come down to:
- publishers have to produce two equivalent wheels (and users are never made to upgrade)
- users have to upgrade installer to get the latest wheels
- publishers have to wait 4 years for users to upgrade first
None of these are ideal. The second is the fastest, while the third is likely to be the least problematic. You have to choose one 
Agreed. I’d like to see the installation procedure specified in the metadata itself, though any new procedure would require an interoperability PEP, so the burden of adding one would be fairly high.
1 Like
Here’s one more option: Publishers can publish two equivalent wheels during a transition period, with a shim inserted into the whl version that emits a DeprecationWarning on import. A tool that produces both versions might be able to do this automatically.
This allows package maintainers to adopt to the new format at their own pace, in case changing or updating their build process might be a lot of work. Some example use-cases:
- Package A is pure python and moves rapidly, they just bump the minor version so that users know to upgrade their tools.
0.5.1is the last to use.whland has a.wheelalongside it,0.5.2is.wheel-only. - Package B is cautious and emits both
.whland.wheelfor many releases, because they know there are a lot of users who can’t upgrade easily. They don’t add the warning until their final.whlupload. - Package C has a really convoluted build process, and sticks with
whlfor a couple years, but they jump to.wheelwhen a new version allows them to do something slick, like compress a lot of bulky data or provide several variants. By the time they do, nearly everyone has already updated their tools, but they still add a warning to catch the extreme cases.
I’d pick the second one. pip already only supports the latest release, and has been consistently designed so that all of its dependencies that aren’t in the standard library are vendored. Upgrading pip should not be a painful process for users.
While I can’t say quite the same of all installers categorically, it’s my understanding that other widely adopted package management tools in python are sitting in a similar position where upgrading should not be blocked by other dependencies and only the tool of choice itself being ready.
2 Likes
I think “users have to upgrade” is definitely the way to go, but there should still be some way to provide them the information that they need to do so. Just giving them the last available whl will lead to a very long tail of confusing bug reports.
A more insistent version of the proposal above would be a) last working whl emits DeprecationWarning on import, and b) the next version has a stub whl that raises ImportError with an informative message about how out-of-date their installer is.
As I was thinking of ways to handle this, I realized a security question exists here too.
A new extension would allow going back to old releases and adding a .whlx (or whatever actual new extension we agree on) for the same platform and version.
Maybe the right way here is just use keep using .whl and installers should error when they encounter one they can’t handle.
2 Likes
Could this be explicitly forbidden by PyPI? Maybe allow a whlx for the last-uploaded release but nothing before that.
Agreed. Particularly as we now have the zipapp distribution of pip, there should never be an insurmountable reason for not using a sufficiently new version of pip. I exclude non-technical reasons like “my corporate policy doesn’t allow using a version of pip that isn’t approved” because (a) the approval process should take into account the limitations of the older version, and (b) corporate policies can do arbitrarily stupid things, so we shouldn’t let them dictate our process.
And uv is similar - you can download the latest version of uv and run it from anywhere.
What we should be doing is ensuring that users who haven’t upgraded:
- Get told that they need to, and why.
- Get an acceptably degraded experience if they don’t upgrade (I’d consider “can’t use versions that don’t ship old-style wheels” as “acceptably degraded”).
And finally, there’s no way this change isn’t going to be disruptive, no matter what we do. So we need to put as much effort into publicising the change, and the reasons for it, as we do into technical design. Python packaging has a bad reputation, and we need to put effort into ensuring that this change doesn’t reinforce that.
10 Likes
It’s possible, but anything we do on that might need to be specified in a better way than this. We could make it an error to provide multiple binary distributions for the same target.
it’s also already possible to backfill wheels, what’s new here is a situation of picking between 1 old and 1 new binary distribution.
It’s also possible there are valid use cases for this such as backfilling a handful of supported versions with a more efficiently stored binary distribution that was built from the same sources.
I’m not sure forbidding this is the right answer, only that it poses a question that we should probably address. I know pdm allows some level of reproducibility by specifying a date and nothing newer than that date is used. The existence of such tools might indicate that this isn’t something pypi needs to solve, only that tools need to be aware is possible for users who consider this part of their threat model
1 Like
That’s essentially what I’m suggesting for PEP 759. Packaging tools could easily turn a .whl into a .rim but it’s just as easy for an external tool to do it as well.
“Never” is a long time! I suspect that there will be an inflection point at which v1 .whl publishing drops off significantly and most of the ecosystem either willingly or by necessity makes the switch. I don’t think it will be as long as Python 2 → 3 transition, but I really don’t know whether that will result in a faster transition than any other option.
Old pips already issue a notice suggesting users upgrade, and it won’t matter if that’s just to get the latest pip or to support a new wheel version.
Also, if/when PEP 694 is implemented, all of those .whl and .whlx uploads can essentially be published atomically. I’m not saying we should but we could disallow uploading .whlx for any published .whl, nudging people to using PEP 694 to accomplish this.
I dislike options that rely on hypothetical tools (or hypothetical changes to PyPI or the repository protocols). I would count this as the same as my first option, even though the wheels aren’t equivalent in this proposal.
If the fundamental change is the metadata version, and we expect old-style wheels to keep working until the end of time, then we can start the change immediately and nothing really breaks until we have a reason to break it. The important decision to make now is that installers should read the metadata version before installing, should fail if it’s “incompatible” (to be defined), and should watch for later PEPs defining new behaviour tied to new metadata versions.
1 Like
The “hypothetical tool” in this case is “a tool that builds wheels.” I assume that support by those tools is a requirement for this PEP to go anywhere. And the difference between this and your first option is that there’s a path to deprecation, so users are forced to upgrade if they want new versions.
In terms of PEP language, this doesn’t need to be a MUST requirement for package maintainers[1], but I think it could be highly recommended. Likewise it doesn’t need to be a hard requirement that any wheel-builder must provide the option to emit parallel old and new wheels. It just seems like an obviously-useful feature that one would want in one’s tooling.
I don’t think “until the end of time” should be the goal. Moving to a better format should be the goal, but it can take a long time. So it seems like a good idea to make a plan for the transition now. I think that’s the whole motivation for PEP 777: introducing the next format in the most painless way possible. There will still be a little disruption, but it can be minimized.
i.e. you could just switch from
whltowheelwith no warning or transition, since you’re the one who will have to deal with your confused users! ↩︎
It could be very problematic, though. The wheel builder in question is the build backend, but the PEP 517 build_wheel interface only allows for returning a single wheel file. And a frontend like pip only expects one file. If I’m installing a sdist, and I call build_wheel and get two wheels back, which one do I use? In theory, we could require that build backends must ensure that multiple wheels are equivalent, but that will be very hard to do in practice, once we start adding features to the new format.
2 Likes