PEP 777: How to Re-invent the Wheel

Packaging Standards
183

While a package like that could exist, I wouldn’t want tools auto updating (CI that explicitly updates when triggered is not automatic, but explicitly as part of the job). That should still be opt-in behavior not an external package automatically fetched for compatibility.

The most likely to entirely avoid a surprise incompatibility as a single measure on your list there is tying it to the python version, but that won’t prompt people to update pip and would create a forcible lag on adoption of years.

2 Likes
184

If we tell them and provide examples, we can ensure it’s a learning experience, shouldn’t be a repeat experience, and it won’t be just something to throw their hands in the air, exasperated about.

I’m okay with some amount of easily resolved breakage that informs the people effected how to get the working outcome because the only people broken are doing something subpar, and we have a simple solution available so it doesn’t need to be an ordeal. Maybe a note in every python release should remind people to also update their installer and remind them that installers are versioned separately from python.

I’m Pretty sure any feature that could be done that way would fall under the current wheel’s minor version, and that for all of the things that have been blocked, this would be impossible.

While @mikeshardmind accurately pointed out that all hypothetical wheel formats can be transformed between*, the things that need a major version need it because they change something that the tools to unpack the wheel would require new knowledge.

* Unless wheels stop being prebuilt binary distributions.

I’m interested in the prototype of this. Iterating on it could provide a better long term in the “break the format once” scenario if it is standardized.

From the 2 package structure, it seems like your plan you’re really just intending to abuse the fact that you can use one wheel to distribute based on platform and another to hook in somewhere that would allow modifying the file layout, but it’s unclear exactly how or what the limitations of this approach would be.

If we were to instead standardize on future, whlx outcome wheels also providing a package that does the unpacking, we no longer need any coordination for future upgrades, people can go wild and make their own formats or use someone else’s. apis for this would basically be the inverse of pep 517’s apis, and it would be less of a hack than whatever you have in mind to use one package to modify another.

185

The kind of surprising breakage (broken automatic CI jobs) are the kind that are likely to break one day anyway. Unfortunately, there’s really no good way to propagate the opinion of “using a pinned installer with unpinned dependencies is a recipe for breaking one day,” or that the fix is to either pin both or pin neither. The best way we have is for the old installer to refuse to install the newer packages and explain that upgrading the installer or constraining the package are the two correct ways forward.

I like the --use-deprecated option for the “automatically downgrade dependencies” path. That is easy enough to use as a workaround, while making it clear that there are non-deprecated ways to get the same result (i.e. pin your dependencies).

But given we’ve gone so long with people finding it safe to put themselves in this circumstance, our only real choice is a “scream test” (or sometimes “brownout”) where we deliberately break that case for a short period and then restore it, so that anyone running into it is only temporarily blocked but also informed. This, however, is incredibly hard to handle in OSS, and doesn’t really make sense for this situation.

Having the tools ready with a more detailed message for when it actually occurs is likely the best we can do: “Wheel {filename} uses wheel format {version}, which is not supported by this version of {installer}. You should either upgrade {installer}, specify an earlier version of {package}, pass {option} to have {installer} search for the last version of {package} that can be installed, or pass {option} to build from source.”

(Edited to add I also like the feature flag idea, but would see it as more options in the WHEEL metadata file, such as symlink-handling = symlink/copy/ignore or compression = zstd/etc., which are ignored when the wheel version is less than whenever they were added.)

4 Likes
186

Python has already solved this problem and recently in-fact IMHO, Take the introduction of ARM 64 as a CPU architectures. For the libraries that didn’t ship it the installers jumped into action and built from source.

This happened because pip was able to communicate or detect the specific architecture needed for that host.

This strikes me as how we could do it here, either:

  • pypi hides wheel2 files unless pip requests them (Pypi already supports a way to request a new API format, why not a way to say “i only want wheels v2” for example ? “application/vnd.pypi.simple.v1+html+wheelsv2;q=0.2"”)
  • Or we borrow the idea of triple from C/Linux, i.e an architecture for a wheel could be more than just architecture of CPU but also things like the ABI of the packaging format (i.e at https://pypi.org/pypi/django/json the json key “packagetype” which is now set to"bdist_wheel" becomes “bdist_wheel2”
187

Relevant to this point, I opened an issue on the website to workshop the language for such a page. The bare minimum is just to establish the URL so that tools can start linking to it, but more background might be useful.

This raises a question: why does this PEP jump to a new major version? The current version is 1.0, and the required behavior of pip and uv is to error on a new major version and warn on a new minor version. So bumping the minor version would, over time, lead to backwards-compatible warnings for any users who have an out-of-date installer.

So, here’s an alternative version of PEP 777, in terms of tasks:

  1. Add a page to packaging.python.org about wheel v1.1 [1]
  2. Add messages to pip [2] and uv [3] to point users to 1. when encountering a new wheel.
  3. Specify wheel version v1.1. I believe that a minor version could add wheel-specific metadata while staying in the v1 specification.
  4. Update tooling to generate and install v1.1 wheels.
  5. Monitor the PyPI stats to verify that installers are able to handle v1.1 and that maintainers are uploading them.
  6. (some time later) wheel v1.2, or v2.0, or whatever is desired, is much simpler to roll out.

I think steps 1 and 2 can be accomplished now, without a lot of further deliberation (beyond some bikeshedding of the precise text). Step 3 can take longer, but this is fine–the longer the installers are out there with a useful warning message, the more people will see that message in the future.

  1. issue ↩︎

  2. issue, pr ↩︎

  3. they can make their own PR ↩︎

9 Likes
188

Not that I like this idea[1], but if installers could auto-upgrade then conceivably situations which today would generate an error (“sorry, I don’t understand this wheel version”) could instead upgrade themselves to a newer installer that did understand the new wheel format.

  1. a previous employer tried this with one of their tools and customers by and large really disliked it ↩︎

189

Not that I like this idea[1], but
if installers could auto-upgrade then conceivably situations which
today would generate an error (“sorry, I don’t understand this
wheel version”) could instead upgrade themselves to a newer
installer that did understand the new wheel format.

This is sort of the approach Tox takes, as described earlier. If you
have a tox.ini which declares a higher minversion than the tox you
invoked, it downloads a new version and runs that instead (though it
technically doesn’t upgrade the version you’ve invoked).

  1. a previous employer tried this with one
    of their tools and customers by and large really disliked it ↩︎

190

That approach might work but it’s almost moot, because existing installers don’t do this, and people will keep using their current installer for years.

Any proposed change to the tools is only starting the clock on a migration plan. If we’re planning on a 4 year transition there are a lot of options for maintaining compatibility, but it’s a long time to wait.

3 Likes
191

I’ll say that a 4 year transition is almost a 5 year transition and, if we’re considering timelines like that, it’s probably reasonable to couple this with a Python release which is one of the rejected ideas in the PEP right now, due to the slowness of that rollout mechanism (paraphrasing). That approach does have a clear “We have completed the transition” point.

3 Likes
192

Yeah, to be clear I don’t think it’s the best option–I think a new version can be introduced faster than that. I was just trying to say that any plan that relies on adding a new feature to installers, then waiting for the new version to be widely available, is going to take 4+ years.

The problem is sort of like this scene from Apollo 13: we want to introduce a new wheel in a timely fashion, with a reasonable user experience for nearly all users, but only relying on the tools they already have installed.

2 Likes
193

I think there are ways to accelerate / incentivise uptake. Especially if you’re alluding to the creativity from Apollo 13. :wink:

Not all of these are equally realistic (or even desirable), but just thinking out loud.

  • Regular libraries could start warning during installation if they are being installed by a tool that doesn’t raise an informative error when fed a dummy v2 wheel (or for future-proofing, advertise compliance with v2). The idea here being to regularly make users come across reminders to upgrade their installer.
  • Do the above but package it as wheel2-compat-warn so that other libraries can depend depend on it, if they want to participate in the effort of spreading the word.
  • Vendor that package into patch-releases of supported Python versions
  • A “hotfix” package that changes behaviour of other packages (e.g. monkey-patch the installer). FWIW, I consider this too intrusive, but listing it since I remember a concrete example.
  • Do a “wheel2-readiness” effort (e.g. track usage of installer versions + v2-compatibility à la Manylinux Timeline, and declare a goal like “90% of users on non-EOL python versions” when we pull the trigger) – something publicly visible will spur grassroots advocacy, especially if there are tangible features attached that people want.
  • [your idea here]

Obviously these are all various degrees of repulsive from the POV of proper software engineering, but then we’re paying for our own mistake of breaking rule 0 of making a standard (“how do I version myself?”). Tangentially, this question should probably become a more prominent checkbox on any new packaging effort, because we’re in a similar situation with pyproject.toml, script metadata (PEP 723), lockfiles (PEP 751), etc… :roll_eyes:

9 Likes
194

How did we break that rule? The wheel spec is versioned, it’s included in each wheel, and we’ve even defined the behaviour of tools in the presence of potentially/definitely incompatible versioned wheels.

1 Like
195

Arguably, we’ve broken it by not simply adhering to that. This pep calls into question if we have actually versioned ourselves properly from the outset if it is impossible to use that versioning[1] and we have to have a new extension.

  1. I don’t think it is, but history has examples where it effectively hasn’t been something people will be willing to do. ↩︎

2 Likes
196

Correct me if I’m wrong, but pretty much the entire discussion is about avoiding breakage by starting to release wheels in a putative v2 format. Which is – IIUC – because existing tools would just crash and burn, rather than raise an informative error (e.g. “the wheel you’re trying to install is a v2 wheel; this version of $installer only supports v1; please update $installer”).

If we had enforced more strongly that this kind of error gets implemented in all installers, we wouldn’t need to have this discussion. This lack of future-proofing is the “our own mistake” I was referring to w.r.t. self-versioning.

3 Likes
197

The only thing missing from pip’s message is “please update $installer”. The rest of the message is there, almost exactly as you propose it.

We can’t enforce this any better than putting it in the wheel specification, which we did.

What’s trying to be achieved now is to change unpack step b from the existing spec from “Check that installer is compatible with Wheel-Version. Warn if minor version is greater, abort if major version is greater.” into “if major version is greater, treat the file as if it does not exist”.

Which is a fine proposal to make (I’ll still oppose it), but it can’t be justified by saying that we don’t have any way to handle major version increases.

3 Likes
198

OK, point taken, the installer side is covered. First off, I’m not pointing fingers here when I say “mistakes were made”. That’s normal, and we’ve gotten very far based on the current designs.

That said, the fact that we’re having this discussion means that the future-proofing part of “how do I version myself, and what happens when I increase that version” was still not complete.

I mean, I personally would be fine to basically say “if existing installers abort on a new major version, it’s not catastrophically disruptive for wheel v2 installs to fail, users just have to update to a capable installer. :person_shrugging:”

However, the entire rationale of the PEP is that this is not enough. I can certainly see where that sentiment is coming from (even though I’d be less prone to bending over backwards for people on outdated installers…), in that it would be beneficial if old installers just keep working in the presence of v2 wheels.

In any case, apologies if I gave offense, that was not my intention.

2 Likes
199

I realized in this discussion I was losing track of which problems under discussion are problems the PEP tried to solve and which possible are solutions are ones the PEP actually proposed. I decided to go back over the PEP and see if I can be more specific about the aspects of it that make me think that the bigger problems are with the installer-index interface and not with the wheel format. A lot of the discussion here has been about the idea of a new .wheel extension, but actually a lot of the PEP is just proposing standards changes of the form “installers/resolvers MUST do X”. For the most part those make a lot more sense to me, and my question is whether we would even need to do the extension change if those changes were made. Here are some relevant bits from the PEP:

However, resolvers do not currently exclude wheels with an incompatible wheel version. There is also currently no way for a resolver to check a wheel’s version without downloading the wheel directly.

These are problems with resolvers and/or indexes and not with the wheel format. It would be good to fix these problems even if no changes are made to the wheel format.

Resolvers, in the process of selecting a wheel to install, MUST check a candidate wheel’s Wheel-Version, and ignore incompatible wheel files.

Therefore, installers SHOULD emit a warning if, in the process of resolving packages, they come across an incompatible wheel and skip it.

These changes seemed to have more support in the discussion, and are not actually about the wheel format; they’re just about installer/resolver behavior.

It could take upwards of four years before the majority of users are on updated resolvers, based on current data about PyPI installer usage

To allow for experimentation and faster adoption of 2.0 wheels, this PEP proposes a change to the file extension of the wheel file format, from .whl to .whlx for all future wheel versions.

Is the only reason for the extension change to avoid having to wait four years (or more)? It seems a bit odd to me to have borne with all these wheel-related issues for many years and now suddenly decide we have to switch the extension because we can’t wait any longer.

For the goal of experimentation, it seems like it should be possible to set up some kind of test index where people can upload wheels with new versions, test them with new installers, etc.

The main compatibility limitation of this PEP is for projects that start publishing solely new wheels alongside a source distribution. If a user on an older installer tries to install the package, it will fall back to the source distribution, because the resolver will skip all newer wheels.

This is a problem with sdists and installers/resolves, not with the wheel format. The problem has been discussed many times. It would be beneficial to fix the problem of unexpected sdist build attempts even if no changes are made to the wheel format.[1]

Earlier I suggested that we should think about what wheel 2.0 will actually be like before we know if this PEP is a good idea, but now I’m kind of swinging the other way. :slight_smile: What if this PEP removed the proposal to change the wheel extension and just proposed the changes above related to what counts as standards-compliant installer/resolver behavior? (Or this could be broken out into a separate PEP.) It seems like there was much broader consensus in this thread that making those changes is a good idea. In way the PEP currently combines two things: an attempt to fix problems with existing installer assumptions, and an attempt to look forward to a future wheel format. Both may be good, but maybe we can do the former even if we don’t agree about how to do the latter.

  1. I still think the best solution is immediate deprecation of consideration of sdists by resolvers. Maybe actual behavior changes can come later, but as soon as possible people should start seeing warnings saying “in the future, resolvers will only consider artifacts that do not require a build”. ↩︎

3 Likes
200

Then we’d focus the argument down on “silently downgrading packages is better than aborting”, on which there was significant disagreement.

If consensus goes towards silently downgrading, the next argument is “silently downgrading packages must be efficient” which I also disagree with, but if there’s consensus then it justifies changing the repository protocol to include the wheel version (which is less controversial).

And there’s a side argument about how to handle silent downgrading without the current behaviour of trying to build from the sdist, for which there’s at least a few entirely separate threads already.

As I keep saying, lest people believe that because I can frame the arguments positively that I must agree with them, I don’t think we need to do any of what I just set out above. But they are the core points that every other argument flows from[1], so if we can resolve them we should have a way forward.

  1. Apart from “the pain will be worth it”, which we exclude by excluding specific wheel 2.0 features from the discussion. Which I suspect means we’ll get to have exactly the same compatibility argument at feature time… ↩︎

5 Likes
201

I have one comment on this point. But before I make it, I want to be clear that I am currently undecided on basically all of the questions you asked. I think they are an extremely good summary of what we need to get consensus on in order to move forward, but right now, I’m willing to be persuaded either way.

With that said “silently downgraded” is an extremely loaded way of presenting the “ignore incompatible wheels” option. Nothing is being downgraded, it’s simply that the installer will not be aware of newer versions if there’s only new-style wheels[1]. That could more accurately (still somewhat loaded, but in the other direction) be presented as “old installers will continue to give the same result”.

IMO, the first thing we need to do is agree on a uniform way of describing the options, that is neutral on their desirability. Otherwise, we’ll continue using soundbites as arguments and people will become entrenched, rather than moving towards consensus.

Is it fair to say, though, that if we ever want to improve the wheel format, we need to come to some consensus on how we handle the transition process? Whether that’s getting agreement that the existing specification can be made to work, or a specification change to require something that we do have consensus on, it seems to me that doing nothing will leave us blocked the next time we try to move forward.

  1. and no sdist, but that’s a side argument, as you said ↩︎

8 Likes
202

It’s possible to achieve consensus on the status quo, which is basically that the first major package to start releasing 2.0 wheels is probably going to surprise any of their users who aren’t regularly updating their installers. That more or less amounts to “doing nothing”.

There seems to be an undercurrent of “within X time, all wheels should be 2.0 wheels” which doesn’t strike me as at all necessary. But I wonder if that’s driving some of the urgency here?

Installers are almost certainly not going to be able to drop 1.0 wheels for much longer than any transition will take (and since 1.0 wheels amount to “extract and you’re done”, there’s probably not much gained from dropping that anyway). And we don’t need to drop 1.0 wheels at all, particularly, except to force users to migrate forward (which I don’t like, either as a user or as a standards contributor).

So if we can also find a way to agree on “packages must update to wheel 2.0 asap” vs “packages can update to wheel 2.0 on their own schedule or not at all”, that may also help clarify what the transition process could look like.

2 Likes