Speculative: Wheel 2.0 and migration strategies

I think the only way to make a transition like this work is for pip to have near universal support (i.e. all in-use versions have 2.0 support), and PyPI and other third-party feed providers support the new format. Without that, publishers will need to pin to an older version.

In particular, you can’t have a transition point for backends that is before PyPI supports publishing them and pip supports installing them. Otherwise, literally everyone will pin to an old backend and potentially never update again.

The sooner we agree on a mechanism whereby Wheel 2.0 files can be identified by pip, the sooner we can roll out informative warnings and at least get those universally accessible. At that point, at least users will get a clear “you need to update pip” warning rather than whatever they’d see today (broken install?). That’ll give us the best chance to enable any new format and have publishers be able to use it.

Uh, yes, this was stupid of me. PyPI needs to be before backends. I did put pip (consumers in general) first. I don’t consider “all in-use versions have 2.0 support” to be an issue, as we don’t support older versions of pip, and our response to anyone complaining that pip doesn’t recognise a 2.0 wheel would be “please upgrade pip”. So IMO, the natural delays involved in the rollout I suggested would be sufficient here.

That would be useful, but to be perfectly honest, pip’s behaviour in this area isn’t good anyway. We often report a bland “no files available” when files are available but are rejected for some reason (often incompatible Python version requirements). That’s not an excuse for giving bad messages when version 2.0 wheels are ignored, but it would call into question any claims that poor reporting was a showstopper…

That may be your response, but it’s the publisher’s response that I’m worried about. “Can you please publish a Wheel 1.0 as well as we aren’t able to update pip yet” is likely to be a common request, and I’m sure at least a few will just decide it’s easier to pin their backend so that their wheels work everywhere (unless, as I mentioned above, there’s some really beneficial feature in Wheel 2.0, in which case they’ll happily tell their users to figure out how to update pip :wink: ).

Sure, the show will go on. No reason we can’t try and be better this time though :slight_smile:

Doesn’t such a mechanism already exist and is implemented? The WHEEL metadata file in the dist-info contains a Wheel-Version key, and per the Wheel spec, installers should warn if the minor version is higher than the maximum they support, and must raise an error if the major version is higher. So long as installers (pip, installer, etc) are following the spec, and the error they raise is suitably clear and informative, we should already be covered on this point without any additional action.

Indeed, I tested pip (22.3.1), and I do get a yellow warning (four of them to be exact) when installing a wheel with a higher minor version:

$ pip install ~/Downloads/pyroma-4.1-py3-none-any/pyroma-4.1-py3-none-any.whl
Processing c:\users\c. a. m. gerlach\downloads\pyroma-4.1-py3-none-any\pyroma-4.1-py3-none-any.whl
  WARNING: Installing from a newer Wheel-Version (1.9)
Requirement already satisfied: pygments in c:\miniconda3\envs\tools\lib\site-packages (from pyroma==4.1) (2.13.0)
[More requirements here...]
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in c:\miniconda3\envs\tools\lib\site-packages (from packaging>=19.0->build->pyroma==4.1) (3.0.9)
WARNING: Installing from a newer Wheel-Version (1.9)
Installing collected packages: pyroma
  WARNING: Installing from a newer Wheel-Version (1.9)
  WARNING: Installing from a newer Wheel-Version (1.9)
Successfully installed pyroma-4.1

and with a higher Wheel major version, it errors out immediately with a red warning:

$ pip install ~/Downloads/pyroma-4.1-py3-none-any/pyroma-4.1-py3-none-any.whl
Processing c:\users\c. a. m. gerlach\downloads\pyroma-4.1-py3-none-any\pyroma-4.1-py3-none-any.whl
ERROR: pyroma has an invalid wheel, pyroma's Wheel-Version (2.0) is not compatible with this version of pip

Are you suggesting something different here?

5 Likes

Nope, that seems perfectly sufficient. I just hadn’t tested it, and if anyone mentioned it in this thread then I missed it.

1 Like

A wheel 2.0 would be smaller and faster. In a previous thread (no longer hosted of course),

Since this idea represents exactly the same information (down to the individual ZipInfo objects in the nested zip, compared to even using a nested tar), this is like an encoding change - you could losslessly translate back to the old format and use the old tools and vice versa.

I don’t know how many people have trouble using the current version of wheel for size and speed reasons. But pypi’s bandwidth would get a large percentage of the benefit if the only wheel 2.0 package was tensorflow.

The less developed ideas would have something to do with representing new features, for example a more flexible installation scheme (fix the data-files feature) or maybe something to do with linking to system packages.

1 Like

I’d really like for this topic to not become a discussion of what we’d change in a wheel 2.0. We have other two topics for that discussion – let’s keep any details/discussion about what could change in those topics.


I think there’s consensus that:

  • 3 tooling categories are involved: installers, build backends and package indexes.
  • 2 user groups are involved: consumers and publishers.
  • installers need to be first within the tooling ecosystem to implement support.
  • For things to not be disruptive for consumers, they’ll need to be on a new-enough version of their installers to support installing from said wheel. The best way to deal with this is time – letting the newer version of the installers propagate through the ecosystem.

It’s not clear to me from context and this discussion which between build-backends vs package-indexes should come first. The obvious thing here is that the build-backends need to be able to build the new format before the publishers can publish it to the package index – but that doesn’t mean that they need to do it by default IMO. Personally, I’m fine with deferring to each build-backend to decide how they’ll handle the transition, and I imagine that all of them would change the default after the PyPI adds support; but that they can allow building Wheel 2.0 wheels before PyPI supports it (for other package indexes).

So… here’s more leading questions:

  1. How do folks feel about coupling the format change in build-backends + package-indexes with a Python version? :slight_smile:
  2. What is a reasonable amount of timeframe after the first release of (say) pip with wheel v2 support for a publisher to start publishing using it?

My thoughts:

  1. It’s… an interesting idea; especially if we do it after an year+ of the format being supported in installers. It gives us the new format organically when the “new” Python version is being bootstrapped, and there’s no major backwards-compatibility concerns (py3-none-any wheels are the only thing I can think of, and the answer there is for the publisher to either build with an older version of Python, or push users to upgrade their installers). I imagine that the installers would still support installing wheel v1 anyway though, so it’s not going to be a strong argument.
  2. @dholth said 1 year. I’m guessing 18 months. That said, seeing how pip versions roll out for PyPI (based on what-performed-request, not which-version-of-pip-was-downloaded) would be a good proxy/metric to guesstimate this.

Hmm… these error message should be updated – “invalid wheel” is the wrong language to use here.

I can – user shows up on the issue tracker (without mentioning that they’re using $lts-distro) that can’t use this open source project on it due to having an old version of pip; and project has a compelling reason to try and support them. Granted, the brute-force approach works here. :slight_smile:

As for the “type 1 change” no new features just compression wheel 2, you could have the build frontend build · PyPI do the format change. You could even point old pip at a local proxy that translated pypi’s wheel 2’s to local wheel 1’s, to be able to install wheel 2’s with old installers.

As for the “type 2 change” new features that cannot be represented in wheel 1, the build backend would need to care.

  1. -1 on this. It’s nothing to do with the Python version, we shouldn’t artificially tie it like this. Also, this would just add one more roadblock for projects releasing wheels for a new Python version in a timely manner - “we can’t release wheels for Python 3.12 yet, as our build backend doesn’t support Wheel 2.0”.
  2. I honestly have no idea. I don’t personally think it needs to be long, as we expect people to upgrade pip pretty eagerly. But I’ve already been told off once today for that view, so I’ll let the people who want to support their users on old pip versions give their views on this.

Here’s a question. Do we anticipate projects releasing multiple wheel versions? They wouldn’t be able to for a wheel 1.0 → 1.1 transition, as the filename would be the same. Do we want to require that the wheel version is part of the filename[1], so that multiple versions can be published? I’m personally -1 on such a requirement, even if we end up with 2.0 including a filename format change for other reasons.

I would personally much prefer to encourage, or even require, that projects only release one wheel version. PyPI will explode if something like pyAgrum-nightly started publishing twice as many wheels…


  1. Implied or explicit. ↩︎

1 Like

I expect we won’t change the file naming scheme (otherwise it won’t be a wheel 2.0, but a new format), so no.

So, it seems like we can do this, albeit with a migration timeline and some details to figure out still around the specifics of how we’d deal with the adoption curve/pain.

Oh, I just realised that I hadn’t pushed back on this idea. :sweat_smile:

I don’t know the distinction between technical features / non-technical features in a file format – but, I do feel that doing only things like compression changes or whatnot is a waste of the opportunity to, for example, add fix the weirdness around extras, or add default optional dependencies, or add metadata/semantics for GPU/CPU specialisation etc. I don’t wanna build a bucket list here (again, specifics of what we change can be discussed in other topics!) but doing a major version bump gives us the opportunity to add and change things, and it may be worthwhile to solve a few problems together.

That said, that path lies PEP 426/Python 3000 so we’d also want to be wary of trying a “let’s fix everything in one go”. :slight_smile:

Agreed. I’m on record as saying I’m confused by the distinction between major and minor versions in file format specifications, but if we take the view you stated elsewhere, that minor versions can only remove things, and all additions are major version bumps, then I think we need to accept that either we bundle as much as possible into Wheel 2.0, or we have rather a lot of major version bumps, one for each change (or group of changes) in your bucket list. I’m not sure which of those two options is the more palatable to me…

FYI Hatchling was designed with build targets changing over time in mind so it has the concept of versions already :slightly_smiling_face:

2 Likes

I’ve been enormously frustrated with what seems to be a “no incremental improvements” policy.

We could do pretty well with only zstandard (if we can get it past the stdlib) and symlinks (to avoid duplicated shared libraries to provide ld.so, ld.so.1, ld.so.1.1 aliases) and only on the 10 largest projects on pypi, we could probably save 50% of pypi’s total bandwidth (do the math?), some end users might like it too.

Some of the things you mentioned (extras) probably go in a metadata spec and would be independent of the wheel spec.

3 Likes

That’s an exceedingly strong word.

I have stated my opinion, which is not encoded in any sort of systemic manner to (a) enforce what directions others put in effort and (b) affect the decision making process beyond the “Pradyun has a different opinion to everyone else”. To call my opinion here a policy is a stretch.

If you were commenting on a broader trend you’re noticing, then… I don’t see it? Nearly every packaging PEP I’ve seen in the last few years has been oriented around being incremental changes and being non-disruptive. It’s certainly not written anywhere that incremental changes will be avoided (pypa.io does say something about focusing on incremental changes intentionally). I’m well aware that communities end up with informal encoded behaviours (i.e. things a community does but hasn’t written down formally) but, even so, it’s not a no incremental changes informal policy either (see start of para).

The wheel format is by design locked out of those kinds of incremental improvements and there’s a churn for the ecosystem that comes with each backwards-incompatible change roll-out.

Right, and that’s in line with what I said: just compression algorithm changes isn’t a particularly strong argument and having some other functionality improvements along with is what makes it compelling to do a major version bump.

If you think better compression is sufficient on its own and good-enough to roll out as a backwards-incompatible change, you have a different opinion to me and that’s fine. I don’t operate a package index to care about bandwidth but I agree that reducing it is a good thing.

The exact list will vary for people with different opinions, and I don’t want to get into a discussion about that here (please feel welcome to start a separate new thread to talk about what goes into wheel 2.0).

Taking a step back, I’ve gotten the answers I was looking for from this discussion. :slight_smile:

The rollout/migration for a new wheel version that changes the compression is a tractable problem, with a long tail and an unknown shape of the adoption curve. We’d likely have to wait on the order of a year or more for making the switch and pip/installer implementing+releasing support is what will start that counter.

As @dholth noted, certain approaches may be available, depending on whether we add features to the wheel 2.0 that aren’t equivalently translatable to wheel 1.0. Those can be discussed in context once there’s a discussion of what changes go into wheel 2.0.

Thanks for the discussion here folks!


To be clear, this isn’t meant to shut down further discussion but I wanted to note that I have gotten everything I was looking for from this (selfishly, for future me).

Note: @pradyunsg posted after I’d started this comment. And I agree with what he says, that the core question for this topic is answered. But I will post this anyway, as I do want to address the question of incremental changes.

tl;dr; I think we can (and should) allow certain types of new feature in minor version changes, and I’m very uncomfortable with the idea that all new features require a major version, as it pushes us towards a state where gradual improvement isn’t possible.

(Previously written message follows)

I think there is some truth in the idea that packaging changes (and in particular changes to the wheel format) are slowing down, and we’re losing some of the agility around smaller, incremental changes. I don’t think it’s a formal policy, but it is a consequence of our policies (the PEP process can be rather heavyweight for small changes, and we don’t really have a good policy on how to exempt “small enough” changes from needing a PEP).

I think we do have a need to be cautious about compatibility and breakages when we release changes, but we also have a responsibility to not let things stagnate. And to be perfectly honest, I think we are tending to be over-cautious these days. It would be nice if the work on a “Packaging Vision” that @smm is leading, could result in a clearer statement of how we expect to handle “legacy” project distributions, workflows, and formats, so that we could realistically make decisions on what sort of breakage is acceptable when proposing new standards.

I’m not entirely sure what type of “design” you mean when you say the wheel format is locked out of incremental improvements. Is it purely the statement that installers must warn on minor changes and error on major ones? Because if so, then we only have one wheel format right now, so now is the perfect time to change this - I’d happily approve (if people consider that it’s up to me to do so) a change to that statement as a “minor modification” to the spec, so we can just get it done.

If you have something different in mind for why the wheel spec is locked out of incremental changes, can you explain? And in particular, how is the wheel spec more problematic than (for example) the sdist spec, or pyproject.toml, or any of our other standards? Because if we’re inadvertantly designing standards that can’t be improved, let’s work out what we’re doing wrong and fix it!

To give a concrete example (so this isn’t all just handwaving :wink:), adding symlink support to wheels would need to cater for what installers must do if the target environment doesn’t support symlinks. Fail, presumably. So the failure modes are:

  1. Wheel contains symlinks
    a. No symlink support, new installer - install fails.
    b. OS symlink support, new installer - install succeeds.
    c. OS symlink support, old installer, major version bump - install fails.
    d. OS symlink support, old installer, minor version bump - install fails because the symlinks (noted in RECORD) are missing.
  2. Wheel does not contain symlinks. OS Symlink support is irrelevant
    a. New installer - install succeeds.
    b. Old installer, major version bump - install fails.
    c. Old installer, minor version bump - install succeeds with warning.

Of these scenarios, only (2b) seems bad to me. Which says this should be a minor version bump. So why can’t we do a minor version increase to add symlink support, and why do you say the wheel format is by design locked out of such incremental improvements?

The trade-offs are very different for changes where all new-format wheels are unreadable by old-format installers (where case (2) above does not exist). That is what I’d consider to be a major change, though, and is where we should focus on thinking about migration strategies (which is what I’d thought this topic was intended to be about, but we’ve drifted somewhat :slightly_frowning_face:)

To be clear, I’m not trying to say that you (@pradyunsg) are blocking incremental increments. But I do think that this whole discussion is tending to see the risks as more important than the benefits, and we’re becoming too risk-averse as a consequence. Although not many people are actively participating in the discussion here, so maybe the majority opinion is different.

6 Likes

Well, we also have an order(s?) of magnitude more existing-wheels in the ecosystem; it’d be a bad idea to have a lot of churn in this area compared to 5 years ago even. :slight_smile:

I’m not sure I follow how this will fail, TBH. Last I checked, this was a silent success.

Yes.

Right now, as I see it, we don’t have any way to indicate to the installer that this wheel needs symlink support and you should fail if you don’t have that and this wheel uses it. We can either blanket warn (i.e. unactionable / noisy warnings) or blanket fail (i.e. fail even if the new functionality isn’t being used).

It doesn’t really matter how much nicer we can make the experience on the newer versions; the failure modes with the older versions of installers is what locks us into certain behaviours.

Which… is expected? I mean you’re right but I don’t see how this could be different.

It’s basically about “hey, what’s the challenges and how do we deal with them?”. I’ve literally being insisting the whole time that we don’t talk about the “cool” new features we could add, and talk about the things that are not as “exciting” (like :sparkles: rollout schedules :sparkles: and :rainbow: version numbers :rainbow:).

That’s intentional – I couldn’t figure out a way for us to roll out changes incrementally that doesn’t have suboptimal semantics. Based on the discussion so far, I figure we need to either (a) change that or (b) realise that and operate with that understanding. You’ve just suggested (a) and I’ve been operating under the assumption that it’s not an option because people still install via pip 9. :slight_smile:


To throw a design out there, I’ve wondered if we should add a Wheel-Features-Needed header to indicate that a wheel needs support for a certain feature (eg: symlink support) to enable rolling out changes such that only wheels that use a new feature would fail on an old installer; and that might be feasible to do without a version bump to the format (or make that a part of the major version bump; idk).

Just double-checked and having extra entries in RECORD is not an error with pip.

❯ unzip -l dist/livereload-2.6.3-py2.py3-none-any.whl                              
Archive:  dist/livereload-2.6.3-py2.py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      366  12-21-2022 20:45   livereload/__init__.py
     1381  12-21-2022 20:45   livereload/cli.py
     6699  12-21-2022 20:45   livereload/handlers.py
        0  12-21-2022 21:04   livereload/management/
        0  12-21-2022 21:04   livereload/management/commands/
        0  07-22-2022 22:41   livereload/management/commands/__init__.py
     1720  12-21-2022 20:45   livereload/management/commands/livereload.py
    12884  12-21-2022 20:45   livereload/server.py
        0  12-21-2022 21:04   livereload/vendors/
    37411  12-21-2022 20:45   livereload/vendors/livereload.js
     7726  12-21-2022 20:45   livereload/watcher.py
     1510  12-21-2022 20:46   livereload-2.6.3.dist-info/LICENSE
     7388  12-21-2022 20:46   livereload-2.6.3.dist-info/METADATA
     1216  12-21-2022 21:04   livereload-2.6.3.dist-info/RECORD
      110  12-21-2022 20:46   livereload-2.6.3.dist-info/WHEEL
       51  12-21-2022 20:46   livereload-2.6.3.dist-info/entry_points.txt
       11  12-21-2022 20:46   livereload-2.6.3.dist-info/top_level.txt
---------                     -------
    78473                     17 files

❯ unzip -p dist/livereload-2.6.3-py2.py3-none-any.whl livereload-2.6.3.dist-info/RECORD
livereload/__init__.py,sha256=tWkNElZzfhLSZkrHP6BgcG7mzPYVB4SPYqFZ8GM-3sI,366
livereload/cli.py,sha256=Kpq7ixzxDI2vM9vZxiJGO7sjKvt4K_BJJJi8l3kwfkU,1381
livereload/handlers.py,sha256=jXsQWXIpQXM7sl2Dh6f-RsZhDMoL_qCfOdTBRpTf3dc,6699
livereload/server.py,sha256=kCekDMLBb3r3oYvek4pjfrtG501QexCQWtMPaLQkuFk,12884
livereload/watcher.py,sha256=pxRq2FOdiXpFoHP1UfzkdsHD-ykyyJG9BSHtEVYnDIQ,7726
livereload/management/commands/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
livereload/management/commands/livereload.py,sha256=4t_lgabMDdHHu8L4C-z_Ty_ApP2w93deFUUrhdTsjFI,1720
livereload/this-file-does-not-exist.txt,,
livereload/vendors/livereload.js,sha256=AvKz3V9PnpyGqifSzRRfVNKPSJGJml6D-8SZH90huCY,37411
livereload-2.6.3.dist-info/LICENSE,sha256=I8xrY0r-MxIhFTGWIizqnKoSaJdhCm-3iUU2cbe4wts,1510
livereload-2.6.3.dist-info/METADATA,sha256=qU9KyFhJaxTxX0960qvr-YLQ-NpWof_bKqHI7nhnFfE,7388
livereload-2.6.3.dist-info/WHEEL,sha256=bb2Ot9scclHKMOLDEHY6B2sicWOgugjFKaJsT7vwMQo,110
livereload-2.6.3.dist-info/entry_points.txt,sha256=3YgxN8AYXhj9NgTXLUPCtjbZISosc_WujYztH4sfKd8,51
livereload-2.6.3.dist-info/top_level.txt,sha256=UV3wC1nFjYsFOIbaCNjQpT5p1ueRUQV2meP2BNUplFk,11
livereload-2.6.3.dist-info/RECORD,,

(notice livereload/this-file-does-not-exist.txt,,)

❯ pip install dist/livereload-2.6.3-py2.py3-none-any.whl --no-deps -vvv
Using pip 22.3.1 from /Users/pradyunsg/Developer/github/python-livereload/.venv/lib/python3.11/site-packages/pip (python 3.11)
Non-user install because user site-packages disabled
Created temporary directory: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h
Initialized build tracking at /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h
Created build tracker: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h
Entered build tracker: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h
Created temporary directory: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-install-p33a_hym
Created temporary directory: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-ephem-wheel-cache-ciwp6fx1
Processing ./dist/livereload-2.6.3-py2.py3-none-any.whl
  Added livereload==2.6.3 from file:///Users/pradyunsg/Developer/github/python-livereload/dist/livereload-2.6.3-py2.py3-none-any.whl to build tracker '/private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h'
  Removed livereload==2.6.3 from file:///Users/pradyunsg/Developer/github/python-livereload/dist/livereload-2.6.3-py2.py3-none-any.whl from build tracker '/private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h'
Created temporary directory: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-unpack-hce6ld60
Installing collected packages: livereload

  changing mode of /Users/pradyunsg/Developer/github/python-livereload/.venv/bin/livereload to 755
Successfully installed livereload-2.6.3
Remote version of pip: 22.3.1
Local version of pip:  22.3.1
Was pip installed by pip? True
Removed build tracker: '/private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h'

IMO that’s a bug in pip. It certainly doesn’t count as the wheel format being “by design locked out of those kinds of incremental improvements”.