Speculative: Wheel 2.0 and migration strategies

That’s an exceedingly strong word.

I have stated my opinion, which is not encoded in any sort of systemic manner to (a) enforce what directions others put in effort and (b) affect the decision making process beyond the “Pradyun has a different opinion to everyone else”. To call my opinion here a policy is a stretch.

If you were commenting on a broader trend you’re noticing, then… I don’t see it? Nearly every packaging PEP I’ve seen in the last few years has been oriented around being incremental changes and being non-disruptive. It’s certainly not written anywhere that incremental changes will be avoided (pypa.io does say something about focusing on incremental changes intentionally). I’m well aware that communities end up with informal encoded behaviours (i.e. things a community does but hasn’t written down formally) but, even so, it’s not a no incremental changes informal policy either (see start of para).

The wheel format is by design locked out of those kinds of incremental improvements and there’s a churn for the ecosystem that comes with each backwards-incompatible change roll-out.

Right, and that’s in line with what I said: just compression algorithm changes isn’t a particularly strong argument and having some other functionality improvements along with is what makes it compelling to do a major version bump.

If you think better compression is sufficient on its own and good-enough to roll out as a backwards-incompatible change, you have a different opinion to me and that’s fine. I don’t operate a package index to care about bandwidth but I agree that reducing it is a good thing.

The exact list will vary for people with different opinions, and I don’t want to get into a discussion about that here (please feel welcome to start a separate new thread to talk about what goes into wheel 2.0).

Taking a step back, I’ve gotten the answers I was looking for from this discussion. :slight_smile:

The rollout/migration for a new wheel version that changes the compression is a tractable problem, with a long tail and an unknown shape of the adoption curve. We’d likely have to wait on the order of a year or more for making the switch and pip/installer implementing+releasing support is what will start that counter.

As @dholth noted, certain approaches may be available, depending on whether we add features to the wheel 2.0 that aren’t equivalently translatable to wheel 1.0. Those can be discussed in context once there’s a discussion of what changes go into wheel 2.0.

Thanks for the discussion here folks!


To be clear, this isn’t meant to shut down further discussion but I wanted to note that I have gotten everything I was looking for from this (selfishly, for future me).

Note: @pradyunsg posted after I’d started this comment. And I agree with what he says, that the core question for this topic is answered. But I will post this anyway, as I do want to address the question of incremental changes.

tl;dr; I think we can (and should) allow certain types of new feature in minor version changes, and I’m very uncomfortable with the idea that all new features require a major version, as it pushes us towards a state where gradual improvement isn’t possible.

(Previously written message follows)

I think there is some truth in the idea that packaging changes (and in particular changes to the wheel format) are slowing down, and we’re losing some of the agility around smaller, incremental changes. I don’t think it’s a formal policy, but it is a consequence of our policies (the PEP process can be rather heavyweight for small changes, and we don’t really have a good policy on how to exempt “small enough” changes from needing a PEP).

I think we do have a need to be cautious about compatibility and breakages when we release changes, but we also have a responsibility to not let things stagnate. And to be perfectly honest, I think we are tending to be over-cautious these days. It would be nice if the work on a “Packaging Vision” that @smm is leading, could result in a clearer statement of how we expect to handle “legacy” project distributions, workflows, and formats, so that we could realistically make decisions on what sort of breakage is acceptable when proposing new standards.

I’m not entirely sure what type of “design” you mean when you say the wheel format is locked out of incremental improvements. Is it purely the statement that installers must warn on minor changes and error on major ones? Because if so, then we only have one wheel format right now, so now is the perfect time to change this - I’d happily approve (if people consider that it’s up to me to do so) a change to that statement as a “minor modification” to the spec, so we can just get it done.

If you have something different in mind for why the wheel spec is locked out of incremental changes, can you explain? And in particular, how is the wheel spec more problematic than (for example) the sdist spec, or pyproject.toml, or any of our other standards? Because if we’re inadvertantly designing standards that can’t be improved, let’s work out what we’re doing wrong and fix it!

To give a concrete example (so this isn’t all just handwaving :wink:), adding symlink support to wheels would need to cater for what installers must do if the target environment doesn’t support symlinks. Fail, presumably. So the failure modes are:

  1. Wheel contains symlinks
    a. No symlink support, new installer - install fails.
    b. OS symlink support, new installer - install succeeds.
    c. OS symlink support, old installer, major version bump - install fails.
    d. OS symlink support, old installer, minor version bump - install fails because the symlinks (noted in RECORD) are missing.
  2. Wheel does not contain symlinks. OS Symlink support is irrelevant
    a. New installer - install succeeds.
    b. Old installer, major version bump - install fails.
    c. Old installer, minor version bump - install succeeds with warning.

Of these scenarios, only (2b) seems bad to me. Which says this should be a minor version bump. So why can’t we do a minor version increase to add symlink support, and why do you say the wheel format is by design locked out of such incremental improvements?

The trade-offs are very different for changes where all new-format wheels are unreadable by old-format installers (where case (2) above does not exist). That is what I’d consider to be a major change, though, and is where we should focus on thinking about migration strategies (which is what I’d thought this topic was intended to be about, but we’ve drifted somewhat :slightly_frowning_face:)

To be clear, I’m not trying to say that you (@pradyunsg) are blocking incremental increments. But I do think that this whole discussion is tending to see the risks as more important than the benefits, and we’re becoming too risk-averse as a consequence. Although not many people are actively participating in the discussion here, so maybe the majority opinion is different.

6 Likes

Well, we also have an order(s?) of magnitude more existing-wheels in the ecosystem; it’d be a bad idea to have a lot of churn in this area compared to 5 years ago even. :slight_smile:

I’m not sure I follow how this will fail, TBH. Last I checked, this was a silent success.

Yes.

Right now, as I see it, we don’t have any way to indicate to the installer that this wheel needs symlink support and you should fail if you don’t have that and this wheel uses it. We can either blanket warn (i.e. unactionable / noisy warnings) or blanket fail (i.e. fail even if the new functionality isn’t being used).

It doesn’t really matter how much nicer we can make the experience on the newer versions; the failure modes with the older versions of installers is what locks us into certain behaviours.

Which… is expected? I mean you’re right but I don’t see how this could be different.

It’s basically about “hey, what’s the challenges and how do we deal with them?”. I’ve literally being insisting the whole time that we don’t talk about the “cool” new features we could add, and talk about the things that are not as “exciting” (like :sparkles: rollout schedules :sparkles: and :rainbow: version numbers :rainbow:).

That’s intentional – I couldn’t figure out a way for us to roll out changes incrementally that doesn’t have suboptimal semantics. Based on the discussion so far, I figure we need to either (a) change that or (b) realise that and operate with that understanding. You’ve just suggested (a) and I’ve been operating under the assumption that it’s not an option because people still install via pip 9. :slight_smile:


To throw a design out there, I’ve wondered if we should add a Wheel-Features-Needed header to indicate that a wheel needs support for a certain feature (eg: symlink support) to enable rolling out changes such that only wheels that use a new feature would fail on an old installer; and that might be feasible to do without a version bump to the format (or make that a part of the major version bump; idk).

Just double-checked and having extra entries in RECORD is not an error with pip.

❯ unzip -l dist/livereload-2.6.3-py2.py3-none-any.whl                              
Archive:  dist/livereload-2.6.3-py2.py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      366  12-21-2022 20:45   livereload/__init__.py
     1381  12-21-2022 20:45   livereload/cli.py
     6699  12-21-2022 20:45   livereload/handlers.py
        0  12-21-2022 21:04   livereload/management/
        0  12-21-2022 21:04   livereload/management/commands/
        0  07-22-2022 22:41   livereload/management/commands/__init__.py
     1720  12-21-2022 20:45   livereload/management/commands/livereload.py
    12884  12-21-2022 20:45   livereload/server.py
        0  12-21-2022 21:04   livereload/vendors/
    37411  12-21-2022 20:45   livereload/vendors/livereload.js
     7726  12-21-2022 20:45   livereload/watcher.py
     1510  12-21-2022 20:46   livereload-2.6.3.dist-info/LICENSE
     7388  12-21-2022 20:46   livereload-2.6.3.dist-info/METADATA
     1216  12-21-2022 21:04   livereload-2.6.3.dist-info/RECORD
      110  12-21-2022 20:46   livereload-2.6.3.dist-info/WHEEL
       51  12-21-2022 20:46   livereload-2.6.3.dist-info/entry_points.txt
       11  12-21-2022 20:46   livereload-2.6.3.dist-info/top_level.txt
---------                     -------
    78473                     17 files

❯ unzip -p dist/livereload-2.6.3-py2.py3-none-any.whl livereload-2.6.3.dist-info/RECORD
livereload/__init__.py,sha256=tWkNElZzfhLSZkrHP6BgcG7mzPYVB4SPYqFZ8GM-3sI,366
livereload/cli.py,sha256=Kpq7ixzxDI2vM9vZxiJGO7sjKvt4K_BJJJi8l3kwfkU,1381
livereload/handlers.py,sha256=jXsQWXIpQXM7sl2Dh6f-RsZhDMoL_qCfOdTBRpTf3dc,6699
livereload/server.py,sha256=kCekDMLBb3r3oYvek4pjfrtG501QexCQWtMPaLQkuFk,12884
livereload/watcher.py,sha256=pxRq2FOdiXpFoHP1UfzkdsHD-ykyyJG9BSHtEVYnDIQ,7726
livereload/management/commands/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
livereload/management/commands/livereload.py,sha256=4t_lgabMDdHHu8L4C-z_Ty_ApP2w93deFUUrhdTsjFI,1720
livereload/this-file-does-not-exist.txt,,
livereload/vendors/livereload.js,sha256=AvKz3V9PnpyGqifSzRRfVNKPSJGJml6D-8SZH90huCY,37411
livereload-2.6.3.dist-info/LICENSE,sha256=I8xrY0r-MxIhFTGWIizqnKoSaJdhCm-3iUU2cbe4wts,1510
livereload-2.6.3.dist-info/METADATA,sha256=qU9KyFhJaxTxX0960qvr-YLQ-NpWof_bKqHI7nhnFfE,7388
livereload-2.6.3.dist-info/WHEEL,sha256=bb2Ot9scclHKMOLDEHY6B2sicWOgugjFKaJsT7vwMQo,110
livereload-2.6.3.dist-info/entry_points.txt,sha256=3YgxN8AYXhj9NgTXLUPCtjbZISosc_WujYztH4sfKd8,51
livereload-2.6.3.dist-info/top_level.txt,sha256=UV3wC1nFjYsFOIbaCNjQpT5p1ueRUQV2meP2BNUplFk,11
livereload-2.6.3.dist-info/RECORD,,

(notice livereload/this-file-does-not-exist.txt,,)

❯ pip install dist/livereload-2.6.3-py2.py3-none-any.whl --no-deps -vvv
Using pip 22.3.1 from /Users/pradyunsg/Developer/github/python-livereload/.venv/lib/python3.11/site-packages/pip (python 3.11)
Non-user install because user site-packages disabled
Created temporary directory: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h
Initialized build tracking at /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h
Created build tracker: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h
Entered build tracker: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h
Created temporary directory: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-install-p33a_hym
Created temporary directory: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-ephem-wheel-cache-ciwp6fx1
Processing ./dist/livereload-2.6.3-py2.py3-none-any.whl
  Added livereload==2.6.3 from file:///Users/pradyunsg/Developer/github/python-livereload/dist/livereload-2.6.3-py2.py3-none-any.whl to build tracker '/private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h'
  Removed livereload==2.6.3 from file:///Users/pradyunsg/Developer/github/python-livereload/dist/livereload-2.6.3-py2.py3-none-any.whl from build tracker '/private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h'
Created temporary directory: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-unpack-hce6ld60
Installing collected packages: livereload

  changing mode of /Users/pradyunsg/Developer/github/python-livereload/.venv/bin/livereload to 755
Successfully installed livereload-2.6.3
Remote version of pip: 22.3.1
Local version of pip:  22.3.1
Was pip installed by pip? True
Removed build tracker: '/private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-tracker-weypz10h'

IMO that’s a bug in pip. It certainly doesn’t count as the wheel format being “by design locked out of those kinds of incremental improvements”.

TBH, I’m not sure this discussion is going in a productive direction. I think it’s ultimately down to whoever writes a PEP for a new wheel feature to deal with questions like this, and I’m sure that any subsequent discussion will cover questions about versioning, migration, etc., in endless depth…

Trying to speculate any further on this in the absence of a concrete proposal just feels frustrating to me, so I’m going to drop out of this discussion now.

I am disappointed at how much pushback there seems to be over trying to add long-requested features to the wheel format like symlink support. I was pretty close to working on a PEP for it myself, but frankly I don’t think I have the energy for it after this discussion, so I’ll leave it to someone else to pick it up.

2 Likes

I have noted your suggestion regarding legacy projects. I will add the topic to the agenda.

1 Like