PEP 777: How to Re-invent the Wheel

[emmatyping (Emma Smith)]

I view the problem of future wheel compatibility as a two stage process:

Stage 1: How to introduce the first breaking change without breaking too much

and

Stage 2: how to make it easy (or at least, as easier) to bump the wheel major version to add independent new features to the wheel format

To me, the question of whether to rename the wheel file extension is pertinent to stage 1. However, I’m going to punt on that because stage 2 is probably more important long term. Defining how to handle upgrading the wheel version is central to making both stages work well.

So let’s focus on just the question of installer behavior for now.

One of the central arguments to having installers raise an error on unrecognized wheel versions is that users should be able to easily upgrade their installer to resolve the error, which minimizes the impact. After a lot of thought on this, I’ve come to the conclusion that either this greatly limits what future wheel versions can do, or is just not going to be the case in the future.

Here are a couple of examples of where a user cannot simply upgrade their installer to avoid an error:

1. The wheel being installed uses symlinks (and requires them!), yet the filesystem on the user’s machine, e.g. NFS, does not support symlinks.
2. The wheel being installed uses lzip (for arguments sake), an lzma based compression format, but the user did not build their CPython with lzma.

In both of these cases, the installer can be impacted by features (or lack thereof) in the user’s environment. A user cannot simply upgrade their way out of seeing an error with these wheels.

So how do we resolve this? One proposed workaround would be to enhance repository metadata to indicate how fallbacks should be handled. But this doesn’t work if I’m invoking pip --find-links with a directory containing a mix of wheels. It also doesn’t handle situations where users have multiple indices. How do we handle the instance where there are 2.0 wheels in some 3rd party index, but 1.0 wheels on PyPI?

For what it’s worth, I originally agreed that not aborting was a problem:

@emmatyping

I do worry that if we go down the route of “try not to error up front and instead try to get existing installers to choose old wheels or sdists” that we will end up with a manylinux scenario, where users are confused about why they are doing a source build or getting an older version. At minimum, I think in any world where pip considers Wheel-Version in selection, it should at least warn if it comes across a major version it is incompatible with.

But as I’ve thought about various future wheel changes people want, I have come to believe that the wheel compatibility story must be handled by revolvers. We already have a robust mechanism by which tools select wheels that are compatible with your environment, it makes sense to me we should leverage that when talking about selecting wheels compatible with your installation environment, which can be influenced by your runtime environment.

I’ve also seen people suggest that we could define weaker versions of the standards, such that symlinks be an optional feature. But that isn’t realistically workable either in my opinion. If I ship some libraries that I want users to be able to load at runtime but also link to dynamically, I really want both libfoo.so and libfoo.so.1 in my site-packages, and I want to be able to rely on that. If I can’t, I have to include patches to my build process or loading code, which I’d have to do without symlinks anyway, so I win very little. Features that depend on the operating environment should be able to be mandatory, and I think we would limit ourselves otherwise.

---

I also think we can mitigate concerns about installers ignoring unrecognized/incompatible wheel versions.

Regarding sdists, we can either solve the problem wholesale (e.g. the discussions in Provide a way to signal an sdist isn't meant to be built?) or simply specify in PEP 777 that if incompatible wheels were skipped for a given version, resolvers should skip sdists of the same version. This will force resolvers to backtrack to an older version that should have a compatible wheel, else the installer can indicate that no compatible wheels were found and suggest trying to upgrade as a resolution (or some other resolution dependent on the feature).

I also think with a big loud warning about skipped wheels suggesting people upgrade, people will upgrade when they need to.

@steve.dower

And we don’t need to drop 1.0 wheels at all, particularly, except to force users to migrate forward (which I don’t like, either as a user or as a standards contributor).

I wholeheartedly agree, I don’t think we should expect everyone to be using the same wheel version, or even encourage it. We practically can’t anyway, since there are 12 million and counting files on PyPI today that aren’t going to be upgraded