How to reinvent the wheel

[emmatyping (Emma Smith)]

@pf_moore

I think the first thing we need to look at is how things work today. That imposes some rather strong (and not ideal) constraints on any solution.

• If pip (and uv) chooses to install a particular version of a package, and the selected wheel is greater than version 1.0, the install fails. The installer doesn’t backtrack, or try other versions or other wheels.

Thanks Paul! This is exactly the kind of thing I’m looking to discuss in this thread.

I figured pip didn’t take into consideration the wheel major version when resolving packages but it’s good to confirm that.

I do worry that if we go down the route of “try not to error up front and instead try to get existing installers to choose old wheels or sdists” that we will end up with a manylinux scenario, where users are confused about why they are doing a source build or getting an older version. At minimum, I think in any world where pip considers Wheel-Version in selection, it should at least warn if it comes across a major version it is incompatible with.

@pf_moore

@steve.dower

So basically, I think anything that doesn’t change the nature of installing the package itself (i.e. the ZIP extract into site-packages [1]), but would otherwise provide the installer more useful information about what else to do, could pass as a minor update to the spec.

Thanks. I’m inclined to wonder if symlinks could fit this form. Stage one, require copies rather than symlinks as now, but add a LINKS file that installers can use to replace all but one of the copies with symlinks. Stage two, make LINKS authoritative and allow omission of the copies. Stage three, allow links that don’t simply point to copies within the wheel.

I am somewhat leaning towards it being better to surface the issue of an out of date installer to the user sooner rather than later. I don’t want a 2 → 3 scenario but I also don’t want confusing breakage or unexpected package resolution. We could recommend projects to do a version bump if starting to use wheel 2.0 wheels so users can at least constrain the version to below the version they start using those wheels if their installer still doesn’t support it.

@pf_moore

But the feedback we got on the symlinks-in-wheels thread is that this isn’t the reality. So someone will need to make an assessment of what we should really use as a transition timescale.

I think one easy improvement would be to have pip add a call to action in the error message about major version. Right now it says “the wheel major version is higher than pip can handle” but if a user doesn’t know the implications of wheel 1.0 vs 2.0, they have no path to fixing the error. Explicitly saying that updating pip should fix this issue will get people to update more quickly.

I also think the update check could be made more assertive. The current warning is

WARNING: You are using pip version 21.2.3; however, version 24.0 is available.
You should consider upgrading via...

This does not express any urgency to updating in my reading, other than the “warning.” It reads to me like “oh hey we have a newer version, you might want to try it out!” But not “we highly recommend upgrading to the latest version to avoid issues.”

@pf_moore

The alternative is to simply publish the wheel 2.0 spec, let tools and indexes implement support for it, and manage the damage by getting users to upgrade as needed. But this feels to me like the sort of “big bang, break the world” approach that the Python 2-to-3 transition took, and I doubt users will have the appetite for anything like this (I know that I don’t, personally )

I absolutely agree we don’t want to just change things and break them. If we do the “upgrade the spec with several new features” I would include a multi-year plan of how to handle the migration trying to avoid breakage as much as possible.

@pf_moore

@rgommers

But either way it’d be great to commit to a wheel 2.0 version, conditional on enough features (two? symlinks and better compression are two great candidates already) and an acceptable rollout plan.

Agreed, with one proviso, that one of the features we include has to be a mechanism to ensure that future version bumps are not this painful.

I’d also like to do this in wheel 2.0, glad we are on the same page here. I think the hard part will be figuring out how to avoid having to do this in future releases.

Fundamentally, any feature that adds additional requirements on installer behavior is a “breaking change” in the regard that if we don’t bump the major version, installers would silently do the wrong thing.

@steve.dower

Back on the “how do we avoid this in future”, I think we probably need a “critical features” mechanism in the wheel metadata. So a particular wheel can say “I use symlink and it’s critical” or “I use symlink and it’s optional”, and the installer can fail or warn based on a specific feature. Wheels that don’t use links won’t declare anything.

Hm, how do build backends decide if something is optional or required? If it isn’t used then the wheel probably doesn’t need to declare the feature all, so I’m curious what an optional requirement on a feature might look like.