Ways to update pip (and when updating is a good idea)

Packaging
1

Off-topic question from a pip maintainer here - why isn’t pip’s self check message enough to tell users to keep their copy of pip up to date? We don’t support anything but the latest version of pip, so it’s important to us that people get the message about keeping their copy of pip up to date…

2

I think a lot of users probably don’t realize that. Especially considering it doesn’t auto-update, an old version will commonly be bootstrapped into venvs, the Internet is swamped with existing information about fixing problems including examples in older versions…

If the default behaviour were to do the self-check first, and also offer to self-update before doing what was requested, I bet you’d see much higher rates of updating. (Although you’d also probably see a lot more complaints…)

1 Like
3

I also usually recommend users pin their copy of pip in CI (usually to whichever is installed by default) so that new pip releases don’t cause CI failures. Updating the version then requires a PR, which could fail, but in the context of “is pip about to fail” so it’s somewhat expected.

However, it requires effort, and so doesn’t get done that often :slight_smile:

1 Like
4

In general, I think it’s reasonable for people who have ever gotten their Python environment in a bad setup (so, most people using Python packages :slight_smile: ) to be conservative in what they change. If it looks like things are installing fine with the version of pip they already have, why risk it?

Concretely:

  • At my old day job, we build basically the entire standard scientific/deep-learning Python stack from source at pinned versions, ship it in a read-only Python environment, and let users build a system-site-packages venv on top of that. This means they get a base environment that we’ve tested and hopefully dealt with all the weird binary compatibility problems you can get, but you can also pip install things without having to bother anyone if you just want to try things out. In that setup I think it’s perfectly reasonable for people to try to leave packages in the base environment alone. As a specific example I recently had a user do a pip install -U torch and get a wheel that needed a different CUDA version than the one our environment had, and they were very easily convinced that they could use the slightly older version of PyTorch in the base environment. So people learn to leave packages alone, which naturally includes pip. (I could be convinced we’re doing our users a disservice but I don’t think we are.)
  • Also our users will do reproducible research by picking one of our base environments and sticking with that for several months.
  • I think there’s an analogous argument if you get pip from your OS. For me personally, if I’m making a venv based on Apple’s /usr/bin/python3, I’m usually not going to upgrade pip unless I know I need to, because I haven’t thought very hard about how Apple’s build of Python works and I’d prefer not to think about it.
  • Should you upgrade pip globally or in a venv? In general we recommend only doing stuff inside a venv, no? So if you want a newer pip, you’re going to version/lock pip for your project in the way you lock everything else, which means you’re not always getting the newest version.
  • Also that means that the first time you run pip install in a venv it’s an older version. Yes, you can do pip install -U pip; pip install -r requirements.txt but people don’t usually do that.

One thing that might help is to add something in pip now to specifically encourage you to upgrade pip if it sees a wheel 2.0. Right now if I hack one up I get

$ bin/pip install attrs-23.2.0-py3-none-any.whl 
Processing ./attrs-23.2.0-py3-none-any.whl
ERROR: attrs has an invalid wheel, attrs's Wheel-Version (2.0) is not compatible with this version of pip

and I’m not sure how it handles wheel 2.0 on an index - does it produce this error or does it silently install an older version with a wheel 1.0, the way it does for unknown platform tags?

EDIT: It does produce the same error on an index, and moreover if you’re not running the latest version of pip, the version check happens right after this error, so I think this is actually doing exactly what is needed. Cool!

3 Likes
5

I think it might be a good idea to split this topic out into its own thread. I do have ideas about it, but I don’t want to muddle the current conversation.

2 Likes
6

Self-updating is problematic because it fails on Windows if you do pip install -U pip using the wrapper. And ensurepip (which is what bootstraps into venvs) uses a fixed (often older) version because the core devs wanted a policy of not needing an internet connection to run ensurepip. With my pip maintainer hat on I’m not sure I agree with that policy, but I don’t have the power to change it.

CI is different, and I agree with your recommendation (as long as it’s coupled with “but when a new version of pip comes out, test it against your CI and update your pin as soon as you know the new version doesn’t cause you problems”).

OK. This is a side issue, so I don’t want to derail things too much. I hear what people are saying, and I accept it’s a reality. But it remains true that you won’t get support (beyond “please try again with the current version of pip”) from the pip maintainers for older versions.

As far as this PEP is concerned, I’m fine if you want to make sure it “fails safe” by doing something sensible with older versions of pip. As you say, people are using such versions in the wild, and we can’t ignore that fact.

You need to remember that we’re no longer in a situation where pip is the only installer. People could be using uv. But in general, I don’t know what you intend in the way of a preparatory change to pip, but I’m mildly against it. We really don’t want to add support for PEPs that might be approved in the future.

I did a test. If there’s wheels for test version 0.2 (wheel version 2) and test version 0.1 (wheel version 1) then both pip install test and uv pip install test will fail with an error saying that the wheel version is invalid/incompatible. It will not install either version of test.

So as soon as any project publishes a version 2.0 wheel, all current versions of both uv and pip will stop being able to install that project.

That doesn’t sound ideal…

But it is essentially what the current wheel spec says - “Check that installer is compatible with Wheel-Version. Warn if minor version is greater, abort if major version is greater”. So the PEP introducing Wheel 2.0 needs to come up with a transition plan that handles this.

7

Yes, that’s one of the main sources of potential complaints I imagined :wink:

Would it work to make it possible to opt in to updating the bootstrap source, when one does have a connection? Then future venv creation wouldn’t require it, but could still get updated (up to that point) Pip even without the Internet.

1 Like