How should virtualenv behave respective of seed packages (e.g. pip) by default

In Fedora, the system-installed virtualenv will use the system-installed version of the seed packages (which are available as wheels on the filesystem). So, it’s a bit like the “middle ground” option, except updates are handled system-wide, so from virtualenv’s point of view it looks like “embedded wheel”.
I don’t see this changing. I expect system-installed software to not download and install stuff from the 'net – unless it’s a tool that clearly needs to do that, like pip itself.
Can that be possible without patching/breaking things too much?

1 Like

The introduction of this feature will likely mean the addition of a auto-update default value for the seed packages (see https://virtualenv.pypa.io/en/latest/cli_interface.html#pip), so you can probably patch that to bundle?

Definitely. The patch will then be this, adding /usr/share/python-wheels to the wheel search path, and removing the bundled wheels.

If you can have a note in documentation that the flag’s default may vary in distros, it could reduce user confusion.

It’s the same on Debian and that’d be my concern too. Our patching of virtualenv is now pretty minimal and I hope we can keep it that way.

Well as you can see above @dstufft really does not want any distributions to do any patching and views it as bad for the ecosystem :man_shrugging:

I couldn’t find the argument above :‍(
Which ecosystem are we talking about here? Python’s, virtualenv’s, or the distro’s?

And then at https://github.com/pypa/virtualenv/issues/1821#issuecomment-625895863

Though with the upgrade model of Fedora this is less pronounced than Debian.

Agile development and frequent releases are great, until the point where users don’t like it (which turns out is most non-web apps and also many web apps).

Luckily for most of our community, there are other people who worry about that for us :slightly_smiling_face: And we’re all here right now discussing it.

So, given that the upstream projects generally consider a version dead after the next is released, what do the distributors require in order to more efficiently maintain the prior versions so they largely keep working? Off the top of my head:

  • patchable configuration (as mentioned by Petr)
  • roadmap/forward notification of changes (e.g. changing compression in pip)
  • backportable patches
  • …?

Projects of course have no obligation to provide these, but then distributions have no obligation to include projects, so it’s best to find the middle ground that works for everyone.

In this case, a patchable install command for putting pip into a new virtualenv seems like a good way forward (and also helpful for testing :wink:).

Addressing pip’s frequent release rate is a big and tricky issue, I know. I’d be interested to hear the distro’s concerns about updating (hypothetically) pip far more frequently, so at least their users who install updates can get them. If the reason is time/resources, or if it’s stability, or something else, there’s likely a way the upstream project can help and hence ensure that they aren’t overly prevented from innovating because of the distribution network (I’d say the same applies to cloud and CI providers updating preconfigured VM images or PaaS images).

2 Likes

Created a rough PR for this: https://github.com/pypa/virtualenv/pull/1831

The goal of the PR is to try to pull in new pip versions, however:

  • do not degrade virtual environment creation performance significantly,
  • do not trigger during CI runs (the feature is aimed mostly at end-users, that install virtualenv themselves).

The proposed change would make pip upgrade periodically:

  • every two weeks try to upgrade pip to a newer version

  • the upgrade runs not inline, but instead as a new background process (this makes the performance overhead not significant)

  • upgrade the seeded pip to a new version if there’s a newer version that satisfies the following criteria

    • was discovered at least an hour ago (this is to ensure that CI environments don’t start with an old version and mid-way through they switch to a newer version)
    • was released at least 28 days ago (allow 4 weeks for pip to do a bugfix release).

I consider this a reasonable middle-ground for the outcome of the poll. Outcome of the vote as of writing this is 42 votes: 45% always install the bundled one, 38% periodically upgrade (every two week - upgrade 10 days older), 17% download should be true by default. We will mostly use bundled versions (and almost always in the CI), but for end-users, once a pip version is out there for a long time we do switch to it, as we consider that a proof of stability.

People who would like to always get the latest version should either set the environment variable VIRTUALENV_DOWNLOAD=1 or set download inside the virtualenv.ini file (or pass in the --download flag).

People who would like to always pin to a given version should either set the environment variable VIRTUALENV_PIP=20.1 or set pip inside the virtualenv.ini file (or pass in the --pip flag).

4 Likes

A brief update on this: due to unforeseen events I’ll step back from this topic and leave it to the other maintainers to address it as they see it fit.