In Fedora, the system-installed virtualenv will use the system-installed version of the seed packages (which are available as wheels on the filesystem). So, it’s a bit like the “middle ground” option, except updates are handled system-wide, so from virtualenv’s point of view it looks like “embedded wheel”.
I don’t see this changing. I expect system-installed software to not download and install stuff from the 'net – unless it’s a tool that clearly needs to do that, like pip itself.
Can that be possible without patching/breaking things too much?
1 Like
The introduction of this feature will likely mean the addition of a auto-update default value for the seed packages (see https://virtualenv.pypa.io/en/latest/cli_interface.html#pip), so you can probably patch that to bundle?
Definitely. The patch will then be this, adding /usr/share/python-wheels to the wheel search path, and removing the bundled wheels.
If you can have a note in documentation that the flag’s default may vary in distros, it could reduce user confusion.
It’s the same on Debian and that’d be my concern too. Our patching of virtualenv is now pretty minimal and I hope we can keep it that way.
Well as you can see above @dstufft really does not want any distributions to do any patching and views it as bad for the ecosystem 
I couldn’t find the argument above :(
Which ecosystem are we talking about here? Python’s, virtualenv’s, or the distro’s?
And then at Ensure bundled pip does not get too out of date · Issue #1821 · pypa/virtualenv · GitHub
Though with the upgrade model of Fedora this is less pronounced than Debian.
Agile development and frequent releases are great, until the point where users don’t like it (which turns out is most non-web apps and also many web apps).
Luckily for most of our community, there are other people who worry about that for us 
And we’re all here right now discussing it.
So, given that the upstream projects generally consider a version dead after the next is released, what do the distributors require in order to more efficiently maintain the prior versions so they largely keep working? Off the top of my head:
- patchable configuration (as mentioned by Petr)
- roadmap/forward notification of changes (e.g. changing compression in pip)
- backportable patches
- …?
Projects of course have no obligation to provide these, but then distributions have no obligation to include projects, so it’s best to find the middle ground that works for everyone.
In this case, a patchable install command for putting pip into a new virtualenv seems like a good way forward (and also helpful for testing 
).
Addressing pip’s frequent release rate is a big and tricky issue, I know. I’d be interested to hear the distro’s concerns about updating (hypothetically) pip far more frequently, so at least their users who install updates can get them. If the reason is time/resources, or if it’s stability, or something else, there’s likely a way the upstream project can help and hence ensure that they aren’t overly prevented from innovating because of the distribution network (I’d say the same applies to cloud and CI providers updating preconfigured VM images or PaaS images).
2 Likes
Created a rough PR for this: https://github.com/pypa/virtualenv/pull/1831
The goal of the PR is to try to pull in new pip versions, however:
- do not degrade virtual environment creation performance significantly,
- do not trigger during CI runs (the feature is aimed mostly at end-users, that install virtualenv themselves).
The proposed change would make pip upgrade periodically:
-
every two weeks try to upgrade pip to a newer version
-
the upgrade runs not inline, but instead as a new background process (this makes the performance overhead not significant)
-
upgrade the seeded pip to a new version if there’s a newer version that satisfies the following criteria
- was discovered at least an hour ago (this is to ensure that CI environments don’t start with an old version and mid-way through they switch to a newer version)
- was released at least 28 days ago (allow 4 weeks for pip to do a bugfix release).
I consider this a reasonable middle-ground for the outcome of the poll. Outcome of the vote as of writing this is 42 votes: 45% always install the bundled one, 38% periodically upgrade (every two week - upgrade 10 days older), 17% download should be true by default. We will mostly use bundled versions (and almost always in the CI), but for end-users, once a pip version is out there for a long time we do switch to it, as we consider that a proof of stability.
People who would like to always get the latest version should either set the environment variable VIRTUALENV_DOWNLOAD=1 or set download inside the virtualenv.ini file (or pass in the --download flag).
People who would like to always pin to a given version should either set the environment variable VIRTUALENV_PIP=20.1 or set pip inside the virtualenv.ini file (or pass in the --pip flag).
4 Likes
A brief update on this: due to unforeseen events I’ll step back from this topic and leave it to the other maintainers to address it as they see it fit.
1 Like
Nobody can speak for all the distros, so I’ll speak for the ones I’m familiar with.
For Fedora, a faster-moving distro: please make it easy to keep the system updater in charge of updates. Don’t call home, don’t download code unless requested. If that’s not possible, make it easy to patch that out. (Of course, once a user installs stuff with pip, then pip is in charge – for that environment.)
I’m personally involved in the Python community, so I know I can trust it (and how much). But I wouldn’t want the Rust- or JS- or Perl-based parts of my system doing any of the above. I, as a user, rely on my distro’s guidelines to keep developers from doing only what’s convenient to them.
For RHEL, a slower-moving distro: Don’t bundle stuff, so there’s only one place on the system where the next decade’s invevitable security bugs need to be fixed. If that is not possible, make it easy to unbundle.
Don’t expect users to use new versions and features, unless forced to do so.
If possible, don’t do needless refactorings, so patches written a decade from now can be applied to old versions.
(But, don’t do this because of RHEL: the commercial distro does not serve the community first, and it’s unfair to ask the community to support it. Rather think of “enterprise” users in general, and how much you want to help them.)
For LTS distros:
Some users want to run stable (a.k.a. “pinned” or “old”) versions of software. A good LTS distro will serve those users. If there were no LTS distros, something else (and probably very similar) would fill that niche. Not updating is not so much a question of resources, but of not wanting to update a system that workeds just fine.
When you have no need for the latest features, updating only has downsides. Especially if your environment dictates that each new version will need to be reviewed/vetted/certified in some way.
Faster-moving distros can update more frequently, of course, so they shouldn’t be a bottleneck.
1 Like
There’s an explicit request for better handling of bundled wheels - see https://github.com/pypa/virtualenv/issues/1840. I’ll refactor the code and add documentation on how distributions can replace the built-in logic with their own (which is updates with the OS) if they wish to do so. Such patches could also disable the default update periodically feature, but still allowing the user if they wish to request one of these (for the case when you’re stuck on centos, that will never upgrade pip, but you really need a newer one).
A changeset went out with today’s release 20.0.24. More in detail about the used changeset at https://virtualenv.pypa.io/en/latest/user_guide.html#wheels
3 Likes
Thanks @bernatgabor for all of your work on this! ^.^
1 Like