PEP 704 - Require virtual environments by default for package installers

The recommended way for this to happen is to have the other package manager install into a location that isn’t the preferred scheme in sysconfig.

Well, pip is already able to recognise packages that are managed by other package managers (via the INSTALLER file in .dist-info). So as long as conda is following the standard and creating that, pip should respect it (if that’s not the case, it’s a pip bug that I’d expect to get fixed with no major issues). If conda doesn’t create an INSTALLER file, the breakage is basically their own fault.

Conversely, pip writes the INSTALLER file when it installs packages - if conda doesn’t respect that, then IMO that’s a bug in conda. If they don’t want to fix that, then having distinct install locations is an alternative approach, but it does have a downside in that there’s a possibility of the same package being installed in both locations, not necessarily the same version. So respecting INSTALLER is better, but both ensure tools don’t tread on each other’s toes.

If conda isn’t comfortable with either of these two options, I think it’s time to stop worrying and simply accept that they will just “do their own thing”. I certainly don’t think adding a third option because they didn’t like the first two is particularly reasonable.

1 Like

Using INSTALLER to refuse uninstallation would go directly against the “multiple interoperating tools” idea.

If a package can’t be uninstalled using standard metadata – the RECORD file – then can be installed without the RECORD file. (That went in with PEP 627. Should it be clarified?)

4 Likes

As designed, I don’t think INSTALLER was intended to reflect “don’t touch me with something else” information. We could add a per-package EXTERNALLY-MANAGED to reflect that but skipping RECORD and storing that info externally is equivalent-ish.

1 Like

Inspired by this proposal, we now changed maturin’s behavior for maturin develop (which acts like pip install -e) so that after checking for a virtualenv/conda env, it will now next look for a virtualenv .venv in the current or any parent directory and use that if it exists (and error if it didn’t find any environment).

We’re also discussing a maturin run <args> command that acts like python <args> [1] but picks the right python interpreter through the same strategy, avoiding both accidentally launching system python and manual venv wrangling. imho this would also a good strategy for a general python launcher.


  1. execv the python binary on linux and a subcommand on windows, to be exact ↩︎

I’ve gone ahead and initiated a conversation with Conda folks on their discussion forum (it’s on Matrix, in case you are wondering, and I can’t figure out how to link to it).

I presented them with three directions that the pip + conda pieces of this could take:

  • Adopt the same behaviour as OS Python installations, moving to require virtual environments by default.
  • Add the opt-out to conda environments (except base).
  • Change nothing (i.e. withdraw this PEP because conda)

While the discussion is still ongoing, the general sense is towards the second bullet, which can be achieved using pip’s configuration stack + opt out.

I still need to catch up on the single vs multiple virtual environments aspect of this (I’ll reiterate that picking a default doesn’t make other approaches illegal or whatever, and they’re not mutually exclusive workflows either).

Awesome. This could also be an opportunity to get input on the related but maybe thornier question raised by @steve.dower in another thread:

What I’ve not seen written down before is an actual spec from the Conda side of how they would like pip to behave in the presence of conflicts when merging repositories.

Edit: looks like @rgommers already started this conversation a couple days ago, which is great.

Here’s a link to the initial message[1] in the chat room from @rgommers, the referenced discussion follows in several chunks and subthreads (that can be unfolded like in slack).

BTW, conda has a discourse now too, and @rgommers posted a summary (of the related discussions on DPO) there (I mention it as it’s easier to reference)


  1. you can click on a message and then there’s an option to create a link under the three dots. ↩︎

1 Like

That link sent me to a blank page with just a background image… (on iPad, if it matters).

I don’t really understand matrix yet, but it seems it can be accessed through several apps (the link sends me to a disambiguition screen to decide between them; I created it from getting to the right place through the original gitter-link that has been superseded but auto-forwards). FWIW, here is another link to that comment in a more(?) canonical app.

Yeah, on Windows the first link expects me to download an app (I only tried the default, maybe others are web-based). And the second takes me to somewhere I need to sign in to view.

So I guess no, if there’s no simple public viewing in a browser, I’ll pass. Thanks for hunting out the second alternative for me, though. If I care enough at some stage, I’ll see if I can find a route in that suits me.

There are some web-based apps available; sign-in with github should work out of the box AFAICT.

Slightly OT, but It would be great if PEP 582 could be brought up there too. PEP 704 seems to be getting most of the attention on the Conda front and I’ve seen much less attention to the Conda implications of PEP 582 despite potentially having at least a similar level of disruption, at least if not mitigated/prepared for, since it’s essentially forcing the same pip/conda prefix split as discussed here in addition to several other potential impacts.

You should probably make sure @kushaldas is aware of any such discussion. I don’t get the impression he follows the broader packaging discussions, so keeping him in the loop explicitly (on the dedicated PEP 582 thread) is probably advisable.

1 Like

I guess I was more thinking of ensuring the Conda folks were fully aware of the implications of PEP 582 as well, so they could reach out on the PEP 582 thread, as opposed to relying on Kushal to initiate that. But I’ll mention that over on that thread as well, thanks.

I don’t think PEP 582 has a similar impact at all. PEP 704 contains two topics:

  1. A name for a default virtual env - chosen to be .venv. This is fairly similar to PEP 582’s __pypackages__.
  2. A change to pip and other Python package installers to only install into virtual envs.

(2) has no analog in PEP 582 (unless I am completely misreading that PEP) and is a major backwards compatibility break. (2) does not work at all for Conda, so needs a global opt-out (not per user, but something conda can set permanently). In this thread it has already be identified that (2) also doesn’t work in very common scenarios like CI config scripts and for Docker use. It similarly won’t work for pip usage in Spack and other packaging environments, nor for packagers who use pip in their distro packaging workflows.

I don’t have a strong preference for (1) between what PEP 582 and PEP 704 propose, but (2) is major breakage for many scenarios for both end users and packagers, and therefore I think that it should simply be removed from PEP 704.

2 Likes

Unless “virtual env” is loosened to something that can be filled by a conda environment, then it would actually help a lot IMO, because always[1] having (some form of) an environment would be very helpful IMO.

To be honest, that’s how I initially interpreted the PEP as it grew out of the previous discussions. If it was always ever only intended to cover the venv case, then I’d agree with Ralf that it’ll break many things.


  1. even cases that opt out from creating an environment would still have an environment – the base install. ↩︎

This proposal (the original pip feature request, and the “part (2)” section of this PEP) resulted from real-world issues caused by people installing stuff into their system environment.

But it was originally proposed before PEP 668 (“externally managed” environments) was available. Maybe it’s not needed any more? People will still be able to install into their Windows system environment (because the Windows Python installers don’t set EXTERNALLY-MANAGED) and maybe also into conda base environments or spack environments, depending on what those distributors decide to do about PEP 668. But maybe that’s good enough?

Pip can’t solve all of the problems of the world, maybe we need to accept that “people not using environments and isolation options appropriately” is just not pip’s issue? Maybe it’s up to the core Python team and the virtualenv developers to address the (percieved) usability issues with virtual environments? PEP 582 can be considered as part of that discussion, I guess…

4 Likes

Right, this part of this PEP merely proposes a standard name for an in-tree virtual env, which is essentially harmless from Conda’s perspective (whereas it is item 2 that of course presents the major challenges and concerns with this PEP, as discussed here). But it’s not totally comparable to what PEP 582 is proposing; there the default name is merely incidental to the behavior changes for installers and for Python itself, which is more analogous to the problems with the previous proposal to separate Conda and pip-installed packages into separate prefixes. There’s some more detail in my reply to that thread (to avoid straying too far OT on this one):

For Conda, it would indeed seem to be good enough as far as this department is concerned—see conda/conda#12245, which @pradyunsg actually opened as a result of me suggesting the potential benefits for Conda of this PEP covering Conda envs too, i.e.

As Conda would be able to achieve the same effect with EXTERNALLY-MANAGED, just under it’s control, this PEP isn’t needed to achieve the same effect (as Pradyun himself pointed out to me).

I understand that; the reason I’m interested in PEP 704 is in order to establish a global (i.e. not just conda or whichever tool) baseline or at least default to install packages into an environment [1].

Just having that aspect (even if the environment in question ends up using a different tool/implementation) would IMO be beneficial, because it would just remove one huge and unproductive detail to explain and document.

And if we’re thinking longer-term, then that would pave the way for convergence on several axes, not least by not having to constantly implement everything twice for the env & non-env case, as well as the interaction of the those two modes.


  1. even if it ends up being the system level environment, after explicitly opting into manipulating that. ↩︎