PEP 704 - Require virtual environments by default for package installers

Packaging
5

It’s somewhat bikeshedding so it’s up to you :slight_smile: I would say though that we enforce compliance with those other PEPs by saying that if you don’t follow those PEPs, then your thing is unlikely to continue to work. Like we can’t force people to format their versions correctly, but if they don’t then it’s unlikely a large set of tools will function.

This is inherently different, since it’s not saying that virtual environments are the only way to install something, it’s just changing the defaults, the UX of installers, and in the past we’ve gone to lengths not to dictate what UX the various tools provide.

1 Like
6

The installer MUST also provide an command line option to disable this behaviour…

If I decide to write my own installer that only works with virtual environments, this PEP dictates I need to implement installation to non-virtual environments as well. That seems rather unfortunate. Should this requirement be limited to installers that previously supported that and now want to follow this PEP?

2 Likes
7

@hroncok Fair point – amended to:

The installer SHOULD also provide a mechanism to disable this behaviour

This relaxes the requirement, and tools can opt-out on the basis that they don’t wish to support non-virtualenv models. I can make that explicit, if anyone feels strongly about that. :slight_smile:

1 Like
8

I think this is the most reasonable path forward; fantastic solution!

9

What does it mean for this to be “a standard”? If a tool author decides that they have a better idea for how their tool should work to serve their users, then is this PEP saying that we all agree to go harass them until they take down their tool and go away, or… what?

2 Likes
10

That is a good question. I think the PEP should state something about that explicitly but how I interpreted it (although it did not state it) was that if a tool for example has a Python setup they want to install into they would set the VIRTUALENV environment variable.

So for example our Agent at Datadog distributes a Python installation that most of the Agent Integrations use. Starting with Python 3.13 we would modify our installation command (which just uses pip currently but that’s irrelevant) to set that environment variable first thing.

11

As long as there are tools that allow an opt-out, this sounds like hyperbole to me. Furthermore, no-one is talking about harassing them, they just would not be a model citizen of the packaging ecosystem.

  • If their idea is really that good, take it up with PyPA and improve the standard or make the case that change is beneficial.
  • If their idea is not that great, avoiding further balkanization of the packaging landscape is a good thing.[1]

In both cases, the emergence of minimal standards in this space is to be celebrated IMO, and should not be obstructed by entirely hypothetical scenarios.

  1. It’s completely justifiable that introducing a new tool (which will want users, thus cannibalizing existing solutions) – rather than improving existing ones – has some basic hurdles to clear, because as user responses plainly show, each additional tool has a very real cost in confusing/upsetting users. ↩︎

12

To be up front, I would personally love it if everyone always used some form of environment isolation mechanism when installing Python packages (at least in typical, non-container, use cases with pip and conda) or other given the countless hours of my life that I would save helping others solve the issues that resulted when they didn’t do this. That said, at this point, I’m currently undecided as to my feelings on the central approach of this PEP.

A couple big questions:

How would Conda deal with this? If installers don’t detect Conda environments being activated (as pip currently doesn’t, last time I tested enabling pip’s implementation of this), this means that any installer that implements this PEP will not work by default with Conda (unless Anaconda/Conda-Forge patches the relevant versions of that installer, and users are using a Conda-patched version, which sometimes they are and sometimes they aren’t).

On top of the base level of user frustration this would likely cause by users having to figure out how to bypass this (assuming they even can in all cases, which the PEP now does not require, and the mechanism may not always be accessible in all contexts). If people get a pip error in a conda env that they need to active a venv and follow the mandated

shell-specific instructions on how to create and activate a virtual environment named .venv

now they are going to have a separate venv activated inside their conda env, which is a sure recipe for chaos and confusion.

Also, wrt virtual environment directory names, I’m all in favor of standardization on this front, at least in terms of giving unnamed venvs (i.e. for which their enclosing directory namespaces them from all other venvs) a standardized name, as the current hodge-podge of conventions is maddening and makes things complicated for users (and everyone else) for no benefit. However, there are two drawbacks to consider when considering this vs. named venvs, i.e. venvs that have a meaningful specific name:

  • This means only one venv can be placed in a given directory, which may or may not interoperate with existing centralized management approaches, as now all conforming venvs will need a unique parent directory too. I’m not sure how much of an impact this may have on existing tooling that uses this approach, as I’m not too familiar with it (I mostly use either conda or add-hoc venv management).
  • The name of the venv directory is what is shown as part of the prompt when the venv is activated. For me, this is the key reason I give my venv directories unique names, giving me an instantly visible confirmation of what venv I’m in. While this PEP protects against installing outside a venv, this would make it much easier to install/uninstall in the wrong venv, which is something I got bitten by several times before I adopted unique venv names.

I’ve also reviewed the PEP from a PEP editor perspective, including a handful of comments regarding the PEP clearly and precisely expressing what it intends to in terms of terminology and rationale, that may stray a bit into the gray area of content/subject matter expertise. If you feel any of them stray too far, I can defer them and bring them up here instead.

7 Likes
13

FWIW, I just noticed this Maturin issue that got linked to the PEP PR thread; Maturin apparently already enforces using venvs by default, which brings up some of the issues involved in considering this PEP:

14

I think there is a terminology ambiguity here that is likely to cause confusion, that is with the term “installer”. Not surprisingly, when I read the PEP title, my first thought was that it referred to the Windows and macOS installers for CPython itself that we provide on python.org for each release. I’m sure some other readers would make the same mistake. I suggest changing the title to use “package installer” and also add some clarification text in the PEP intro.

3 Likes
15

Thanks; I added a suggestion implementing your feedback on the PR.

And as for the broader issue of the precise meaning of the term “installer”, that’s actually something that bothered me as well, and I previously raised on the PR (and will elevate to here, if appropriate):

What counts as “an installer”? Obviously, pip counts (as its mentioned by name), and I’m assuming Hatch, PDM and Poetry count as well when used in that capacity, while apt, dnf, brew and choco don’t, though that isn’t explicitly stated.

If you limit it to “Python-specific” installer, what about tools like shiv, pex, etc.? And what about, of course, Conda? The PEP should provide a precise definition of that term and examples of what tools would and would not quality (similar to PEP 668 for what qualifies as an externally-managed environment).

1 Like
16

Thinking about this more, I actually think this is a bad idea.

One of the key things we found in the EXTERNALLY-MANAGED PEP was that there are use cases where installing into the “base” environment directly is the right thing to do. For instance, the python docker containers have a custom version of Python that you’re intended to install into directly, and while you can use a virtual environment, doing so would be pointless-- and many do not, so this would break all of those users for no real reason.

12 Likes
17

Would my interpretation of what the PEP meant assuage your concerns? PEP 704 - Require virtual environments by default for installers - #10 by ofek

edit: so for example the container would set that environment variable to the installation path

18

I don’t think so? For one the VIRTUALENV environment variable only exists when a virtual environment is activated, not when you’re doing something like .venv/bin/pip, so it would fail for that.

Ultimately I think the problem here is that while it’s best practice to isolate installs from one another, there isn’t a singular way of doing that, but this PEP, if accepted, if basically saying that all isolation mechanisms that aren’t stemming from venv or virtualenv are second class citizens that are equally as “bad” as just YOLO installing everything into a single shared environment.

Off the top of my head, I can think of:

  • venv / virtualenv
  • Conda environments
  • Docker Images
  • Separate “concrete” environments
    • Think of things like tools that ship with their own Python install
  • Symlink trees (used by things like Bazel, etc)
  • PYTHONPATH or sys.path manipulation (think of things like the original multi version egg installs, but this could also be things like zipapp, etc).

All of those are equally isolated, in some cases even more isolated than venv / virtualenv, but this PEP would REQUIRE that tools generate an error by default when operating under these conditions, and I feel like the “win” is very minimal? Even in the PEP’s own motivation it says:

This creates a barrier to entry for many new users, since virtual environments serve as a pre-requisite for typical Python development workflows and how virtual environments work is a lot of information for anyone new. It can take a lot of extra time and effort to explain them.

Which then the PEP goes on to then say we should force users into learning that information, which we just declared is a lot of information for somebody new.

So this doesn’t appear to actually make the amount of information for new users less, it just forces them to learn it up front, even if they don’t need it because they’re using another isolation mechanism.

It does prevent people from mucking with a system managed Python by default… but so does PEP 668, which PEP 704 mentions, but says it’s only “partially” mitigated. I’m curious what gaps we think exist in PEP 668, other than it’s still in the process of rolling out. From my POV, if we assume a world where PEP 668 has been fully rolled out to Linux distros, etc… the only cases that aren’t mitigated are places where we don’t know whether it’s the preferable thing or not.

9 Likes
19

Just a further bit of thought:

I think there is an argument here that this might make things more confusing for new users, not less. I’m not someone that spends a lot of time teaching people, so maybe one of them can correct me if I’m wrong, but I suspect if someone new is presented with an error like this PEP suggests, they will[1] promptly blindly run whatever commands they were told to “just make the damn thing work”, without really delving into what it is they’ve just done [2]. They’ll then exit out of their shell for the day, come back at a later time, forget entirely about all that virtual environment stuff that they blindly ran, and then get really confused when all of the things they’ve installed are gone.

It feels like it adds, or at least makes more likely, a class of new user confusion where they get confused about what environment they’re operating in and under what conditions. Dictating the .venv name does help some (you can imagine different tools deciding on different names, and a new user getting conflicting information) but I don’t think it mitigates it entirely.

  1. Or at least, a non trivial number of them will. ↩︎

  2. See all the people who blindly slapped a sudo in front of their pip install for no reason. ↩︎

8 Likes
20

This PEP IMO misinterprets the nature of the existing user confusion. An implicit concept without activation like PEP 582 just makes more sense to most people familiar with any other language, requires less learning up front for beginners, and reduces the number of tools one is required to know about.

This PEP just suggests more frustrating errors. Honestly it feels a bit like “the beatings will continue until morale improves”. :slight_smile:

If you double down in this direction please consider some changes:

  • Instead of an error, require that a venv is automatically created.
    • Not doing so seems frankly user hostile.
  • Recommend never activating a venv.
    • Activation is an advanced, unneeded concept that is never needed and just confuses most people for many reasons.
  • Change the venv layout so commands like python, pip etc. are in a consistent location across platforms.
    • Preferably directly in the venv directory, to further reduce confusion and make things easier to find, type and remember.
  • Make venvs relocatable.
    • (Reducing another source of confusion.)
  • Change the default name from .venv to __pypackages__ :slight_smile:
    • (Windows Explorer refuses to work with folders with a leading dot, giving rather cryptic error messages (“you must type a filename”), and hides file extensions by default, so the name .venv is potentially inviting new confusion. At least I thought this was the case, but maybe this was solved in Windows 10 or 11?)
14 Likes
21

I agree with @petersuter here. This does not solve any of the problems with virtual environments. People will still upgrade their OS and end up with a broken venv with broken symlinks with no easy way to fix them. People will rename or move the project directory and will end up with a broken venv. People will still forget to activate venvs, or activation will still fail, and end up with weird errors. None of those problems occur with PEP 582 (assuming installing the older Python version counts as easy).

9 Likes
22

I agree that the defaults should not require having to learn about virtual envs, or making choices about them. IMO it should just be there (in a standard location, I wouldn’t suggest within a given project folder), unless you want to start renaming your envs etc. but then you can look up how to do that.

Re some of @dstufft’s points:

  • I think docker is special in the sense that image recipes there are already full of special gymnastics to keep the footprint as small as possible, so adding one flag to the python install would not be a big burden IMO (and it would work without that flag)
  • A putative PEP668 + PEP704 should work together in the sense that it should be possible to tell the installer to accept & use the conda env (and conda could probably hook into this by default)
  • Tools that ship their own Python install – this is an advanced case, and such authors can just opt out out of venvs when they package their own Python
  • I think it would be fine to not explicitly support any PATH manipulation things. Those are hacks IMO that should be replaced by better tools/options.

So I think the idea needs to connect a bit more to existing use cases, and polish the UX a bit (less errors, more happy defaults), but I still think it’s the best direction so far.

23

What does “run an installer” mean, exactly?
With require-virtualenv = true, pip currently allows pip list or pip hash, but not pip freeze, pip download or pip install --target.

24

Shouldn’t this be limited to systems that actually have a system python? There’s no sense in hassling people who are in complete control of their own install.

How about keeping PIP_REQUIRE_VENV so that the opposite preference can be set with PIP_REQUIRE_VENV=0?

1 Like