Pip. Make PIP_REQUIRE_VIRTUALENV the default?

Packaging
1

Hi.

I thought better a discussion, rather than an straight-to-issue.

Inspired by a discussion on Twitter, something that’s been on my mind for a while, is can we PIP_REQUIRE_VIRTUALENV default to true?

I kind of think, this would solve 90% of the complaints folks have with venvs (not understanding them, forgetting to set them up, and so on…) If, from day one, unless you explicitly passed a flag to do otherwise, pip would have your back in making sure you had a venv active, with an awesome Elm-like error message, saying just what you needed to do.

Jannis mentioned (on the discussion) that this was considered originally:

On hindsight practicality should have beat purity.

I know it would be a change, but one for the good, I think. Would it be possible? Are they strong reasons why not?

Thanks for the thoughts.

Kind Regards,

Carlton

2

I want to say yes, but also imagine this would make a lot of users grumpy, especially those relying on system-provided pip (e.g. python-pip from APT). Global pip “just works” for them, and it would be a problem if they feel we’re force-feeding virtual environments to them. This is especially true in the current age of containers. (Side: IMO it’s still a good idea to use virtual environments in a container, but that is definitely not the mainstream view.)

3 Likes
3

By far the biggest issue would be backward compatibility. This would break a lot of container builds, CI jobs and similar. Certainly all those users could switch to --no-require-virtualenv (or whatever form we chose to add for an “allow outside of a virtual environment” flag), but that’s a lot of people impacted.

Also, Windows users need pip to manage their system Python.

And as a final point, I’m not sure how widespread use of virtualenv actually is. As part of the packaging community, I get the impression that most people use them, but every so often I encounter experienced Python users who have never used virtual environments, and I realise that they may not be as universal as I realised.

So while I think it would probably have been great if we’d made this decision from the start, I don’t know if it’s practical now.

Maybe a long deprecation (warn for N releases, “you are using pip outside a virtual environment - you should use --no-require-virtualenv, as this will become mandatory in future”) would help, at least to judge user reaction, but my instinct is that you’d get a lot of complaints.

7 Likes
4

Yes, thanks Paul. That is the worry.

I guess my thought is that I’d trade some (many?) grumbles from experienced hands, who are perfectly capable of setting an environment variable, or using a command line option, in order to make life easier for the armies of beginners who come to Python and immediately run into the pip/venv learning curve. A lot of ink (and energy) is spent on this. (XKCD right). Often the response is It’s not that hard… but quite probably it is.

I can understand if the conclusion is No, we can’t change it but I put-forward that a different default here (I want to say a sane default here…) would make enough of a difference to justify the churn. It would smooth the default pathway for the vast majority of users, especially new ones.

I’ll let it sit. Thanks again!

5

These people tend to be very-very vocal. Drawing from my experience with virtualenv, that was mostly backwards compatible.

3 Likes
6

These days, I mostly use pip inside a conda environment. I wouldn’t want pip to bother me about that…

2 Likes
7

@carltongibson Thanks for suggesting. Like Antoine, I use a combination of pip and conda in my development and this would complicate things for some users.

Some would find this change good and others would not.

I do like @pf_moore’s points too.

8

How about a checkbox to set the environment variable in the python installer that’s checked by default?
Pros:

  • Allows opt-out
  • Funnels new users to best practices

Cons:

  • Effects all python versions installed
  • Not all platforms have a python installer?
9

No. This is a pip option, so the sensible default should be part of pip.

They could prompt the first time you go to install into a non-venv and save the response in their configuration file.

This does not belong in only one of the range of installers that people use.

3 Likes
10

Grrr. OK. As expected I guess, this doesn’t look like something we can change now. (Maybe someone has a good idea but…)

I think we’re going to need to beat the drum that the option exists, particularly in tutorials and things. (From the discussion yesterday, even very old hands didn’t know about it.)

Thanks everyone for replies!

11

pip doesn’t treat conda as a “virtual environment” - as far as I can see from the pip code, because conda doesn’t set sys.prefix and sys.base_prefix in a way that “normal” virtual environments do.

So as things stand, --require-virtualenv would block use of pip within conda environments. Addressing this would require either a conda change (to act more like a standard venv) or a pip change (to check for a conda-specific indicator that we’re in a conda environment, assuming such an indicator exists).

1 Like
12

Can you elaborate on how pip detects it’s in a virtualenv?
(also, why does pip actually need to detect this?)

1 Like
13

Please, no, do not set a default for pip that requires virtual environments. Beginners that use Windows (i.e. no pre-installed Python) should not be required to have to set up a virtual environment before installing a package with pip. They should be able to download and install Python (from python.org), do “pip install pygame” and follow a simple tutorial. There are tons of tutorials aimed at beginners that would have instructions such as these.

André Roberge

2 Likes
14

Can you elaborate on how pip detects it’s in a virtualenv?

(also, why does pip actually need to detect this?)

Mainly to decide where to put things like headers or VCS checkouts (for editable installs) and what to allow to uninstall (for instance inside a virtualenv with global site-packages you usually don’t want pip to mess with global site-packages).

15

It will be hard to detect whether you are in a conda environment without explicitly checking for the use of conda as each new conda environment is always a fresh Python installation. Technically, this is the same case as if you were using docker for separating your environments and thus also the same approach should be used. It is unclear to me though what a clean approach could be.

16

sys.prefix != sys.base_prefix basically - the same as venv. (Or for older versions of virtualenv, the presence of sys.real_prefix).

Mostly just to implement the --require-virtualenv option :slightly_smiling_face: and to warn about some weird interactions around --user.

Edit: The stuff @xafer mentioned is the main reason, in fact, I just missed it because it’s only a very minor variation in terms of pip’s code paths.

17

Yes, that’s actually why I stopped using it. Unfortunately the maintainers make a concerted effort to conceal exactly what Conda looks for to detect activation: Spurious activation warning · Issue #8424 · conda/conda · GitHub

18

Looks like you(?) caught Ray on a bad day there :slight_smile:

Conda detects that it hasn’t loaded properly, so it’s not really a case of turning off the warning, it’s a case of fixing how you load it. Unfortunately, they do make it hard to run from outside a supported interactive shell, which I’ve discussed with them in the past but I don’t think it’s turned into a real feature (to export the required environment variables that have to be set for paths, DLLs, etc).

Pip shouldn’t need to distinguish between a Conda environment and a custom install of Python, and right now it can’t. Neither is a venv, but both are valid install targets, and provided pip can get the ABI tag required and an install location, it’s fine.

If the goal is to simplify the user experience, people should have been more supportive of PEP 582 or something that actually simplifies things. More band-aids (like the original proposition here) just make things more complicated.

3 Likes
19

Can you clarify here, please? I’m not trying to be argumentative - I’m genuinely not entirely sure what you mean here. I was mildly in favour of PEP 582 when it was proposed, but I wouldn’t have said it makes a lot of difference to this debate, which suggests that maybe I missed something then (which might have resulted in me being more supportive).

My understanding is:

  1. This proposal is trying to make it harder for people to “accidentally” alter their system environment with pip, by forcing people to explicitly say if they want to install outside of a virtual environment.
  2. This fails, because too many things are legitimate “not virtual environments” where people still want to install things with pip (conda environments, Windows installs, docker containers).
  3. PEP 582 would add another “not virtual environment” thing that people would want to install in, but they specify this not by adding a flag, but by creating a __pypackages__ directory - which would IMO make it more likely that they’d accidentally run pip install somewhere where the install would go to the system Python (forgot to create the directory, typo in the name, …)

So my first impression is that PEP 582 would strengthen the case for something like this1. What am I missing?

1 Amended to add “in a location with a __pypackages__” to the things that pip counts as a virtualenv…

20

Yes, it would. But it would also mean that the default behavior could reasonably be “create the directory”, as opposed to “go figure out venv and come back later, then figure out activation”.

Blocking use of one tool by default until you’ve done something inexplicable with another tool, and then it looks like it did nothing until you’ve done something else inexplicable with another tool isn’t actually making things easier.

Prompting to create a local directory that then gets picked up automatically, does.

But for the context of this thread, I definitely vote for doing nothing. We’re not going to actually improve things with the current set of options.

4 Likes