PEP 704 - Require virtual environments by default for package installers

[CAM-Gerlach]

@rgommers

If you want to include conda environments in your picture here, then you could:

• rely on the externally managed designator for installers (good idea for the base env at least, and there’s an active discussion on that),
• treat non-base conda envs like virtualenvs rather than like system envs, either by special-casing conda envs or by generalizing whatever you do to “user-activated environments” (I’d quite like the latter),

As a sidenote, the reason for opening the discussion on the former approach was due to the latter approach being initially suggested to @pradyunsg (by me) in order to for this PEP to better interoperate with Conda envs.

@pradyunsg

OK, my understanding based on my limited use, Anaconda | Using Pip in a Conda Environment, frequent communication that pip-directly-in-Conda can (and does!) break conda environments, and the fact that Conda’s pip interoperability is experimental is that we don’t want people mixing them.

Yes, it most certainly can and does, to the point where we made our very first Spyder Says video about it:

Spyder says: Don't mix pip and conda

However, it is necessary for using PyPI-only packages, editable installs and various other scenarios, and can typically be avoided if you are careful, know what you are doing and understand the basic mechanics involved (namely, installing only “tip-of-the-stack” packages with pip, and avoiding further updates to the existing env). And if things do go wrong, it is cheap to remove and recreate the env—so long as it is not base.

That 's the reason I originally brought up the above—the real harm is pip install-ing into base, and if that could be prevent by default (with EXTERNALLY-MANGED or this PEP) without interfering with pip install-ing into non-base envs (or maybe adding an opt-out warning) then that would be a big win for Conda users, IMO (considering how many bork their whole Conda install because of it).

The unfortunate truth is that many/most “regular” Conda users are students, scientists, engineers, analysts, etc. and not programmers, and thus don’t have the knowledge and experience to know this (because many/most of the people they’re learning from don’t know themselves, because no one taught it to them).

In fact, even the basic concept of what an environment is and why you should always use one is a difficult one to teach for the first time to students who are graduate-level scientists but with limited programming backgrounds—like I struggled with doing for a student just this Friday, who had run into issues with her base environment. This is especially when those students are typically already busy and often come to me tired, frustrated and just want to get their project/research done and go home, for whom Python is but one means to an end.

Which I suppose, circling back, is pretty relevant to think about here—while I remain a strong advocate of teaching users (how to) use environments early and often, due to the countless times I’ve seen users struggle with the aftermath of not doing so, it is equally important that we make them as easy and painless to use as practical if we are requiring them.

@pradyunsg

I just tried this (again), and it seemed to work fine. Am I missing something?

Because all that is being tested is just that Conda-installed packages in the outer Conda env can be imported inside it’s Python with the venv when --system-site-packages is set on creation and both envs are activated, in the correct order.

Where things tend to go wrong is everything that comes after: installing further packages in the venv and/or the conda env, attempting to update one or the other, installing duplicate packages in one or the other, activating one but not the other, installing packages with incompatible binary deps, etc., etc. I believe things are less broken than they were in the past, where I’m not even totally sure that the above would work right, but the things that can go wrong in this setup are close to a superset of the issues with installing PyPI packages directly into the Conda env, the limited protections and warnings on the Conda side don’t work, the likelihood of user error is much higher, and problem recovery still requires recreating both (vs. just the Conda env).

@pf_moore

Currently, and as part of the language definition , this is done by checking if sys.base_prefix != sys.prefix. I don’t know if conda adheres to this approach, but if it doesn’t, then my first thought would be that they should (I’m taking your comment that conda envs are much more like virtualenvs than system envs as the basis of this view). If it does, then there’s nothing more to be said - we can just use that.

Unfortunately, its the former and that check won’t work:

$ python
Python 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:30:19) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.base_prefix
'C:\\Miniconda3\\envs\\sphinx-5'
>>> sys.prefix
'C:\\Miniconda3\\envs\\sphinx-5'

@pf_moore

If, for some reason, conda can’t use the difference between sys.prefix and sys.base_prefix the way it’s documented, then IMO we need a new definition, that is agreed between everyone. And that will require conda to explain what they need that the current approach doesn’t do.

Per the definitions of sys.prefix and sys.base_prefix in the Python docs, sys.prefix/sys.base_prefix is set at install time to a “value for the Python installation”. Since each Conda environment contains its own install of Python (unlike with a venv), this path is thus naturally that of the Python installed into that environment.

And also, per those definitions, site.py will change those paths when a “virtual environment” is in use, a term which links to Python’s venv module, and which the site.py code has a bespoke check for specifically virtual environment as created by venv (and similar). Therefore, in both cases, the current behavior seems to be the closest match to what is actually documented, and also the behavior of an unpatched Python installed inside a Conda environment.

That being said, I’m not an expert on any of this. Perhaps @jezdez could comment more.