PEP 704 - Require virtual environments by default for package installers

I agree, this is not the right thing for conda to do. (If only because getpath.py is holding together with really really cheap and nasty duct tape, and any attempt to induce it to do something specific is going to be disastrous.)

It is a real concept, it’s just so widely ignored due to how Linux distros lay out their Python installs that sys.prefix isn’t actually a good way to tell you where your isolated Python install is (contrast with Windows and Conda, where sys.prefix is entirely sufficient to tell you where your isolated Python is).

But this isn’t the axis that’s relevant here. The problem is the same as it’s always been - when a Python environment is externally managed then it shouldn’t be managed by tools that don’t know how to manage it.

Whether it’s “isolated” or not really just depends on whether you’re going to use it for multiple apps - it’s not a function of Python itself, but the user’s intent (or the *waves hands* administrator’s intent, e.g. if a Linux distro wants a Python environment just for their in-built apps, then they intend it to be isolated, and need to take their own steps to achieve that).

2 Likes

FYI this is off-topic for this PEP. If people want to discuss it then I am going to ask they start a new topic.

1 Like

The Wanting a singular packaging tool/vision thread is literally on this topic, so it can simply move there (and anyone wanting to discuss it should review the ground we’ve covered so far :wink: )

1 Like

Yes, that’s EXTERNALLY-MANAGED.

That isn’t what the conda devs want though (outside the base environment), since they want to allow pip et al to still to be used to manage non-conda packages - the breakage in PEP 704 for conda is that pip would stop installing into conda environments (since they don’t register as virtual environments in the venv sense, and making them do so would require contorting conda’s Python installation in admittedly weird and wacky ways).

I’m honestly not sure what aspect of the problem space PEP 704 is designed to address given that PEP 668 has been accepted and the implementation is being rolled out:

  • environments that don’t want pip (et al) to be used at all can set EXTERNALLY-MANAGED
  • environments that do want pip (et al) to be used may want to restrict or influence what Python-specific tools do, but requiring the providers of those environments to create or emulate a virtual environment just to get back the functionality they already have today would be a pointless inconvenience rather than something that improved their level of influence or control

Rejecting or withdrawing the PEP on that basis would make more sense to me than trying to morph it into covering a different idea (and I say that as someone that has published PEPs where the idea that I ended up submitting was wildly different from what I proposed in the initial drafts. Sometimes it’s useful to go that route, but I don’t think this is one of those cases, since the presented idea is quite clear. I just don’t think it will help, and it will definitely hurt).

The rejection/withdrawal might provide inspiration for other ideas, but those can always be pursued separately.

1 Like

My feeling is that the PEP covered two different purposes:

  1. Putting in writing somewhere the common convention that a project virtual environment should be called .venv and be located in the project directory.
  2. Checking whether pip’s plan to disable installing into the system environment by default had any major flaws.

For (1), I think it’s a worthy goal to try to formulate some “best practices” in this area, so that tools and IDEs can set comfortable defaults. I’m not sure there’s sufficient consensus for this to be a standard, though (PEP 704 doesn’t attempt to do that, to be clear) so I feel that a PEP is the wrong place for such a guideline (much like we wouldn’t try to make the “src layout” for projects a PEP).

For (2), I think it’s clear that pip doing this would cause problems, for conda at least. I’m not sure that the “pip design discussion via PEP” approach was ideal, and I’d be reluctant to use it in future, but it’s got the information that we need. At this point, I think the PEP can be withdrawn or rejected and the pip developers can take the information and decide how to proceed. To be clear, though, I do not think that rejection would imply “pip must not do this” - pip’s feature set is not controlled by PEPs in that way. But withdrawal is probably better to avoid any confusion.

1 Like

Indeed. My plan is to withdraw the PEP at this point – I mostly need to find time to write around the withdrawal notice that I’d place on the PEP when I do so. :slight_smile:

3 Likes

functionally, mostly. Though the wording of the error message (set by pip, not the one that can be overridden in the EXTERNALLY-MANAGED file) is a bit specific (I’ve put in a PR for that), and the flag name is ever worse.

But it’s actually kinda the opposite of what I (not necessary others in the conda community) want – EXTERNALLY-MANAGED means “disable pip altogether” – which is pretty much what we all do want for “base” environments, but for other conda environments, maybe we don’t want disabled pip, but have pip behave differently – a “minimal pip”.

Yes, but I at least would like to see it used in a more controlled way – but that’s a conda problem :slight_smile:

Anyway, I think that the recent addition of the ability to override pip defaults may work for what I have in mind – experiments underway. And if so, then there’s nothing left to be done by the pip folks.

@pf_moore and @pradyunsg: Sounds great – thanks I think this PEP did prompt some great discussion and better understanding among all.

If this is something people want, I’m obviously interested/motivated to help with this. I tried to get some consensus in Setting up some guidelines around discovering/finding/naming virtual environments - #57 by pradyunsg , but it didn’t get far as it was discussing too many use cases at once.

python -m venv currently prints an error:

venv: error: the following arguments are required: ENV_DIR

Could it instead default to behave like python -m venv .venv? (If that is / will be the recommendation.)

That would look less confusing and overwhelming to beginners, is easier to type, reduces the amount of details to lookup or remember, feels more “official”, and has no obvious downsides or impact on more advanced use cases or users that prefer a different name.

1 Like

If the community is going to standardize on .venv as the default name, we should ask virtualenv to update their docs. They currently use venv as the default:

Another place I’ve seen venv instead of .venv is we could also ask PyCharm to change their default (when it’s not already using a workflow tool such as Poetry):

I suspect PyCharm would do this if there was an accepted PEP that decreed this.

Why not just raise a request/PR asking them to change? If they refuse, asking them what it would take to change their minds would be a useful way of determining what needs to happen next.

@bernatgabor

Possibly/probably, but we would need to get .venv written down as the “official” recommendation first.

I wonder about .venv for newbies.

“dotfiles”, are (I"m pretty sure) hidden by default on all *nix and Macs, and shown by default on Windows.

Most *nix users will expect this, but Mac users might get pretty confused.

Newbies have enough trouble with Windows hiding the filename extension by default :frowning:

Yes, and for Windows users. The “dotfile” convention is very unixy-centric. I don’t really use venv so I’m not sure, but it does sound like .venv is kind of standard already.

I’m not really happy with the wording in “Opting out of virtual environments”. it MUST provide an opt-out. What if if you are installing an OS package globally (let’s say something like youtube-dl, Graphor, etc…) ? What if I don’t want to use VE at all?

Since VS Code adopted it we have not run into any confusion or concerns from beginners, just advanced users being concerned for beginners (and that’s a serious comment; we have gotten way more complaints from advanced users wanting to search their virtual environment for some code and having some tool ignore . stuff by default). I suspect most beginners don’t even think about it being some convention or question it; to them I suspect it’s “just how things are,” and move on; they are just following the instructions. I also think you’re assuming beginners are doing raw terminal work and not some tool which will show things differently than e.g. the Finder in macOS, and once again, with no instructions on e.g. activation next or something else they are going to just blindly type in the terminal.

Honestly, the much bigger problems for beginners are putting their code in their virtual environment or committing it to version control (which is why VS Code’s Create Environment command drops a .gitignore file in the virtual environment directory).

6 Likes

I’m not assuming that, but I’m a bit confused – the Finder in OS-X hides dotfiles by default. I think that’s a greater potential source of confusion than Windows, where, IUUC, there’s nothing special about dotfiles. This is no different than, say a __init__.py file being something special.

Are you suggesting that folks don’t use Finder for looking at their projects?

It’s a bit of a tangent, but I often wish that pip install had a --local switch akin to --user that installed packages into __pypackages__, say, in the local folder. This local installation would be lightweight and completely transparent, without explicit activation, and would work on top of venv, conda, etc. As such, it would be part of the import system in Python rather than any specific package manager.

In the spirit of this PEP, if global installation is impossible, e.g. because of EXTERNALLY-MANAGED, then pip could say: “You can install packages locally into the current folder using pip install --local.” This would not elevate venv or any other tool above any others.

Ideally, the local install would keep a list of installed packages that can be committed, as well as the actual binaries separated by Python ABI, so that you can just reinstall the local packages with some command should you switch machines or Python version.

NB, this seems a lot more relevant to the PEP 582 thread instead

1 Like