PEP 832: virtual environment discovery

PEPs
1

The abstract:

This PEP sets out to help make the discovery of a project’s virtual environment easier for tools by providing a default location as well as a way for a project to point to its preferred virtual environment when it differs from the default location.

This is not a packaging PEP due to the proposed changes to venv and dev workflow stuff like this is ambiguous with no WG to specifically go to.

I acknowledge that the PEP targets Python 3.15 and the chances of getting a discussion done quickly and the SC approving the PEP is slim, but one can dream. :grin:

24 Likes
FYI Virtual environment discovery PEP
Setting up some guidelines around discovering/finding/naming virtual environments
FYI Virtual environment discovery PEP
2

There’s no packaging aspect to “we propose that the venv module will default to .venv and if you change it will write a file to .venv with the path you chose, and we recommend all tools should inspect .venv to find the environment” anyway. This is just a core change that formalises a convention and provides an interface - whether other tools choose to follow that or not is up to them.

(I’m not a huge fan of abusing our “official” privilege like this - it’s a bit like if we announced that PEP 8 is actually the style guide for all Python code - but the next best mechanism here is for 2-3 of the popular IDEs to announce that they’re only going to follow this convention and tools that want to make their user’s lives easier should play along. I’m sure either can be arranged, but at least having the discussion here is in the open, whereas the IDE discussion would probably be backchannels.)

FWIW, I’m fine with what I wrote above, and my reading of the PEP is that it’s consistent with my summary, so if people want to argue about specifics you can probably just delete details and then treat them as implementation rather than specification.

The only point I’d argue about is anywhere you’ve written “tools MUST” - we can’t impose specification on the tools trying to use our interface, so it should really be something like “well-behaved tools will …”. But it’s a semantic point.

1 Like
3

I had a quick look and I’m not entirely sure what the PEP is proposing or what the impact would be. Probably I agree with it but I want to share a common peeve I have with the PEP format or the way that PEPs are written.

It would be much better if the abstract of a PEP succinctly explains what it proposes like “this changes X to Y to improve A and B”. It is late here but I read most of the way through this PEP and it still seems like I should probably give up for now and read it tomorrow to figure out what is being proposed and why it may or may not matter to me.

5 Likes
4

Is there an search path/discovery algorithm for finding the appropriate .venv? As far as I know, there’s no official definition of a “project” (the PEP implies that pyproject.toml does not define the presence/absence of a project), so are tools expected to search recursively up the tree until they find one, or assume that the current directory is the root, or do something else (IDEs usually have the concept of a current project, but CLI tools don’t usually)?

I presume based on Setting up some guidelines around discovering/finding/naming virtual environments - #51 by brettcannon that the interaction between an “activated” venv and .venv is up to individual tools, should that be mentioned explicitly in the PEP?

5

How is this intended to interact with projects that use multiple venvs to run tests and other tooling under different supported conditions? What about monorepos containing related microservices, each with its own venv?

I don’t think it’s a benefit to be prescriptive about venv location; It places artificial limits on what an appropriate project is. It may be a benefit to explore a standard for workflow tools to record information about the venvs in use by a project, but my experience with this is that it’s entirely unnecessary. The workflow tools that manage venvs also handle invoking things in the correct venvs.

2 Likes
6

Why .venv that is a hidden file?

Persanally I use tmp.venv and like that its visible when I ls etc.

When I use the python -m venv with an explicit venv name I assume it will just work as it aways did. Is that what you expect?

1 Like
7

Some initial thoughts.

The PEP discusses “projects”, but doesn’t define what a project is. This is fairly reasonable, as everyone has an intuitive understanding of what a Python project is, but it does leave things open to confusion in a number of cases. Specifically, in the case of a monorepo, is the monorepo itself the “project”, or is each individual subproject its own “project”?

This ties in with my point above. I assume the idea is that I’m working with a tool on a “project”, and the point of this PEP is for the tool to be able to find my venv. But without some agreed way of knowing what the project I’m working on is, it’s not possible to do that discovery. And more importantly, it’s possible for the tool’s view to differ from my mental image - to give a simple example, many command line tools just work based on the CWD, so if I’m in a subdirectory of what I think of as the project, how do they do discovery? Given that the project might not have a venv, a simple search upwards doesn’t work - it could fail to spot that the current project has no venv because a containing directory for some reason does have a .venv file or directory in it.

Obviously, this issue can be fixed by stating that we only support certain layouts/workflows - but it’s a significant extension of the scope of the proposal if we’re now trying to define what’s “official”.

I agree. This PEP reads very much like a packaging interoperability spec. But I don’t think that core PEPs in general can make such statements about how people use the language - they certainly haven’t in the past (to my knowledge). Unlike Steve, I’m not sure I am happy with the core statement he summarised in his post - my feeling is that without the authority of a standard saying that users must work in a certain way, people will simply carry on the way they currently do, and tools won’t be able to assume things work any differently than they do now.

And the problem is that I don’t think the core team and the SC have the community support or authority to make statements about how people “must” use Python, or structure their workflow[1].

Maybe this is just because of the way the PEP is worded, but it feels like it’s trying to impose a “best practice” that would be easier for tools to support onto end users. If, instead, it presented the case more neutrally, maybe it would work better. How about the following structure:

  1. The PEP proposes an extension to how venv stores virtual environments (a default name, and a “redirection” mechanism).
  2. It is assumed that other venv creation tools will implement the same mechanism, for compatibility as well as to provide the benefits of those mechanisms, provided below.
  3. Users will benefit from the change as follows (easier to find the venv because the default name is now standardised, a mechanism to put venvs elsewhere while still having a discoverable pointer to them).
  4. The proposal does not offer any new features[2] for people or tools who use multiple venvs for a single project.
  5. The proposal is opt-in - users who don’t change how they invoke venv will see no change[3].
  6. For users who do use the new features, tools that need to discover “the correct” virtual environment to use will be able to do so more easily. By following the existing convention of using the name .venv, the discovery process will also work for many users who do not opt into the new mechanism.
  7. Tools which need to discover virtual environments may choose whether to continue supporting other conventions (assuming they do at the moment) - the PEP cannot influence that choice.
  8. Tools which expect .venv to be a virtual environment may be confused by the new redirection mechanism. Any confusion should be minor, though, as finding a file rather than a directory should already be an error.

Put like that, it doesn’t feel like it’s making statements it can’t deliver on. But equally, the argument feels a lot less compelling - without an assumption that users will follow the recommended layout, tools can’t actually do anything differently from how they currently work. Which is the reality of what’s being proposed, but it’s harder to spot with the PEP as it’s currently written…

I should say, though - this PEP is more or less irrelevant to me as a user, because I already name my virtual environments .venv. So I don’t need the new features of venv, nor am I going to change my workflow in any way as a result of this. I suspect that’s true of a lot of people, but not everyone. As a result, getting feedback from those who are affected will be tricky.

  1. Unlike packaging, where the packaging community has effectively granted that authority to interoperability specs ↩︎

  2. Or if it does, explain them here! ↩︎

  3. Other virtual environment creation tools may or may not change their defaults, the PEP can’t mandate anything there ↩︎

5 Likes
8

One reason I often keep the environment out of the project is that I don’t want it affected by things like git clean. Having a gitignored .venv file in-repo isn’t really much better in that regard than having the venv itself being taken out.

(That said, I doubt I’ll ever use a tool that adopts this convention so it doesn’t matter much to me.)

4 Likes
9

That’s what git will do to find a .git folder, so it’s not unreasonable to expect that to happen.

2 Likes
10

Hopefully know to stop at the user $HOME

1 Like
11

Nope

$ git init
$ mkdir -p a/b/c/d
$ cd a/b/c/d
$ HOME=$(realpath ../) git add a
$ HOME=$(realpath ../) git status
On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
	new file:   a
12

We’re talking about tools with more context than just the command line here, though, so many will have other ways to figure it out (e.g. something like VS Code has the workspace root, where it’ll likely stop searching).

There’s nothing in this proposal to make plain python commands do any searching, and if other tools search in an unsafe way then they can also come up with their own heuristics to avoid risks.

13

To me that’s supporting the open issue of using “MAY” everywhere.

I already had that private discussion to get the list of supporting projects to even be motivated to do this PEP.

Look at the PoC for venv.executable().

Correct.

I think that’s being too prescriptive.

If you have a default virtual environment then specify it. Otherwise the PEP takes no position or you like the open issue on the topic.

But not all tools that use a project’s virtual environment are the same tool that created the virtual environment (e.g. your code editor).

That’s covered in the PEP twice.

And I didn’t want to be the one to define it either since I knew the monorepo users would have differing opinions.

In my mind, it’s the directory you open in your editor.

If you look at the PoC for venv.executable() you will see it looks at a starting point and goes up the directory hierarchy from there. I suspect for any tools consuming a .venv file it will be a search from the current working directory or the appropriate directory the tool is being pointed at.

I will say all the workflow tools that are likely to write out a .venv file were not concerned about this.

Yeah, that’s a whole other thing that I’m purposefully avoiding.

But then no one has authority, and so that means trying to make any guidelines are also futile.

That’s the “MAY” open issue to me.

That’s the other open issue.

Same here (hence why the Python Launcher has been using any .venv it found from the start).

And yet beyond the tools listed in the PEP their hasn’t been a supportive post yet.

2 Likes
14

We really aren’t, though. The PEP doesn’t specify which tools, and while I agree that an IDE like VS Code has more context, many other tools (especially command line ones) just don’t. For example, if someone wanted to propose that pip default to installing into a discovered venv, pip has no useful context on where to stop such a search.

For tools that don’t have the right context, this means that it’s possible to fail so that rather than raising an error “This project doesn’t have a venv”, the tool uses (in pip’s case that would mean installing packages into) a venv that is completely unrelated to the project.

Of course, none of this is a new problem. Tools can (and do!) implement this sort of discovery already. All the PEP offers is some vague sense that it’s “official” - without really making it likely that users will change their habits much.

2 Likes
15

My point is that people who won’t be affected are probably just saying nothing, because they gain nothing as well as losing nothing. So you’re probably not going to get supportive posts[1]. The problem is you also won’t get many negative posts from people who are negatively impacted, because there aren’t many of them (especially if you’re restricted to DPO participants).

So you’re left with people like me and Steve, who aren’t personally affected, but who are trying to channel people who are.

I’d be very curious to know what the projects who support the PEP expect to get from it. My gut feeling is that they likely feel that it gives them license to focus only on people who follow the .venv convention - but that’s pure speculation, and I’d much rather have some facts. As I noted, I don’t actually see anything in the PEP that would provide a real benefit, so knowing what I’ve missed would be helpful to me.

  1. Let’s be honest, PEPs rarely do, but if they do it’s from people who gain something, not from people who are unaffected ↩︎

5 Likes
16

I assumed they basically just agreed that .venv is a fine default name and weren’t too worried beyond that.

Any tool running in the context of a terminal environment should be checking the VIRTUAL_ENV environment variable (or PATH) before doing any search for this, or are already running inside a specific environment and so don’t have to worry about finding one. The launcher in the Python install manager won’t change its behaviour here, since it deliberately only reacts to an active venv, while this addition helps choose which one to activate.

(The most valuable tool that could be added is a generic activate script that figures out what to activate based on the CWD, which would save people from having to type .venv/bin/activate or .venv/Scripts/activate. That’s the kind of 3P tool that would have to do a search.)

And just to be clear, I’m +0 on the proposal. My only pushback is that the text can probably specify less and still capture the essentials that need SC approval. But if the over-specified parts aren’t causing too much conflict, then there’s no real harm.

2 Likes
17

Sorry, I shouldn’t have misrepresented your position.

1 Like
18

I’ve posted on Mastodon and Bluesky and it’s all been positive (albeit not a ton of comments, but I’m used to getting at least some negative to my PEPs :sweat_smile:).

A good example is Python Environments - Visual Studio Marketplace which exists because it’s so much work to find one’s virtual environment when you don’t mandate a workflow tool. You can’t assume someone uses a terminal to launch VS Code, so you can’t expect an activated environment. A lot of tools support an environment in .venv, but most don’t create one there by default. And so you need bespoke support for each tool that creates a virtual environment to know where to look (whether that’s from support being baked into the extension or using a provided API to give those details to the extension via another extension); and this all only works for VS Code. And then there are tools which don’t provide a way to point at what environment to use by default when there are multiple options

This PEP would give tools an agreed-upon way for tell e.g. editors what environment to use regardless of whether they want it in a .venv directory or not. It also gives them a reasonable default if they don’t already have a preference. And this wouldn’t be specific to any one editor (or even editors, period).

5 Likes
20

This view irks me a little, because it only works that way when viewing it as if all projects will only have one relevant venv. If we need a way to handle this, it should handle it for all reasonable workflows. this proposal writes that off explicitly, which is taking a stance on which projects (including ones people already have proper support for in python) are somehow wrong for using the tools a specific way.

There’s also no reason to hijack the convention of a common directory (.venv) to make it a standard that it should be an informational file instead here when we have a mechanism for project information already.

The obvious answer to me is to add to pyproject.toml and to declare that tools may configure the venv path they should use in pyproject.toml, may allow configuring multiple venvs for different purposes, and declare a project level key that can be read as a default venv path by any tool needing it if present.

That leaves all of the complexity of the multi-venv use to the tools that need it, without implicitly saying a workflow involving multiple venvs isn’t important by convention, while still providing the simple answer in a value belonging to the project, not to specific tools, for the tools that only need to know what the project wants used for any tool not needing further configuration.

2 Likes
21

So are you saying you like the open issue about this? And do you expect every tool that needs to know what virtual environment to use to ask the user which one to use? This is why I have been saying “default” over words like “relevant”.

Can you please phrase this as a response to the open issue in the PEP?