PEP 832: virtual environment discovery

[mikeshardmind (Michael H)]

@brettcannon

@mikeshardmind

This view irks me a little, because it only works that way when viewing it as if all projects will only have one relevant venv.

So are you saying you like the open issue about this? And do you expect every tool that needs to know what virtual environment to use to ask the user which one to use?

Not as presented, but more than the actual solution posed that hijacks a well-known directory for another purpose. As for each tool asking, no, and I covered that already in my response here:

@mikeshardmind

and declare a project level key that can be read as a default venv path by any tool needing it if present.

The part preceding this regarding what tools may do in tool tables is to ensure that whatever is landed on remains clear that tools are allowed to configure more without violating this standard, while the project level key is intended for the tools that don’t need more than whatever default makes sense for the project, covering all of the cases that the current proposal would be capable of.

[mikeshardmind (Michael H)]

@brettcannon

@mikeshardmind

The obvious answer to me is to add to pyproject.toml

Can you please phrase this as a response to the open issue in the PEP?

I’m not sure the best way to phrase this in more detail while in response to that question right now. I’m not ignoring this, but I’ll have to come back to it tommorow.

[aragilar (James Tocknell)]

@brettcannon

A good example is Python Environments - Visual Studio Marketplace which exists because it’s so much work to find one’s virtual environment when you don’t mandate a workflow tool. You can’t assume someone uses a terminal to launch VS Code, so you can’t expect an activated environment. A lot of tools support an environment in .venv, but most don’t create one there by default. And so you need bespoke support for each tool that creates a virtual environment to know where to look (whether that’s from support being baked into the extension or using a provided API to give those details to the extension via another extension); and this all only works for VS Code. And then there are tools which don’t provide a way to point at what environment to use by default when there are multiple options

Is the target audience for this PEP then IDE plugin developers and environment managers? I feel there’s nothing inherently wrong with the PEP, it just feels a bit vague (e.g. .venv is a fine default, but there’s nothing about multiple venvs as would occur when you’re using tools like tox).

I think if this is actually about making it easier to use IDEs with various tools, then maybe this should be stated more clearly in the PEP?

Assuming this is for IDE ↔ environment manager interop, maybe creating a file where tools can put standard hooks (e.g. lookup venvs, get default venv, create new venv) would be a more future proof way of handling this interop? This would also avoid issues around git clean deleting what is effectively a config file.

[jeff5 (Jeff Allen)]

@mikeshardmind

There’s also no reason to hijack the convention of a common directory (.venv) to make it a standard that it should be an informational file instead here when we have a mechanism for project information already.

Yes, this seems a clumsy way to provide an alternative name. Tools that expect .venv (or any venv) to be a directory will fail in obscure ways if there is a (hidden) file of that name instead. Arguably it’s a bug in the tool, but I’m thinking here of simple home-made tools like a bash script that has worked for years until some IDE creates that file.

My practice is to have several virtual environments in a single project, with meaningful names. E.g. a one (from a pip freeze) for reapeatable results and another for analysis that can be freely updated. Having them all end in .venv seems a useful convention, and privileging the one with no name-part.

[steve.dower]

@aragilar

Assuming this is for IDE ↔ environment manager interop, maybe creating a file where tools can put standard hooks (e.g. lookup venvs, get default venv, create new venv) would be a more future proof way of handling this interop?

Yes, but we’ve discussed it multiple times in the past and it always fails (for much the same reasons we’re seeing here - people can’t tell the difference between “a basic default” and “the full scope of potential tasks”).

This PEP is Brett deliberately scoping out all those other things, because the last attempts didn’t get anywhere, and getting a predictable way to find the right virtual environment for a directory is more important.

[mikeshardmind (Michael H)]

@brettcannon

@mikeshardmind

The obvious answer to me is to add to pyproject.toml

Can you please phrase this as a response to the open issue in the PEP?

I don’t think the solution I view as obvious is a response to the open issue about an API

• Add default-venv-path key to the project table in pyproject.toml, that indicates the default venv path for the project absent tool specific configuration.
• Explicitly make clear that tools are still allowed to do something else the user has requested, and that this is just a default in the absence of other more specific information.

This leaves out all the ways to create a venv, ask some workflow tool, etc, this is just the expected default path, in the file we already have for project settings, while leaving it crystal clear that this isn’t meant to replace anything more complex that tools may already be capable of and leaving those tools without an implied “this use isn’t standards compliant”

[brettcannon]

@mikeshardmind

but more than the actual solution posed that hijacks a well-known directory for another purpose.

If you don’t like having redirect files be named .venv then that’s fine, but see below for my motivation (which is in the PEP).

@mikeshardmind

The part preceding this regarding what tools may do in tool tables is to ensure that whatever is landed on remains clear that tools are allowed to configure more without violating this standard

I didn’t intend to have the PEP suggest that having more than one virtual environment was improper somehow. Is there some specific wording that made you think that? Or are you just asking for an explicit sentence stating multiple virtual environments are fine?

@aragilar

Is the target audience for this PEP then IDE plugin developers and environment managers?

They are one of the target audiences, but not the sole audience.

And I will say that PyCharm supports the PEP. I’ve asked them to please participate in the discussion here.

@aragilar

but there’s nothing about multiple venvs as would occur when you’re using tools like tox

Correct, because you are typically only using a single virtual environment directly at a time. For example, when you run mypy, do you point it at multiple environments simutaneously or just one? Just like you don’t type python and have it work against multiple Python interpreters simultaneously, same goes for anything else that needs some environment to work.

Also note that @bernatgabor , as the maintainer of tox, supports this PEP.

@aragilar

Assuming this is for IDE ↔ environment manager interop, maybe creating a file where tools can put standard hooks (e.g. lookup venvs, get default venv, create new venv) would be a more future proof way of handling this interop?

Please read the open issues section where there’s a proposal for an API. If you still prefer your file idea then please rephrase it as a response to that.

@jeff5

Yes, this seems a clumsy way to provide an alternative name. Tools that expect .venv (or any venv) to be a directory will fail in obscure ways if there is a (hidden) file of that name instead.

That’s acknowledged in the Backwards Compatibility section (as is what I say below).

So let’s say we change the name used to something other than .venv (e.g. .venv-location). Those tools that would have “fail[ed] in obscure ways” with a .venv file would also not notice there was a virtual environment to use and thus assume one didn’t exist, so now you may have two virtual environments. So yes, it’s “clumsy”, but it also oddly “backwards compatible” in that tools won’t accidentally ignore a virtual environment either that exists elsewhere.

@jeff5

Having them all end in .venv seems a useful convention, and privileging the one with no name-part.

But that only works if you keep the virtual environments together in your project which some people really hate doing.

@mikeshardmind

Add default-venv-path key to the project table in pyproject.toml

I wouldn’t add it to [project] as I don’t think that’s an appropriate place, but @cjames23 of Hatch asked privately for equivalent support in the workflow.environments key from the open issue.

But here’s the question/concern with that idea: should projects be able to dictate your workflow to that extent? The PEP lets you keep your virtual environment wherever you want with a small file pointing to it. But what you’re proposing mandates where the virtual environment goes in the file sytem. And based on the years I have spent talking to people about this topic (and I’m not exaggerating on that time scale), some folks have strong feelings about their virtual environments all living under a single directory somewhere away from a project’s code. Now I get that in a controlled environment like a company, you can make such demands. But that doesn’t extend to all Python users wanting that.

[mikeshardmind (Michael H)]

@brettcannon

@mikeshardmind

The part preceding this regarding what tools may do in tool tables is to ensure that whatever is landed on remains clear that tools are allowed to configure more without violating this standard

I didn’t intend to have the PEP suggest that having more than one virtual environment was improper somehow. Is there some specific wording that made you think that? Or are you just asking for an explicit sentence stating multiple virtual environments are fine?

An explicit sentence that whatever the mechanism ends up on, that other cases not covered by this should continue using whatever methods are appropriate.

Unfortunately, I’ve dealt with situations where one blessed way, and other paths unmentioned leads to people being more averse to using other paths even when they are the right choice, out of (IMO, excessive) concern that their way may be broken in the future. I’d just want to head off anyone needing to deal with such.

@brettcannon

But here’s the question/concern with that idea: should projects be able to dictate your workflow to that extent?

Since we’re only configuring a default, why not? I assume that configuring a venv management tool to place venvs in a separate directory, separate from code is an explicit signal not to use the default.

I think what I’m missing here is why you see venvs as something that are both something you should be able to provide a default path for in a project, but that the default shouldn’t belong to the project. If I shared your view of it, then I think it would be easier to view this as something that should be a seperate file.

@brettcannon

@mikeshardmind

but more than the actual solution posed that hijacks a well-known directory for another purpose.

If you don’t like having redirect files be named .venv then that’s fine, but see below for my motivation (which is in the PEP).

The problem I see with this, is that it actually promotes any existing venv named .venv to default status, which might not be the user’s intent in a multi-venv situation, or in a situation where they invoke some tools through a workflow tool that keeps per-tool venvs in a managed directory.

We should use something less likely to conflict with existing use, even if it makes sense to have it be a user-provided additional file not belonging to the project.

[brettcannon]

@mikeshardmind

An explicit sentence that whatever the mechanism ends up on, that other cases not covered by this should continue using whatever methods are appropriate.

I can do that when I do my first round of updates.

@mikeshardmind

I assume that configuring a venv management tool to place venvs in a separate directory, separate from code is an explicit signal not to use the default.

I don’t think that’s necessarily true. Just because e.g. virtualenv stores named virtual environments in a global directory doesn’t mean there isn’t a specific one the user plans to use with a project, it’s just how virtualenv chooses to store virtual environments by default.

@mikeshardmind

I think what I’m missing here is why you see venvs as something that are both something you should be able to provide a default path for in a project, but that the default shouldn’t belong to the project.

Because not all projects mandate a workflow in order to work with them. As such, users can do what they want. Some people love that while others hate it. This PEP tries to support either as the Python community doesn’t have a blessed workflow or tool, which is why this PEP (and most of my packaging PEPs) are trying to come up with something that tools can agree to so you can use what tool you want and I can use what I want and yet we can still work on the same code base.

So my answer is that unless a project has a mandated tool or some other reason to restrict where the virtual environment is, enough people seem to think it isn’t any of our business where they put their virtual environment.

If I had my way it would be .venv, period, and you recreate as necessary (like what uv does by default). But based on my conversations with people, that wouldn’t be acceptable, hence this solution of a cheap redirect file at a well-known location.

@mikeshardmind

The problem I see with this, is that it actually promotes any existing venv named .venv to default status, which might not be the user’s intent in a multi-venv situation, or in a situation where they invoke some tools through a workflow tool that keeps per-tool venvs in a managed directory.

If .venv. isn’t meant to be the virtual environment to pick up, you’re already working against a broadly accepted convention (which I’m trying to codify), so I think you’re already working against things in that instance. This also means me suggesting another name is a more uphill battle.

As for separate environments for tools, I don’t think this plays into this either as those wouldn’t be environments you would be installing e.g. pytest in along with your project’s code or expect your auto-complete or linter to examine for what’s installed (like mypy saying it can’t find the types for something because it isn’t installed or Pylance not giving completions).

[mikeshardmind (Michael H)]

@brettcannon

@mikeshardmind

An explicit sentence that whatever the mechanism ends up on, that other cases not covered by this should continue using whatever methods are appropriate.

I can do that when I do my first round of updates.

Thanks.

I don’t fully agree with the perspectives that seem to contribute to this specific solution, but I appreciate the time spent discussing the specific bits of the rationale, and it makes enough sense that so long as it is clear in purpose and in scope, I’m fine with it even if I don’t think it’s the best option.

There’s probably a better way here involving well-known user config locations since this is basically saying the user owns their workflows, which wouldn’t place loose config files that dont belong to a project in a project’s directory, but then it’s also probably something more involved to propose, and probably requires more work to design. I know this has been something where some solution is better than waiting on full agreement on a perfect outcome, so this may be the best option for progress available right now.

[aragilar (James Tocknell)]

I want to emphasise that I’m in no way opposed to the PEP (I’m probably +0.5, given I’ve seen the mess people make of their systems when they try to configure their IDE, so my comments should be read as suggestions not objections), I just think there’s some editing that can be done to make it clearer what the intent is.

@brettcannon

Correct, because you are typically only using a single virtual environment directly at a time. For example, when you run mypy, do you point it at multiple environments simutaneously or just one? Just like you don’t type python and have it work against multiple Python interpreters simultaneously, same goes for anything else that needs some environment to work.

Not sure if this will help or not, but my default workflow is to run static analysis tools like mypy via tox (and if I was concerned about per-Python-version aspects of the tool, I’d use a venv per Python version). If I wanted to inspect a particular tox-created environment I’d activate it, and for doing one-off tests/experiments I’d use virtualenvwrappers temporary venvs (and if I was doing multiple experiments, I’d create a shell and temp venv per experiment to keep things clean).^[1] I don’t see the proposed causing any issues for me, assuming tools which use a single venv continue to use the active venv over creating their own/searching for a venv.

---

1. For what is probably best described as “data science tasks” I do use the named virtualenvs from virtualenvwrapper, this is for long running (e.g. multi-day) tasks where I do want to keep the venv around for comparison between versions. So there would be multiple venvs activated as well. ↩︎

[j-carson (Janet Carson)]

Yes, I also have different venvs – It’s not stated explicitly but presumably my “venv-switch” command would change the pointer in the .venv file to the correct .venv-purpose directory for the venv that I need to switch to?

I think a few more details added to the proposal that explain the UX for users who need multiple venvs for different purposes would be helpful.

[tiangolo (Sebastián Ramírez)]

This is great!

I think it’s great having a default location for venvs defined as a standard.

It would be quite useful for tools I’m working on, related to FastAPI.

Additionally, and I think more importantly, I think it would be very important for making Python more welcoming to newcomers.

I also think it’s useful for AI stuff.

Details

New tools for FastAPI and others

I’m currently working on tools to inspect AI “agent skills” by libraries, starting with work for FastAPI, for this I need to make the tool capable of running on its own temporary venv (e.g. via uvx), but be able to discover some specific project’s venv to inspect the libraries installed there.

Having a standard for this would simplify heavily that type of work.

Right now I would need to do a bunch of manual work to try and cover many use cases, and have lower certainty that I’m even following the “right” conventions.

Welcoming newcomers

One of the biggest struggles I have seen from people coming to Python (learning and adopting it) over the years has been installing Python and setting up virtual environments (at least before uv came along).

When I’ve done Python (and FastAPI) workshops, there’s a non-negligible amount of time (hours) dedicated to solving the audiences’ mess of venvs they end up with while trying to set up things.

One of the (very few) complains I hear about Python from newcomers is handling, understanding, and using venvs. E.g. in JS there’s just node_modules and that’s it, people can npm install and expect a node_modules directory, users and tools can assume it’s there (including any caveats too, but it’s always consistent). There are no cognitive branches (e.g. is the venv there? Somewhere else? There’s no venv? etc).

If tools, editors, and users can (soft) assume a standard default, it improves things all around.

AI Agents and stuff

Independent on everyone’s views on AI, agents and all that stuff, in the end they are probabilistic, sophisticated tools. Those work much better with standards.

[cjames23 (Cary Hawkins)]

@brettcannon

But here’s the question/concern with that idea: should projects be able to dictate your workflow to that extent?

If you are pulling down a project to work on it, my opinion here is a strong yes. My first jump into open source development was Airflow and their opinionated stance on what you do when developing against their code base makes it far easier for newer contributors in my opinion.

In fact I have been thinking about how to add some more opinionated workflow related concepts for hatch for contributions in the same way to make it easier for newcomers to get started with contributing. It also gives me more assurance as a maintainer that the contributor’s code has followed steps that make it easier to avoid nitpick comments.

[pf_moore (Paul Moore)]

@cjames23

If you are pulling down a project to work on it, my opinion here is a strong yes. My first jump into open source development was Airflow and their opinionated stance on what you do when developing against their code base makes it far easier for newer contributors in my opinion.

Having a default workflow seems OK to me, but dictating a workflow is more problematic. Suppose I’m working on a project, and my project directory is on a disk with some sort of constraint on it (maybe a fairly small space quota, for example). I could have very good reasons not to want my project venv to be in the project directory. With PEP 832, I can just create a .venv file pointing to where I want. If the venv location was held in pyproject.toml, I couldn’t do that, without maintaining a local patch to pyproject.toml overriding the project policy^[1].

Yes, people with special requirements are relatively rare. But when they do come up, “just create a file pointing to your preferred location” is far more beginner^[2] friendly (as well as being something an IDE could help you with) than “change pyproject.toml, but be careful not to accidentally commit that change or include it in PRs”.

---

1. Of course the same is true if the project checks a .venv file into the VCS, but the PEP strongly implies that would be considered bad practice. ↩︎

2. not just beginner, actually! ↩︎

[mikeshardmind (Michael H)]

I do still think some of the parts that are open to future improvements/work because the pep is intentionally narrowly scoped to get a solution for the immediate term moving, hints at a stronger reason to not treat .venv as the place this exists at. I would prefer .venv.info or something else of that sort, with tools allowed to assume that in the absence of that file or any tools specific configuration, and the presence of a venv located at .venv, that the venv at .venv is the default.

This should also behave better if this is ever expanded to better interact with more complex needs, without being specifically more complicated for the current scope.

@pf_moore

Yes, people with special requirements are relatively rare. But when they do come up, “just create a file pointing to your preferred location” is far more beginner friendly (as well as being something an IDE could help you with) than “change pyproject.toml, but be careful not to accidentally commit that change or include it in PRs”.

To me, there’s an orthogonal solution for problems shaped like this: An overrides file, eg. pyproject.local.toml, for expressing local overrides to pyproject.toml. I don’t have enough info and haven’t heard enough to think this is needed currently, but a solution like this allows ignoring the local overrides from version control, without requiring users manually track their local changes that shouldn’t make their way into commits, and would avoid any other issues where we can’t place a key in pyproject.toml due to the difficulty of overriding locally (allowing user > project > defaults while continuing to use the standard mechanisms)

[pf_moore (Paul Moore)]

@mikeshardmind

To me, there’s an orthogonal solution for problems shaped like this: An overrides file, eg. pyproject.local.toml, for expressing local overrides to pyproject.toml.

Agreed, with the amount of use nowadays of pyproject.toml for purposes other than the original one of holding project metadata (which generally shouldn’t be overridden by the user) I think that such functionality would be useful. It would be a different PEP, of course, and would probably spill over into the whole “projects that aren’t intended to be built as a wheel” debate. So I can understand people being reluctant to get into that one