PEP 832: virtual environment discovery

PEPs
42

I would expect it to.

The PEP purposefully does not dictate UX as that’s a tool’s choice. I’m happy to say the PEP is entirely neutral on having multiple virtual environments. In fact, I would argue the PEP helps with the support of multiple virtual environments by making them work better with tools than now where there’s no way to help guide a tool to know which environment to use.

A symlink to a file or a directory? I don’t quite follow this since .venv is an entry in the file system and I’m trying to make the PEP say “take that entry as it is and work with it accordingly”. That takes away any preference in format and questions of how tools should look for .venv in parent directories.

:+1:

Let me think on it. I’m at least open to saying “first line” since you can do head -n 1 .venv | tr -d '\n' or Get-Content .venv -TotalCount 1.

That’s great, but that doesn’t solve the problem I’m trying to tackle here. Let’s say you launch Emacs (and for those of you who don’t know, Barry is a fan), VS Code, or PyCharm from a shortcut on your desktop. That means there’s no activated virtual environment or any other environment variables to give you hints as to what environment to use. With your editor now launched, you open your project’s folder. Now, how is your editor to know what virtual environment to use for anything (which matters for things like auto-complete)? There might not be any hint you use Hatch. As well, even if your editor can deduce you use Hatch, your editor would have to bake the knowledge in. And as I listed earlier, that would mean 3 different editors all coming up with some way to detect Hatch usage and knowing where virtual environments are kept (or the Hatch maintainers having to do that work somehow), let alone then prompting the user to choose which virtual environment to use (and we all know there are more than 3 code editors being used by people in the community, so I don’t view this as scalable based on my experience with trying to make this all work with VS Code).

Now ,one way to solve this in a very generic fashion is the open issue around some defined API in pyproject.toml to get structured results, but my admittedly parent-to-a-2-year-old brain can’t remember anyone liking that idea (assuming everyone read to the end of the PEP), plus that is heading towards a “project dictates workflow” approach without defining a separate pyworkflow.toml or pyproject.local.toml as suggested for this sort of thing.

So my question to you (and anyone who doesn’t like this PEP) is how do you solve the problem outlined above? How do you get something like a code editor the information of what virtual environment to use without making them have to duplicate/replace your workflow tool? And BTW, workflow tools don’t have a guarantee of installing the same thing, so there’s the potential for differences in the virtual environments.

I brought that idea to the SC during OH in Feb (which you attended), and the response I got was “that’s outside of the purview of the SC”. Based on that (lack of) guidance, I started that work with this PEP so I could at least get virtual environment locations standardized so that wouldn’t be a controversial thing to add to the Python Launcher for when long-lasting virtual environments are required. I don’t think using ephemeral virtual environments for everything works unless you have a way to give the location to tools that need the resulting virtual environment based on what your installer will install.

2 Likes
43

Agreed, given that PEP 582 was widely unpopular because it didn’t use a “real” virtual environment, it seems a normal venv is going to be needed for any smooth workflow from any of the launchers (also for Barry’s suggestion, the idea of python doing anything more than running the interpreter in its current environment was also rejected, so I wouldn’t put any weight on that - it’s py or nothing/someone else).

1 Like
44

Now we’re getting somewhere! I alluded to this problem in my original message, but this really does clarify the problem the PEP is trying to solve, because I do exactly have this problem. Perhaps more narrowly focusing the PEP on this problem[1] would help?

When I launch Emacs from my macOS desktop[2], I want to run eglot as my language server client, and pyrefly as the LSP server. I have some custom elisp which knows how to invoke pyrefly through a particular hatch environment name, using hatch run syntax. There are pros and cons.

The cons are fairly obvious: the hatch run command only works in my own personal projects where I control both the use of environment manager (i.e. hatch) and the environments I define for that project in my pyproject.toml. It’s exactly why I get errors when I’m editing Python code in “foreign” repos such as CPython or other projects I don’t maintain.

The pro is that on the IDE[3] side, I don’t have to expose my desktop’s peculiarities to anybody else, because the connection between the project repo and my editor is local to my own configs, as IMHO, it should be.

Note that I don’t need an activated venv to make this work, I just need a way to invoke a command from a venv, which in my case is through hatch. Which kind of gets at my other comment about invocation through py or somesuch, but we can come back to that later.

So now thinking about this PEP in the context of my own enviroment, I would of course like to have a way to connect some venvs with the project or its tools with my IDE. In my case, I need a command to run so a directory alone doesn’t help. Or maybe it does if I can turn the contents of that file into a shell command that appends the venv’s bin directory and invokes the command. Which is likely easy for me and trivial for $coding-agent-of-the-day.

Although come to think about it, there really are better ways to do what I’m doing anyway since none of that is really tied to a project’s venv. I can imagine though that in some cases, you do need tools or scripts that are tied to the project (e.g. perhaps a CLI that the project defines) and you need a way to hook them together.

Oh pshaw that Steering Council :laughing:. Just as a developer I can’t shake the thought that there’s a there there if we stirred all the existing ingredients just right.

  1. although it doesn’t have to be super tightly focused on IDEs ↩︎

  2. using Quicksilver but really that’s immaterial ↩︎

  3. to classify Emacs generously :winking_face_with_tongue: ↩︎

45

You’re likely right. python should Python and leave it at that. FWIW, uv already does this, but uv does a lot else too, so some narrowly focused shebang runner that handles 723 metadata and can be configured to do a thing with a local editable, would be very cool. py to me is the natural UX for that.

1 Like
46

To answer part of this, specifically hatch maintainers having to do that work. I would say we do have some part in this, we have an open issue requesting changes to env show in hatch for the json args from PyCharm for their hatch environment detection logic to improve.

I will say as one of the maintainers that IDEs not always properly detecting hatch environments for a given project was why I was in support of this PEP. I am a maintainer and have problems with PyCharm and VSCode based IDEs not always detecting the correct or even a single hatch environment, if I have this problem then someone using my tool definitely has faced these problems.

5 Likes
48

I take that back. After experimenting with a few alternatives, I think this does in fact need to be tied to the project’s venvs in some way.

49

To a directory.
I’m asking if a symlink to ../shared-venv should work the same as a text file containing ../shared-venv. The difference is a flag that tools could take advantage of (in incompatible ways), and I’d be happier if the PEP explicitly said they shouldn’t.

1 Like
50

What difference in interpretation could there be? Whether to display the path to the venv as the path to .venv or the path to its target?

Brett can answer definitively, but I would treat a symlink as if it’s the real file/directory (and let the OS figure it out when it appears in paths). So that’s a vote for “work the same as if it was a directory if it’s a link to a directory or the same as a file if it’s a link to a file” - in other words, don’t treat it specially at all.

4 Likes
51

That’s what I’m assuming.working towards.

OT and one thing at a time; I have plans and this is an early step in those plans. :wink:

I can at least add it to the Motivation section explicitly.

And as the person who used to manage the team that handled this for VS Code, I have seen the problems as well. Virtual environments and any tools that create/manage them can both sit at the top of one’s toolchain stack (i.e. running the tools in the terminal) or be one step down/parallel to other tools (i.e. theiir outputs being used by editors), which makes this whoel situation:

  1. A pain that plenty of people see.
  2. Something everyone has an opinion on how to solve since this touches people’s DX tools.

That’s what I intended, so I will clarify that in the next/first update.

2 Likes
52

I’ve updated the PEP to cover some of the things discussed here:

  • Add PyCharm and library-skills as supporters
  • Have redirect files read up to the first newline (@encukou )
  • Clarify there is no opinion to having multiple virtual environments (@mikeshardmind )
  • Explicitly use the code editor example for the motivation (@barry )
  • Have venv.executable() be configurable for the virtual environment name
  • Clarify symlinks are not to be treated in any special way (@encukou )
  • Move the Rationale after the Specification and simplify the latter by moving details to the former
  • Loosened things involving “MAY”, “SHOULD”, and “NOT” so tools are not required to do anything beyond how they interpret a redirect file (@pf_moore , @steve.dower )
  • Removed the previous open issues and replaced them with one about listing all virtual environments in the file (I thought someone made this suggestion, but now I can’t find a post about it)
  • Clarify no direct stance is taken on what a “project” is (@pf_moore , @aragilar )

There’s now one open issue around whether to have stuff passed the first newline mean anything. For the multiple environment situation I could see the .venv file listing all of them while having the first entry be the default one to use. You could even squint and view everything passed the first line as a TSV of virtual environment path and some label you may want some tool like your code editor to show people to help in choosing (e.g. virtualenvwrapper could append to the file any time the virtual environment swapped in order to list all virtual environments that were used and use the name for the virtual environment). You could also get fancy and make it JSON or JSONL instead of a TSV if it was warranted.

But this is only worth talking about if the people who were objecting to this PEP over the lack of multiple virtual environment support would actually change their objections with some change like this. If it won’t then it’s just added complexity that’s not worth talking about and can be left as a postponed idea.

5 Likes
53

I’ve been away for 13 days on a work trip (& mini-vacation) to Paris and just returned yesterday. I don’t have much time to organize a proper response here but I wanted to provide context for my opposition to the proposal so I’m going to heavily copy my comments from the pre-PEP discussion with the mentioned stakeholders. Any text I post that does not come from me is present to maintain the trail of thought and is anonymized.

TL;DR:

I view the best course of action that would satisfy everyone to be the construction of an API for environment managers themselves. There was actually a proposal several years ago (pre-uv though) with feedback/buy-in from several maintainers.

My opinion

N.B.: The opinion of @cjames23 should carry more weight than mine as he has more time to maintain Hatch these days.

I very strongly dislike the original proposal for the following reasons:

  • The .venv path suddenly becoming a file will surely break users and their tools in ways we cannot foresee. More importantly though, the proposal doesn’t specify requirements for the “default” environment. It says the virtual environment a workflow tool that is external to the one that wrote the .venv file is expected to use with the example that it should match the behavior of hatch run.

    Hatch uses the default environment unless specified by the user. It’s often the case that projects define specific environments without ever configuring the default and inheriting that base configuration. Are we fine with the default environment being unable to run tests, or even having the project installed should the project configure it as such?

  • The lack of support for multiple virtual environments leaves a feature gap between tools that would be confusing and unwelcome to users. It’s only possible as a rejected idea because uv doesn’t currently support the concept but unless I am severely mistaken that will happen eventually and therefore this proposal is not future-proof.

    Consider a very simple project that has an environment for every supported minor version of Python (py3.14, py3.13, …) with a non-negligible risk of behavioral differences between them. How would a user in an IDE, which is the primary use case here, select the environment they’re currently having issues with and press the test buttons? If the answer involves manual interaction with the environment manager to update the state of the discovery tool then we have failed.

    Also, as [redacted] mentioned, we appear to be not considering projects with complex requirements, monorepos or environments that are not on the host. As an example, consider that Airflow already experiences dependency conflicts and would benefit from easy selection of specific environments rather than their current recommendation to manually enter a workspace member for targeted testing.

I love the new pull-based concept however and would be 100% in favor if:

  1. The file idea was removed.

  2. Multiple environments were supported such that existing custom logic for the various environment managers in VS Code could be implemented with the new mechanism. The most important initial functionality would be the environment/Python dropdown selector.

  3. The expected capabilities of the default environment are specified with the SHOULD keyword.

I would potentially also be in favor of another PEP that specified user-defined tasks that could be executed within certain environments. @bernatgabor and I have been very much against attempts to standardize command execution without regard for where nor how it happens. If we have an API for tools to trigger the environment orchestrator then we unlock a ton of cool stuff.

Pull-based concept

What I referenced above was an idea from the thread:

If we go from the PEP’s “push” approach (i.e. workflow tools push out where the virtual environment) to a “pull” approach (tools pull where the virtual environment), then I think that could address these concerns, but at a price of more configuration.

So, one possibility is to introduce a [workflow] table to pyproject.toml. In that table, you can specify a command to use to get the virtual environment path from stdout when the status code is 0.

[workflow]
virtual-environment = {tool = "pdm", cli = ["pdm", "venv", "--path"], shell = false}

I could see this supporting string substitution for things like the path to the pyproject.toml file or its containing directory.

The tool key is just informational like pylock.toml’s created-by.

shell is for those that use some shell thing and would be false by default.

Having people add this to their pyproject.toml, but I think most of the tools here have an init/quickstart which would write this out.

Extended version I support:

Let’s look at what “complicated” might look like:

[workflow]
environments = {tool = "...", command = ["..."]}

That gives you a command to run in a subprocess with a cwd of where the pyproject.toml file is (we could add shell = "..." if people wanted), along with a tool name to include in any error message if the tool wasn’t installed. That command prints out JSON to stdout:

{
  "version": "1.0",  // ... or whatever to version the JSON schema.
  "default": 0,  // Optional
  "environments": [
    {
      "path": ".venv",
      "python_version": "...",  // Optional
      "name": "..."  // Optional
    }
    ...
  ]
}

default gives you an index into environments to find what environment should be used if you don’t want to bother asking the user what environment to use. environments.path points to the location of a virtual environment. The other stuff is for tools to show the user if they were to make a choice of which environment to use. Tools are allowed to create virtual environments as a side-effect of all of this. I assume “name” would have any details like what extras and dependency groups are installed in the environment if a tool cared to share such details.

And if workflow.environments isn’t defined, look for .venv next to pyproject.toml.

Now, I’m going on the assumption people are not running workflow tools out of the virtual environment as that’s a bootstrapping issue. But there could be stuff in workflow.environments to say to install certain dependency groups into .venv to then have the tool available if people wanted, then the tool can fill in anything else they wanted installed when asked to list the virtual environments.

Another addition could be letting people list a path instead of a command because they work in a controlled environment.

Purpose extrapolation

A comment and response by others:

There may be a benefit if further features are added to the [workflow] table as proposed by ofek (if I understand correctly).

I think that’s the question: is flexibility worth the complexity for this situation.

Someone answered:

  • Automatic virtual environment creation as a side-effect of requesting the location

    • This can lead to easier bootstrapping for tools like editors or AI since their request for virtual environments would also create the environment

    • With the PEP, something like an editor opening a fresh checkout wouldn’t have a virtual environment set up

    • E.g. VS Code or AI could get a virtual environment created on their behalf without having to first notice there’s no virtual environment and then either inferring – incorrectly – how to create an environment or having to ask how to create one or ask the user to do it for them

  • Supporting multiple virtual environments

    • E.g. VS Code or AI would know about alternative virtual environments upfront

    • The PEP as it stands doesn’t support this […]

I answered:

  • Similar to your first point, my view is that we should meet the user where they’re at and never require changing their workflow. If they want to use a GUI 100% of the time then so be it, and the same goes for CLI, AI agents, etc.

  • Multiple environments are a hard requirement for many projects so that’s important to me.

  • The ability to have non-local Python environments e.g. in a Docker container running on your host, on a VM in the cloud, etc. and non-virtual environments like Conda. I think we should remove the word “virtual” from the PEP title and have the concept broadly be about “Python environments”. This may sound like too much complexity but it’s entirely possible and already something users rely on this moment with Hatch environment plugins. As long as you come up with an adequate high-level API the internals don’t matter.

  • The ability to instruct the environment orchestrator to run user-defined tasks/commands within certain environments would be great. I’m lukewarm on the idea of specific functionality carveouts just for running tests and building docs as folks have asked for in the past. However, I’m 100% in favor of allowing for arbitrary tasks. I’d imagine IDEs that have similar concepts would integrate with such standards where the list of user-defined commands populates what is available. Another benefit would of course be redistributors who would no longer have to re-implement test setups of every package and could rely more on projects’ chosen orchestrators to do the work after they add support for the orchestrator to their build system.

Follow-ups to my opinion

Will it be just a case of hoping that the tool is already available on the Path environment variable?

I think this is the best option indeed.

The expected capabilities of the default environment are specified with the SHOULD keyword.

I don’t quite follow this. What “capabilities” are you referring to since I don’t think of environments as having capabilities?

Assuming the API requires passing an explicit directory, something like:

If the workspace directory contains a pyproject.toml file defining a valid [project] table as specified by PEP 621, then the default environment provided by the environment orchestrator:

  1. SHOULD have the project installed in editable mode i.e. runtime imports always reflect the current state of the code but not necessarily project metadata
  2. MAY be capable of running the project’s tests
7 Likes
54

For those that didn’t notice, the proposal that Ofek is referencing is from me for an extension mechanism for the Python Launcher for Unix as another approach to tackle this problem. I ultimately didn’t pursue it do to complexity, but I’m not opposed to it either if people wanted something like it.

To summaize the proposal, it was to have some naming convention for executables on $PATH (e.g. _py_launcher_<tool name> from the discussion) for which executing would get you a list of interpreters/environments that were available . It was designed for the Python Launcher for Unix to have assistance from any workflow tools in finding interpreters and especially environments. As well, it was designed to also cover the data that one may want to see in VS Code when selecting an interpreter/environment such that VS Code could ship the Python Launcher for Unix to do this discovery (or re-implement the logic in TS).

There are two drawbacks to it. One is asking workflow tools to ship another command. It’s not the worst thing, but it’s not nothing either. Two, launching a subprocess or two isn’t cheap (since this only occurs for any workflow tools you have installed, I don’t expect people to have 10 of these; probably 1 typically, but at worst for like 5 you really like workflow tools). Once again it’s probably not that costly, but it isn’t as cheap as reading a file either.

This was the API which I had proposed as an open issue that people didn’t seem to latch on to.

1 Like
55

I understand that having “project” undefined for long has had sprawling consequences and made it quite difficult to define now. But I’m confused how this PEP can both not define “project” and include project_root in the stdlib venv module. Once project is in the stdlib due to a PEP, doesn’t it have both a de-facto and de-jure definition?

I help maintain a build tool that among other things supports “Python monorepos”, parameterization across multiple Python versions, etc. “How to set up your IDE” is a meaningful source of friction so I read the draft through that lens. I understand “monorepos” are out of scope, but I’m not sure even in the “single venv” case how much this helps the “tool doing stuff with venvs”<–>IDE handoff. Say a user opened up the IDE, read a .venv file, and now has made a change to a requirement, so the build tool has changed the contents of .venv. How does the IDE know that happened? If the contents of the .venv file have not changed but the directory it points to has, how is that signaled? Does updating a .venv directory need to be atomic? How does the IDE say “Hey I’m running the tests right now, wait before updating?” I get the motivation for a faster “time to IDE open”, but I’m having trouble seeing how enough building blocks are specified to actually “do” anything once open.

1 Like
56

Presumably the same way they do now when the venv path is configured through the IDE. This proposal is only changing the “the environment is here” part. “This environment needs reindexing” and “this environment is busy” will carry on being handled with whatever watchdoggy magic the IDEs already use. Nobody seems to be complaining that that doesn’t work.

2 Likes
57

What about the way project_root is used in the ‘venv’ changes defines what “project” means? In all cases it’s just a directory looking for .venv. There’s no introspection or anything about what’s in the directory other than .venv. “project” in project_root is just to signify that that’s where .venv will (start to be) looked for.

:backhand_index_pointing_down:

58

As a long time virtualenvwrapper user, this would be a very helpful addition. (I hang out on the repo there but don’t maintain it but maybe @dhellmann might indicate support?)

virtualenvwrapper already has support mapping from a venv to the project. A standard mechanism for going the other way is something that’s come up (for me at least) several times over the years.

With regards to multiple venv support, and only a half-idea but, what comes to mind is pyenv’s ability to activate multiple Python versions via multiple lines in the `.python-version` file it uses. Essentially the first line gives the default Python, which others being available in the order specified. Here, the first venv listed would be used by default, with others available to tools if they which to list or select from a range of options. (:person_shrugging:)

In summary though, yes, this would solve a real need for me that’s come up many times. +1

1 Like
59

virtualenvwrapper is largely unmaintained these days, but I would love to have someone who is still actively using it contribute updates and potentially take over maintenance.

60

I’ll ping you on the repo.

61

I think you just explained the open issue in the PEP on supporting multiple virtual environments at PEP 832 – Virtual environment discovery | peps.python.org ?

You can thank Jeff Triplett for making sure virtualenvwrapper users are always in the back of my head.

There might be some interest from the conda community in this PEP (/cc @jezdez ), so I do want to give them some time to participate, which very likely means this PEP will miss Python 3.15 (which really only impacts the venv module, but I will add appropriate code to microvenv).

If conda support were to come in, then I think that would warrant going from a raw string to JSON, JSONL, or TOML with a simplified version of what I outlined in Provide structured output for environment/interpreter discovery · brettcannon/python-launcher · Discussion #168 · GitHub being written to the redirect file instead (the idea predates build-details.json, so a bunch of stuff isn’t warranted anymore). The file name would also need to change since “venv” is virtual environment-specific (recommending .venv for where to put a virtual environment would stay).

2 Likes
62

FTR, with my conda hat on, I’d be up for supporting and implementing this PEP in the conda CLI analogous to pixi (as linked above, I’ll leave a comment there, too) both in the currently incubating conda-workspaces (to be shipped in the future as part of conda CLI) and maybe in conda’s create command (see below). Ideally, there’d be a CEP (Conda Enhancement Proposal) to make sure the three different conda-type tools (pixi, mamba, and conda) are following the same patterns.

The tricky thing is that conda environments aren’t actual venvs, they look a lot like them, but there are some semantic differences, e.g., containing non-Python software, python.exe living in the root directory on Windows and not in Scripts, while entry point scripts are there. POSIX conda envs look a lot like regular venvs.

I’m mentioning conda-workspaces because it’s essentially porting most of pixi’s workspace pattern (~a .venv dir in a project root) to good ol’ conda. More info about it can also be found in the blog post draft for the upcoming 0.4.0 release. Reminder for all Python packaging users: conda/pixi are more like Homebrew than pip/uv/etc., userland distribution tools.

For the main conda create command, I’m not sure if it would make sense to implement the PEP, given it stores environments elsewhere on disk and not in the project dirs. But it’s not difficult to add an option to write a file or whatever to detect it easily. I’d think the conda-type tools would want to coordinate on this as well.

I’d appreciate it if the marker file were easy to parse; the conda ecosystem has been starting to adopt TOML more, e.g., pixi’s pixi.toml, but historically was very YAML-centric (sob). So how about a pyenvironment.toml or a new environment item in pyproject.toml as a marker?

5 Likes