PEP 832: virtual environment discovery

PEPs
53

I’ve been away for 13 days on a work trip (& mini-vacation) to Paris and just returned yesterday. I don’t have much time to organize a proper response here but I wanted to provide context for my opposition to the proposal so I’m going to heavily copy my comments from the pre-PEP discussion with the mentioned stakeholders. Any text I post that does not come from me is present to maintain the trail of thought and is anonymized.

TL;DR:

I view the best course of action that would satisfy everyone to be the construction of an API for environment managers themselves. There was actually a proposal several years ago (pre-uv though) with feedback/buy-in from several maintainers.

My opinion

N.B.: The opinion of @cjames23 should carry more weight than mine as he has more time to maintain Hatch these days.

I very strongly dislike the original proposal for the following reasons:

  • The .venv path suddenly becoming a file will surely break users and their tools in ways we cannot foresee. More importantly though, the proposal doesn’t specify requirements for the “default” environment. It says the virtual environment a workflow tool that is external to the one that wrote the .venv file is expected to use with the example that it should match the behavior of hatch run.

    Hatch uses the default environment unless specified by the user. It’s often the case that projects define specific environments without ever configuring the default and inheriting that base configuration. Are we fine with the default environment being unable to run tests, or even having the project installed should the project configure it as such?

  • The lack of support for multiple virtual environments leaves a feature gap between tools that would be confusing and unwelcome to users. It’s only possible as a rejected idea because uv doesn’t currently support the concept but unless I am severely mistaken that will happen eventually and therefore this proposal is not future-proof.

    Consider a very simple project that has an environment for every supported minor version of Python (py3.14, py3.13, …) with a non-negligible risk of behavioral differences between them. How would a user in an IDE, which is the primary use case here, select the environment they’re currently having issues with and press the test buttons? If the answer involves manual interaction with the environment manager to update the state of the discovery tool then we have failed.

    Also, as [redacted] mentioned, we appear to be not considering projects with complex requirements, monorepos or environments that are not on the host. As an example, consider that Airflow already experiences dependency conflicts and would benefit from easy selection of specific environments rather than their current recommendation to manually enter a workspace member for targeted testing.

I love the new pull-based concept however and would be 100% in favor if:

  1. The file idea was removed.

  2. Multiple environments were supported such that existing custom logic for the various environment managers in VS Code could be implemented with the new mechanism. The most important initial functionality would be the environment/Python dropdown selector.

  3. The expected capabilities of the default environment are specified with the SHOULD keyword.

I would potentially also be in favor of another PEP that specified user-defined tasks that could be executed within certain environments. @bernatgabor and I have been very much against attempts to standardize command execution without regard for where nor how it happens. If we have an API for tools to trigger the environment orchestrator then we unlock a ton of cool stuff.

Pull-based concept

What I referenced above was an idea from the thread:

If we go from the PEP’s “push” approach (i.e. workflow tools push out where the virtual environment) to a “pull” approach (tools pull where the virtual environment), then I think that could address these concerns, but at a price of more configuration.

So, one possibility is to introduce a [workflow] table to pyproject.toml. In that table, you can specify a command to use to get the virtual environment path from stdout when the status code is 0.

[workflow]
virtual-environment = {tool = "pdm", cli = ["pdm", "venv", "--path"], shell = false}

I could see this supporting string substitution for things like the path to the pyproject.toml file or its containing directory.

The tool key is just informational like pylock.toml’s created-by.

shell is for those that use some shell thing and would be false by default.

Having people add this to their pyproject.toml, but I think most of the tools here have an init/quickstart which would write this out.

Extended version I support:

Let’s look at what “complicated” might look like:

[workflow]
environments = {tool = "...", command = ["..."]}

That gives you a command to run in a subprocess with a cwd of where the pyproject.toml file is (we could add shell = "..." if people wanted), along with a tool name to include in any error message if the tool wasn’t installed. That command prints out JSON to stdout:

{
  "version": "1.0",  // ... or whatever to version the JSON schema.
  "default": 0,  // Optional
  "environments": [
    {
      "path": ".venv",
      "python_version": "...",  // Optional
      "name": "..."  // Optional
    }
    ...
  ]
}

default gives you an index into environments to find what environment should be used if you don’t want to bother asking the user what environment to use. environments.path points to the location of a virtual environment. The other stuff is for tools to show the user if they were to make a choice of which environment to use. Tools are allowed to create virtual environments as a side-effect of all of this. I assume “name” would have any details like what extras and dependency groups are installed in the environment if a tool cared to share such details.

And if workflow.environments isn’t defined, look for .venv next to pyproject.toml.

Now, I’m going on the assumption people are not running workflow tools out of the virtual environment as that’s a bootstrapping issue. But there could be stuff in workflow.environments to say to install certain dependency groups into .venv to then have the tool available if people wanted, then the tool can fill in anything else they wanted installed when asked to list the virtual environments.

Another addition could be letting people list a path instead of a command because they work in a controlled environment.

Purpose extrapolation

A comment and response by others:

There may be a benefit if further features are added to the [workflow] table as proposed by ofek (if I understand correctly).

I think that’s the question: is flexibility worth the complexity for this situation.

Someone answered:

  • Automatic virtual environment creation as a side-effect of requesting the location

    • This can lead to easier bootstrapping for tools like editors or AI since their request for virtual environments would also create the environment

    • With the PEP, something like an editor opening a fresh checkout wouldn’t have a virtual environment set up

    • E.g. VS Code or AI could get a virtual environment created on their behalf without having to first notice there’s no virtual environment and then either inferring – incorrectly – how to create an environment or having to ask how to create one or ask the user to do it for them

  • Supporting multiple virtual environments

    • E.g. VS Code or AI would know about alternative virtual environments upfront

    • The PEP as it stands doesn’t support this […]

I answered:

  • Similar to your first point, my view is that we should meet the user where they’re at and never require changing their workflow. If they want to use a GUI 100% of the time then so be it, and the same goes for CLI, AI agents, etc.

  • Multiple environments are a hard requirement for many projects so that’s important to me.

  • The ability to have non-local Python environments e.g. in a Docker container running on your host, on a VM in the cloud, etc. and non-virtual environments like Conda. I think we should remove the word “virtual” from the PEP title and have the concept broadly be about “Python environments”. This may sound like too much complexity but it’s entirely possible and already something users rely on this moment with Hatch environment plugins. As long as you come up with an adequate high-level API the internals don’t matter.

  • The ability to instruct the environment orchestrator to run user-defined tasks/commands within certain environments would be great. I’m lukewarm on the idea of specific functionality carveouts just for running tests and building docs as folks have asked for in the past. However, I’m 100% in favor of allowing for arbitrary tasks. I’d imagine IDEs that have similar concepts would integrate with such standards where the list of user-defined commands populates what is available. Another benefit would of course be redistributors who would no longer have to re-implement test setups of every package and could rely more on projects’ chosen orchestrators to do the work after they add support for the orchestrator to their build system.

Follow-ups to my opinion

Will it be just a case of hoping that the tool is already available on the Path environment variable?

I think this is the best option indeed.

The expected capabilities of the default environment are specified with the SHOULD keyword.

I don’t quite follow this. What “capabilities” are you referring to since I don’t think of environments as having capabilities?

Assuming the API requires passing an explicit directory, something like:

If the workspace directory contains a pyproject.toml file defining a valid [project] table as specified by PEP 621, then the default environment provided by the environment orchestrator:

  1. SHOULD have the project installed in editable mode i.e. runtime imports always reflect the current state of the code but not necessarily project metadata
  2. MAY be capable of running the project’s tests
6 Likes