API for discovering environments for a project

[brettcannon]

In PEP 832: virtual environment discovery - #71 by ofek, @ofek said that he “only and ardently support[s my] alternative pull-based idea from the private pre-PEP discussion where the desired environment manager is defined by metadata in pyproject.toml files.” That is not the current approach that PEP 832 takes, not because it’s wrong, but because I don’t know if such a fancy solution is necessary. But I think it’s worth talking about the idea to see if I’m wrong and should change PEP 832 accordingly.

So what is this idea that I had which Ofek is referring to? Basically it’s to specify a CLI call to make in a configuration file that returns JSON to say where the environments for a project are. Unfortunately there are a lot of “firsts” in that short idea. Let’s break this all down.

Why

So why even do this fancy idea? One is to support multiple environments easily. PEP 832 is purposefully simple, and so it has avoided supporting multiple environments. But if we go this fancier solution, then we might as well support them upfront.

Two, supporting different types of environments (i.e. virtual and conda). Reasons here are the same as multiple environments.

Three, it moves to a pull model. With PEP 832 and .venv redirect files and such, you need to push the environment by creating it first. By having you e.g. code editor pull the info by calling a tool, it gets rid of any bootstrap issues of having not created a .venv file yet. And by having the tool you call that can create any environments that are needed to start work, it helps with the bootstrapping problem of creating the environment first before launching your editor (but creation is not required).

Where

You need to write down somewhere this CLI call you want to make. In a [workflow] table in pyproject.toml is the simplest, but that only works if the project mandates a specific tool to use (e.g. you use a plug-in with Hatch for your project). What if my project has no opinion about what tool to use to manage your environments? Do I still have to make a decision for you to provide this even if it would be arbitrary in that case?

I would argue there should be more flexibility to specify the CLI call just for yourself, but that gets us to a “first” as there is no such thing as a “local” pyproject.toml. Do we have tools potentially read from a pyproject.local.toml that’s next to pyproject.toml, or maybe a pyworkflow.toml? How about a per-user or machine-wide config that gets overridden the more specific you get?

And do you support overriding anything in a project’s pyproject.toml? Could you override [project] or even per-key like project.version? Or only [workflow] and [tool]?

My gut says:

• pyproject.local.toml
• Support specifying next to pyproject.toml, per-user, and per-machine
• You can override keys in [workflow] and [tool] via machine | user | pyproject | local (pyproject.local.toml in the project > pyproject.toml > per-user > per-machine)

And I don’t view having some support for setting things locally as optional.

How

The next question is how do you specify the CLI to call? I think we gain a [workflow] table and it has an environments key. That key takes a table with the following keys:

• cmd or shell
  • cmd is an array of strings of a subprocess command to run
  • shell is a string to run in a subshell
  • Only one of these two keys can be specified, but one is required
• requires is optional and specifies the requirement for the tool so you could install it from e.g. PyPI as necessary

This is another “first” as we don’t have any CLI API anywhere.

When the tool is called, the PY_PROJECT_PATH environment variable holds the path to the pyproject.toml or the pyproject.local.toml file for the project. It’s required that one of those files to be found, else there’s no anchor point to be working off of to know what project you’re working on for selecting the proper environment(s). A search from the current working directory up for those files works for me as it’s simple, easy to understand, and doesn’t get the per-user or per-machine config in the way most of the time.

What

Now that we know how to call a tool, what should it do? Well, it should return JSON on stdout about the environments for the project:

{
  "interpreters": [
    "path": "...",
    "type": "...",  // "conda", "virtual", "global", "x-..."
    "name": "..."  // Optional
    "run": {  // Optional
      // Requires one of "cmd" or "shell"
      "cmd": ["..."],
      "shell": "..."
    }
  ],
  "default": 0  // Optional
}

• “interpreters” is an array of objects
  • “path” represents the path to the environment
    • It should be unique to the environment
    • If possible, point to the Python interpreter
  • “type” is the type of environment
    • “conda” is for a conda environment
    • “virtual” is a virtual environment
    • “global” is a globally installed Python interpreter
    • Anything with an “x-” prefix is some custom type of environment
  • “name” is a string acting as a label to help identify the environment to the user
  • “run” is optional and specifies how to run the environment
    • If omitted then “path” is assumed to point at the Python interpreter
    • “cmd” is an array of strings for a subprocess
    • “shell” is a string to run in a subshell
    • Only one of “cmd” or “shell” can be specified
    • Arguments to the Python interpreter are expected to be appended at the end (e.g. -m ... would be added appropriately at the end of “path”, “cmd”, or “shell”)
    • Useful for requiring e.g. conda run to run the interpreter
• “default” is optional and an integer indexing into “interpreters” to specify the default environment
  • It’s possible to pipe the result into jq to at least select the default interpreter: jq '.default as $i | .interpreters[$i]'

The tool being called may create environments before returning the JSON. In fact, I would encourage creating a single environment instead of returning an empty result of “{}”.

Who

So far, Hatch seems to really want this.

---

So that’s the proposal. Good and worth pursuing? Bad and stick with what’s in PEP 832 today? It’s going to take a good amount of support to pivot PEP 832 since this is:

1. More work for me and everyone else who wants to support this
2. I already have a decent amount of tool support for PEP 832 already

[henryiii (Henry Schreiner)]

I don’t think you can use stdout. If you are installing a large environment, a conda environment, or (most relevant to me!) building a binary package that takes time, you need to be able to communicate with the user and provide feedback. You could require all communication be directed to stderr, but that doesn’t seem ideal. And also PyPy will break this every other patch version with forgotten debug statements. I think it would need to write out some sort of file, and maybe that could be a form of caching.

And, with this design, since it returns multiple environments, I think it would need to make all the environments, as the tool requesting the environments isn’t requesting a specific one. That could be quite expensive.

I like seeing an example, so I’ve made what I think this would look like:

User writes:

[workflow.environments]
requires = ["hatch"]
cmd = ["hatch", "envtool"]

Then something (I’m not sure what) triggers this, installing all of hatch’s environments, and returns information about them, including indicating one is default, perhaps. I feel like the other proposal had a clear “let’s standardize uv run” feel to it. This one isn’t very clear in who would actually be asking “what are all the environments of the project” - I don’t see a use for multiple environments without commands (tasks) assigned to each one. I think there at least has to be some way for the triggering tool to indicate which venv it wants populated. And the whole lookup is complex, would this all be helpers in the stdlib, or a new CLI in the stdlib?

I don’t think this needs to block the .venv proposal. This is a proposal in how to make venv’s, while that proposal is just setting up a “default” venv to use. If you added “make a .venv that points at the default environment created” to this proposal, it becomes a followup to the other one, I think?

PS: I don’t think PEP 832 goes as far as to standardize part of uv run. though? Basically a py launcher that takes into account .venv. If it did, the this could follow on that and give the py launcher a way to make .venv if it doesn’t exist.

[brettcannon]

@henryiii

If you are installing a large environment, a conda environment, or (most relevant to me!) building a binary package that takes time, you need to be able to communicate with the user and provide feedback.

Feedback for what? This is just to list the environments you have for a project and it’s tool-to-tool, so there shouldn’t be a person in the loop.

@henryiii

And, with this design, since it returns multiple environments, I think it would need to make all the environments

Why do you think that? If I don’t need some environment, why create it? If I need a specific one you can ask you tool of choice to make what you want.

@henryiii

Then something (I’m not sure what) triggers this

Think your code editor to list the environments that you have available to choose from (if you happen to be a VS Code user, this would be used to populate the list of Python environments to choose from). Otherwise @ofek or @cjames23 might have some ideas as they have said Hatch already has similar support for this.

@henryiii

And the whole lookup is complex

There’s a reason why there’s a bar to clear for me to take this on for PEP 832.

@henryiii

would this all be helpers in the stdlib, or a new CLI in the stdlib?

Much like packaging, the stdlib very likely wouldn’t be involved.

@henryiii

This is a proposal in how to make venv’s

If that’s what you think then I’ve phrased something poorly. It can lead to environment creation, but the main motivator is listing all the environments you have.

I’ll try to clarify this in my statement above.

@henryiii

If you added “make a .venv that points at the default environment created” to this proposal, it becomes a followup to the other one, I think?

Yes, this could be a follow-up, but Ofek very clearly said he doesn’t like the .venv redirect file idea and thinks this proposal is a better solution.

@henryiii

I don’t think PEP 832 goes as far as to standardize part of uv run. though?

No, it does not.

@henryiii

Basically a py launcher that takes into account .venv.

The Python Launcher for Unix already does.

@henryiii

If it did, the this could follow on that and give the py launcher a way to make .venv if it doesn’t exist.

Correct, and part of the motivation for me with PEP 832 is so .venv is standardized enough that having the Python Launcher use PEP 832 isn’t viewed as hoisting a workflow on to anyone. And as you have suggested, it opens the door for a py run-like experience.

[steve.dower]

I’ve said it before and I guess I’ll say it again - this should be no more than a build-system-shaped hook in the pyproject.toml that says “if you trigger <new generic command> then this project will execute <specific command>” (by installing, importing, and calling its hook).

Want something different? Don’t run that generic command.

A project wants different default settings for different tools? Include different config files/tool sections (aka. the status quo).

There’s no reason to try and specify the detail you’re going into here - overrides/etc. should be entirely in the realm of tooling (thinking IDEs here, but also CLI-based workflow tools such as the Python Launcher). There are legitimate reasons to want alternative settings to be more local (e.g. pyproject.local.toml) or more global (e.g. an enterprise whitelisting workflow tools).

The only gap is a way to programmatically (i.e. without a human reading the dev docs) determine what tool/invocation can be used to initialise the workspace, and if you want to solve that then it’d better be done before everyone gets used to letting the AIs read the README.md to figure it out

@brettcannon

part of the motivation for me with PEP 832 is so .venv is standardized enough that having the Python Launcher use PEP 832 isn’t viewed as hoisting a workflow on to anyone

-m venv .venv && .venv/bin/python ...^[1] is a perfectly fine default for a generic tool, IMHO. And checking for a section in a pyproject.toml and invoking the tool listed there instead of -m venv feels like sufficient customisation. We don’t need 100 choices; we need one good enough choice that projects can adapt to using or else ignore and do their own thing.

---

1. With suitable x-plat adjustments ↩︎

[Jost (Jost Migenda)]

I had a couple of thoughts while reading this proposal:

Is there demand from projects to mandate a specific environment management tool?

None of the projects I maintain or contribute to have particular requirements on the location of the environment or the tool^[1] used to create this. Admittedly, most of those are fairly simple projects; but even the developer docs of some very complex projects like Airflow or NumPy are agnostic in both regards.

I’m aware that some development teams within companies standardise on a specific workflow & environment managers; but I have no idea whether they’d be interested in this at all—I’d guess they already have custom tooling (or MDM software or …) to enforce that anyway.

Without clear demand for such a [workflow] table in pyproject.toml, I don’t think the significant added complexity of adding such a table and the corresponding override mechanism (pyproject.local.toml, plus per-user and per-machine equivalents) is justified.

---

Supporting different types of environments (i.e. virtual and conda)

I see this as an important point; if the current approach in PEP 832 doesn’t allow for this (which is a discussion that I’ll put in the other thread), that would be a significant downside.

---

@brettcannon

Now that we know how to call a tool, what should it do? Well, it should return JSON on stdout about the environments for the project:

I’m thinking about how this would work for people using workflows where all environments are stored in a central folder. In that case, there probably won’t be a mapping of environments to projects. I guess conda could return a JSON listing of all environments (instead of “the environments for the project”); but would something similar work for all tools?

---

1. There’s one exception here—a particle physics software stack I’ve worked on, which required conda because it needed to integrate complex non-Python dependencies. ↩︎

[brettcannon]

@steve.dower

I’ve said it before and I guess I’ll say it again - this should be no more than a build-system-shaped hook in the pyproject.toml that says “if you trigger then this project will execute ” (by installing, importing, and calling its hook).

Ignoring the CLI-vs-hook difference, what are you suggesting here? We start standardizing the CLI for workflow tools overall in key scenarios for the “generic command”? Or are you thinking in terms of more like the Python Launcher has the generic command and tools like Hatch can optionally provide the “specific command” that the Launcher uses when provided?

And how is that different than what I’m proposing here as a structured py list --json? The fact that this proposal doesn’t specifically say, “py list --json will use this if provided, otherwise do its own thing”?

@steve.dower

overrides/etc. should be entirely in the realm of tooling (thinking IDEs here, but also CLI-based workflow tools such as the Python Launcher).

So you’re saying we shouldn’t write down how to resolve things and leave it to the tools to do their own thing?

@steve.dower

-m venv .venv && .venv/bin/python ... is a perfectly fine default for a generic tool, IMHO.

Same here, hence PEP 832 as it is effectively standardizing it as the baseline with the redirect files as an escape hatch (and I know you’re aware of this; saying this for anyone else who may not know this view).

@Jost

Is there demand from projects to mandate a specific environment management tool?

Enterprises that control everything do. There’s also projects that use e.g. uv.lock and so mandate the tool being used.

@Jost

Supporting different types of environments (i.e. virtual and conda)

I see this as an important point; if the current approach in PEP 832 doesn’t allow for this

It doesn’t explicitly, but introspection might allow for it (at least it’s easy to tell if something is a virtual environment thanks to pyvenv.cfg).

It also probably would suggest not using .venv for the redirect file name.

@Jost

I’m thinking about how this would work for people using workflows where all environments are stored in a central folder. In that case, there probably won’t be a mapping of environments to projects.

Depends on the tool. It could keep a file in the project to record what has been selected before. It could record per-environment what projects it’s used for. There isn’t a technical reason the issue couldn’t be solved in some way.

[cjames23 (Cary Hawkins)]

@brettcannon

Think your code editor to list the environments that you have available to choose from (if you happen to be a VS Code user, this would be used to populate the list of Python environments to choose from). Otherwise @ofek or @cjames23 might have some ideas as they have said Hatch already has similar support for this.

This is exactly what hatch env show --json is used for right now. We in fact still have an open issue to address a bug around this when integrating PyCharm "env show --json" fails with an error if some plugin is not installed · Issue #2011 · pypa/hatch · GitHub which was opened from https://youtrack.jetbrains.com/projects/PY/issues/PY-81270/

When you run this command you get an output like

{"default":{"type":"virtual"},"hatch-build":{"skip-install":true,"installer":"uv","dependencies":["build[virtualenv]>=1.0.3"],"scripts":{"build-all":["python -m build"],"build-sdist":["python -m build --sdist"],"build-wheel":["python -m build --wheel"]},"type":"virtual"},"hatch-static-analysis":{"skip-install":true,"installer":"uv","dependencies":["ruff==0.12.0"],"scripts":{"format-check":["ruff format --check --diff ."],"format-fix":["ruff format ."],"lint-check":["ruff check ."],"lint-fix":["ruff check --fix ."]},"type":"virtual","config-path":"none"},"hatch-test.py3.13":{"installer":"uv","dependencies":["coverage-enable-subprocess==1.0","coverage[toml]~=7.4","pytest~=8.1","pytest-mock~=3.12","pytest-randomly~=3.15","pytest-rerunfailures~=14.0","pytest-xdist[psutil]~=3.5"],"scripts":{"run":["pytest --rootdir=. --junitxml=test-results.xml --cov --no-cov-on-fail --cov-report=lcov --cov-report=term-missing"]},"type":"virtual","features":["test"],"python":"3.13"},"hatch-uv":{"skip-install":true,"installer":"uv","type":"virtual"}}

Which then allows an editor to switch between the different environments using hatch. This could be formalized into a better API that also provides the location of each environment. Right now it is a two hop call, call hatch env show to get the list of environments and then call hatch env find with the environment name to see if it exists.

I think if the schema of the JSON is standardized that allows tools to do the right thing, and I do not think based on private conversations that there would be any strong opposition from the popular tools towards providing this as a command with a standardized JSON schema for the output.

I am very open to adding locations of the environment in hatch’s env show JSON output. That is not a hard thing to add. And from the consuming side I would suggest looking at the output and any empty location field as an indicator that the environment has not been created yet.

[steve.dower]

@brettcannon

We start standardizing the CLI for workflow tools overall in key scenarios for the “generic command”? Or are you thinking in terms of more like the Python Launcher has the generic command and tools like Hatch can optionally provide the “specific command” that the Launcher uses when provided?

Basically, yeah. We did it for “build” (and created build to implement it), so why not create “setup” or “prepare”^[1] and let frontends such as the Python Launcher start supporting it?

@brettcannon

how is that different than what I’m proposing here as a structured py list --json?

You can only do something useful with that information if you created it or if someone specified it all the way through. The latter case stifles innovation, and the former is already possible.

If it’s limited to “things that py can launch” then it works, but that doesn’t need anything more than an implementation in py, because anyone who wants to participate is literally only interoperating with one tool. Trying to extend it to “anything at all” is biting off way more than you want to chew

@brettcannon

So you’re saying we shouldn’t write down how to resolve things and leave it to the tools to do their own thing?

Pretty much. Or write it down as “in the absence of user preferences, do it this way, but you can do anything else you want if the user tells you to”. Too many of these specs are designed to constrain the user (indirectly via tools, since that’s the only mechanism we have), but fundamentally we want to enable users to do anything they need to do, which means allowing tools to do anything their users want to do, which means drawing as small a boundary as possible around the interoperability point.

In this case, we’re saying “how do we have a single way to find [and implicitly, create if not found] the environments for this project”. My proposal is that the interoperability point looks like “here’s the tool and its config that knows how to do it for this project”, so anyone can easily find and initially invoke the tool, but the actual behaviour is entirely up to the tool.

---

1. “workspace” implied ↩︎

[brettcannon]

@steve.dower

Basically, yeah. We did it for “build” (and created build to implement it), so why not create “setup” or “prepare” and let frontends such as the Python Launcher start supporting it?

I don’t see anything explicitly against this right now, but I think you’re promoting more of a PEP 517 approach that uses an code API and declaration of what’s needs to be installed for the API to work for the calling tool to use (along with reasonable defaults for tools to follow if nothing is declared).

I would expect there to be an API to find out about all environments (either existing or are possible based on a flag), and another to get a single environment with a flag as to whether it should be created (and either based on some key/name or “give me the default”).

I’m not opposed to the idea, but there’s 2 potential sticking points. One is I don’t think most workflow tools have a Python API that treats them like a library, so that would be a change. Two, this does necessitate the caller handle the installation of the underlying environment management tool while my current approach can at least lean on $PATH to have the management tool already installed.

I don’t think either is insurmountable, but it is a shift.

@steve.dower

@brettcannon

how is that different than what I’m proposing here as a structured py list --json?

You can only do something useful with that information if you created it

Who is the “you” in that sentence? Me the user, or the environment management tool (e.g. Hatch)?

Regardless, yes, someone inevitably needs to create an environment.

@steve.dower

or if someone specified it all the way through. The latter case stifles innovation

“Specified all the way through” as in put in the config details into pyproject.toml?

@steve.dower

If it’s limited to “things that py can launch” then it works, but that doesn’t need anything more than an implementation in py, because anyone who wants to participate is literally only interoperating with one tool.

Yes, but after talking to the SC about getting the Launcher to do more workflow stuff I got the feeling they have trepidation on dictating workflows in some way. Hence trying to get PEP 832 so there isn’t any accusations of that. So “interoperating with one tool” I don’t think works from the perspective of “py can come up with its own thing and people just work with that” without a PEP or something else people have signed off on (if that’s what you mean).

@steve.dower

Trying to extend it to “anything at all” is biting off way more than you want to chew

There’s a reason why I said from the outset there’s a bar to clear before I change PEP 832.

@steve.dower

write it down as “in the absence of user preferences, do it this way, but you can do anything else you want if the user tells you to”. Too many of these specs are designed to constrain the user (indirectly via tools, since that’s the only mechanism we have), but fundamentally we want to enable users to do anything they need to do, which means allowing tools to do anything their users want to do, which means drawing as small a boundary as possible around the interoperability point.

I’m not married to trying to override/merge details in some clever way, but I do want some way to be able to set what tool to at least use if the project didn’t specify a preference. Now you might be saying, “let the tool making the call handle how that gets resolved and supported” (e.g. VS Code just does its own thing)? In which case are you advocating for how to specify the API, how it works, and how write it down in pyproject.toml, but any other fanciness is a per-tool thing (e.g. specifying the per-user default in your personal VS Code settings)?

@steve.dower

In this case, we’re saying “how do we have a single way to find [and implicitly, create if not found] the environments for this project”. My proposal is that the interoperability point looks like “here’s the tool and its config that knows how to do it for this project”, so anyone can easily find and initially invoke the tool, but the actual behaviour is entirely up to the tool.

I’m trying to tease apart how fundamentally “find and initially invoke the tool” is different from what I’m proposing. Is it the fact I’m asking for a list instead of just asking for whatever environment the tool deems reasonable? Is “find” for you more PEP 517-like than what I have for workflow.environments.requires? Same for “invoke” being a code API instead of the proposed CLI API?

[ofek (Ofek Lev)]

@brettcannon

What if my project has no opinion about what tool to use to manage your environments? […] My gut says:

• pyproject.local.toml
• Support specifying next to pyproject.toml, per-user, and per-machine
• You can override keys in [workflow] and [tool] via machine | user | pyproject | local (pyproject.local.toml in the project > pyproject.toml > per-user > per-machine)

I like enabling more control and I agree with the chosen override precedence. However, can we please punt on overriding anything other than the workflow table? I foresee that causing significant issues for this proposal being accepted.

@brettcannon

I think we gain a [workflow] table and it has an environments key. That key takes a table with the following keys:

• cmd or shell
  • cmd is an array of strings of a subprocess command to run
  • shell is a string to run in a subshell
  • Only one of these two keys can be specified, but one is required
• requires is optional and specifies the requirement for the tool so you could install it from e.g. PyPI as necessary

1. I think we should only allow cmd and require PATH being properly set up such that one never needs shell functionality to execute the binary.
2. I know we technically did it for the build system but I worry that the complexity of standardizing the behavior of requires may impact acceptance of this proposal. I’m fine with keeping it though.

@brettcannon

Now that we know how to call a tool, what should it do? Well, it should return JSON on stdout about the environments for the project

@henryiii

I don’t think you can use stdout […] you need to be able to communicate with the user […] I think it would need to write out some sort of file

I agree that the output should be a file whose location is determined by the caller. The location must be passed as a command line argument rather than an environment variable so that we support non-local execution.

@henryiii

since it returns multiple environments, I think it would need to make all the environments, as the tool requesting the environments isn’t requesting a specific one

Querying metadata should definitely not require manifesting the actual environment. As Cary mentioned, Hatch provides all of this information with or without the environment being created and for any type of environment.

---

I’ll provide a bit more feedback tomorrow as it’s getting late here. Thanks Brett!

[brettcannon]

@ofek

However, can we please punt on overriding anything other than the workflow table?

Fine by me as so far both you and Steve are leery of going too far with overrides (albeit at different levels).

@ofek

I think we should only allow cmd and require PATH being properly set up such that one never needs shell functionality to execute the binary.

The only reason shell support is there is for virtualenvwrapper and in case conda needed it for some reason (they do register shell stuff, but I would assume it wouldn’t come into play here).

@ofek

I know we technically did it for the build system but I worry that the complexity of standardizing the behavior of requires may impact acceptance of this proposal.

I’m up for taking it out if people prefer.

@ofek

I agree that the output should be a file whose location is determined by the caller.

How would you expect for that to be specified? Tacked on to the end of the command? Add support for some placeholder like “{output}”?

@ofek

The location must be passed as a command line argument rather than an environment variable so that we support non-local execution.

I don’t understand how environment variables don’t work in that situation. If I’m making a subprocess call, how does that not support using an environment variable? Or are you thinking of some future RPC support where it isn’t the tool being called that’s making the RPC call, so you’re trying to future-proof for some unforeseen scenario? In that instance, wouldn’t that mean no environment variable being used for anything?

[steve.dower]

@brettcannon

both you and Steve are leery of going too far with overrides (albeit at different levels).

I’m just leery of overspecifying them. If you write it in a PEP, then tools must do it, but they should be allowed to do whatever is best for their users. We should only be specifying the author’s (of the configuration) intent, and tools can choose whether to honour that intent or not by themselves (based on what their end-user wants).

@brettcannon

@ofek

I know we technically did it for the build system but I worry that the complexity of standardizing the behavior of requires may impact acceptance of this proposal.

I’m up for taking it out if people prefer.

In the same vein, specifying the intent that “these things are required” is totally fine by me. Specifying that tools must figure out how to install those things is going too far. Let tools choose to honour the intent however they see fit.

@brettcannon

@ofek

The location must be passed as a command line argument rather than an environment variable so that we support non-local execution.

I don’t understand how environment variables don’t work in that situation. If I’m making a subprocess call, how does that not support using an environment variable?

I agree with Brett, if there’s going to be commands specified, then environment variables are a must. They’re the only reliable way to avoid shell injection and escaping issues (writing via a file introduces permissions and TOCTOU issues).

Still, I’d rather not deal with subprocess calls here at all - call into a Python API with trustworthy and reliable Python objects and let the wrapper figure out how best to invoke whatever tool it’s calling.

@brettcannon

I’m trying to tease apart how fundamentally “find and initially invoke the tool” is different from what I’m proposing. Is it the fact I’m asking for a list instead of just asking for whatever environment the tool deems reasonable?

We’re talking about it at different levels, I think. I mean “find” like the instructions on this page, which has already mostly hidden the work behind a single nox command already, but we should be able to hide it behind a generic py hypothetical-set-up-my-project command that can discover that it needs nox itself.^[1]

The things you seem to be thinking about - finding/creating Python executables - are to me the responsibility of nox (in this example), and once we’ve launched that we stay out of it. Trying to standardise the steps inside nox is what I want to avoid [having you waste your precious time on ]. There are too many possible workflows out there, including many we’ve never even considered, and we don’t solve that by choosing one universal one - we solve it by making most people not have to research which one is being used right now.

---

1. And whether it installs it when it’s missing or just reports an error is left up to the tool. ↩︎

[brettcannon]

@steve.dower

In the same vein, specifying the intent that “these things are required” is totally fine by me. Specifying that tools must figure out how to install those things is going too far. Let tools choose to honour the intent however they see fit.

And I wasn’t planning on having this proposal require installation for calling a CLI; I was treating it as a hint as to what was expected to exist at worst, or to let a tool do some installation if they so desired.

@steve.dower

We’re talking about it at different levels, I think. I mean “find” like the instructions on this page, which has already mostly hidden the work behind a single nox command already

And that’s why tox supports PEP 832 as it currently sits for the exact same workflow: environment creation is not standardized, just how you write down where the environment is so other tools can be let in on that bit of knowledge.

@steve.dower

but we should be able to hide it behind a generic py hypothetical-set-up-my-project command that can discover that it needs nox itself.

Yes, it’s at a different level as you surmised. Your view aligns more with what’s in PEP 832: simple case of “tell me where the/an environment is”. But this proposal is “surface all details that may need to be surfaced in e.g. an editor”. Probably the biggest difference between PEP 832 and what you’re after here is the Python API instead of redirect files.

I think if I were to take what you’re proposing it would be to:

1. Keep .venv as the suggested default for a single virtual environment
2. Keep the concept of environment redirect files, but maybe change the file name if conda would use them
3. Add in your proposed PEP 517-like Python API for setup/bootstrapping the first time you start using a project
4. Come up with some guidelines to set up per-user, and per-machine specifications when the project lacks them (I’m skipping over per-project on purpose; I just want to at this point let users say, “in the face of no preference, use Hatch” or something with the assumption and that per-project doesn’t make sense for that and you would edit pyproject.toml in that case anyway).

That lets the detection of an environment be somewhat of a cheap check if things are set up (if tools chose to work that way; it wouldn’t be mandated but the PEP would say it’s okay as well). And having the command be a setup/bootstrap would give leeway in terms of doing more than creating an environment (e.g. downloading stuff).

To be clear, this is not what I’m explicitly proposing as coming out of this conversation, just where my head is at if Steve’s ideas win out.

@steve.dower

There are too many possible workflows out there, including many we’ve never even considered

And I think that’s the view @ofek is taking: if there’s going to be any attempt at this, it should be broad and be flexible for future changes.

[lucascolley]

@brettcannon

@Jost

Is there demand from projects to mandate a specific environment management tool?

Enterprises that control everything do. There’s also projects that use e.g. uv.lock and so mandate the tool being used.

Using a lock file does not imply a mandated tool. In SciPy we use pixi.lock for the majority of CI workflows, for example, but developers may still choose to use pip venvs with their IDE — nothing is stopping that.

@brettcannon

@Jost

Supporting different types of environments (i.e. virtual and conda)

I see this as an important point; if the current approach in PEP 832 doesn’t allow for this

… It also probably would suggest not using .venv for the redirect file name.

Yeah, probably something more obviously generic would be better.

[tmk (Thomas Kehrenberg)]

@brettcannon

By having you e.g. code editor pull the info by calling a tool, it gets rid of any bootstrap issues of having not created a .venv file yet. And by having the tool you call that can create any environments that are needed to start work, it helps with the bootstrapping problem of creating the environment first before launching your editor

If environment creation could be automated, that would be great, but this functionality doesn’t have to be bundled with the rest of the proposal. I think a combination of a push-based approach for environment discovery and a pull-based approach for environment creation makes sense, because the former is done often and should be as cheap as possible, while the latter is done rarely, such that it is okay if it is more expensive.

As for the issue of discovering multiple environments, I think a push-based solution can be found.

[brettcannon]

@lucascolley

Using a lock file does not imply a mandated tool. In SciPy we use pixi.lock for the majority of CI workflows, for example, but developers may still choose to use pip venvs with their IDE — nothing is stopping that.

I would argue that’s true because you control the CI and so daily development doesn’t require it to run tests locally. But if you expected me to keep that lock file updated then you have mandated the tool, you just happen to mandate it in a way where I can avoid it if I wanted to.

But I do get your point it isn’t a forgone conclusion that a specific tool is necessary even in the face of tool-specific files for all projects.

@lucascolley

@brettcannon

… It also probably would suggest not using .venv for the redirect file name.

Yeah, probably something more obviously generic would be better.

While I have the attention of a conda-knowledgeable person, @lucascolley , if I pointed you at a directory and asked, “is this a conda environment”, could that be answered purely by the file system or file contents (like you can with virtual environments by the pyvenv.cfg file)?

@tmk

If environment creation could be automated, that would be great, but this functionality doesn’t have to be bundled with the rest of the proposal.

No, but the question becomes whether I have it in me to do a separate PEP for project setup.

I also want to hear from @steve.dower that this idea reflects what he’s suggesting and from @ofek as he said …

@ofek

I’ll provide a bit more feedback tomorrow as it’s getting late here.

… 5 days ago.

@tmk

As for the issue of discovering multiple environments, I think a push-based solution can be found.

Oh, definitely! It’s more of a question as to whether I have it in me to solve it in this PEP or leave it for someone else to solve.

[lucascolley]

@brettcannon

While I have the attention of a conda-knowledgeable person, @lucascolley , if I pointed you at a directory and asked, “is this a conda environment”, could that be answered purely by the file system or file contents (like you can with virtual environments by the pyvenv.cfg file)?

Yes, from ceps/cep-0032.md at main · conda/ceps · GitHub

A conda environment is defined as a directory that contains, at least, a ./conda-meta/history file.

[steve.dower]

@brettcannon

@tmk

If environment creation could be automated, that would be great, but this functionality doesn’t have to be bundled with the rest of the proposal.

No, but the question becomes whether I have it in me to do a separate PEP for project setup.

I also want to hear from @steve.dower that this idea reflects what he’s suggesting

It kind of does, because I think a key point is that only the tool that created an environment can really identify it (in a useful way), and so your choice is either to define a way to delegate all the steps to e.g. conda^[1], or to define exactly how all tools must create their environments so that you can identify them without delegating to the tool.^[2]

My idea is to do the former, which doesn’t require you to write another PEP for project setup.

---

1. Convenient example since you also just had to ask how Conda identifies its own environments. ↩︎

2. One way to do this could be to make pyvenv.cfg “mandatory” and add new fields to let the runtime ignore it when it’s not a venv environment. I’m specifically not advocating this. ↩︎

[brettcannon]

@steve.dower

My idea is to do the former, which doesn’t require you to write another PEP for project setup.

I talked with @steve.dower offline and his idea for the API is to probably support (assuming I’m not misrepresenting what Steve is suggesting):

1. Listing the environments
2. Setting up the project (which probably implies setting up the environments)
3. Running a command using one of the environments

This would abstract out a lot of details so that tools working at this level could do something like:

1. Setup working with a project when you open a project in your editor
2. Get a list of environments to work with so your editor knows what you want to work with
3. Run code with a selected environment so your editor can launch you test suite

But it’s abstracted out such that the environment could be remote or something.

This can all be driven by some CLI tool directly or by some tool on your behalf. Steve also suggested having some project that defines a reasonable fallback that tools could use. That is where Steve would have .venv as a “I have no preference” default live and not have that in the PEP. Steve also doesn’t like the redirect file idea since you could just ask for the list rather than writing it down.

While I can see this working in an ideal scenario (especially if it expands to cover more situations over time), the question I’m ruminating about is whether this will lead to actual adoption of this approach or if people would balk at the idea of having their workflow tool abstracted away this much.

[pf_moore (Paul Moore)]

@brettcannon

While I can see this working in an ideal scenario (especially if it expands to cover more situations over time), the question I’m ruminating about is whether this will lead to actual adoption of this approach or if people would balk at the idea of having their workflow tool abstracted away this much.

For me, I think I’d just use whatever tool I preferred, and wouldn’t see having to add config to say “my tool is XXX” (which I assume I’d need to to for the API to work) as being any better than the current situation. I wouldn’t use a generic API for my normal workflow, it would be solely for IDEs and the like.

Conversely, a naming convention like .venv is (for me, at least) a zero-friction way to work in a way that IDEs can just use, without me needing to do anything.