brettcannon | 2026-05-27 20:50:33 UTC | #1 In https://discuss.python.org/t/pep-832-virtual-environment-discovery/106998/71, @ofek said that he "only and ardently support[s my] alternative pull-based idea from the private pre-PEP discussion where the desired environment manager is defined by metadata in pyproject.toml files." That is **not** the current approach that [PEP 832](https://peps.python.org/pep-0832/) takes, not because it's wrong, but because I don't know if such a fancy solution is necessary. But I think it's worth talking about the idea to see if I'm wrong and should change PEP 832 accordingly. So what is this idea that I had which Ofek is referring to? Basically it's to specify a CLI call to make in a configuration file that returns JSON to say where the environments for a project are. Unfortunately there are a lot of "firsts" in that short idea. Let's break this all down. # Why So _why_ even do this fancy idea? One is to support multiple environments easily. PEP 832 is purposefully simple, and so it has avoided supporting multiple environments. But if we go this fancier solution, then we might as well support them upfront. Two, supporting different types of environments (i.e. virtual and conda). Reasons here are the same as multiple environments. Three, it moves to a pull model. With PEP 832 and `.venv` redirect files and such, you need to _push_ the environment by creating it first. By having you e.g. code editor _pull_ the info by calling a tool, it gets rid of any bootstrap issues of having not created a `.venv` file yet. And by having the tool you call that _can_ create any environments that are needed to start work, it helps with the bootstrapping problem of creating the environment first before launching your editor (but creation is **not** required). # Where You need to write down _somewhere_ this CLI call you want to make. In a `[workflow]` table in `pyproject.toml` is the simplest, but that only works if the project mandates a specific tool to use (e.g. you use a plug-in with Hatch for your project). What if my project has no opinion about what tool to use to manage your environments? Do I still have to make a decision for you to provide this even if it would be arbitrary in that case? I would argue there should be more flexibility to specify the CLI call just for yourself, but that gets us to a "first" as there is no such thing as a "local" `pyproject.toml`. Do we have tools potentially read from a `pyproject.local.toml` that's next to `pyproject.toml`, or maybe a `pyworkflow.toml`? How about a per-user or machine-wide config that gets overridden the more specific you get? And do you support overriding **anything** in a project's `pyproject.toml`? Could you override `[project]` or even per-key like `project.version`? Or only `[workflow]` and `[tool]`? My gut says: - `pyproject.local.toml` - Support specifying next to `pyproject.toml`, per-user, and per-machine - You can override keys in `[workflow]` and `[tool]` via `machine | user | pyproject | local` (`pyproject.local.toml` in the project > `pyproject.toml` > per-user > per-machine) And I don't view having _some_ support for setting things locally as optional. # How The next question is _how_ do you specify the CLI to call? I think we gain a `[workflow]` table and it has an `environments` key. That key takes a table with the following keys: - `cmd` or `shell` - `cmd` is an array of strings of a subprocess command to run - `shell` is a string to run in a subshell - Only one of these two keys can be specified, but one is required - `requires` is optional and specifies the requirement for the tool so you could install it from e.g. PyPI as necessary This is another "first" as we don't have any CLI API anywhere. When the tool is called, the `PY_PROJECT_PATH` environment variable holds the path to the `pyproject.toml` or the `pyproject.local.toml` file for the project. It's required that one of those files to be found, else there's no anchor point to be working off of to know what project you're working on for selecting the proper environment(s). A search from the current working directory up for those files works for me as it's simple, easy to understand, and doesn't get the per-user or per-machine config in the way most of the time. # What Now that we know how to call a tool, _what_ should it do? Well, it should return JSON on stdout about the environments for the project: ```jsonc { "interpreters": [ "path": "...", "type": "...", // "conda", "virtual", "global", "x-..." "name": "..." // Optional "run": { // Optional // Requires one of "cmd" or "shell" "cmd": ["..."], "shell": "..." } ], "default": 0 // Optional } ``` - "interpreters" is an array of objects - "path" represents the path to the environment - It should be unique to the environment - If possible, point to the Python interpreter - "type" is the type of environment - "conda" is for a conda environment - "virtual" is a virtual environment - "global" is a globally installed Python interpreter - Anything with an "x-" prefix is some custom type of environment - "name" is a string acting as a label to help identify the environment to the user - "run" is optional and specifies how to run the environment - If omitted then "path" is assumed to point at the Python interpreter - "cmd" is an array of strings for a subprocess - "shell" is a string to run in a subshell - Only one of "cmd" or "shell" can be specified - Arguments to the Python interpreter are expected to be appended at the end (e.g. `-m ...` would be added appropriately at the end of "path", "cmd", or "shell") - Useful for requiring e.g. `conda run` to run the interpreter - "default" is optional and an integer indexing into "interpreters" to specify the default environment - It's possible to pipe the result into `jq` to at least select the default interpreter: `jq '.default as $i | .interpreters[$i]'` The tool being called **may** create environments before returning the JSON. In fact, I would encourage creating a single environment instead of returning an empty result of "{}". # Who So far, Hatch seems to really want this. ----- So that's the proposal. Good and worth pursuing? Bad and stick with what's in PEP 832 today? It's going to take a good amount of support to pivot PEP 832 since this is: 1. More work for me and everyone else who wants to support this 2. I already have a decent amount of tool support for PEP 832 already ------------------------- henryiii | 2026-05-05 04:55:25 UTC | #2 I don't think you can use stdout. If you are installing a large environment, a conda environment, or (most relevant to me!) building a binary package that takes time, you need to be able to communicate with the user and provide feedback. You could require all communication be directed to stderr, but that doesn't seem ideal. And also PyPy will break this every other patch version with forgotten debug statements. :) I think it would need to write out some sort of file, and maybe that could be a form of caching. And, with this design, since it returns multiple environments, I think it would need to make all the environments, as the tool requesting the environments isn't requesting a specific one. That could be quite expensive. I like seeing an example, so I've made what I think this would look like: User writes: ```toml [workflow.environments] requires = ["hatch"] cmd = ["hatch", "envtool"] ``` Then something (I'm not sure what) triggers this, installing all of hatch's environments, and returns information about them, including indicating one is default, perhaps. I feel like the other proposal had a clear "let's standardize `uv run`" feel to it. This one isn't very clear in who would actually be asking "what are all the environments of the project" - I don't see a use for multiple environments without commands (tasks) assigned to each one. I think there at least has to be some way for the triggering tool to indicate which venv it wants populated. And the whole lookup is complex, would this all be helpers in the stdlib, or a new CLI in the stdlib? I don't think this needs to block the `.venv` proposal. This is a proposal in how to make venv's, while that proposal is just setting up a "default" venv to use. If you added "make a .venv that points at the default environment created" to this proposal, it becomes a followup to the other one, I think? PS: I don't think PEP 832 goes as far as to standardize part of `uv run`. though? Basically a `py` launcher that takes into account `.venv`. If it did, the this could follow on that and give the `py` launcher a way to make `.venv` if it doesn't exist. ------------------------- brettcannon | 2026-05-05 21:12:47 UTC | #3 [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] If you are installing a large environment, a conda environment, or (most relevant to me!) building a binary package that takes time, you need to be able to communicate with the user and provide feedback. [/quote] Feedback for what? This is just to list the environments you have for a project and it's tool-to-tool, so there shouldn't be a person in the loop. [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] And, with this design, since it returns multiple environments, I think it would need to make all the environments [/quote] Why do you think that? If I don't need some environment, why create it? If I need a specific one you can ask you tool of choice to make what you want. [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] Then something (I’m not sure what) triggers this [/quote] Think your code editor to list the environments that you have available to choose from (if you happen to be a VS Code user, this would be used to populate the list of Python environments to choose from). Otherwise @ofek or @cjames23 might have some ideas as they have said Hatch already has similar support for this. [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] And the whole lookup is complex [/quote] There's a reason why there's a bar to clear for me to take this on for PEP 832. 😉 [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] would this all be helpers in the stdlib, or a new CLI in the stdlib? [/quote] Much like packaging, the stdlib very likely wouldn't be involved. [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] This is a proposal in how to make venv’s [/quote] If that's what you think then I've phrased something poorly. It _can_ lead to environment creation, but the main motivator is listing all the environments you have. I'll try to clarify this in my statement above. [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] If you added “make a .venv that points at the default environment created” to this proposal, it becomes a followup to the other one, I think? [/quote] Yes, this could be a follow-up, but Ofek very clearly said he doesn't like the `.venv` redirect file idea and thinks this proposal is a better solution. [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] I don’t think PEP 832 goes as far as to standardize part of `uv run`. though? [/quote] No, it does not. [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] Basically a `py` launcher that takes into account `.venv`. [/quote] The Python Launcher for Unix already does. [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] If it did, the this could follow on that and give the `py` launcher a way to make `.venv` if it doesn’t exist. [/quote] Correct, and part of the motivation for me with PEP 832 is so `.venv` is standardized enough that having the Python Launcher use PEP 832 isn't viewed as hoisting a workflow on to anyone. And as you have suggested, it opens the door for a `py run`-like experience. ------------------------- steve.dower | 2026-05-05 21:29:43 UTC | #4 I've said it before and I guess I'll say it again - this should be no more than a build-system-shaped hook in the `pyproject.toml` that says "if you trigger \ then this project will execute \" (by installing, importing, and calling its hook). Want something different? Don't run that generic command. A project wants different default settings for different tools? Include different config files/`tool` sections (aka. the status quo). There's no reason to try and specify the detail you're going into here - overrides/etc. should be entirely in the realm of tooling (thinking IDEs here, but also CLI-based workflow tools such as the Python Launcher). There are legitimate reasons to want alternative settings to be more local (e.g. `pyproject.local.toml`) or more global (e.g. an enterprise whitelisting workflow tools). The _only_ gap is a way to programmatically (i.e. without a human reading the dev docs) determine what tool/invocation can be used to initialise the workspace, and if you want to solve that then it'd better be done before everyone gets used to letting the AIs read the `README.md` to figure it out ;) [quote="Brett Cannon, post:3, topic:107186, username:brettcannon"] part of the motivation for me with PEP 832 is so `.venv` is standardized enough that having the Python Launcher use PEP 832 isn’t viewed as hoisting a workflow on to anyone [/quote] `-m venv .venv && .venv/bin/python ...`[^1] is a perfectly fine default for a generic tool, IMHO. And checking for a section in a `pyproject.toml` and invoking the tool listed there instead of `-m venv` feels like sufficient customisation. We don't need 100 choices; we need one good enough choice that projects can adapt to using or else ignore and do their own thing. [^1]: With suitable x-plat adjustments ------------------------- Jost | 2026-05-05 22:14:17 UTC | #5 I had a couple of thoughts while reading this proposal: **Is there demand from projects to mandate a specific environment management tool?** None of the projects I maintain or contribute to have particular requirements on the location of the environment or the tool^[There’s one exception here—a particle physics software stack I’ve worked on, which required conda because it needed to integrate complex non-Python dependencies.] used to create this. Admittedly, most of those are fairly simple projects; but even the developer docs of some very complex projects like [Airflow](https://discuss.python.org/t/pep-832-virtual-environment-discovery/106998/40) or [NumPy](https://numpy.org/devdocs/dev/development_environment.html#using-virtual-environments) are agnostic in both regards. I’m aware that some development teams within companies standardise on a specific workflow & environment managers; but I have no idea whether they’d be interested in this at all—I’d guess they already have custom tooling (or MDM software or …) to enforce that anyway. Without clear demand for such a `[workflow]` table in `pyproject.toml`, I don’t think the significant added complexity of adding such a table and the corresponding override mechanism (`pyproject.local.toml`, plus per-user and per-machine equivalents) is justified. --- **Supporting different types of environments (i.e. virtual and conda)** I see this as an important point; if the current approach in PEP 832 doesn’t allow for this (which is a discussion that I’ll put [in the other thread](https://discuss.python.org/t/pep-832-virtual-environment-discovery/106998/78)), that would be a significant downside. --- [quote="Brett Cannon, post:1, topic:107186, username:brettcannon"] Now that we know how to call a tool, *what* should it do? Well, it should return JSON on stdout about the environments for the project: [/quote] I’m thinking about how this would work for people using workflows where all environments are stored in a central folder. In that case, there probably won’t be a mapping of environments to projects. I guess conda could return a JSON listing of *all* environments (instead of “the environments *for the project*”); but would something similar work for all tools? ------------------------- brettcannon | 2026-05-05 22:20:22 UTC | #6 [quote="Steve Dower, post:4, topic:107186, username:steve.dower"] I’ve said it before and I guess I’ll say it again - this should be no more than a build-system-shaped hook in the `pyproject.toml` that says “if you trigger then this project will execute ” (by installing, importing, and calling its hook). [/quote] Ignoring the CLI-vs-hook difference, what are you suggesting here? We start standardizing the CLI for workflow tools overall in key scenarios for the "generic command"? Or are you thinking in terms of more like the Python Launcher has the generic command and tools like Hatch can optionally provide the "specific command" that the Launcher uses when provided? And how is that different than what I'm proposing here as a structured `py list --json`? The fact that this proposal doesn't specifically say, "`py list --json` will use this if provided, otherwise do its own thing"? [quote="Steve Dower, post:4, topic:107186, username:steve.dower"] overrides/etc. should be entirely in the realm of tooling (thinking IDEs here, but also CLI-based workflow tools such as the Python Launcher). [/quote] So you're saying we shouldn't write down how to resolve things and leave it to the tools to do their own thing? [quote="Steve Dower, post:4, topic:107186, username:steve.dower"] `-m venv .venv && .venv/bin/python ...` is a perfectly fine default for a generic tool, IMHO. [/quote] Same here, hence PEP 832 as it is effectively standardizing it as the baseline with the redirect files as an escape hatch (and I know you're aware of this; saying this for anyone else who may not know this view). [quote="Jost Migenda, post:5, topic:107186, username:Jost"] Is there demand from projects to mandate a specific environment management tool? [/quote] Enterprises that control everything do. There's also projects that use e.g. `uv.lock` and so mandate the tool being used. [quote="Jost Migenda, post:5, topic:107186, username:Jost"] **Supporting different types of environments (i.e. virtual and conda)** I see this as an important point; if the current approach in PEP 832 doesn’t allow for this [/quote] It doesn't explicitly, but introspection might allow for it (at least it's easy to tell if something is a virtual environment thanks to `pyvenv.cfg`). It also probably would suggest not using `.venv` for the redirect file name. [quote="Jost Migenda, post:5, topic:107186, username:Jost"] I’m thinking about how this would work for people using workflows where all environments are stored in a central folder. In that case, there probably won’t be a mapping of environments to projects. [/quote] Depends on the tool. It could keep a file in the project to record what has been selected before. It could record per-environment what projects it's used for. There isn't a technical reason the issue couldn't be solved in some way. ------------------------- cjames23 | 2026-05-05 22:26:35 UTC | #7 [quote="Brett Cannon, post:3, topic:107186, username:brettcannon"] Think your code editor to list the environments that you have available to choose from (if you happen to be a VS Code user, this would be used to populate the list of Python environments to choose from). Otherwise @ofek or @cjames23 might have some ideas as they have said Hatch already has similar support for this. [/quote] This is exactly what `hatch env show --json` is used for right now. We in fact still have an open issue to address a bug around this when integrating PyCharm https://github.com/pypa/hatch/issues/2011 which was opened from https://youtrack.jetbrains.com/projects/PY/issues/PY-81270/ When you run this command you get an output like ``` {"default":{"type":"virtual"},"hatch-build":{"skip-install":true,"installer":"uv","dependencies":["build[virtualenv]>=1.0.3"],"scripts":{"build-all":["python -m build"],"build-sdist":["python -m build --sdist"],"build-wheel":["python -m build --wheel"]},"type":"virtual"},"hatch-static-analysis":{"skip-install":true,"installer":"uv","dependencies":["ruff==0.12.0"],"scripts":{"format-check":["ruff format --check --diff ."],"format-fix":["ruff format ."],"lint-check":["ruff check ."],"lint-fix":["ruff check --fix ."]},"type":"virtual","config-path":"none"},"hatch-test.py3.13":{"installer":"uv","dependencies":["coverage-enable-subprocess==1.0","coverage[toml]~=7.4","pytest~=8.1","pytest-mock~=3.12","pytest-randomly~=3.15","pytest-rerunfailures~=14.0","pytest-xdist[psutil]~=3.5"],"scripts":{"run":["pytest --rootdir=. --junitxml=test-results.xml --cov --no-cov-on-fail --cov-report=lcov --cov-report=term-missing"]},"type":"virtual","features":["test"],"python":"3.13"},"hatch-uv":{"skip-install":true,"installer":"uv","type":"virtual"}} ``` Which then allows an editor to switch between the different environments using hatch. This could be formalized into a better API that also provides the location of each environment. Right now it is a two hop call, call `hatch env show` to get the list of environments and then call `hatch env find` with the environment name to see if it exists. I think if the schema of the JSON is standardized that allows tools to do the right thing, and I do not think based on private conversations that there would be any strong opposition from the popular tools towards providing this as a command with a standardized JSON schema for the output. I am very open to adding locations of the environment in hatch's env show JSON output. That is not a hard thing to add. And from the consuming side I would suggest looking at the output and any empty location field as an indicator that the environment has not been created yet. ------------------------- steve.dower | 2026-05-05 22:44:59 UTC | #8 [quote="Brett Cannon, post:6, topic:107186, username:brettcannon"] We start standardizing the CLI for workflow tools overall in key scenarios for the “generic command”? Or are you thinking in terms of more like the Python Launcher has the generic command and tools like Hatch can optionally provide the “specific command” that the Launcher uses when provided? [/quote] Basically, yeah. We did it for "build" (and created [build](https://pypi.org/project/build/) to implement it), so why not create "setup" or "prepare"[^1] and let frontends such as the Python Launcher start supporting it? [^1]: "workspace" implied [quote="Brett Cannon, post:6, topic:107186, username:brettcannon"] how is that different than what I’m proposing here as a structured `py list --json`? [/quote] You can only do something useful with that information if you created it or if someone specified it all the way through. The latter case stifles innovation, and the former is already possible. If it's limited to "things that `py` can launch" then it works, but that doesn't need anything more than an implementation in `py`, because anyone who wants to participate is literally only interoperating with one tool. Trying to extend it to "anything at all" is biting off way more than you want to chew ;) [quote="Brett Cannon, post:6, topic:107186, username:brettcannon"] So you’re saying we shouldn’t write down how to resolve things and leave it to the tools to do their own thing? [/quote] Pretty much. Or write it down as "in the absence of user preferences, do it this way, but you can do anything else you want if the user tells you to". Too many of these specs are designed to constrain the user (indirectly via tools, since that's the only mechanism we have), but fundamentally we want to _enable_ users to do anything they need to do, which means allowing tools to do anything their users want to do, which means drawing as small a boundary as possible around the interoperability point. In this case, we're saying "how do we have a single way to find [and implicitly, create if not found] the environments for this project". My proposal is that the interoperability point looks like "here's the tool and its config that knows how to do it for this project", so _anyone_ can easily find and initially invoke the tool, but the actual behaviour is entirely up to the tool. ------------------------- brettcannon | 2026-05-06 23:06:13 UTC | #9 [quote="Steve Dower, post:8, topic:107186, username:steve.dower"] Basically, yeah. We did it for “build” (and created [build](https://pypi.org/project/build/) to implement it), so why not create “setup” or “prepare” and let frontends such as the Python Launcher start supporting it? [/quote] I don't see anything explicitly against this right now, but I think you're promoting more of a PEP 517 approach that uses an code API and declaration of what's needs to be installed for the API to work for the calling tool to use (along with reasonable defaults for tools to follow if nothing is declared). I would expect there to be an API to find out about all environments (either existing or are possible based on a flag), and another to get a single environment with a flag as to whether it should be created (and either based on some key/name or "give me the default"). I'm not opposed to the idea, but there's 2 potential sticking points. One is I don't think most workflow tools have a Python API that treats them like a library, so that would be a change. Two, this does necessitate the caller handle the installation of the underlying environment management tool while my current approach can at least lean on `$PATH` to have the management tool already installed. I don't think either is insurmountable, but it is a shift. [quote="Steve Dower, post:8, topic:107186, username:steve.dower"] [quote="brettcannon, post:6, topic:107186"] how is that different than what I’m proposing here as a structured `py list --json`? [/quote] You can only do something useful with that information if you created it [/quote] Who is the "you" in that sentence? Me the user, or the environment management tool (e.g. Hatch)? Regardless, yes, someone inevitably needs to create an environment. [quote="Steve Dower, post:8, topic:107186, username:steve.dower"] or if someone specified it all the way through. The latter case stifles innovation [/quote] "Specified all the way through" as in put in the config details into `pyproject.toml`? [quote="Steve Dower, post:8, topic:107186, username:steve.dower"] If it’s limited to “things that `py` can launch” then it works, but that doesn’t need anything more than an implementation in `py`, because anyone who wants to participate is literally only interoperating with one tool. [/quote] Yes, but after talking to the SC about getting the Launcher to do more workflow stuff I got the feeling they have trepidation on dictating workflows in some way. Hence trying to get PEP 832 so there isn't any accusations of that. So "interoperating with one tool" I don't think works from the perspective of "`py` can come up with its own thing and people just work with that" without a PEP or something else people have signed off on (if that's what you mean). [quote="Steve Dower, post:8, topic:107186, username:steve.dower"] Trying to extend it to “anything at all” is biting off way more than you want to chew :wink: [/quote] There's a reason why I said from the outset there's a bar to clear before I change PEP 832. 😉 [quote="Steve Dower, post:8, topic:107186, username:steve.dower"] write it down as “in the absence of user preferences, do it this way, but you can do anything else you want if the user tells you to”. Too many of these specs are designed to constrain the user (indirectly via tools, since that’s the only mechanism we have), but fundamentally we want to *enable* users to do anything they need to do, which means allowing tools to do anything their users want to do, which means drawing as small a boundary as possible around the interoperability point. [/quote] I'm not married to trying to override/merge details in some clever way, but I do want some way to be able to set what tool to at least use if the project didn't specify a preference. Now you might be saying, "let the tool making the call handle how that gets resolved and supported" (e.g. VS Code just does its own thing)? In which case are you advocating for how to specify the API, how it works, and how write it down in `pyproject.toml`, but any other fanciness is a per-tool thing (e.g. specifying the per-user default in your personal VS Code settings)? [quote="Steve Dower, post:8, topic:107186, username:steve.dower"] In this case, we’re saying “how do we have a single way to find [and implicitly, create if not found] the environments for this project”. My proposal is that the interoperability point looks like “here’s the tool and its config that knows how to do it for this project”, so *anyone* can easily find and initially invoke the tool, but the actual behaviour is entirely up to the tool. [/quote] I'm trying to tease apart how fundamentally "find and initially invoke the tool" is different from what I'm proposing. Is it the fact I'm asking for a list instead of just asking for whatever environment the tool deems reasonable? Is "find" for you more PEP 517-like than what I have for `workflow.environments.requires`? Same for "invoke" being a code API instead of the proposed CLI API? ------------------------- ofek | 2026-05-07 07:48:56 UTC | #10 [quote="Brett Cannon, post:1, topic:107186, username:brettcannon"] What if my project has no opinion about what tool to use to manage your environments? \[...\] My gut says: * `pyproject.local.toml` * Support specifying next to `pyproject.toml`, per-user, and per-machine * You can override keys in `[workflow]` and `[tool]` via `machine | user | pyproject | local` (`pyproject.local.toml` in the project > `pyproject.toml` > per-user > per-machine) [/quote] I like enabling more control and I agree with the chosen override precedence. However, can we please punt on overriding anything other than the `workflow` table? I foresee that causing significant issues for this proposal being accepted. [quote="Brett Cannon, post:1, topic:107186, username:brettcannon"] I think we gain a `[workflow]` table and it has an `environments` key. That key takes a table with the following keys: * `cmd` or `shell` * `cmd` is an array of strings of a subprocess command to run * `shell` is a string to run in a subshell * Only one of these two keys can be specified, but one is required * `requires` is optional and specifies the requirement for the tool so you could install it from e.g. PyPI as necessary [/quote] 1. I think we should only allow `cmd` and require PATH being properly set up such that one never needs shell functionality to execute the binary. 2. I know we technically did it for the build system but I worry that the complexity of standardizing the behavior of `requires` may impact acceptance of this proposal. I'm fine with keeping it though. [quote="Brett Cannon, post:1, topic:107186, username:brettcannon"] Now that we know how to call a tool, *what* should it do? Well, it should return JSON on stdout about the environments for the project [/quote] [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] I don't think you can use stdout \[...\] you need to be able to communicate with the user \[...\] I think it would need to write out some sort of file [/quote] I agree that the output should be a file whose location is determined by the caller. The location must be passed as a command line argument rather than an environment variable so that we support non-local execution. [quote="Henry Schreiner, post:2, topic:107186, username:henryiii"] since it returns multiple environments, I think it would need to make all the environments, as the tool requesting the environments isn't requesting a specific one [/quote] Querying metadata should definitely not require manifesting the actual environment. As Cary mentioned, Hatch provides all of this information with or without the environment being created and for any type of environment. --- I'll provide a bit more feedback tomorrow as it's getting late here. Thanks Brett! ------------------------- brettcannon | 2026-05-07 22:24:40 UTC | #11 [quote="Ofek Lev, post:10, topic:107186, username:ofek"] However, can we please punt on overriding anything other than the `workflow` table? [/quote] Fine by me as so far both you and Steve are leery of going too far with overrides (albeit at different levels). [quote="Ofek Lev, post:10, topic:107186, username:ofek"] I think we should only allow `cmd` and require PATH being properly set up such that one never needs shell functionality to execute the binary. [/quote] The only reason shell support is there is for virtualenvwrapper and in case conda needed it for some reason (they do register shell stuff, but I would assume it wouldn't come into play here). [quote="Ofek Lev, post:10, topic:107186, username:ofek"] I know we technically did it for the build system but I worry that the complexity of standardizing the behavior of `requires` may impact acceptance of this proposal. [/quote] I'm up for taking it out if people prefer. [quote="Ofek Lev, post:10, topic:107186, username:ofek"] I agree that the output should be a file whose location is determined by the caller. [/quote] How would you expect for that to be specified? Tacked on to the end of the command? Add support for some placeholder like "{output}"? [quote="Ofek Lev, post:10, topic:107186, username:ofek"] The location must be passed as a command line argument rather than an environment variable so that we support non-local execution. [/quote] I don't understand how environment variables don't work in that situation. If I'm making a subprocess call, how does that not support using an environment variable? Or are you thinking of some future RPC support where it isn't the tool being called that's making the RPC call, so you're trying to future-proof for some unforeseen scenario? In that instance, wouldn't that mean no environment variable being used for anything? ------------------------- steve.dower | 2026-05-08 09:38:49 UTC | #12 [quote="Brett Cannon, post:11, topic:107186, username:brettcannon"] both you and Steve are leery of going too far with overrides (albeit at different levels). [/quote] I'm just leery of overspecifying them. If you write it in a PEP, then tools _must_ do it, but they should be allowed to do whatever is best for their users. We should only be specifying the author's (of the configuration) intent, and tools can choose whether to honour that intent or not by themselves (based on what their end-user wants). [quote="Brett Cannon, post:11, topic:107186, username:brettcannon"] [quote="ofek, post:10, topic:107186"] I know we technically did it for the build system but I worry that the complexity of standardizing the behavior of `requires` may impact acceptance of this proposal. [/quote] I’m up for taking it out if people prefer. [/quote] In the same vein, specifying _the intent_ that "these things are required" is totally fine by me. Specifying that tools _must_ figure out how to install those things is going too far. Let tools choose to honour the intent however they see fit. [quote="Brett Cannon, post:11, topic:107186, username:brettcannon"] [quote="ofek, post:10, topic:107186"] The location must be passed as a command line argument rather than an environment variable so that we support non-local execution. [/quote] I don’t understand how environment variables don’t work in that situation. If I’m making a subprocess call, how does that not support using an environment variable? [/quote] I agree with Brett, if there's going to be commands specified, then environment variables are a must. They're the only reliable way to avoid shell injection and escaping issues (writing via a file introduces permissions and TOCTOU issues). Still, I'd rather not deal with subprocess calls here at all - call into a Python API with trustworthy and reliable Python objects and let the wrapper figure out how best to invoke whatever tool it's calling. [quote="Brett Cannon, post:9, topic:107186, username:brettcannon"] I’m trying to tease apart how fundamentally “find and initially invoke the tool” is different from what I’m proposing. Is it the fact I’m asking for a list instead of just asking for whatever environment the tool deems reasonable? [/quote] We're talking about it at different levels, I think. I mean "find" like [the instructions on this page](https://github.com/pyca/cryptography/blob/main/docs/development/getting-started.rst), which has already mostly hidden the work behind a single `nox` command already, but we should be able to hide it behind a generic `py hypothetical-set-up-my-project` command that can discover that it needs `nox` itself.[^2] The things you seem to be thinking about - finding/creating Python executables - are to me the responsibility of `nox` (in this example), and once we've launched that we stay out of it. Trying to standardise the steps _inside_ `nox` is what I want to avoid [having you waste your precious time on ;) ]. There are too many possible workflows out there, including many we've never even considered, and we don't solve that by choosing one universal one - we solve it by making _most_ people not have to research which one is being used right now. [^2]: And whether it installs it when it's missing or just reports an error is left up to the tool. ------------------------- brettcannon | 2026-05-08 21:58:55 UTC | #13 [quote="Steve Dower, post:12, topic:107186, username:steve.dower"] In the same vein, specifying *the intent* that “these things are required” is totally fine by me. Specifying that tools *must* figure out how to install those things is going too far. Let tools choose to honour the intent however they see fit. [/quote] And I wasn't planning on having this proposal require installation for calling a CLI; I was treating it as a hint as to what was expected to exist at worst, or to let a tool do some installation if they so desired. [quote="Steve Dower, post:12, topic:107186, username:steve.dower"] We’re talking about it at different levels, I think. I mean “find” like [the instructions on this page](https://github.com/pyca/cryptography/blob/main/docs/development/getting-started.rst), which has already mostly hidden the work behind a single `nox` command already [/quote] And that's why tox supports PEP 832 as it currently sits for the exact same workflow: environment creation is not standardized, just how you write down where the environment is so other tools can be let in on that bit of knowledge. [quote="Steve Dower, post:12, topic:107186, username:steve.dower"] but we should be able to hide it behind a generic `py hypothetical-set-up-my-project` command that can discover that it needs `nox` itself. [/quote] Yes, it's at a different level as you surmised. Your view aligns more with what's in PEP 832: simple case of "tell me where the/an environment is". But this proposal is "surface all details that may need to be surfaced in e.g. an editor". Probably the biggest difference between PEP 832 and what you're after here is the Python API instead of redirect files. I think if I were to take what you're proposing it would be to: 1. Keep `.venv` as the suggested default for a single virtual environment 2. Keep the concept of environment redirect files, but maybe change the file name if conda would use them 3. Add in your proposed PEP 517-like Python API for setup/bootstrapping the first time you start using a project 4. Come up with some guidelines to set up per-user, and per-machine specifications when the project lacks them (I'm skipping over per-project on purpose; I just want to at this point let users say, "in the face of no preference, use Hatch" or something with the assumption and that per-project doesn't make sense for that and you would edit `pyproject.toml` in that case anyway). That lets the detection of an environment be somewhat of a cheap check if things are set up (if tools chose to work that way; it wouldn't be mandated but the PEP would say it's okay as well). And having the command be a setup/bootstrap would give leeway in terms of doing more than creating an environment (e.g. downloading stuff). To be clear, this is **not** what I'm explicitly proposing as coming out of this conversation, just where my head is at if Steve's ideas win out. [quote="Steve Dower, post:12, topic:107186, username:steve.dower"] There are too many possible workflows out there, including many we’ve never even considered [/quote] And I _think_ that's the view @ofek is taking: if there's going to be any attempt at this, it should be broad and be flexible for future changes. ------------------------- lucascolley | 2026-05-10 16:28:27 UTC | #14 [quote="Brett Cannon, post:6, topic:107186, username:brettcannon"] [quote="Jost, post:5, topic:107186"] Is there demand from projects to mandate a specific environment management tool? [/quote] Enterprises that control everything do. There’s also projects that use e.g. `uv.lock` and so mandate the tool being used. [/quote] Using a lock file does not imply a mandated tool. In SciPy we use `pixi.lock` for the majority of CI workflows, for example, but developers may still choose to use pip venvs with their IDE — nothing is stopping that. [quote="Brett Cannon, post:6, topic:107186, username:brettcannon"] [quote="Jost, post:5, topic:107186"] **Supporting different types of environments (i.e. virtual and conda)** I see this as an important point; if the current approach in PEP 832 doesn’t allow for this [/quote] ... It also probably would suggest not using `.venv` for the redirect file name. [/quote] Yeah, probably something more obviously generic would be better. ------------------------- tmk | 2026-05-11 12:52:08 UTC | #15 [quote="Brett Cannon, post:1, topic:107186, username:brettcannon"] By having you e.g. code editor *pull* the info by calling a tool, it gets rid of any bootstrap issues of having not created a `.venv` file yet. And by having the tool you call that *can* create any environments that are needed to start work, it helps with the bootstrapping problem of creating the environment first before launching your editor [/quote] If environment creation could be automated, that would be great, but this functionality doesn't have to be bundled with the rest of the proposal. I think a combination of a *push*-based approach for environment *discovery* and a *pull*-based approach for environment *creation* makes sense, because the former is done often and should be as cheap as possible, while the latter is done rarely, such that it is okay if it is more expensive. As for the issue of discovering multiple environments, I think a push-based solution can be found. ------------------------- brettcannon | 2026-05-11 23:26:23 UTC | #16 [quote="Lucas Colley, post:14, topic:107186, username:lucascolley"] Using a lock file does not imply a mandated tool. In SciPy we use `pixi.lock` for the majority of CI workflows, for example, but developers may still choose to use pip venvs with their IDE — nothing is stopping that. [/quote] I would argue that's true because you control the CI and so daily development doesn't require it to run tests locally. But if you expected me to keep that lock file updated then you have mandated the tool, you just happen to mandate it in a way where I can avoid it if I wanted to. But I do get your point it isn't a forgone conclusion that a specific tool is necessary even in the face of tool-specific files for all projects. [quote="Lucas Colley, post:14, topic:107186, username:lucascolley"] [quote="brettcannon, post:6, topic:107186"] … It also probably would suggest not using `.venv` for the redirect file name. [/quote] Yeah, probably something more obviously generic would be better. [/quote] While I have the attention of a conda-knowledgeable person, @lucascolley , if I pointed you at a directory and asked, "is this a conda environment", could that be answered purely by the file system or file contents (like you can with virtual environments by the `pyvenv.cfg` file)? [quote="Thomas Kehrenberg, post:15, topic:107186, username:tmk"] If environment creation could be automated, that would be great, but this functionality doesn’t have to be bundled with the rest of the proposal. [/quote] No, but the question becomes whether I have it in me to do a separate PEP for project setup. I also want to hear from @steve.dower that this idea reflects what he's suggesting and from @ofek as he said ... [quote="Ofek Lev, post:10, topic:107186, username:ofek"] I’ll provide a bit more feedback tomorrow as it’s getting late here. [/quote] ... 5 days ago. 😉 [quote="Thomas Kehrenberg, post:15, topic:107186, username:tmk"] As for the issue of discovering multiple environments, I think a push-based solution can be found. [/quote] Oh, definitely! It's more of a question as to whether I have it in me to solve it in this PEP or leave it for someone else to solve. 😉 ------------------------- lucascolley | 2026-05-12 07:47:23 UTC | #17 [quote="Brett Cannon, post:16, topic:107186, username:brettcannon"] While I have the attention of a conda-knowledgeable person, @lucascolley , if I pointed you at a directory and asked, “is this a conda environment”, could that be answered purely by the file system or file contents (like you can with virtual environments by the `pyvenv.cfg` file)? [/quote] Yes, from https://github.com/conda/ceps/blob/main/cep-0032.md#specification: > A conda environment is defined as a directory that contains, at least, a `./conda-meta/history` file. ------------------------- steve.dower | 2026-05-12 15:19:38 UTC | #18 [quote="Brett Cannon, post:16, topic:107186, username:brettcannon"] [quote="tmk, post:15, topic:107186"] If environment creation could be automated, that would be great, but this functionality doesn’t have to be bundled with the rest of the proposal. [/quote] No, but the question becomes whether I have it in me to do a separate PEP for project setup. I also want to hear from @steve.dower that this idea reflects what he’s suggesting [/quote] It kind of does, because I think a key point is that only the tool that _created_ an environment can really _identify_ it (in a useful way), and so your choice is either to define a way to delegate all the steps to e.g. conda[^1], or to define exactly how all tools must create their environments so that you can identify them without delegating to the tool.[^2] My idea is to do the former, which doesn't require you to write another PEP for project setup. [^1]: Convenient example since you also just had to ask how Conda identifies its own environments. [^2]: One way to do this could be to make `pyvenv.cfg` "mandatory" and add new fields to let the runtime ignore it when it's not a `venv` environment. I'm specifically _not_ advocating this. ------------------------- brettcannon | 2026-05-12 23:02:20 UTC | #19 [quote="Steve Dower, post:18, topic:107186, username:steve.dower"] My idea is to do the former, which doesn’t require you to write another PEP for project setup. [/quote] I talked with @steve.dower offline and his idea for the API is to probably support (assuming I'm not misrepresenting what Steve is suggesting): 1. Listing the environments 2. Setting up the project (which probably implies setting up the environments) 3. Running a command using one of the environments This would abstract out a lot of details so that tools working at this level could do something like: 1. Setup working with a project when you open a project in your editor 2. Get a list of environments to work with so your editor knows what you want to work with 3. Run code with a selected environment so your editor can launch you test suite But it's abstracted out such that the environment could be remote or something. This can all be driven by some CLI tool directly or by some tool on your behalf. Steve also suggested having some project that defines a reasonable fallback that tools could use. That is where Steve would have `.venv` as a "I have no preference" default live and not have that in the PEP. Steve also doesn't like the redirect file idea since you could just ask for the list rather than writing it down. While I can see this working in an ideal scenario (especially if it expands to cover more situations over time), the question I'm ruminating about is whether this will lead to actual adoption of this approach or if people would balk at the idea of having their workflow tool abstracted away this much. ------------------------- pf_moore | 2026-05-13 09:56:27 UTC | #20 [quote="Brett Cannon, post:19, topic:107186, username:brettcannon"] While I can see this working in an ideal scenario (especially if it expands to cover more situations over time), the question I’m ruminating about is whether this will lead to actual adoption of this approach or if people would balk at the idea of having their workflow tool abstracted away this much. [/quote] For me, I think I'd just use whatever tool I preferred, and wouldn't see having to add config to say "my tool is XXX" (which I assume I'd need to to for the API to work) as being any better than the current situation. I *wouldn't* use a generic API for my normal workflow, it would be solely for IDEs and the like. Conversely, a naming convention like `.venv` is (for me, at least) a zero-friction way to work in a way that IDEs can just use, without me needing to do anything. ------------------------- steve.dower | 2026-05-13 14:52:49 UTC | #21 [quote="Paul Moore, post:20, topic:107186, username:pf_moore"] I think I’d just use whatever tool I preferred, and wouldn’t see having to add config to say “my tool is XXX” (which I assume I’d need to to for the API to work) as being any better than the current situation. [/quote] The main difference from the current situation is (using VS Code as the example front-end) right now the VS Code _user_ has to specify (and determine) which tool to use, while with the API the project _maintainer_ can specify it and the user doesn't have to specify anything. If you are both of these personas, then yes, nothing changes. And most people who are going to be here discussing this are either both or just the maintainer, so the only friction they'll have experienced is having to explain to contributors how to set things up. The friction being smoothed here is _the user's friction_, not the maintainer's. Most people here shouldn't expect their lives to change significantly with an idea like this (but if it sounds like your life is getting *worse*, then definitely speak up!) ------------------------- pf_moore | 2026-05-13 15:13:57 UTC | #22 [quote="Steve Dower, post:21, topic:107186, username:steve.dower"] while with the API the project *maintainer* can specify it and the user doesn’t have to specify anything. [/quote] But the project maintainer doesn't know what tool I want to use to work on a project. This comes back to the question of whether environment management is a project choice or a user choice, and I'm strongly in favour of the idea that it's a user choice (with the project able to recommend a workflow if they choose). If having a CLI API means accepting that it's a project choice what workflow gets used, count me as a strong -1. Edit: And speaking as a maintainer, I explicitly don't want to choose the user's workflow for them, so my life is made worse by the fact that I can no longer choose a workflow for my personal use on this project *without* it being imposed on my project users. ------------------------- steve.dower | 2026-05-13 15:20:44 UTC | #23 [quote="Paul Moore, post:22, topic:107186, username:pf_moore"] I explicitly don’t want to choose the user’s workflow for them, so my life is made worse by the fact that I can no longer choose a workflow for my personal use on this project *without* it being imposed on my project users. [/quote] And yet... https://github.com/pypa/pip/blob/main/docs/html/development/getting-started.rst But I agree, none of this is meant to become mandatory. It's a discovery tool, so that someone who clones a repo to start contributing can just "give me the default the project recommends" without having to find that document themselves. If you don't want it... just... don't do it? And I've already argued that tools should feel free to ignore this metadata if the user has overridden (at any level). The precedence/priority there doesn't need to be specified - we don't need to tell VS Code what to do when a user has said "ignore the spec" - but I assume we do need to explicitly say "it's okay to ignore this without having the OSS community bite your head off". ------------------------- davidism | 2026-05-13 15:22:31 UTC | #24 For what it's worth, Flask uses uv, and documents how to use uv. (Previously it was pip and pip-tools, and briefly PDM.) We absolutely do want to choose our contributors' workflow. We pin an exact set of tools for our development environment, and instruct users how to use that, so that every contributor is using the same starting point. They can certainly try to use something else, but that's on them, we won't be able to support it or help at sprints. ------------------------- davidism | 2026-05-13 15:26:40 UTC | #25 And from a contributor perspective, I would much rather be told exactly what tools to use to contribute to a project. I occasionally still run into projects that just say "here's a python project, run the tests, submit a PR" and then have to figure out exactly how to do that myself, and it's a giant pain. Projects should absolutely be picking one set of tools and using them. That doesn't mean other projects can't pick other tools, only that each project should pick one. I'm fine if one project is PDM, one is Conda, and one is uv, as long as each tells me exactly how to use that to develop and contribute. ------------------------- pf_moore | 2026-05-13 15:46:29 UTC | #26 [quote="Steve Dower, post:23, topic:107186, username:steve.dower"] And yet… [pip/docs/html/development/getting-started.rst at main · pypa/pip · GitHub](https://github.com/pypa/pip/blob/main/docs/html/development/getting-started.rst) [/quote] That's advisory, not required (yes, the wording is a bit stronger than that, but we don't actually care in the majority of cases). [quote="Steve Dower, post:23, topic:107186, username:steve.dower"] If you don’t want it… just… don’t do it? [/quote] I thought that at least part of the idea was that IDEs like VS Code could discover where I keep the virtual environments I'm using to work on the project? If `pyproject.toml` says they should be in Poetry, but I'm actually using venv, what do I do? My impression was that I have to add an override file somewhere to say I'm not using the project default. Which, as I said originally, feels like it's no improvement over the current situation where I have to select the environment I'm using. [quote="David Lord, post:25, topic:107186, username:davidism"] And from a contributor perspective, I would much rather be told exactly what tools to use to contribute to a project. [/quote] There's clearly different viewpoints here. I personally *really* dislike Poetry, and on at least one project where I've been required to use it, it's been a significant disincentive to contribute. Also, there's a configuration issue here. I'll continue to use Poetry as my example because the problem is worse for tools written in Python. But let's assume the project config says that you list the project environments using `poetry env list` (for example). But I don't use Poetry, and don't want it globally installed. So when *I* use Poetry, I use it via `uvx poetry`. Do I need to override the project config just because I choose to *invoke* Poetry via a different mechanism than the project developers do? I don't want to make too much of this - for the average user, none of these concerns are that important. But for people who *do* have non-standard needs, a CLI API is often fiddly to set up. And yes, the affected users have the knowledge to deal with this, but often *tools* don't cover all the bases. It just feels like a lot more work for very little benefit compared to PEP 832. ------------------------- steve.dower | 2026-05-13 15:49:15 UTC | #27 [quote="Paul Moore, post:26, topic:107186, username:pf_moore"] My impression was that I have to add an override file somewhere to say I’m not using the project default. Which, as I said originally, feels like it’s no improvement over the current situation where I have to select the environment I’m using. [/quote] Correct, the case where you are not using the default remains the same as today (where there is no default). If the default happens to align with your preference, things get easier, which is also what would happen if we mandated that the only possible default was venv+`.venv`+CPython. We don't want to mandate that, which means there'll be a maintainer choice of default, and if you ignore it, then you do exactly what you do today (and maybe it gets worse if VS Code decides to make it more complicated to manually invoke the tools, but you'll have to take it up with the front-end developers). That feels like a long way to say "if you ignore this, you ignore it" again, but that's the idea. Like I said, the improvement is meant for the users who no longer have to find, read, and copy-paste the commands from the getting-started page. ------------------------- pf_moore | 2026-05-13 15:52:44 UTC | #28 [quote="Steve Dower, post:27, topic:107186, username:steve.dower"] Like I said, the improvement is meant for the users who no longer have to find, read, and copy-paste the commands from the getting-started page. [/quote] (One last comment, then I'll drop this as I think it's getting unproductive). If we have a way for projects to put workflow setup into `pyproject.toml`, I imagine they will *stop* documenting the recommended workflow in the contributor docs. That means that for people like me who prefer to use my standard workflow and adapt it to make sure I'm not being an inconvenience to the project, I no longer have documentation to work from, and I'm reduced to reverse-engineering the config file. Which I won't do, I'll likely just not contribute instead. ------------------------- steve.dower | 2026-05-13 15:56:35 UTC | #29 [quote="Paul Moore, post:28, topic:107186, username:pf_moore"] I’m reduced to reverse-engineering the config file. Which I won’t do, I’ll likely just not contribute instead. [/quote] I already reverse engineer the CI definitions for half the projects I contribute to, since they're _always_ more up to date that the documentation. And the more projects switch to "just use uv" (or more complicated tox/nox/etc.-based projects), the more you have to know how to reverse engineer it back to the basics. Maybe that gets worse when simple projects no longer have to explain even venv/pip, I'm not sure. I suspect projects who notice that they're losing skilled contributors by only offering the one-click setup will add more details for those contributors. Those that don't notice, well, won't care. Their loss, just like any other project that makes itself difficult to contribute to. ------------------------- davidism | 2026-05-13 16:00:42 UTC | #30 All I’d add to our docs is “if you are using an IDE that supports it, the environment will be set up automatically. Otherwise follow these steps.” ------------------------- fungi | 2026-05-13 16:30:02 UTC | #31 >If we have a way for projects to put workflow setup into `pyproject.toml`, I imagine they will *stop* documenting the recommended workflow in the contributor docs. If it were projects *I* work on, the documentation would at least say that the recommended workflow setup is managed by that mechanism with its configuration in the accompanying pyproject.toml file, or whatever, so that newcomers know what they're expected to run and where to look to find out what it does. I don't think changing workflows means you stop documenting them. Of course, projects that never bothered to document their recommended developer environment setup will probably continue to not do so. ------------------------- brettcannon | 2026-05-13 21:02:17 UTC | #32 [quote="Paul Moore, post:26, topic:107186, username:pf_moore"] If `pyproject.toml` says they should be in Poetry, but I’m actually using venv, what do I do? [/quote] This is one of the reasons I'm tempted to keep the redirect file idea from PEP 832 around; it's simple and can act as a short-circuit for tools to skip calling any API for the virtual environment. [quote="Paul Moore, post:26, topic:107186, username:pf_moore"] But I don’t use Poetry, and don’t want it globally installed. So when *I* use Poetry, I use it via `uvx poetry`. Do I need to override the project config just because I choose to *invoke* Poetry via a different mechanism than the project developers do? [/quote] In this case having a PEP 517-like API instead of a CLI one would help as e.g. your editor could just have a cached install it chose to use. ------------------------- brettcannon | 2026-05-14 18:56:21 UTC | #33 [quote="Paul Moore, post:22, topic:107186, username:pf_moore"] This comes back to the question of whether environment management is a project choice or a user choice, and I’m strongly in favour of the idea that it’s a user choice (with the project able to recommend a workflow if they choose). If having a CLI API means accepting that it’s a project choice what workflow gets used, count me as a strong -1. [/quote] I was thinking about this last night and trying to articulate in my head why you and others feel this way. I want to be upfront this is not a leading question or meant to suggest that it's an incorrect view. I'm just trying to think through _why_ some of us think this, perhaps to include it in the PEP. When it comes to other things like test runners (e.g. pytest), task runners (e.g. tox and nox), or pretty much anything else where you technically don't have to use what the project provides, people in general don't seem to have the same reaction of "don't tell me what to do" as they do to virtual environments. Now I don't think it's due to disk use or anything as tox and now definitely put files on your disk. So what makes virtual environments special in this regard? Is it a "get off my lawn" reaction for those of us doing this manually for so long that we just like our workflow a lot and don't want to be forced out of it at a loss of personal efficiency? Or is it more like picking a code editor and interacting with your virtual environment feels closer to you than other things and thus more personal? Or is it a belief/assumption most projects don't really need to have a preference, so why give them the ability to somewhat arbitrarily choose one that gets hoisted upon contributors (while admitting some projects _do_ have reasons to require a specific tool)? Or something else I'm not thinking of? As I said, I don't view any of this as wrong. I'm just trying to think through where the feeling comes from to help shape where I should take things and the motivation behind any decisions I have to make. ------------------------- bwoodsend | 2026-05-14 19:41:40 UTC | #34 [quote="Brett Cannon, post:33, topic:107186, username:brettcannon"] When it comes to other things like test runners (e.g. pytest), task runners (e.g. tox and nox), or pretty much anything else where you technically don’t have to use what the project provides, people in general don’t seem to have the same reaction of “don’t tell me what to do” as they do to virtual environments. Now I don’t think it’s due to disk use or anything as tox and now definitely put files on your disk. So what makes virtual environments special in this regard? [/quote] At least for me, the answer is that they aren't special. I boycott tox, uv, precommit, all the venv alternatives and any other tool that simply installs^[often in its own cold cache so everything takes forever to setup] and wraps its own preferred interface^[which I then have to learn -- usually inadequately so that I end up having to debug via print statements or some such because I can't figure out where the `-m pdb` goes] around other tools^[which I probably already have installed globally] the same. They're all akin to pushing a specific IDE or keyboard+mouse+monitor setup to me. I've never not regretted humouring a contributing guide that told me to use its preferred environment management. It's at the top of my mental list of things to avoid when deciding whether to submit a pull request to a project or just dump it on their issue tracker and guiltily expect the maintainer to do the rest. What can make things OK is if the configurations for these wrapper tools are easily unpacked. If `pip install -e .` still works and the raw test command+dependencies are easy to find and it's easy to figure out "ok, this project lints with black and flake8" then I don't mind coexisting with this workflow tools since there's still a user level choice to not use them. ------------------------- pf_moore | 2026-05-14 21:08:24 UTC | #35 [quote="Brett Cannon, post:33, topic:107186, username:brettcannon"] Or something else I’m not thinking of? [/quote] For me, it's that I want to choose where my venvs go. By putting them in my project directory, I can see them (I'm on Windows, so a directory named `.venv` is visible), they get deleted with the project, and I can delete or recreate them myself when I want to. I can also use them in an adhoc manner just by running `.venv/Scripts/python.exe`. With tools that hide venvs away somewhere, I lose that control. The environments don't retain any link to the project directory (or if they do, it's some tool-specific data that I probably don't know about), and when I delete the project, the orphaned venvs remain, taking up disk space^[I know disk space isn't that important, but unrecoverable garbage gradually filling my disk is (to me at least).]. And to start a Python interpreter in the venv I have to either use a tool-specific command, or find the executable which is likely in some obscure directory. So the `.venv` proposal works great for me, because it leaves me in control. Whereas this CLI API proposal is *designed* to hide the details from me, so that I lose that control. (And the side discussion about projects that mandate a particular tool ties into that, because they might require a tool that uses "hidden" environments). To be honest, it's not just about venvs. I'd feel the same way about a project that mandated hatch for its test runner. It's the same problem - the environments used for testing get stored in a central directory, and when I delete the project they will remain unless I remember to deliberately hunt them down and remove them. And as I'm not a hatch user, I won't be familiar with the need to do that, so I'll almost certainly forget. ------------------------- steve.dower | 2026-05-14 21:16:43 UTC | #36 FWIW, I share the same dislike of tools that install to secret directories when there's an obvious place to put it.[^1][^2] I'm not concerned that this proposal makes that any worse, though, just makes it easier to have one of those tools cluttering up unrelated locations. Maybe if we're all forced to deal with it more often then we'll complain to the projects/tools instead of just avoiding them and doing it manually? No telling how many people would prefer them to change their default behaviour... [^1]: I need that caveat because I _do_ like that the Python install manager uses a "secret" directory, but also has a `--target` option for when there's an obvious place to put the runtime. If it gained a `py run`-like option, then I'd strongly lean towards putting it in the project directory by default. [^2]: Including a cache directory of files that gets linked into my project-specific directory. ------------------------- ofek | 2026-05-17 18:12:46 UTC | #37 [quote="Ofek Lev, post:10, topic:107186, username:ofek"] [quote="brettcannon, post:1, topic:107186"] * \[...\] `pyproject.local.toml` in the project > `pyproject.toml` > per-user > per-machine [/quote] \[...\] I agree with the chosen override precedence. [/quote] Upon further consideration, I think we should adopt the strategy of most AI agents and increase the priority of machine configuration for easier managed installations. I'd go with: `pyproject.local.toml` > per-machine > `pyproject.toml` > per-user [quote="Brett Cannon, post:11, topic:107186, username:brettcannon"] [quote="ofek, post:10, topic:107186"] The location must be passed as a command line argument rather than an environment variable so that we support non-local execution. [/quote] I don't understand how environment variables don't work in that situation. \[...\] are you thinking of some future RPC support where it isn't the tool being called that's making the RPC call, so you're trying to future-proof for some unforeseen scenario? [/quote] Yes, I was thinking about scenarios where environments would be managed remotely. Thank you for being able to extrapolate through my lack of specificity :slight_smile: After thinking more about this, I'm okay with saying that a CLI will forever be the only communication mechanism. [quote="Brett Cannon, post:13, topic:107186, username:brettcannon"] [quote="steve.dower, post:12, topic:107186"] There are too many possible workflows out there, including many we've never even considered [/quote] And I *think* that's the view @ofek is taking: if there's going to be any attempt at this, it should be broad and be flexible for future changes. [/quote] Yes, that's exactly my view. I especially appreciate Steve explaining that our preference was operating at a different level than what some folks were internalizing. We don't want to prescribe what workflow tools do so that consumers can rely on invariants to hold when introspecting but rather request a high-level operation that's more of a black-box e.g. we want to tell an API to run a command in the context of a certain environment rather than finding the Python binary within an environment on disk in order to run a command. [quote="Brett Cannon, post:19, topic:107186, username:brettcannon"] I talked with @steve.dower offline and his idea for the API is to probably support (assuming I'm not misrepresenting what Steve is suggesting): 1. Listing the environments 2. Setting up the project (which probably implies setting up the environments) 3. Running a command using one of the environments [/quote] I've thought a lot about this the past two weeks and I think I landed on the optimal design that would satisfy everyone. There are two aspects: the API and how to communicate with the environments manager. * Communication: I think workflow tools should behave like language servers. A consumer like an IDE would start the process (`hatch env server`, `uv …`, etc.) and send API requests to its stdin while receiving responses on stdout. This is much cleaner and more extensible than mandating a subcommand structure with certain flags for each command. It would also eliminate consideration for the frequency of operations in our design choices since startup would occur once (e.g. no more optimizing for environment discovery because users would perceive the call to list local environments as no slower than reading known paths from disk). We should also make some sort of fast exit path option for times when you want to run only a single operation without process management which is useful for AI agents in change+validation loops, redistributors like Conda or Debian who want to properly run a package's test suite, etc. This also provides a way to reduce background memory consumption for potentially suboptimal implementations. * API: I think one server process should be capable of managing environments for an arbitrary number of projects so every operation would accept an optional `cwd` string field referring to an absolute path that would default to the current working directory. I prefer future-proofing here by acting upon the project root directory rather than a `pyproject.toml` file specifically. Operations that target environments would accept an optional `envs` field referring to a non-empty array of environment names that would default to a single environment chosen by the tool, or an error if tool-specific project configuration enforces explicit selection. The initial operations would be: * `create` - options: `cwd`, `envs` * `remove` - options: `cwd`, `envs` * `exec` - options: `cwd`, `envs`, `cmd` array of strings e.g. `[”coverage”, "run", "-m", "pytest"]` * `list` - options: `cwd` | response: `envs` array of strings representing the available environments * `inspect` - options: `cwd`, `envs` | response: `env_info` map of environment names to a semi-structured mapping containing details about the environment * (future wish list) `lsp` - options: `cwd`, `env` (required) a single environment | response: `cmd` array of strings e.g. `[”pyrefly”, "lsp"]` This API unlocks really cool workflow-level functionality as future PEPs like the definition of user-defined `tasks` which would run commands in the context of certain environments. We could then reserve the name of certain tasks so that, if we copy the reserved `extras` names as examples, the `test` task would run tests and the `doc` task would build documentation. [quote="Brett Cannon, post:19, topic:107186, username:brettcannon"] While I can see this working in an ideal scenario (especially if it expands to cover more situations over time), the question I'm ruminating about is whether this will lead to actual adoption of this approach or if people would balk at the idea of having their workflow tool abstracted away this much. [/quote] I think anyone balking would be doing so out of ignorance of the benefits that standardized environment management interoperability will bring to users. We should definitely express this in documentation but otherwise I don't think we should care much about such sentiment. [quote="Brett Cannon, post:19, topic:107186, username:brettcannon"] Steve also suggested having some project that defines a reasonable fallback that tools could use. That is where Steve would have `.venv` as a "I have no preference" default live and not have that in the PEP. [/quote] I'm okay with a `.venv` directory alongside a `pyproject.toml` file acting as the default IFF it's a non-empty directory and there is no workflow tool defined anywhere. Consumers can assume the directory structure is that of a standard virtual environment but they should not create it on behalf of the user. An improperly configured environment is far more user-hostile than no environment. [quote="Brénainn Woodsend, post:34, topic:107186, username:bwoodsend"] I boycott tox, uv, precommit, all the venv alternatives and any other tool that simply installs and wraps its own preferred interface around other tools the same. They're all akin to pushing a specific IDE or keyboard+mouse+monitor setup to me. I've never not regretted humouring a contributing guide that told me to use its preferred environment management. It's at the top of my mental list of things to avoid when deciding whether to submit a pull request to a project or just dump it on their issue tracker and guiltily expect the maintainer to do the rest. [/quote] [quote="Brett Cannon, post:33, topic:107186, username:brettcannon"] [quote="pf_moore, post:22, topic:107186"] This comes back to the question of whether environment management is a project choice or a user choice, and I'm strongly in favour of the idea that it's a user choice (with the project able to recommend a workflow if they choose). If having a CLI API means accepting that it's a project choice what workflow gets used, count me as a strong -1. [/quote] I was thinking about this last night and trying to articulate in my head why you and others feel this way. I want to be upfront this is not a leading question or meant to suggest that it's an incorrect view. [/quote] I'm under the assumption that as a primary goal we all want to maximally reduce user friction while simultaneously allowing for freedom of choice. As such, and knowing that I have good rapport with Paul and Brénainn, I'll take the opposite stance and assert that these are generally incorrect views along two different axes :wink: Incidentally, @davidism already expressed views above that are perfectly aligned with the goal for both. The first is the experience of users or contributors: [quote="David Lord, post:25, topic:107186, username:davidism"] And from a contributor perspective, I would much rather be told exactly what tools to use to contribute to a project. I occasionally still run into projects that just say "here's a python project, run the tests, submit a PR" and then have to figure out exactly how to do that myself, and it's a giant pain. Projects should absolutely be picking one set of tools and using them. That doesn't mean other projects can't pick other tools, only that each project should pick one. I'm fine if one project is PDM, one is Conda, and one is uv, as long as each tells me exactly how to use that to develop and contribute. [/quote] This is critical. Most users lack the knowledge, time and compelling motivation to use anything but what a project recommends. The vast majority of users want a workflow that is functional out-of-the-box with minimal effort. It's well-known that I made Hatch in part due to my dislike of Poetry. I was never forced to use it as part of several contributions; I did so several times willingly. Not once did I look through a project's Poetry, CI and other configuration in order to reverse-engineer what would give me a working setup just to use my preferred workflow or shell aliases. My time on this planet is limited and I have no desire to waste it in such a way when I can simply install a tool to run one or two of its commands. [quote="David Lord, post:24, topic:107186, username:davidism"] For what it's worth, Flask uses uv, and documents how to use uv. (Previously it was pip and pip-tools, and briefly PDM.) We absolutely do want to choose our contributors' workflow. We pin an exact set of tools for our development environment, and instruct users how to use that, so that every contributor is using the same starting point. They can certainly try to use something else, but that's on them, we won't be able to support it or help at sprints. [/quote] The second aspect here is about correctness. Projects often have logic encoded in their recommended development workflows that are difficult or impossible to express otherwise. Using a strawman to illustrate, consider someone desiring to build a project in the Google3 monorepo with something other than Blaze. For a more realistic example, take a look at the [Breeze](https://github.com/apache/airflow/tree/593621176989b8506e220ef311d55b063b6688c7/dev/breeze/doc#airflow-breeze-ci-environment) developer environment offered by Apache Airflow. Manually configuring a parallel setup capable of running integration tests will quickly lose parity with their CI and waste users' time when change validation results differ from what they work on locally. [quote="Paul Moore, post:28, topic:107186, username:pf_moore"] If we have a way for projects to put workflow setup into `pyproject.toml`, I imagine they will *stop* documenting the recommended workflow in the contributor docs. [/quote] I understand where you're coming from as I think the opening post only showed example output and there hasn't been much about how to invoke the CLI. However, even before my final JSON API idea it was never my intention to have the standardized commands match the day-to-day workflow of a user nor did I expect that tools would drop their custom UX in favor of such generalization. To me, the purpose of environment standardization has always been purely for tooling interoperability so that everything interacting with a project behaves the same way by default while users passively gain a better experience. Sorry if I didn't make that clear before! [quote="Paul Moore, post:26, topic:107186, username:pf_moore"] I don't use Poetry, and don't want it globally installed. So when *I* use Poetry, I use it via `uvx poetry`. Do I need to override the project config just because I choose to *invoke* Poetry via a different mechanism than the project developers do? [/quote] This is a good call out. I haven't thought too much about this but I think it would fit nicely in the user customization scheme to have a way to define arguments that prefix the workflow tool command. Perhaps we could even support environment variables (order of precedence TBD). [quote="Paul Moore, post:35, topic:107186, username:pf_moore"] [quote="brettcannon, post:33, topic:107186"] Or something else I'm not thinking of? [/quote] For me, it's that I want to choose where my venvs go. [/quote] As Steve said a few times, nothing would change for folks who wish to avoid certain tool behavior but rather it would ease the default experience for most. One's preferred workflow could still be used since nothing mandates specific user interaction. In the worst case scenario, an editor calls the configured tool when an action is performed and an environment is unexpectedly created elsewhere. Users with such preferences would either read how to configure the tool or override/disable it as the environment manager. I consider transient yet avoidable friction for power users worth the price of improving the default experience for most users and reducing the maintenance burden of those in other ecosystems who must consume our projects. ------------------------- pf_moore | 2026-05-17 20:59:28 UTC | #38 [quote="Ofek Lev, post:37, topic:107186, username:ofek"] This is a good call out. I haven’t thought too much about this but I think it would fit nicely in the user customization scheme to have a way to define arguments that prefix the workflow tool command. [/quote] There's a lot to digest in your post and I need to take some time to consider it, but I think this is something that I really struggle with. As a community, we've started to take a view that "standards shouldn't dictate UI", and while I agree broadly with that, I think that we make life unnecessarily difficult if we try to debate standards in the abstract, *without* considering how they will look to the user. In this case, I'd like to properly understand how this proposal would affect me. At the moment, the abstractions mean I can't work that out for myself. My workflow is likely unusual (I have issues with nearly every "opinionated" tool out there :slightly_smiling_face:) so I don't expect to get away with doing *nothing*, but conversely right now I deliberately do almost nothing to configure my tools, and I get a "good enough" (for me) experience. For day to day use, I use "raw" pip and either virtualenv or venv to manage environments. Both pip and virtualenv are installed centrally as the zipapp versions - I try to avoid installing pip in my virtual environments as much as possible. For the "main" virtual environment in my projects, I call it `.venv`. For other environments, I either use adhoc names or I use a tool like nox (generally run via `uvx nox`) to manage them. I *never* activate environments^[Exception: VS Code activates an environment for me, but I tend to ignore that when working at the command line], instead I invoke commands from the venv using the full (relative) path - `.\.venv\Scripts\python`. So there's no "workflow tool" that I use, and therefore there's no obvious tool that I could configure as a "workflow server". In addition, I occasionally use tools like hatch, PDM, or uv. This is partly because I'm a dabbler and keep trying out new approaches, but sometimes it's because I need to keep up with how the various tools work (those projects tend to be "toy" ones). So in those cases I need to use a tool that *isn't* my "normal workflow". Again, I tend to use these tools via `uvx ` because I'm experimenting, and don't want to install yet another tool globally. The "project dictates a workflow" case is similar to what's described in the previous paragraph, but made worse by the fact that it's not always clear where to draw the line - if the project says "install Poetry globally" and I use `uvx`, I'm already not following the instructions. So is it really so bad if I run manual tests in my own venv rather than following the project documentation? I know that I won't ask the project for help if I do so - it's my choice and they have no obligation to support me - so why does it matter to anyone but me? In terms of *consumers*, I use both VS Code and Vim as editors. Right now, I do barely any configuration, and things just work for me (presumably because "raw venv" is special cased enough that I don't *need* anything special). I don't have any feel for what consumers *other* than IDEs and editors this proposal is intended to target. Given the above, what would I expect to change in my workflow under this proposal? Presumably I'd need to specify my "workflow server" in various config files? Would I continue to customise any tools I use (for example, telling hatch not to store environments externally) in the normal way, or would I need to add "workflow server" config as well? And what would I gain? Would VS Code "magically" know what environment to use (like it does now with .venv), or would it ask me to pick from a list? What are the create/remove operations targetted at? My editors don't create or delete environments for me at the moment. Would they gain new options to do so? Would I be required to use them rather than manually creating environments? It's possible I simply have a very Luddite attitude to managing my environments and workflow. I certainly don't have any objection to making it easier for people to write tools that manage everything for their users. But if that comes with the cost of no longer supporting users who *like* the ability to control everything for themselves^[Or even just making life harder for such users, because they now have to fight the expectations built into their tools. It's a bit like AI skeptics having to switch off AI features everywhere - the existence of those features doesn't harm them *directly*, but it does add friction for them from having to continually "opt out".], then I think we should clearly acknowledge that we're choosing to do so. ------------------------- ofek | 2026-05-18 03:26:20 UTC | #39 [quote="Paul Moore, post:38, topic:107186, username:pf_moore"] In this case, I'd like to properly understand how this proposal would affect me. [/quote] I think a concrete example of a non-default user scenario is a great idea! However, before answering I'd like to hone in on a sentiment that is causing friction in this thread. [quote="Paul Moore, post:38, topic:107186, username:pf_moore"] So is it really so bad if I run manual tests in my own venv rather than following the project documentation? I know that I won't ask the project for help if I do so - it's my choice and they have no obligation to support me - so why does it matter to anyone but me? [/quote] Ignoring a project's recommended development workflow isn't bad at all if, as you said, there's no impact for others. The central issue I see in this thread is that some are perceiving projects gaining the ability to influence the default behavior of all supported tooling as an encroachment on user choice. I'm honestly not sure where this perception is coming from as none in favor have expressed such intentions and the topic started off by making override mechanisms a hard requirement. [quote="Paul Moore, post:38, topic:107186, username:pf_moore"] \[...\] So there's no "workflow tool" that I use, and therefore there's no obvious tool that I could configure as a "workflow server". \[...\] Given the above, what would I expect to change in my workflow under this proposal? [/quote] I think the impact on your workflow would depend on whether you take advantage of what the proposal offers. ### Option 1: Do nothing This looks exactly like what you do today. Although the actions you take wouldn't change, things responding to your behavior might. For example, if a project configures a tool then an editor would provide functionality like syntax highlighting and static analysis from an environment in a location that may not reside in a `.venv` directory in the project root as you expect. To remedy this you could either read the docs of the project-chosen tool to configure it to your liking, disable usage of the tool via configuration that hasn't yet been discussed (probably an environment variable), or go with option 2. Optional functionality would be exposed to you in various ways by default. Some examples: * A future PEP would specify a `tasks` table that refers to named sequences of commands which are to be executed in the context of one or more environments. Each of these would be available as an action you can trigger directly from your editor like `test-e2e`, `update-deps`, `start-docs-server`, etc. * Consumers like editor extensions or a new blessed `build`-style tool could offer a way to free up resources (disk space, Docker containers, cloud VMs) by removing all environments tied to a project. If at some point you want to try a project's configured tool named `foo` in your terminal that's managed in a unique way, you need only satisfy its `foo`-shaped requirement. For example, to avoid a global installation via `uvx`/`pipx` merely add an equivalently named script on your `PATH` that forwards all arguments to `uvx foo`. This is a common enough scenario though so we should add a way to easily configure this. ### Option 2: Create your own tool Here is where the proposal shines particularly bright! You can create your own workflow tool and override the project default to provide a UX that behaves exactly as you wish without having to configure several different things per project. You can write a small `moore_env` script which you put on PATH that would implement the few required API methods defined by the spec. You can enforce that the first `python` on PATH (or any other of your choosing) will be used to invoke the zipapp versions of `virtualenv` and `pip` (located wherever you decide) for creating virtual environments and subsequent package installation, respectively. The location of a project's default environment would be a `.venv` directory at the root and you can provide additional heuristics for the detection of other environments as you see fit. If there are some projects that you find incompatible with your preferences, then you can hardcode a list of them and have any API method that targets one invoke the actual tool that the project configured. Basically, you can do whatever you want :slight_smile: [quote="Paul Moore, post:38, topic:107186, username:pf_moore"] Would I continue to customise any tools I use (for example, telling hatch not to store environments externally) in the normal way, or would I need to add "workflow server" config as well? [/quote] Yes, users would still configure the tools themselves to modify behavior with very few exceptions. [quote="Paul Moore, post:38, topic:107186, username:pf_moore"] Would VS Code "magically" know what environment to use (like it does now with .venv), or would it ask me to pick from a list? [/quote] The tool would determine which environment acts as the default. In Hatch, it's the `default` environment although if it defines a matrix then we'd have to select one of the generated environments as the default using sane heuristics like the Python version and lexicographical order of their names. I'd imagine that some tools would prefer explicit selection by users at least under certain scenarios and would return an error response. [quote="Paul Moore, post:38, topic:107186, username:pf_moore"] What are the create/remove operations targetted at? My editors don't create or delete environments for me at the moment. Would they gain new options to do so? [/quote] Removal perhaps but definitely creation. The user would no longer have to be the one to create the `.venv` in order for tools to know what to do. [quote="Paul Moore, post:38, topic:107186, username:pf_moore"] Would I be required to use them rather than manually creating environments? [/quote] Nope! ------------------------- pf_moore | 2026-05-18 10:10:31 UTC | #40 [quote="Ofek Lev, post:39, topic:107186, username:ofek"] The central issue I see in this thread is that some are perceiving projects gaining the ability to influence the default behavior of all supported tooling as an encroachment on user choice. I’m honestly not sure where this perception is coming from as none in favor have expressed such intentions and the topic started off by making override mechanisms a hard requirement. [/quote] I think what people see is that having to override changes the dynamic from project preferences being "opt in" (the project asks that users follow their recommended practice) to being "opt out" (the user has to add override configuration). That feels like a change in which group is being prioritised. Your comment "Although the actions you take wouldn’t change, things responding to your behavior might" under option 1 reflects this - I'll now have to "opt out" of project-defined defaults if they don't match my preferred workflow. [quote="Ofek Lev, post:39, topic:107186, username:ofek"] For example, to avoid a global installation via `uvx`/`pipx` merely add an equivalently named script on your `PATH` that forwards all arguments to `uvx foo`. This is a common enough scenario though so we should add a way to easily configure this. [/quote] +1 on making this configurable - expecting a specific command to exist on `PATH` is (IMO) a stumbling block for many CLI-based interfaces. And don't forget Windows, where writing a "custom script" and putting it on `PATH` isn't easy - the OS doesn't treat scripts as first-class objects (`CreateProcess` doesn't recognise Python scripts, for example). The only valid type of script to the OS is a `.cmd` file, and those have problematic semantics (e.g., calling a `.cmd` file from within another `.cmd` file doesn't nest, you have to use the `CALL` statement, which breaks the abstraction that this is "just another command"). [quote="Ofek Lev, post:39, topic:107186, username:ofek"] Here is where the proposal shines particularly bright! You can create your own workflow tool and override the project default to provide a UX that behaves exactly as you wish without having to configure several different things per project. [/quote] This needs to be emphasised a lot more IMO. People think of "write your own workflow tool" as a complicated process, far more inaccessible than "configure the system to match your workflow". I certainly never even considered writing my own tool as an option here. I can come up with a bunch of objections, none of which are showstoppers, but they just reflect that "writing a tool" feels like a non-trivial barrier to entry. To give an example, again on Windows, where would I put such a tool? There's no standard `PATH` location for "user-developed utilities" - should I put it in `C:\Windows\System32`? Surely not. If I put it in my "general stuff" folder, it's likely to get accidentally deleted in a periodic cleanup. And again, there's the whole "making an exe is hard, and `.cmd` files are clunky" problem. To be clear, I love this idea in principle. I'm just concerned that in practice, most people wouldn't think of it, or find it particularly accessible^[If I, as an experienced developer with familiarity with both Windows and Python, struggle to think of a clean way of implementing this, what hope do the sort of new Python users I worked with in my previous job have?]. So if it's to be the solution for non-standard workflows, we need to think about how to promote it (*or* we need to reduce the barrier for building and deploying standalone utilities in Python, which is a much bigger issue that I don't think we can tackle here...) ------------------------- steve.dower | 2026-05-18 13:11:54 UTC | #41 I'm only responding to a few points to help keep discussion focused - not to ignore the rest of the posts, just to avoid repetition or contributing where I don't have anything of value above what's already posted. [quote="Ofek Lev, post:37, topic:107186, username:ofek"] Communication: I think workflow tools should behave like language servers. A consumer like an IDE would start the process (`hatch env server`, `uv …`, etc.) and send API requests to its stdin while receiving responses on stdout. This is much cleaner and more extensible than mandating a subcommand structure with certain flags for each command. [/quote] My suggestion to Brett was to follow PEP 517, but I'm inclined to think this is the better interface in these days of "everything is Rust". Still, wouldn't be hard to make a Python API in a single script that translates to this - I don't think we're too worried about the "zero Python runtimes on the machine" case, but if we are, adopting an _existing_ stdin/out-based protocol would make sense. As long as the interface doesn't involve dealing with shell quoting/etc. Requiring a Python interface also deals with a lot of the "search PATH" and cross-platform issues for more casually installed tools, such as using a setup script that's in the repository itself. So I still favour that approach and figure we'll eventually deal with people who want to use Python without getting Python with relocatable builds. [quote="Ofek Lev, post:37, topic:107186, username:ofek"] The initial operations would be ... [/quote] Agree with this list. I think the `exec` command needs some way to return a `Popen`-equivalent object so the caller can pipe stdin/out and wait for completion, but that's the biggest complexity here. We can't really rely on piping it all back via the interface, so it probably requires more pass-forward arguments to connect up streams. What we can't rely on is having the interface return the full command line and let the caller launch it, unfortunately. Too many environments require setup that can't/shouldn't just be faked with environment variables. So the direction you've proposed is the right one IMHO. [quote="Ofek Lev, post:37, topic:107186, username:ofek"] I’m okay with a `.venv` directory alongside a `pyproject.toml` file acting as the default IFF it’s a non-empty directory and there is no workflow tool defined anywhere. [/quote] My suggestion was that "no workflow tool defined" implies a _specific_ default workflow tool (to be implemented) that uses `venv` to create an environment at `.venv` and can launch it. None of which needs to be overspecified here, as long as people are happy to say "tools that normally use the project's defined workflow tool may use this one if the tool doesn't say so, rather than throwing up your hands and making your user figure it out on their own" (and since this would only be in response to the user clicking a "set it up for me button", I'm quite okay with this as a default). [quote="Ofek Lev, post:37, topic:107186, username:ofek"] it would fit nicely in the user customization scheme to have a way to define arguments that prefix the workflow tool command [/quote] This is one of the things I think we don't need to _define_, we just need to make sure we don't _forbid_ it. UI tools can offer whatever customisation they like, up to and including seeing the name of a defined workflow tool and using something else entirely (if they're willing to deal with upset/confused users, but I can think of valid cases for this, mostly involving constrained enterprise environments). ------------------------- pf_moore | 2026-05-18 13:18:05 UTC | #42 [quote="Steve Dower, post:41, topic:107186, username:steve.dower"] Requiring a Python interface also deals with a lot of the “search PATH” and cross-platform issues for more casually installed tools, such as using a setup script that’s in the repository itself. [/quote] The biggest problem with a Python API is deciding *which* environment the API should be installed into. There's no obvious one, not least because this is an API for *discovering* Python environments. I guess we could require every tool consuming this API to host (or create and manage) a Python environment of its own to hold the API, but you'd then need a mechanism for the user to install code into that environment. I'd like to have a solution to this (because it would be useful in many more contexts than just this one) but I fear it's not as simple to solve as we'd like. ------------------------- steve.dower | 2026-05-18 14:06:12 UTC | #43 [quote="Paul Moore, post:42, topic:107186, username:pf_moore"] The biggest problem with a Python API is deciding *which* environment the API should be installed into. [/quote] Again, I don't think we have to tell VS Code (for example) what to do here. They can create a private venv using the standard library and put the tool in there, and then provide it with the CWD of the user's project and it'll never know the difference. Or they can use pipx, or uvx. Or they can make the user choose a location, or they can try and use the system one (and deal with complaints when it fails). The key point is, we can't choose better now than they can choose when they actually implement it. So we only get ourselves into trouble if we try to design their product for them. Let's just not do that. ------------------------- pf_moore | 2026-05-18 15:27:58 UTC | #44 [quote="Steve Dower, post:43, topic:107186, username:steve.dower"] Again, I don’t think we have to tell VS Code (for example) what to do here. [/quote] Normally I'd agree with you. But if we assume @ofek is right, and "write your own wrapper that encapsulates your personal preferred workflow" is to be seen as normal behaviour, it's worth *thinking* about implementation choices - not as things that need to be standardised, but as context for how well the proposal will work in practice (and food for the "How to teach this" section). [quote="Steve Dower, post:43, topic:107186, username:steve.dower"] So we only get ourselves into trouble if we try to design their product for them. [/quote] But if we define a standard that we can't come up with a reasonable implementation strategy for, then we're just dumping a problem we don't have a solution for onto them. I don't think it's fair to demand that every proposal comes with a working implementation, but on the other hand, I don't think it's fair to make a proposal that you don't have *any* decent implementation for. We don't design their product for them, but we should give them a starting point. ------------------------- steve.dower | 2026-05-18 15:47:14 UTC | #45 [quote="Paul Moore, post:44, topic:107186, username:pf_moore"] We don’t design their product for them, but we should give them a starting point. [/quote] Was there something wrong with all of the five starting points I offered? If we choose one, then we're effectively restricting them, and the one we'd choose for VS Code is not the same one we'd choose for GitHub Actions (for example), so how deep do we have to go vs. saying "the name maps to a package name on PyPI, how it gets installed is up to you"? ------------------------- pf_moore | 2026-05-18 17:22:50 UTC | #47 [quote="Steve Dower, post:45, topic:107186, username:steve.dower"] saying “the name maps to a package name on PyPI, how it gets installed is up to you”? [/quote] Ah, this is where we are misunderstanding each other. I'm trying to understand how the suggestion that @ofek made for my workflow would work. In that situation, the idea is that I'd create my own personal wrapper that implements the discovery API in a way that matches my workflow. Having to publish that wrapper to PyPI is a bigger barrier than I think is reasonable for what is basically just "customising things to support a workflow that isn't just how one of the big tools does things". ------------------------- steve.dower | 2026-05-18 18:55:29 UTC | #48 [quote="Paul Moore, post:47, topic:107186, username:pf_moore"] [quote="steve.dower, post:45, topic:107186"] saying “the name maps to a package name on PyPI, how it gets installed is up to you”? [/quote] Ah, this is where we are misunderstanding each other. [/quote] Sorry, I didn't spell out the second option of "the name maps to a location in the current \*wave hands\*workspace/repository/whatever", which lets you implement the API directly in your workspace (per-project), and I also didn't spell out the "tools can totally ignore the package on PyPI _if the user tells you to do something else_" because I've mentioned that in basically every other post. But that is how you would implement your own personal environment manager for all projects regardless of their declared metadata - you literally just ignore the metadata. There's still a benefit, in that tools that know how to use the new API can probably use any tool providing that API, whereas today they have to implement support for each tool individually and so will _never_ support your custom one. But we can't force third-parties to implement the feature to let their users override metadata in the projects they use, so you won't see it spelled out in a spec. The best we can do is say that it's only as binding on the end user as anything else in Python is, which is "not very". Tools should be respecting that Python idiom rather than strictly enforcing standards *against* their users. It's unfortunate that we've put out a number of very strict sounding PEPs recently that don't seem to allow that leeway, because it trains tools to ignore users in favour of the "official" metadata[^1], but we aren't going to fix that by spelling out every possible preference of every user to make them all official. We need to just encourage the tools we can influence to let their users be the boss and do whatever they need to make things work. [^1]: One of my major reasons for disliking static typing. ------------------------- pf_moore | 2026-05-18 19:05:48 UTC | #49 [quote="Steve Dower, post:48, topic:107186, username:steve.dower"] I didn’t spell out the second option of “the name maps to a location in the current *wave hands*workspace/repository/whatever”, which lets you implement the API directly in your workspace (per-project) [/quote] I still don't see how that lets me configure tools to use my workflow in *all* my projects (unless I specifically choose a different config). I don't want a workflow API script in every project, just one somewhere on my machine to affect everything where I don't override it. (Well, what I actually want is for my personal workflow to be supported without any config, but that's not realistic once we go beyond the `.venv` convention for identifying the default virtual environment, i.e., PEP 832.) ------------------------- steve.dower | 2026-05-18 19:12:27 UTC | #50 [quote="Paul Moore, post:49, topic:107186, username:pf_moore"] I still don’t see how that lets me configure tools to use my workflow in *all* my projects (unless I specifically choose a different config). I don’t want a workflow API script in every project, just one somewhere on my machine to affect everything where I don’t override it. [/quote] Yes, you want a default setting in your UI tool (IDE or whatever). See every part of my post other than what you quoted, since that's the bit where I talked about this. [quote="Paul Moore, post:49, topic:107186, username:pf_moore"] Well, what I actually want is for my personal workflow to be supported without any config [/quote] Right, I get that it's a personal/selfish request for all tools to default to what you do, but that's why the rest of us are saying that we're not interested in defining a spec solely for that single workflow. We'd rather define a generic interface that enables interesting tools to be built on either side of it. Saying "if `.venv` exists you can assume it's a virtual environment" barely qualifies as interesting for tool building, since any tool that will do that can already go "if `.venv/pyvenv.cfg` exists then I assume it's a virtual environment" and you won't notice the difference. It's just not worth defining, and probably the only reason Brett even wrote it up is because people wanted to argue over the `.venv` bit. ------------------------- ofek | 2026-05-20 00:31:54 UTC | #51 [quote="Steve Dower, post:41, topic:107186, username:steve.dower"] I don't think we're too worried about the "zero Python runtimes on the machine" case \[...\] Requiring a Python interface also deals with a lot of the "search PATH" and cross-platform issues for more casually installed tools, such as using a setup script that's in the repository itself. [/quote] I think we should be quite worried about requiring Python for environment management and other project workflows as that introduces a bootstrapping issue. In the early days of Hatch I [learned](https://hatch.pypa.io/latest/blog/2023/12/11/hatch-v180/#installation-made-easy) how critical of an issue that is from watching users. [quote="Steve Dower, post:41, topic:107186, username:steve.dower"] [quote="ofek, post:37, topic:107186"] I'm okay with a `.venv` directory alongside a `pyproject.toml` file acting as the default IFF it's a non-empty directory and there is no workflow tool defined anywhere. [/quote] My suggestion was that "no workflow tool defined" implies a *specific* default workflow tool (to be implemented) that uses `venv` to create an environment at `.venv` and can launch it. [/quote] Ah, I see! I'd be in favor of that as long as we keep it minimal like `build`. [quote="Steve Dower, post:41, topic:107186, username:steve.dower"] [quote="ofek, post:37, topic:107186"] it would fit nicely in the user customization scheme to have a way to define arguments that prefix the workflow tool command [/quote] This is one of the things I think we don't need to *define*, we just need to make sure we don't *forbid* it. UI tools can offer whatever customisation they like, up to and including seeing the name of a defined workflow tool and using something else entirely (if they're willing to deal with upset/confused users, but I can think of valid cases for this, mostly involving constrained enterprise environments). [/quote] The `uvx` example Paul provided is quite common which made me consider special casing that but I find this view compelling. I change my mind :slight_smile: [quote="Paul Moore, post:49, topic:107186, username:pf_moore"] [quote="steve.dower, post:48, topic:107186"] I didn't spell out the second option of "the name maps to a location in the current *wave hands*workspace/repository/whatever", which lets you implement the API directly in your workspace (per-project) [/quote] I still don't see how that lets me configure tools to use my workflow in *all* my projects (unless I specifically choose a different config). I don't want a workflow API script in every project, just one somewhere on my machine to affect everything where I don't override it. [/quote] Yes, although we should offer project-local override configuration it shouldn't be required for configuring a more broad override nor for the actual implementation of workflows. [quote="Paul Moore, post:47, topic:107186, username:pf_moore"] In that situation, the idea is that I'd create my own personal wrapper that implements the discovery API in a way that matches my workflow. Having to publish that wrapper to PyPI is a bigger barrier than I think is reasonable for what is basically just "customising things to support a workflow that isn't just how one of the big tools does things". [/quote] It's not just a barrier for user customization but also for workflow tools that intend to be published. For example, it doesn't make sense for maintainers of a tool written in an another language like uv to create a Python package just to manage itself and accept the compatibility hardship that naturally comes from such dual versioning. ------------------------- steve.dower | 2026-05-20 13:53:58 UTC | #52 [quote="Ofek Lev, post:51, topic:107186, username:ofek"] I think we should be quite worried about requiring Python for environment management and other project workflows as that introduces a bootstrapping issue. [/quote] Like many of the other concerns, I mean we're pushing it to the front-end to solve, rather than expecting the backends to solve it. So if VS Code is going to use this API, _it_ is responsible for solving bootstrapping to get enough Python to use the interface. Regular humans don't use the API themselves, so they are never expected to become the bootstrappers. And anyone who bypasses/ignores the API is in exactly the same position as they are today. So while it does put the work _somewhere_, it's clear about who is responsible, and the burden is in the right place for user customization/overrides (e.g. VS Code already knows whether the user has a default/preferred/system/bundled Python runtime, while an API wrapper doesn't have to figure that out). [quote="Ofek Lev, post:51, topic:107186, username:ofek"] I’d be in favor of that as long as we keep it minimal like `build`. [/quote] That's the wrong analogy - `build` is the front-end, and I'm talking about a default backend. So it's "minimal like `flit-core`". [quote="Ofek Lev, post:51, topic:107186, username:ofek"] It’s not just a barrier for user customization but also for workflow tools that intend to be published. For example, it doesn’t make sense for maintainers of a tool written in an another language like uv to create a Python package just to manage itself and accept the compatibility hardship that naturally comes from such dual versioning. [/quote] I think it totally makes sense to put the "burden" of publishing and versioning on the tool that is being published and versioned. I'm open to other discoverability mechanisms besides "must be on PyPI", but I don't see why we should increase the complexity/decrease the reliability for everyone else for the sake of the handful of tools that would form the backends here. One reasonable example - `uv` could say "when a workflow references `uv-api-wrapper` and you know that `uv` is already installed, you can find a local copy of the wrapper at \". And I've previously mentioned specifying a relative path to the config file for in-tree tools. The _actual_ interpretation of the name can vary in ways that are more convenient, provided the _default_ interpretation (i.e. `pip install `) also works. Besides, I expect most tools are just going to `subprocess.run(...)` themselves out of the wrapper anyway (possibly with a download step right before it). If your command line is changing frequently enough to become a versioning issue, your users are going to quit using you. ------------------------- ofek | 2026-05-20 13:59:59 UTC | #53 Could you please concretely describe what you think the Python API approach would look like here with an example? ------------------------- steve.dower | 2026-05-20 14:19:04 UTC | #54 ``` # pyproject.toml, presumably [workflow] requires=['my-wrapper-module'] api='my_wrapper_module:WorkflowAPI' ``` ``` # my_wrapper_module.py class WorkflowAPI: def __init__(self, cwd): self.cwd = cwd def create_environments(self): # our tool defaults to our config file in cwd subprocess.check_output(["tool", "create"], self.cwd) def list_environments(self): # request a list of envs our tool knows about return json.loads(subprocess.check_output(["tool", "list", "--json"], self.cwd)) def run_command(self, env_id, args): # Run our tool to calculate what an "activated" environ looks like, # because that's all our tool needs. Other tools might do other things, # or they might have a suitable 'tool run ...' command reqd_env = json.loads(run_and_capture(["tool", "calculate-env", env_id], self.cwd)) return subprocess.Popen(args, cwd=self.cwd, env=reqd_env) ``` ------------------------- ofek | 2026-05-20 17:58:27 UTC | #55 Ah okay, thanks for clarifying! I had imagined based on a previous comment of yours that you were now in favor of the language server-like approach and just wanted the Python API for a more deterministic way of acquiring the tool. Now I understand that you are still advocating for a PEP 517-like approach. [quote="Steve Dower, post:52, topic:107186, username:steve.dower"] [quote="ofek, post:51, topic:107186"] I think we should be quite worried about requiring Python for environment management and other project workflows as that introduces a bootstrapping issue. [/quote] Like many of the other concerns, I mean we're pushing it to the front-end to solve, rather than expecting the backends to solve it. So if VS Code is going to use this API, *it* is responsible for solving bootstrapping to get enough Python to use the interface. Regular humans don't use the API themselves, so they are never expected to become the bootstrappers. [/quote] I'm overall against a spec that requires an existing Python distribution in order to install and run something that's meant to manage Python environments. However, you are right that a human wouldn't use the API themselves and therefore I am slightly less hesitant. I do still think that the user experience may be degraded in at least the following ways: * I assumed that AI agents would be a consumer and, after some time, models will know how to interact with projects out-of-the-box without repo-specific instructions for managing environments, running configured tasks like `test`, etc. Requiring Python distribution and environment management for using the workflow tool adds either extra context bloat and processing time, or maintenance effort for the agents to have a built-in deterministic fast path. Or, folks would come up with various skills of varying quality that users would be subjected to. It's likely that the model would be intelligent enough to know that this approach wouldn't be optimal and they would scan the repo contents for the commands that humans are running, which invariably would use what's on PATH anyway. * Although VS Code has significant maintenance resources, other consumers are unlikely to be as fortunate and managing Python distributions imposes a cost regardless of how great one perceives the cost to be. This has the potential to reduce adoption of the functionality and as a result limits the benefit to users. * Not a user issue but I imagine that Debian, Conda, etc. would not like having to go through supporting another standard that may cause bootstrapping issues. Although it wouldn't be as bad as recursive build dependencies, I could see trouble arising when they start using the workflow tool for testing projects and there are cyclic dependencies for that post-build validation step. [quote="Steve Dower, post:12, topic:107186, username:steve.dower"] [quote="brettcannon, post:11, topic:107186"] [quote="ofek, post:10, topic:107186"] I know we technically did it for the build system but I worry that the complexity of standardizing the behavior of `requires` may impact acceptance of this proposal. [/quote] I'm up for taking it out if people prefer. [/quote] In the same vein, specifying *the intent* that "these things are required" is totally fine by me. Specifying that tools *must* figure out how to install those things is going too far. Let tools choose to honour the intent however they see fit. [/quote] If you still have this opinion, what do you think about the following? > If the `requires` field is defined then consumers SHOULD add a directory to the front of PATH for all invocations of the workflow tool. The directory MUST contain only the contents of what a satisfactory dependency resolution would install to the `scripts` [directory](https://docs.python.org/3/library/sysconfig.html#installation-paths) of a virtual environment. Here we both compromise: * You no longer have a Python API but maintain the ability to dictate versioning requirements and reduce nondeterministic tool execution. * I no longer have a spec that's completely free of a dependence on Python distributions but maintain the CLI-based API and support for users like Paul. ------------------------- steve.dower | 2026-05-20 19:27:25 UTC | #56 [quote="Ofek Lev, post:55, topic:107186, username:ofek"] If you still have this opinion, what do you think about the following? > If the `requires` field is defined then consumers SHOULD add a directory to the front of PATH for all invocations of the workflow tool. The directory MUST contain only the contents of what a satisfactory dependency resolution would install to the `scripts` [directory](https://docs.python.org/3/library/sysconfig.html#installation-paths) of a virtual environment. [/quote] I really dislike any assumption about file-system layout here (or assuming that everyone uses/looks like `venv`). What you've proposed is about the only thing that could work, but the only way to figure it out reliably is to run the Python runtime, at which point all the other concerns also go away and it may as well be a small Python shim that knows how to set things up for the particular workflow tool being used. And yeah, I forgot about the LSP-like proposal. My earlier comments on that still stand - it's nice, but could easily be _another_ shim on top of the Python API that provides the LSP translation (essentially a remote procedure call layer) in the exact format desired by the tool using it, just as a Python API could be the layer that wraps up the LSP model so that Python hosts can use it more conveniently. My preference is for the Python API because it's the most convenient for us to define and specify and know that it allows implementers the greatest level of flexibility for cross-plat/unknown-plat/etc. support. (And I also still believe that relocatable builds will happen, which reduces the eventual bloat of a bundled runtime to one of the _smallest_ language runtimes currently in use today - it's about 10MB on Windows.) ------------------------- bwoodsend | 2026-05-20 21:17:55 UTC | #57 [quote="Ofek Lev, post:55, topic:107186, username:ofek"] Not a user issue but I imagine that Debian, Conda, etc. would not like having to go through supporting another standard that may cause bootstrapping issues. Although it wouldn’t be as bad as recursive build dependencies, I could see trouble arising when they start using the workflow tool for testing projects and there are cyclic dependencies for that post-build validation step. [/quote] I don't think this will ever be useable within the repackager's world. Repackagers generally will always have to do the same unpacking/reverse engineering down to the most low level commands that the minimalists like me end up doing. Any tool that needs the internet can't be used. Build isolation blocks distribution-required patches to build backends. Using venvs for testing defeats the purpose of system level testing in the first place since you're not actually testing with the dependencies you're deploying with.^[Although sometimes Fedora do it anyway -- I personally thing they're nuts...] Anything that tries to detect or install Python installations is self defeated in the same way. Lockfiles or pinned dependencies are unusable in an ecosystem where only one version of each package is available. Any test command+dependencies list that uses linters (`pytest-flake8`) or formatters (`pytest-black`), coverage, profiles, benchmarks, heavy fuzz testing or multiple environments needs to be replaced anyway. So tl;dr, I wouldn't worry about it. ------------------------- ofek | 2026-05-20 22:22:40 UTC | #58 [quote="Steve Dower, post:56, topic:107186, username:steve.dower"] [quote="ofek, post:55, topic:107186"] If the `requires` field is defined then consumers SHOULD add a directory to the front of PATH for all invocations of the workflow tool. The directory MUST contain only the contents of what a satisfactory dependency resolution would install to the `scripts` [directory](https://docs.python.org/3/library/sysconfig.html#installation-paths) of a virtual environment. [/quote] I really dislike any assumption about file-system layout here (or assuming that everyone uses/looks like `venv`). [/quote] That's fair, I didn't adequately distill the essence of the requirement. How about this? > If the `requires` field is defined then consumers SHOULD add a directory to the front of PATH for all invocations of the workflow tool. The directory MUST contain all files contained within the `scripts` subdirectory of the `.data` [directory](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-data-directory) of all wheels that are selected from a satisfactory dependency resolution. --- [quote="Steve Dower, post:56, topic:107186, username:steve.dower"] My preference is for the Python API because it's the most convenient for us to define and specify [/quote] I care little about what is expedient for us and a lot about what is best for users. The LSP-like approach provides a much snappier experience for all users, allows users like Paul to more easily tailor their workflow to meet their needs, and potentially gives users more options in the long term due to the reduced requirements of implementation. I think that approach provides a better experience until I hear compelling evidence to the contrary. [quote="Steve Dower, post:56, topic:107186, username:steve.dower"] it allows implementers the greatest level of flexibility for cross-plat/unknown-plat/etc. support [/quote] Surely that can't be true since the number of supported Python platforms is necessarily less than a CLI written in something like Rust? ------------------------- fungi | 2026-05-20 22:37:45 UTC | #59 > Surely that can't be true since the number of supported Python platforms is necessarily less than a CLI written in something like Rust? Did that change recently? Last I recall, there were a lot of platforms with a working C toolchain capable of compiling CPython but no Rust compiler yet. I guess it all depends on what you mean by "supported" and "platform." ------------------------- steve.dower | 2026-05-20 22:58:48 UTC | #60 [quote="Ofek Lev, post:58, topic:107186, username:ofek"] [quote="steve.dower, post:56, topic:107186"] it allows implementers the greatest level of flexibility for cross-plat/unknown-plat/etc. support [/quote] Surely that can’t be true since the number of supported Python platforms is necessarily less than a CLI written in something like Rust? [/quote] I was thinking mostly in terms of not having to get path/shell quoting or environment rules correct at the "frontend" level, delegating that to the "backend", which then only has to worry about its own rules (if the frontend was doing it, it has to worry about the rules of any arbitrary backend). [quote="Ofek Lev, post:58, topic:107186, username:ofek"] allows users like Paul to more easily tailor their workflow to meet their needs [/quote] You're going to have to spell this one out for me, perhaps with an example. I'm really not clear how "implement LSP" is easier for Paul than "implement a class with 3-4 functions" (aka my example). ------------------------- pf_moore | 2026-05-20 23:24:10 UTC | #61 [quote="Steve Dower, post:60, topic:107186, username:steve.dower"] I’m really not clear how “implement LSP” is easier for Paul than “implement a class with 3-4 functions” (aka my example). [/quote] I think the point is that I can write the LSP however I want (python, rust, whatever) and just supply the command line needed to run it. But if I write a Python class, I need to make that loadable in a Python interpreter supplied by the tool - and every tool might need me to do that differently, so there could be multiple copies of my Python file scattered around my machine. ------------------------- brettcannon | 2026-05-21 04:51:37 UTC | #62 First off, thanks to all the thoughtful posts! [quote="Ofek Lev, post:37, topic:107186, username:ofek"] I think we should adopt the strategy of most AI agents and increase the priority of machine configuration for easier managed installations. I’d go with: `pyproject.local.toml` > per-machine > `pyproject.toml` > per-user [/quote] I'm not ready to get into config order, if because I know Steve has argued to leave that up to the front-end tools instead of standardizing. [quote="Ofek Lev, post:37, topic:107186, username:ofek"] I think workflow tools should behave like language servers. A consumer like an IDE would start the process (`hatch env server`, `uv …`, etc.) and send API requests to its stdin while receiving responses on stdout. [/quote] So JSON-RPC with some defined function endpoints where you are specifying how to launch the server. It's an interesting idea! [quote="Ofek Lev, post:37, topic:107186, username:ofek"] We should also make some sort of fast exit path option for times when you want to run only a single operation without process management which is useful for AI agents in change+validation loops, redistributors like Conda or Debian who want to properly run a package’s test suite, etc. [/quote] I'm not sure if this needs to be part of any spec. If you only want to run one command then run one command and exit. If you're talking about surfacing the functionality as individual CLI commands, a CLI tool could provide that and act as a unified CLI API across all tools. That gets you both your scripting and AI support. [quote="Ofek Lev, post:37, topic:107186, username:ofek"] I think one server process should be capable of managing environments for an arbitrary number of projects so every operation would accept an optional `cwd` string field referring to an absolute path that would default to the current working directory. [/quote] That seems reasonable as tools can perform whatever search they want for a `pyproject.toml` file. [quote="Ofek Lev, post:37, topic:107186, username:ofek"] I prefer future-proofing here by acting upon the project root directory rather than a `pyproject.toml` file specifically. [/quote] It also makes it a little nicer to the conda community. [quote="Ofek Lev, post:37, topic:107186, username:ofek"] Operations that target environments would accept an optional `envs` field referring to a non-empty array of environment names that would default to a single environment chosen by the tool, or an error if tool-specific project configuration enforces explicit selection. [/quote] I don't know if always accepting an array makes sense, e.g. ... [quote="Ofek Lev, post:37, topic:107186, username:ofek"] `exec` - options: `cwd`, `envs`, `cmd` array of strings e.g. `[”coverage”, "run", "-m", "pytest"]` [/quote] ... does what when given an array of environments? Runs the same command concurrently? I will also say that taking a JSON-RPC approach for execution will require a broader API surface to handle stdin, killing processes, etc. Steve's PEP 517 approach avoids this somewhat by defining a protocol for interacting with the running command (very likely like `subprocess` and/or [`asyncio` subprocesses](https://docs.python.org/3/library/asyncio-subprocess.html#asyncio-subprocess)). [quote="Ofek Lev, post:37, topic:107186, username:ofek"] I’m okay with a `.venv` directory alongside a `pyproject.toml` file acting as the default IFF it’s a non-empty directory and there is no workflow tool defined anywhere. [/quote] That would be my assumption assuming people don't opt out of having any workflow tool run automatically. [quote="Ofek Lev, post:37, topic:107186, username:ofek"] Consumers can assume the directory structure is that of a standard virtual environment but they should not create it on behalf of the user. [/quote] Is "consumers" front-ends? In that case I would say it's fine to create a virtual environment if no tool is specified. [quote="Ofek Lev, post:37, topic:107186, username:ofek"] [quote="pf_moore, post:26, topic:107186"] I don’t use Poetry, and don’t want it globally installed. So when *I* use Poetry, I use it via `uvx poetry`. Do I need to override the project config just because I choose to *invoke* Poetry via a different mechanism than the project developers do? [/quote] This is a good call out. I haven’t thought too much about this but I think it would fit nicely in the user customization scheme to have a way to define arguments that prefix the workflow tool command. [/quote] I can tell I have thought about this. 😁 Regardless of whether a server protocol or PEP 517 approach is taken, I expect front-ends to be able to run the thing they need if it isn't installed. How they do that can be up to the tool (e.g. create an ephemeral virtual environment, use pipx, etc.). I would expect there to be a way to specify what should be available to run the tool. [quote="Ofek Lev, post:39, topic:107186, username:ofek"] If at some point you want to try a project’s configured tool named `foo` in your terminal that’s managed in a unique way, you need only satisfy its `foo`-shaped requirement. For example, to avoid a global installation via `uvx`/`pipx` merely add an equivalently named script on your `PATH` that forwards all arguments to `uvx foo`. This is a common enough scenario though so we should add a way to easily configure this. [/quote] I agree with Steve's sentiment that ... [quote="Steve Dower, post:41, topic:107186, username:steve.dower"] This is one of the things I think we don’t need to *define*, we just need to make sure we don’t *forbid* it. [/quote] So I have always expected a way to specify what to make available from PyPI if that's where your tool exists. [quote="Ofek Lev, post:39, topic:107186, username:ofek"] You can create your own workflow tool and override the project default to provide a UX that behaves exactly as you wish without having to configure several different things per project. [/quote] This has been a design goal for the PEP from the start. My personal workflow is very much like Paul's workflow and I will make sure I can configure my own setup via my own code so I can have this (and anyone else can have their own). [quote="Steve Dower, post:41, topic:107186, username:steve.dower"] As long as the interface doesn’t involve dealing with shell quoting/etc. [/quote] Not if I can help it. I would assume anything dealing with processes takes a list of arguments so quoting is avoided. [quote="Steve Dower, post:41, topic:107186, username:steve.dower"] My suggestion was that “no workflow tool defined” implies a *specific* default workflow tool (to be implemented) that uses `venv` to create an environment at `.venv` and can launch it. [/quote] I do assume _some_ CLI tool will be created to provide a CLI API for working with whatever approach may be chosen (if any approach is chosen at all by me; I can still choose to stick with PEP 832 as-is an let some other ~~fool~~ person take this more complicated approach on). As such, that CLI tool can provide such a baseline, simple experience when no workflow tool is specified. [quote="Steve Dower, post:43, topic:107186, username:steve.dower"] [quote="pf_moore, post:42, topic:107186"] The biggest problem with a Python API is deciding *which* environment the API should be installed into. [/quote] Again, I don’t think we have to tell VS Code (for example) what to do here. They can create a private venv using the standard library and put the tool in there, and then provide it with the CWD of the user’s project and it’ll never know the difference. Or they can use pipx, or uvx. Or they can make the user choose a location, or they can try and use the system one (and deal with complaints when it fails). [/quote] This is also my assumption. This sort of environment management is necessary for inline script metadata, so I don't think there's anything to concern ourselves with. It's basically an implementation detail for a front-end to handle in whatever way they choose. [quote="Paul Moore, post:49, topic:107186, username:pf_moore"] I still don’t see how that lets me configure tools to use my workflow in *all* my projects (unless I specifically choose a different config). I don’t want a workflow API script in every project, just one somewhere on my machine to affect everything where I don’t override it. [/quote] Both approaches would allow you to do this, they just differ in what it would take to make it happen. And as I said, I will make sure this is possible because I personally want this as well. Otherwise it's either having the front-end tool having a way to choose a per-machine or person config or standardizing how to select such a config. [quote="Ofek Lev, post:51, topic:107186, username:ofek"] I think we should be quite worried about requiring Python for environment management and other project workflows as that introduces a bootstrapping issue. In the early days of Hatch I [learned](https://hatch.pypa.io/latest/blog/2023/12/11/hatch-v180/#installation-made-easy) how critical of an issue that is from watching users. [/quote] [quote="Ofek Lev, post:51, topic:107186, username:ofek"] For example, it doesn’t make sense for maintainers of a tool written in an another language like uv to create a Python package just to manage itself and accept the compatibility hardship that naturally comes from such dual versioning. [/quote] I'll address this below, but I don't think having a Python shim around uv is a big ask since that shim could use `subprocess` to call uv and be its own package. [quote="Steve Dower, post:52, topic:107186, username:steve.dower"] [quote="ofek, post:51, topic:107186"] I’d be in favor of that as long as we keep it minimal like `build`. [/quote] That’s the wrong analogy - `build` is the front-end, and I’m talking about a default backend. So it’s “minimal like `flit-core`”. [/quote] Either way, I would expect a CLI tool that can directly utilize either API for whatever reason someone/thing has to want to call a workflow tool via the API. [quote="Steve Dower, post:56, topic:107186, username:steve.dower"] I also still believe that relocatable builds will happen [/quote] I do as well as I will work to make it happen. 😉 [quote="Paul Moore, post:61, topic:107186, username:pf_moore"] I think the point is that I can write the LSP however I want (python, rust, whatever) and just supply the command line needed to run it. But if I write a Python class, I need to make that loadable in a Python interpreter supplied by the tool - and every tool might need me to do that differently, so there could be multiple copies of my Python file scattered around my machine. [/quote] I would assume you could you use inline script metadata, so I don't think multiple copies; you just configure one path to use. ---- I want to make a few things abundantly clear that are independent of either approach. One is people should have a way to point at something bespoke on their machine that implements the API the way they want to do it. And you should be able to override per-project and you should be able to set a default fallback. Whether this is in the spec or something people expect/demand front-ends is open for debate. Two, I expect there to be a way to declare what would need to be installed. That way the front-end can do what it needs to do to get the tool to run it. Three, I expect there to be a CLI tool that can work with the API. That helps AI models, people who want to script things without being tool-specific, etc. I also expect there to be a supported fallback. Now, what are the key differences between the two approaches? Let's look at it from the angle of Hatch (written in Python) and uv (written in Rust). With Ofek's approach, uv implements a JSON-RPC server. With Hatch it also has to implement such a server. With Steve's approach, uv implements a wrapper. With Hatch it can more directly expose its own code. This then also applies to what it takes for someone to do their own solution. With Ofek's approach people are doing a server however they want, while with Steve's they are implementing a Python class. In terms of bootstrapping, Ofek's expects to (potentially) more self-contained when a workflow tool ships a self-contained binary (but not everyone does). Steve assumes getting Python on a machine is either already done or will be easy enough in the future thanks to relocatable builds eventually coming from python.org (but finding Python on a machine can be hard and I have not made relocatable builds happen yet). I _think_ those are the differentiators. Honestly the key question for me is whether any workflow tools would do either approach and do they have a preference or would refuse to implement one of the approaches. I don't want to pester the other workflow tool maintainers until we agree that what I'm saying is accurate so I can give them a reasonable "what do you think" post to read and give feedback on. ------------------------- ofek | 2026-05-21 13:38:43 UTC | #63 [quote="Brett Cannon, post:62, topic:107186, username:brettcannon"] [quote="ofek, post:37, topic:107186"] We should also make some sort of fast exit path option for times when you want to run only a single operation without process management \[...\] [/quote] I'm not sure if this needs to be part of any spec. If you only want to run one command then run one command and exit. If you're talking about surfacing the functionality as individual CLI commands, a CLI tool could provide that and act as a unified CLI API across all tools. That gets you both your scripting and AI support. [/quote] Yeah, that's a better idea. [quote="Brett Cannon, post:62, topic:107186, username:brettcannon"] I don't know if always accepting an array makes sense, e.g. ... [quote="ofek, post:37, topic:107186"] `exec` - options: `cwd`, `envs`, `cmd` array of strings e.g. `[”coverage”, "run", "-m", "pytest"]` [/quote] ... does what when given an array of environments? Runs the same command concurrently? [/quote] It would run the same command in each environment sequentially. Consumers can optimize for parallelism. [quote="Brett Cannon, post:62, topic:107186, username:brettcannon"] [quote="steve.dower, post:41, topic:107186"] My suggestion was that "no workflow tool defined" implies a *specific* default workflow tool (to be implemented) that uses `venv` to create an environment at `.venv` and can launch it. [/quote] I do assume *some* CLI tool will be created to provide a CLI API for working with whatever approach may be chosen \[...\] As such, that CLI tool can provide such a baseline, simple experience when no workflow tool is specified. [/quote] That sounds reasonable. [quote="Brett Cannon, post:62, topic:107186, username:brettcannon"] Now, what are the key differences between the two approaches? Let's look at it from the angle of Hatch (written in Python) and uv (written in Rust). [/quote] I agree with the assessment that follows but I think you missed a rather large difference in user experience. #### ***To be absolutely clear:*** If we do not go with an approach that allows for interactive bi-directional communication then workflow tools written in Python will always be less responsive and downright sluggish compared to others. Users of such tools will never have a snappy experience when clicking the button next to a test function, running one of their predefined `tasks` with a hotkey, etc. To put it in biased terms, if we do this then there is no way for Hatch to match the user expectations of tools like uv. --- [quote="Brett Cannon, post:62, topic:107186, username:brettcannon"] I expect there to be a way to declare what would need to be installed. That way the front-end can do what it needs to do to get the tool to run it. [/quote] If we use my language here: [quote="Ofek Lev, post:58, topic:107186, username:ofek"] If the `requires` field is defined then consumers SHOULD add a directory to the front of PATH for all invocations of the workflow tool. The directory MUST contain all files contained within the `scripts` subdirectory of the `.data` [directory](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-data-directory) of all wheels that are selected from a satisfactory dependency resolution. [/quote] Then I'd be fine with requiring a list of Python package requirements as long as each spec supports the full syntax i.e. allows for a local path to a directory, a Git VCS URI, etc. It's not that much of a lift to turn a script into a package with a minimal `pyproject.toml` nowadays although I'm curious what Paul thinks about this idea. I think it's a fair compromise. ------------------------- steve.dower | 2026-05-21 14:07:14 UTC | #64 Agreed with the rest of your post broadly. I don't think I need to make my imagined approach any clearer here, so happy to leave it with decision makers and clarify when requested. [quote="Brett Cannon, post:62, topic:107186, username:brettcannon"] [quote="steve.dower, post:41, topic:107186"] My suggestion was that “no workflow tool defined” implies a *specific* default workflow tool (to be implemented) that uses `venv` to create an environment at `.venv` and can launch it. [/quote] I do assume *some* CLI tool will be created to provide a CLI API for working with whatever approach may be chosen ... As such, that CLI tool can provide such a baseline, simple experience when no workflow tool is specified. [/quote] Let's not confuse "tool" with "workflow tool", or as I've been stealing from PEP 517 "frontend" vs "backend" (or the concrete examples, "VS Code" vs "Hatch"). The "CLI tool" is a frontend, and so if _it_ provides the default behaviour, then it's the reference implementation and we expect every other frontend to go off and implement it as well. I suggested publishing a _backend_, so if no more specific backend is provided then _that_ can be installed and it will do the job. Front-ends don't need to implement any default behaviour other than "substitute this name and do my usual thing" (or whatever a user has requested they do). ------------------------- pf_moore | 2026-05-21 14:26:51 UTC | #65 [quote="Ofek Lev, post:63, topic:107186, username:ofek"] It’s not that much of a lift to turn a script into a package with a minimal `pyproject.toml` nowadays although I’m curious what Paul thinks about this idea. I think it’s a fair compromise. [/quote] I can't disagree with the assessment that it's a fair compromise. However, it's almost certainly enough of a barrier that for me, personally, it would stop me from using the feature. I'll explain why, simply for information - I'm not asking for special consideration here. But given that I was surprised when Brett said his workflow was like mine, maybe my reservations here will also resonate with others, which might affect the choices we make. My problem is that when I write a single file script, I can do what I like with it - copy it wherever I need, put it on `PATH` or run it by name from a directory, point to it as config in a tool, etc. Single file scripts need no infrastructure and no workflow. Most critically, I don't need to keep the "source" anywhere - the script *is* the source. When I make a script into a package, I suddenly have a source and a built artifact (probably a wheel, maybe something else). I have to build the package and use *that*, but I also need to retain and manage the source. I need to "publish" the built project, even if that publishing is just putting it in a directory somewhere on my PC. I need to rebuild and redeploy after I change the source. I need to choose a build backend. Heck, I have to choose a *name* for my project (many of my personal scripts are called things as meaningless as `pyr.py` or even `x.py` - don't judge me :slightly_smiling_face:) There's basically a whole workflow needed beyond just writing the script. For small personal projects, I'm generally very bad at these things - the sort of management I do for published projects, or bigger pieces of code, doesn't feel relevant for a single personal script. ------------------------- pf_moore | 2026-05-21 14:31:33 UTC | #66 [quote="Steve Dower, post:64, topic:107186, username:steve.dower"] The “CLI tool” is a frontend, and so if *it* provides the default behaviour, then it’s the reference implementation and we expect every other frontend to go off and implement it as well. [/quote] I assumed it was the other way round. The "CLI tool" was something users would configure with a backend (the same as you would with VS Code, for example), and then you could use it to provide backend-agnostic access to environments - `env-tool list`, `env-tool create xxx`, `env-tool run some command`. Which is to say, please can we be a lot more explicit about terminology, because concepts are getting mixed up and it's getting very hard to follow what people are talking about... ------------------------- ofek | 2026-05-22 00:59:26 UTC | #67 [quote="Paul Moore, post:66, topic:107186, username:pf_moore"] [quote="steve.dower, post:64, topic:107186"] The "CLI tool" is a frontend, and so if *it* provides the default behaviour, then it's the reference implementation and we expect every other frontend to go off and implement it as well. [/quote] I assumed it was the other way round. The "CLI tool" was something users would configure with a backend (the same as you would with VS Code, for example), and then you could use it to provide backend-agnostic access to environments - `env-tool list`, `env-tool create xxx`, `env-tool run some command`. Which is to say, please can we be a lot more explicit about terminology, because concepts are getting mixed up and it's getting very hard to follow what people are talking about... [/quote] I made the same assumptions as Paul and therefore was quite confused reading the past few messages. So everyone is on the same page, there are two parts: 1. A tool that exposes a low-level environment management API. Examples: Hatch, uv, Conda, Nix, etc. 2. A caller of the tool. Examples: VS Code, Jupyter, a CLI wrapping the API, etc. I've been referring to the caller as a "consumer" of the tool's API while inconsistently referring to the tool itself as an "environment manager" or "workflow tool". If we were to reuse build system terminology then the tool would be the backend and the API consumer would be the frontend. I think those terms don't fit quite nicely however and I truly dislike the Python API approach, so I'd like to use different terms. Let's agree on the following split terminology: * CLI API: the tool is a "workflow server" and the caller is a "workflow client" * Python API: the tool is a "workflow backend" and the caller is a "workflow frontend" Those advocating for the latter may choose different terminology of course! That's just the preference I perceived based on the desire to emulate PEP 517. [quote="Paul Moore, post:65, topic:107186, username:pf_moore"] [quote="ofek, post:63, topic:107186"] It's not that much of a lift to turn a script into a package with a minimal `pyproject.toml` nowadays although I'm curious what Paul thinks about this idea. I think it's a fair compromise. [/quote] I can't disagree with the assessment that it's a fair compromise. However, it's almost certainly enough of a barrier that for me, personally, it would stop me from using the feature. [/quote] Got it! I think technically we're still fine then because of this: [quote="Steve Dower, post:12, topic:107186, username:steve.dower"] [quote="brettcannon, post:11, topic:107186"] [quote="ofek, post:10, topic:107186"] I know we technically did it for the build system but I worry that the complexity of standardizing the behavior of `requires` may impact acceptance of this proposal. [/quote] I'm up for taking it out if people prefer. [/quote] In the same vein, specifying *the intent* that "these things are required" is totally fine by me. Specifying that tools *must* figure out how to install those things is going too far. Let tools choose to honour the intent however they see fit. [/quote] Assuming that's still Steve's opinion, then we don't need to make `requires` a required field. However, folks seem to want (and I now agree) for there to be a default workflow server package requirement that: * Uses a virtual environment located in a `.venv` directory at the project root * Uses the standard library's `venv` module for environment creation So I think there would be 3 possible scenarios. 1. `requires` is not defined: the field defaults to an array containing the default server as the only dependency 2. `requires` is defined: here's my earlier comment (note that the logic would be the same for the not defined scenario so we would have to clean up the language of that field): [quote="Ofek Lev, post:58, topic:107186, username:ofek"] If the `requires` field is defined then consumers SHOULD add a directory to the front of PATH for all invocations of the workflow tool. The directory MUST contain all files contained within the `scripts` subdirectory of the `.data` [directory](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-data-directory) of all wheels that are selected from a satisfactory dependency resolution. [/quote] 3. A new field/override config that would disable the usage of the `requires` field so that clients don't modify PATH when invoking the server CLI. What do we think about that? It's possible that if we go with this then Steve's comment about the `requires` field being only for intent might be unnecessary. We could say that clients must respect the dependencies, not just a best-effort thing, unless the user disabled it. ------------------------- cjames23 | 2026-05-22 01:20:28 UTC | #68 [quote="Ofek Lev, post:63, topic:107186, username:ofek"] #### ***To be absolutely clear:*** If we do not go with an approach that allows for interactive bi-directional communication then workflow tools written in Python will always be less responsive and downright sluggish compared to others. Users of such tools will never have a snappy experience when clicking the button next to a test function, running one of their predefined `tasks` with a hotkey, etc. To put it in biased terms, if we do this then there is no way for Hatch to match the user expectations of tools like uv. [/quote] This where Ofek and I are fundamentally aligned. I do not want any proposal that puts hatch at a performance disadvantage before we even write a single line of code to implement it. That is not to say that Rust is the magic bullet here, Ofek and I are very good about writing high performant Python code for hatch but without bi-directional communication then there is going to be a cost to pay that will be much harder to keep up with tools written in other languages. [quote="Ofek Lev, post:67, topic:107186, username:ofek"] However, folks seem to want (and I now agree) for there to be a default workflow server package requirement that: * Uses a virtual environment located in a `.venv` directory at the project root * Uses the standard library’s `venv` module for environment creation [/quote] I agree here and I think this is the right user experience to lean into for a default. Admittedly I am less concerned with Python power users here and more concerned with the casual Python user or new to Python user. That is to say, I have full faith that Paul will find a way to use the workflow that they have already regardless of where we go because of the expertise that they have. But we should ensure that whatever the final form is that we do not make it impossible for Paul to have their standard workflow still work for them. And I do not think anyone in this thread so far has suggested a solution that would be pushing towards making that impossible. ------------------------- steve.dower | 2026-05-22 10:04:03 UTC | #69 [quote="Ofek Lev, post:67, topic:107186, username:ofek"] Assuming that’s still Steve’s opinion, then we don’t need to make `requires` a required field. [/quote] It should still be required in the project metadata - otherwise, there's no context for resolving the name of the API that you're about to invoke. I'm just saying we don't have to mandate "workflow frontends _must_ install exactly these packages from PyPI into an isolated virtual environment". That approach should work,[^1] but we shouldn't try to ban other ways to achieve the same result. [quote="Ofek Lev, post:67, topic:107186, username:ofek"] there to be a default workflow server package requirement [/quote] I don't want a "requirement", I just want the package ;) It can be implemented outside of the API definition and just referenced as "here's our intended default behaviour, frontend tools probably want to use this as the fallback backend to keep your users happy or else they'll get angry at you". That way the actual behaviour, particularly as it relates to edge cases and obscure platforms, can be handled outside of the PEP process. [quote="Cary Hawkins, post:68, topic:107186, username:cjames23"] [quote="ofek, post:63, topic:107186"] #### ***To be absolutely clear:*** If we do not go with an approach that allows for interactive bi-directional communication then workflow tools written in Python will always be less responsive and downright sluggish compared to others. Users of such tools will never have a snappy experience when clicking the button next to a test function, running one of their predefined `tasks` with a hotkey, etc. To put it in biased terms, if we do this then there is no way for Hatch to match the user expectations of tools like uv. [/quote] This where Ofek and I are fundamentally aligned. I do not want any proposal that puts hatch at a performance disadvantage before we even write a single line of code to implement it. [/quote] Why are you both assuming that the Python API can only be used once and then the runtime has to be completely shut down? I deliberately made my example a class because you can instantiate it once and do as many operations as you want - if instantiating it starts a local server and does interactive bi-directional communication, great! That's totally allowed. If a frontend isn't written in Python and wants to do their own bi-directional communication with a persistent Python runtime that is accessing the API, great! If Hatch wants to document a "bypass" interface that is more efficient for frontends that want to directly interact with Hatch's server, then you can do that - just specify the values of metadata that frontends can interpret as "use `hatch --json-rpc ...` directly", and go around helping frontends to use it. Any who don't [care about their users] will stick with the default, multi-hop API, which works but is less efficient. Literally everything I'm pushing for is to allow free choices like this. I hope I don't have to go around imagining how to implement this stuff for everyone, because I can, it's just less efficient than project owners getting that this API is a _minimum_ and not the maximum. If you want to go above and beyond, all you need to do is help the API consumer use it, and can rely on _this_ API to signal _when_ it should be used. The rest is up to you. [^1]: Certainly for any public project your users should complain if it _doesn't_ work. Private projects using private backends and private frontends don't have to be forced into using a public index to claim "compliance with the API". (Also FWIW, I'm not overly attached to the "frontend" and "backend" terminology, but I'm using it as consistently as I can because I think the metaphor to PEP 517 is more helpful right now than trying to invent new words.) ------------------------- brettcannon | 2026-05-22 21:54:59 UTC | #70 [quote="Steve Dower, post:69, topic:107186, username:steve.dower"] (Also FWIW, I’m not overly attached to the “frontend” and “backend” terminology, but I’m using it as consistently as I can because I think the metaphor to PEP 517 is more helpful right now than trying to invent new words.) [/quote] How about VS Code and Hatch for terms? 😉 And I'm only half joking; I'm going to stick to those concrete terms in this discussion going forward as it's unambiguous and people for both tools are represented here. I also don't think anyone will be insulted if we use those two projects as running examples. [quote="Ofek Lev, post:63, topic:107186, username:ofek"] If we do not go with an approach that allows for interactive bi-directional communication then workflow tools written in Python will always be less responsive and downright sluggish compared to others. [/quote] Steve beat me to the response: [quote="Steve Dower, post:69, topic:107186, username:steve.dower"] Why are you both assuming that the Python API can only be used once and then the runtime has to be completely shut down? I deliberately made my example a class because you can instantiate it once and do as many operations as you want - if instantiating it starts a local server and does interactive bi-directional communication, great! That’s totally allowed. If a frontend isn’t written in Python and wants to do their own bi-directional communication with a persistent Python runtime that is accessing the API, great! [/quote] I never assumed the Python API approach would be one-shot runs all the time. I expect VS Code would write its own server layer to handle all of this. Yes, it makes it like the CLI API proposal, but it is still different as its a choice instead of a requirement. [quote="Steve Dower, post:64, topic:107186, username:steve.dower"] The “CLI tool” is a frontend, and so if *it* provides the default behaviour, then it’s the reference implementation and we expect every other frontend to go off and implement it as well. I suggested publishing a *backend*, so if no more specific backend is provided then *that* can be installed and it will do the job. Front-ends don’t need to implement any default behaviour other than “substitute this name and do my usual thing” (or whatever a user has requested they do). [/quote] I honestly was just assuming the CLI tool could play both roles. The key point is I don't see either approach prohibiting accessing the API from a CLI tool. [quote="Paul Moore, post:65, topic:107186, username:pf_moore"] Single file scripts need no infrastructure and no workflow. Most critically, I don’t need to keep the “source” anywhere - the script *is* the source. When I make a script into a package, I suddenly have a source and a built artifact (probably a wheel, maybe something else). [/quote] I don't expect either approach to prevent using a single `.py` file because I don't want to necessarily write anything more complicated. I'll talk about it more below. [quote="Ofek Lev, post:67, topic:107186, username:ofek"] Assuming that’s still Steve’s opinion, then we don’t need to make `requires` a required field. [/quote] I think of it more like "requires if needed" at least for the CLI API case. I'll talk about it more below. ---- OK, let's talk through how each approach may work to get the differences laid out. That way we can focus on that aspect and get more targeted feedback from other workflow tool maintainers. # Getting Hatch ## Python API There would be a requirement definition in `pyproject.toml`. VS Code would create or reuse a virtual environment where Hatch gets installed. This does require finding a Python interpreter on the machine, although long term that shouldn't be a concern due to relocatable builds and VS Code can already find interpreters (or even short term not a problem if you're okay with python-build-standalone). ## CLI API VS Code can check if the first thing listed in the command is installed. If Hatch is already installed -- via e.g. Homebrew, dnf, apt, etc. -- then you're done. There would be a requirement definition in `pyproject.toml`. You could create a virtual environment and find the command in `bin/`. Otherwise you could use pipx to handle launching. # Specifying a local alternative to Hatch ## Python API You should be able to give a path to a `.py` file or maybe a directory as a package. ## CLI API You should be able to give a path to a `.py` file or maybe a directory with a `__main__.py` file. You could also point to any file and run it, e.g. a shell script. # VS Code interacting w/ Hatch ## Python API VS Code would launch a Python server that loaded Hatch and worked with it without having to launch Hatch for every operation. ## CLI API VS Code launches Hatch and interacts with it in its server mode, leaving the server running. # Using the API ## Python API Protocols would be defined. The simple case would be methods that return a mapping. Complicated would be a exec API which returns a subprocess-like object that handles communication, killing the process, return code, etc. ## CLI API JSON-RPC server which defines the expected functions. Simple case is getting JSON back. Complicated case it an exec API where you would need to define functions for each part you need for working with a process (e.g. communication, killing the process, return code, etc.) via some process ID. ------------------------- encukou | 2026-05-25 12:55:51 UTC | #71 [quote="Brénainn Woodsend, post:57, topic:107186, username:bwoodsend"] Any tool that needs the internet can’t be used. Build isolation blocks distribution-required patches to build backends. Using venvs for testing defeats the purpose of system level testing in the first place since you’re not actually testing with the dependencies you’re deploying with. [/quote] Yup, some distros should do this in a “non-isolated” build. Which of course *is* isolated -- so much that to Python tooling it looks like a raw OS-installed Python. IMO, tools that *discover* environments should handle the case of the environment not being *virtual*. And *creating* environments should be configurable enough that you can write a tool that provides an offline container instead of a venv. That's largely equivalent to Paul's “configure tools to use *my* workflow in *all* my projects”, but I think it's a good case to keep in mind when deciding how configurable all this should be. [quote="Brénainn Woodsend, post:57, topic:107186, username:bwoodsend"] Anything that tries to detect or install Python installations is self defeated in the same way. [/quote] *Detection* should be easy. “Installing" a specific Python version means using a specific container to run the tool. [^1] [^1]: Even to “install” a Python package that wasn't pre-declared, you re-run the tool in a container that has the given package installed. (Yeah, also “nuts”, but of a different kind.) [quote="Brénainn Woodsend, post:57, topic:107186, username:bwoodsend"] Lockfiles or pinned dependencies are unusable in an ecosystem where only one version of each package is available. [/quote] Any sufficiently corporate-ish environment will have a filtered list of allowed artifacts and will need to deal with the same issue. [quote="Brénainn Woodsend, post:57, topic:107186, username:bwoodsend"] Any test command+dependencies list that uses linters (`pytest-flake8`) or formatters (`pytest-black`), coverage, profiles, benchmarks, heavy fuzz testing or multiple environments needs to be replaced anyway. [/quote] Which is why linting should be separate from the functional tests, and the test suite should be separate from the environment matrix. IMO that's a reasonable step away from “works on my (CI) machine” -- after all, whenever *you* want to add a new environment to the mix, it's rather handy to be able to run the relevant tests in a given pre-existing environment. ------------------------- brettcannon | 2026-05-25 23:18:54 UTC | #72 OK, I'm putting a time limit of deciding by Friday, June 5th as to which API approach to take back to https://discuss.python.org/t/pep-832-virtual-environment-discovery/106998/ to see if people even want an API approach. As such, I'm now looping in the other tool maintainers to see if they have a preference (to be clear, I'm **not** asking if you prefer PEP 832 as-is or an API approach, just **which** API approach to take back to the PEP discussion). I am also going to summarize things here for how I view the strengths of either approach so the tool maintainers don't need to read through 70 posts 😅 (you can also feed https://discuss.python.org/raw/107186/ into some AI if you want a summary that way). Finally, there's a poll at the end for anyone who wants to express an opinion that way which I will run for a week. /cc @frostming (PDM) @radoering (Poetry) @zanie (uv) @carltongibson (virtualenvwrapper) @bernatgabor (virtualenv/tox) @henryiii (nox) @jezdez (conda) @lucascolley (pixi) --- So this is all about what approach to take for defining an API that workflow tools can implement to expose details about what environments they manage(to start; I'm sure it could be expanded in the future). Don't worry about the exact shape of the API for this discussion; this is about how to expose the functionality. We have two approaches we are considering: a CLI API approach and a Python API approach. The CLI API will have workflow tools implement a JSON-RPC server and `pyproject.toml` will record how to launch the tool as a server (think LSP but for Python workflows). The Python API where workflow tools implement a class with methods and `pyproject.toml` records the entry point (think PEP 517). Let's start with what's the same. Both would allow for the same underlying functionality (e.g. both can handle listing the available environments or running a Python command in a specific environment). Both allow for being run as a server to support a responsive experience (who writes that server is different, though, and will be discussed later). Both approaches will let you specify what workflow tool you want to use, either in `pyproject.toml` or in some way to provide a default or even override (but this is just because the tool calling a workflow tool can make that choice). Both approaches allow someone to write their own workflow tool and specify using that (albeit with different requirements on what they need to do; discussed later). Both approaches could have a CLI made for it for scripting purposes to make calls to the workflow tools. So how do the approaches differ? The key differences are covered in https://discuss.python.org/t/cli-api-for-discovering-environments-for-a-project/107186/70 . I think the two key differences is how people get the workflow tool on to their machine and where does responsibility/flexibility lie? For the CLI API approach, it has a perk that the workflow tool could simply be on the machine already. That's a potential irritant for users if the calling tool using the workflow tool doesn't provide a way to get the workflow tool on their behalf. How big of a deal that is depends on how hard you think it is to find a Python install on a machine or how much faith you have in me getting relocatable builds on python.org to make getting Python as easy as a download. The other key difference is who is in charge of what. The CLI API means every workflow tool needs to implement a JSON-RPC server. That _might_ be easy by relying on another project on e.g. PyPI, but it is another layer of stuff to do. On the other hand, it might be a preference as it's a clear boundary of control between the calling tool and the workflow tool. Compare that to the Python API where some might view it as easier to implement, but not getting to control e.g. the event loop if the API gets imported into some running code might not be desired (if that's even a concern; I don't think build back-ends have shown this to be an issue, but you never know). I think the differences come down to how much control/work the workflow tools want. The CLI API approach gives the workflow tools more control at the cost of more work (i.e. more installation control, how they implement their API behind the JSON-RPC server). Now workflow maintainers might want that control for technical reasons or because they think they need to make the lives of the calling tools as easy as possible to maximize uptake. But workflow maintainers might also not want more work if they can help it or think calling tools want more control instead. There's also whether people think it's easy enough to implement their own bespoke workflow tool on their machine with JSON-RPC involved and behind a command/script (with inline script metadata probably helping a lot), or the Python API is the most they are willing to do. 🤷 Both approaches have merits and I don't see a clear winner when trying to choose between the two (and you might not like either approach and prefer PEP 832 as-is, but that's a separate discussion), hence this post. 😅 [poll type=regular results=on_vote public=true chartType=bar close=2026-06-01T07:00:00.000Z] # Which API approach do you prefer? * CLI API * Python API [/poll] ------------------------- brettcannon | 2026-05-29 21:35:20 UTC | #73 [quote="Brett Cannon, post:72, topic:107186, username:brettcannon"] there’s a poll at the end for anyone who wants to express an opinion that way which I will run for a week. [/quote] A reminder that the poll closes on Monday. ------------------------- pf_moore | 2026-05-29 22:30:25 UTC | #74 I’m still unclear about whether the Python API version will require the API to be made available in a package (like PEP 517) or whether a simple Python module is sufficient. If a package is needed, that’s a difference that would matter to me. ------------------------- steve.dower | 2026-06-01 13:02:47 UTC | #75 [quote="Paul Moore, post:74, topic:107186, username:pf_moore"] I’m still unclear about whether the Python API version will require the API to be made available in a package (like PEP 517) or whether a simple Python module is sufficient. [/quote] My thinking was that an option for a relative path (to add to `sys.path` before importing the API) or even an implied import path of the root directory would cover that, with `requires` then only containing external dependencies (if any). Build backends were a bit different, and wanting to force arbitrary code to come from a "trusted"/vettable source made more sense, but since there's no silent upgrade here[^1]. I certainly would not want "we failed to find your module on PyPI" to be an error condition. "We failed to find it on your machine and you didn't give us a way to find it on PyPI" is okay. But it looks like the preference is for a CLI tool, which means you won't be able to distribute custom instructions in your source repo anyway. [^1]: Unlike someone who was previously `pip install`ing a package getting the new behaviour one day, nobody is currently triggering the frontend tool that uses this API. ------------------------- ofek | 2026-06-01 13:25:11 UTC | #76 [quote="Steve Dower, post:75, topic:107186, username:steve.dower"] But it looks like the preference is for a CLI tool, which means you won't be able to distribute custom instructions in your source repo anyway. [/quote] If you're referring to a means of declaring Python dependencies then I think that would definitely be supported. I mentioned how I see that working here: [quote="Ofek Lev, post:67, topic:107186, username:ofek"] I think there would be 3 possible scenarios. 1. `requires` is not defined: the field defaults to an array containing the default server as the only dependency 2. `requires` is defined: here's my earlier comment (note that the logic would be the same for the not defined scenario so we would have to clean up the language of that field): [quote="ofek, post:58, topic:107186"] If the `requires` field is defined then consumers SHOULD add a directory to the front of PATH for all invocations of the workflow tool. The directory MUST contain all files contained within the `scripts` subdirectory of the `.data` [directory](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-data-directory) of all wheels that are selected from a satisfactory dependency resolution. [/quote] 3. A new field/override config that would disable the usage of the `requires` field so that clients don't modify PATH when invoking the server CLI. [/quote] ------------------------- steve.dower | 2026-06-01 13:27:54 UTC | #77 [quote="Ofek Lev, post:76, topic:107186, username:ofek"] If you’re referring to a means of declaring Python dependencies [/quote] No, I'm referring to putting an entire custom tool into your repository. [Like this example on GH](https://github.com/Azure/azure-sdk-for-python/blob/main/scripts/sdk_init.sh), which could be adapted to have an importable interface that tools would just pick up and use, rather than having to install a separate tool that can be run as a CLI to provide the same specific-to-this-project functionality. ------------------------- pf_moore | 2026-06-01 15:41:58 UTC | #78 [quote="Steve Dower, post:75, topic:107186, username:steve.dower"] My thinking was that an option for a relative path (to add to `sys.path` before importing the API) or even an implied import path of the root directory would cover that [/quote] Thanks. My question was less about needing to publish to PyPI than about whether it would need to be a wheel (which the frontend would then install "somewhere"). If a single importable `.py` file containing the relevant entry points is acceptable (and the PEP *requires* frontends to support it, so it's not just a tool implementation choice) then I'm more OK with the Python API. [quote="Steve Dower, post:75, topic:107186, username:steve.dower"] But it looks like the preference is for a CLI tool, which means you won’t be able to distribute custom instructions in your source repo anyway. [/quote] Yeah, and given that your answer simply shifted me from "slightly in favour of a CLI tool" to neutral, the fact that I asked too late to get an answer before the vote closed doesn't actually matter in the end. ------------------------- brettcannon | 2026-06-01 22:49:28 UTC | #79 Thanks for everyone who voted! It came out 7 to 2 for a CLI API (6 to 2 if you combine the 2 Hatch maintainers both voting). I'll start a conversation over on the PEP topic to see if people even want this. [quote="Steve Dower, post:75, topic:107186, username:steve.dower"] But it looks like the preference is for a CLI tool, which means you won’t be able to distribute custom instructions in your source repo anyway. [/quote] I don't plan on that being true (if we follow through with an API in the PEP; still need to have that discussion). Whether it's via having a "script"/"path" key that can point to a script with inline script metadata or point to a directory with a `__main__.py`, `pyproject.toml`, or `pylock.toml`, there will be _some_ way to have a repo contain custom code. -------------------------