The main difference from the current situation is (using VS Code as the example front-end) right now the VS Code user has to specify (and determine) which tool to use, while with the API the project maintainer can specify it and the user doesn’t have to specify anything.
If you are both of these personas, then yes, nothing changes. And most people who are going to be here discussing this are either both or just the maintainer, so the only friction they’ll have experienced is having to explain to contributors how to set things up.
The friction being smoothed here is the user’s friction, not the maintainer’s. Most people here shouldn’t expect their lives to change significantly with an idea like this (but if it sounds like your life is getting worse, then definitely speak up!)
But the project maintainer doesn’t know what tool I want to use to work on a project.
This comes back to the question of whether environment management is a project choice or a user choice, and I’m strongly in favour of the idea that it’s a user choice (with the project able to recommend a workflow if they choose). If having a CLI API means accepting that it’s a project choice what workflow gets used, count me as a strong -1.
Edit: And speaking as a maintainer, I explicitly don’t want to choose the user’s workflow for them, so my life is made worse by the fact that I can no longer choose a workflow for my personal use on this project without it being imposed on my project users.
But I agree, none of this is meant to become mandatory. It’s a discovery tool, so that someone who clones a repo to start contributing can just “give me the default the project recommends” without having to find that document themselves.
If you don’t want it… just… don’t do it?
And I’ve already argued that tools should feel free to ignore this metadata if the user has overridden (at any level). The precedence/priority there doesn’t need to be specified - we don’t need to tell VS Code what to do when a user has said “ignore the spec” - but I assume we do need to explicitly say “it’s okay to ignore this without having the OSS community bite your head off”.
For what it’s worth, Flask uses uv, and documents how to use uv. (Previously it was pip and pip-tools, and briefly PDM.) We absolutely do want to choose our contributors’ workflow. We pin an exact set of tools for our development environment, and instruct users how to use that, so that every contributor is using the same starting point. They can certainly try to use something else, but that’s on them, we won’t be able to support it or help at sprints.
And from a contributor perspective, I would much rather be told exactly what tools to use to contribute to a project. I occasionally still run into projects that just say “here’s a python project, run the tests, submit a PR” and then have to figure out exactly how to do that myself, and it’s a giant pain. Projects should absolutely be picking one set of tools and using them. That doesn’t mean other projects can’t pick other tools, only that each project should pick one. I’m fine if one project is PDM, one is Conda, and one is uv, as long as each tells me exactly how to use that to develop and contribute.
That’s advisory, not required (yes, the wording is a bit stronger than that, but we don’t actually care in the majority of cases).
I thought that at least part of the idea was that IDEs like VS Code could discover where I keep the virtual environments I’m using to work on the project? If pyproject.toml says they should be in Poetry, but I’m actually using venv, what do I do? My impression was that I have to add an override file somewhere to say I’m not using the project default. Which, as I said originally, feels like it’s no improvement over the current situation where I have to select the environment I’m using.
There’s clearly different viewpoints here. I personally really dislike Poetry, and on at least one project where I’ve been required to use it, it’s been a significant disincentive to contribute.
Also, there’s a configuration issue here. I’ll continue to use Poetry as my example because the problem is worse for tools written in Python. But let’s assume the project config says that you list the project environments using poetry env list (for example). But I don’t use Poetry, and don’t want it globally installed. So when I use Poetry, I use it via uvx poetry. Do I need to override the project config just because I choose to invoke Poetry via a different mechanism than the project developers do?
I don’t want to make too much of this - for the average user, none of these concerns are that important. But for people who do have non-standard needs, a CLI API is often fiddly to set up. And yes, the affected users have the knowledge to deal with this, but often tools don’t cover all the bases. It just feels like a lot more work for very little benefit compared to PEP 832.
Correct, the case where you are not using the default remains the same as today (where there is no default). If the default happens to align with your preference, things get easier, which is also what would happen if we mandated that the only possible default was venv+.venv+CPython. We don’t want to mandate that, which means there’ll be a maintainer choice of default, and if you ignore it, then you do exactly what you do today (and maybe it gets worse if VS Code decides to make it more complicated to manually invoke the tools, but you’ll have to take it up with the front-end developers).
That feels like a long way to say “if you ignore this, you ignore it” again, but that’s the idea. Like I said, the improvement is meant for the users who no longer have to find, read, and copy-paste the commands from the getting-started page.
(One last comment, then I’ll drop this as I think it’s getting unproductive).
If we have a way for projects to put workflow setup into pyproject.toml, I imagine they will stop documenting the recommended workflow in the contributor docs. That means that for people like me who prefer to use my standard workflow and adapt it to make sure I’m not being an inconvenience to the project, I no longer have documentation to work from, and I’m reduced to reverse-engineering the config file. Which I won’t do, I’ll likely just not contribute instead.
I already reverse engineer the CI definitions for half the projects I contribute to, since they’re always more up to date that the documentation. And the more projects switch to “just use uv” (or more complicated tox/nox/etc.-based projects), the more you have to know how to reverse engineer it back to the basics. Maybe that gets worse when simple projects no longer have to explain even venv/pip, I’m not sure.
I suspect projects who notice that they’re losing skilled contributors by only offering the one-click setup will add more details for those contributors. Those that don’t notice, well, won’t care. Their loss, just like any other project that makes itself difficult to contribute to.
If we have a way for projects to put workflow setup into pyproject.toml, I imagine they will stop documenting the recommended workflow in the contributor docs.
If it were projects I work on, the documentation would at least say that the recommended workflow setup is managed by that mechanism with its configuration in the accompanying pyproject.toml file, or whatever, so that newcomers know what they’re expected to run and where to look to find out what it does. I don’t think changing workflows means you stop documenting them.
Of course, projects that never bothered to document their recommended developer environment setup will probably continue to not do so.
This is one of the reasons I’m tempted to keep the redirect file idea from PEP 832 around; it’s simple and can act as a short-circuit for tools to skip calling any API for the virtual environment.
In this case having a PEP 517-like API instead of a CLI one would help as e.g. your editor could just have a cached install it chose to use.
I was thinking about this last night and trying to articulate in my head why you and others feel this way. I want to be upfront this is not a leading question or meant to suggest that it’s an incorrect view. I’m just trying to think through why some of us think this, perhaps to include it in the PEP.
When it comes to other things like test runners (e.g. pytest), task runners (e.g. tox and nox), or pretty much anything else where you technically don’t have to use what the project provides, people in general don’t seem to have the same reaction of “don’t tell me what to do” as they do to virtual environments. Now I don’t think it’s due to disk use or anything as tox and now definitely put files on your disk.
So what makes virtual environments special in this regard? Is it a “get off my lawn” reaction for those of us doing this manually for so long that we just like our workflow a lot and don’t want to be forced out of it at a loss of personal efficiency? Or is it more like picking a code editor and interacting with your virtual environment feels closer to you than other things and thus more personal? Or is it a belief/assumption most projects don’t really need to have a preference, so why give them the ability to somewhat arbitrarily choose one that gets hoisted upon contributors (while admitting some projects do have reasons to require a specific tool)? Or something else I’m not thinking of?
As I said, I don’t view any of this as wrong. I’m just trying to think through where the feeling comes from to help shape where I should take things and the motivation behind any decisions I have to make.
At least for me, the answer is that they aren’t special. I boycott tox, uv, precommit, all the venv alternatives and any other tool that simply installs[1] and wraps its own preferred interface[2] around other tools[3] the same. They’re all akin to pushing a specific IDE or keyboard+mouse+monitor setup to me.
I’ve never not regretted humouring a contributing guide that told me to use its preferred environment management. It’s at the top of my mental list of things to avoid when deciding whether to submit a pull request to a project or just dump it on their issue tracker and guiltily expect the maintainer to do the rest.
What can make things OK is if the configurations for these wrapper tools are easily unpacked. If pip install -e . still works and the raw test command+dependencies are easy to find and it’s easy to figure out “ok, this project lints with black and flake8” then I don’t mind coexisting with this workflow tools since there’s still a user level choice to not use them.
often in its own cold cache so everything takes forever to setup ↩︎
which I then have to learn – usually inadequately so that I end up having to debug via print statements or some such because I can’t figure out where the -m pdb goes ↩︎
which I probably already have installed globally ↩︎
For me, it’s that I want to choose where my venvs go. By putting them in my project directory, I can see them (I’m on Windows, so a directory named .venv is visible), they get deleted with the project, and I can delete or recreate them myself when I want to. I can also use them in an adhoc manner just by running .venv/Scripts/python.exe.
With tools that hide venvs away somewhere, I lose that control. The environments don’t retain any link to the project directory (or if they do, it’s some tool-specific data that I probably don’t know about), and when I delete the project, the orphaned venvs remain, taking up disk space[1]. And to start a Python interpreter in the venv I have to either use a tool-specific command, or find the executable which is likely in some obscure directory.
So the .venv proposal works great for me, because it leaves me in control. Whereas this CLI API proposal is designed to hide the details from me, so that I lose that control. (And the side discussion about projects that mandate a particular tool ties into that, because they might require a tool that uses “hidden” environments).
To be honest, it’s not just about venvs. I’d feel the same way about a project that mandated hatch for its test runner. It’s the same problem - the environments used for testing get stored in a central directory, and when I delete the project they will remain unless I remember to deliberately hunt them down and remove them. And as I’m not a hatch user, I won’t be familiar with the need to do that, so I’ll almost certainly forget.
I know disk space isn’t that important, but unrecoverable garbage gradually filling my disk is (to me at least). ↩︎
FWIW, I share the same dislike of tools that install to secret directories when there’s an obvious place to put it.[1][2] I’m not concerned that this proposal makes that any worse, though, just makes it easier to have one of those tools cluttering up unrelated locations.
Maybe if we’re all forced to deal with it more often then we’ll complain to the projects/tools instead of just avoiding them and doing it manually? No telling how many people would prefer them to change their default behaviour…
I need that caveat because I do like that the Python install manager uses a “secret” directory, but also has a --target option for when there’s an obvious place to put the runtime. If it gained a py run-like option, then I’d strongly lean towards putting it in the project directory by default. ↩︎
Including a cache directory of files that gets linked into my project-specific directory. ↩︎
Upon further consideration, I think we should adopt the strategy of most AI agents and increase the priority of machine configuration for easier managed installations. I’d go with: pyproject.local.toml > per-machine > pyproject.toml > per-user
Yes, I was thinking about scenarios where environments would be managed remotely. Thank you for being able to extrapolate through my lack of specificity
After thinking more about this, I’m okay with saying that a CLI will forever be the only communication mechanism.
Yes, that’s exactly my view. I especially appreciate Steve explaining that our preference was operating at a different level than what some folks were internalizing. We don’t want to prescribe what workflow tools do so that consumers can rely on invariants to hold when introspecting but rather request a high-level operation that’s more of a black-box e.g. we want to tell an API to run a command in the context of a certain environment rather than finding the Python binary within an environment on disk in order to run a command.
I’ve thought a lot about this the past two weeks and I think I landed on the optimal design that would satisfy everyone. There are two aspects: the API and how to communicate with the environments manager.
Communication: I think workflow tools should behave like language servers. A consumer like an IDE would start the process (hatch env server, uv …, etc.) and send API requests to its stdin while receiving responses on stdout. This is much cleaner and more extensible than mandating a subcommand structure with certain flags for each command. It would also eliminate consideration for the frequency of operations in our design choices since startup would occur once (e.g. no more optimizing for environment discovery because users would perceive the call to list local environments as no slower than reading known paths from disk).
We should also make some sort of fast exit path option for times when you want to run only a single operation without process management which is useful for AI agents in change+validation loops, redistributors like Conda or Debian who want to properly run a package’s test suite, etc. This also provides a way to reduce background memory consumption for potentially suboptimal implementations.
API: I think one server process should be capable of managing environments for an arbitrary number of projects so every operation would accept an optional cwd string field referring to an absolute path that would default to the current working directory. I prefer future-proofing here by acting upon the project root directory rather than a pyproject.toml file specifically. Operations that target environments would accept an optional envs field referring to a non-empty array of environment names that would default to a single environment chosen by the tool, or an error if tool-specific project configuration enforces explicit selection. The initial operations would be:
create - options: cwd, envs
remove - options: cwd, envs
exec - options: cwd, envs, cmd array of strings e.g. [”coverage”, "run", "-m", "pytest"]
list - options: cwd | response: envs array of strings representing the available environments
inspect - options: cwd, envs | response: env_info map of environment names to a semi-structured mapping containing details about the environment
(future wish list) lsp - options: cwd, env (required) a single environment | response: cmd array of strings e.g. [”pyrefly”, "lsp"]
This API unlocks really cool workflow-level functionality as future PEPs like the definition of user-defined tasks which would run commands in the context of certain environments. We could then reserve the name of certain tasks so that, if we copy the reserved extras names as examples, the test task would run tests and the doc task would build documentation.
I think anyone balking would be doing so out of ignorance of the benefits that standardized environment management interoperability will bring to users. We should definitely express this in documentation but otherwise I don’t think we should care much about such sentiment.
I’m okay with a .venv directory alongside a pyproject.toml file acting as the default IFF it’s a non-empty directory and there is no workflow tool defined anywhere. Consumers can assume the directory structure is that of a standard virtual environment but they should not create it on behalf of the user. An improperly configured environment is far more user-hostile than no environment.
I’m under the assumption that as a primary goal we all want to maximally reduce user friction while simultaneously allowing for freedom of choice. As such, and knowing that I have good rapport with Paul and Brénainn, I’ll take the opposite stance and assert that these are generally incorrect views along two different axes
Incidentally, @davidism already expressed views above that are perfectly aligned with the goal for both. The first is the experience of users or contributors:
This is critical. Most users lack the knowledge, time and compelling motivation to use anything but what a project recommends. The vast majority of users want a workflow that is functional out-of-the-box with minimal effort.
It’s well-known that I made Hatch in part due to my dislike of Poetry. I was never forced to use it as part of several contributions; I did so several times willingly. Not once did I look through a project’s Poetry, CI and other configuration in order to reverse-engineer what would give me a working setup just to use my preferred workflow or shell aliases. My time on this planet is limited and I have no desire to waste it in such a way when I can simply install a tool to run one or two of its commands.
The second aspect here is about correctness. Projects often have logic encoded in their recommended development workflows that are difficult or impossible to express otherwise. Using a strawman to illustrate, consider someone desiring to build a project in the Google3 monorepo with something other than Blaze.
For a more realistic example, take a look at the Breeze developer environment offered by Apache Airflow. Manually configuring a parallel setup capable of running integration tests will quickly lose parity with their CI and waste users’ time when change validation results differ from what they work on locally.
I understand where you’re coming from as I think the opening post only showed example output and there hasn’t been much about how to invoke the CLI. However, even before my final JSON API idea it was never my intention to have the standardized commands match the day-to-day workflow of a user nor did I expect that tools would drop their custom UX in favor of such generalization.
To me, the purpose of environment standardization has always been purely for tooling interoperability so that everything interacting with a project behaves the same way by default while users passively gain a better experience. Sorry if I didn’t make that clear before!
This is a good call out. I haven’t thought too much about this but I think it would fit nicely in the user customization scheme to have a way to define arguments that prefix the workflow tool command. Perhaps we could even support environment variables (order of precedence TBD).
As Steve said a few times, nothing would change for folks who wish to avoid certain tool behavior but rather it would ease the default experience for most. One’s preferred workflow could still be used since nothing mandates specific user interaction.
In the worst case scenario, an editor calls the configured tool when an action is performed and an environment is unexpectedly created elsewhere. Users with such preferences would either read how to configure the tool or override/disable it as the environment manager.
I consider transient yet avoidable friction for power users worth the price of improving the default experience for most users and reducing the maintenance burden of those in other ecosystems who must consume our projects.
There’s a lot to digest in your post and I need to take some time to consider it, but I think this is something that I really struggle with. As a community, we’ve started to take a view that “standards shouldn’t dictate UI”, and while I agree broadly with that, I think that we make life unnecessarily difficult if we try to debate standards in the abstract, without considering how they will look to the user.
In this case, I’d like to properly understand how this proposal would affect me. At the moment, the abstractions mean I can’t work that out for myself. My workflow is likely unusual (I have issues with nearly every “opinionated” tool out there ) so I don’t expect to get away with doing nothing, but conversely right now I deliberately do almost nothing to configure my tools, and I get a “good enough” (for me) experience.
For day to day use, I use “raw” pip and either virtualenv or venv to manage environments. Both pip and virtualenv are installed centrally as the zipapp versions - I try to avoid installing pip in my virtual environments as much as possible. For the “main” virtual environment in my projects, I call it .venv. For other environments, I either use adhoc names or I use a tool like nox (generally run via uvx nox) to manage them. I never activate environments[1], instead I invoke commands from the venv using the full (relative) path - .\.venv\Scripts\python.
So there’s no “workflow tool” that I use, and therefore there’s no obvious tool that I could configure as a “workflow server”.
In addition, I occasionally use tools like hatch, PDM, or uv. This is partly because I’m a dabbler and keep trying out new approaches, but sometimes it’s because I need to keep up with how the various tools work (those projects tend to be “toy” ones). So in those cases I need to use a tool that isn’t my “normal workflow”. Again, I tend to use these tools via uvx <tool> because I’m experimenting, and don’t want to install yet another tool globally.
The “project dictates a workflow” case is similar to what’s described in the previous paragraph, but made worse by the fact that it’s not always clear where to draw the line - if the project says “install Poetry globally” and I use uvx, I’m already not following the instructions. So is it really so bad if I run manual tests in my own venv rather than following the project documentation? I know that I won’t ask the project for help if I do so - it’s my choice and they have no obligation to support me - so why does it matter to anyone but me?
In terms of consumers, I use both VS Code and Vim as editors. Right now, I do barely any configuration, and things just work for me (presumably because “raw venv” is special cased enough that I don’t need anything special). I don’t have any feel for what consumers other than IDEs and editors this proposal is intended to target.
Given the above, what would I expect to change in my workflow under this proposal? Presumably I’d need to specify my “workflow server” in various config files? Would I continue to customise any tools I use (for example, telling hatch not to store environments externally) in the normal way, or would I need to add “workflow server” config as well? And what would I gain? Would VS Code “magically” know what environment to use (like it does now with .venv), or would it ask me to pick from a list?
What are the create/remove operations targetted at? My editors don’t create or delete environments for me at the moment. Would they gain new options to do so? Would I be required to use them rather than manually creating environments?
It’s possible I simply have a very Luddite attitude to managing my environments and workflow. I certainly don’t have any objection to making it easier for people to write tools that manage everything for their users. But if that comes with the cost of no longer supporting users who like the ability to control everything for themselves[2], then I think we should clearly acknowledge that we’re choosing to do so.
Exception: VS Code activates an environment for me, but I tend to ignore that when working at the command line ↩︎
Or even just making life harder for such users, because they now have to fight the expectations built into their tools. It’s a bit like AI skeptics having to switch off AI features everywhere - the existence of those features doesn’t harm them directly, but it does add friction for them from having to continually “opt out”. ↩︎
I think a concrete example of a non-default user scenario is a great idea! However, before answering I’d like to hone in on a sentiment that is causing friction in this thread.
Ignoring a project’s recommended development workflow isn’t bad at all if, as you said, there’s no impact for others.
The central issue I see in this thread is that some are perceiving projects gaining the ability to influence the default behavior of all supported tooling as an encroachment on user choice. I’m honestly not sure where this perception is coming from as none in favor have expressed such intentions and the topic started off by making override mechanisms a hard requirement.
I think the impact on your workflow would depend on whether you take advantage of what the proposal offers.
Option 1: Do nothing
This looks exactly like what you do today. Although the actions you take wouldn’t change, things responding to your behavior might.
For example, if a project configures a tool then an editor would provide functionality like syntax highlighting and static analysis from an environment in a location that may not reside in a .venv directory in the project root as you expect. To remedy this you could either read the docs of the project-chosen tool to configure it to your liking, disable usage of the tool via configuration that hasn’t yet been discussed (probably an environment variable), or go with option 2.
Optional functionality would be exposed to you in various ways by default. Some examples:
A future PEP would specify a tasks table that refers to named sequences of commands which are to be executed in the context of one or more environments. Each of these would be available as an action you can trigger directly from your editor like test-e2e, update-deps, start-docs-server, etc.
Consumers like editor extensions or a new blessed build-style tool could offer a way to free up resources (disk space, Docker containers, cloud VMs) by removing all environments tied to a project.
If at some point you want to try a project’s configured tool named foo in your terminal that’s managed in a unique way, you need only satisfy its foo-shaped requirement. For example, to avoid a global installation via uvx/pipx merely add an equivalently named script on your PATH that forwards all arguments to uvx foo. This is a common enough scenario though so we should add a way to easily configure this.
Option 2: Create your own tool
Here is where the proposal shines particularly bright! You can create your own workflow tool and override the project default to provide a UX that behaves exactly as you wish without having to configure several different things per project.
You can write a small moore_env script which you put on PATH that would implement the few required API methods defined by the spec. You can enforce that the first python on PATH (or any other of your choosing) will be used to invoke the zipapp versions of virtualenv and pip (located wherever you decide) for creating virtual environments and subsequent package installation, respectively. The location of a project’s default environment would be a .venv directory at the root and you can provide additional heuristics for the detection of other environments as you see fit.
If there are some projects that you find incompatible with your preferences, then you can hardcode a list of them and have any API method that targets one invoke the actual tool that the project configured. Basically, you can do whatever you want
Yes, users would still configure the tools themselves to modify behavior with very few exceptions.
The tool would determine which environment acts as the default. In Hatch, it’s the default environment although if it defines a matrix then we’d have to select one of the generated environments as the default using sane heuristics like the Python version and lexicographical order of their names. I’d imagine that some tools would prefer explicit selection by users at least under certain scenarios and would return an error response.
Removal perhaps but definitely creation. The user would no longer have to be the one to create the .venv in order for tools to know what to do.
I think what people see is that having to override changes the dynamic from project preferences being “opt in” (the project asks that users follow their recommended practice) to being “opt out” (the user has to add override configuration). That feels like a change in which group is being prioritised.
Your comment “Although the actions you take wouldn’t change, things responding to your behavior might” under option 1 reflects this - I’ll now have to “opt out” of project-defined defaults if they don’t match my preferred workflow.
+1 on making this configurable - expecting a specific command to exist on PATH is (IMO) a stumbling block for many CLI-based interfaces. And don’t forget Windows, where writing a “custom script” and putting it on PATH isn’t easy - the OS doesn’t treat scripts as first-class objects (CreateProcess doesn’t recognise Python scripts, for example). The only valid type of script to the OS is a .cmd file, and those have problematic semantics (e.g., calling a .cmd file from within another .cmd file doesn’t nest, you have to use the CALL statement, which breaks the abstraction that this is “just another command”).
This needs to be emphasised a lot more IMO. People think of “write your own workflow tool” as a complicated process, far more inaccessible than “configure the system to match your workflow”. I certainly never even considered writing my own tool as an option here. I can come up with a bunch of objections, none of which are showstoppers, but they just reflect that “writing a tool” feels like a non-trivial barrier to entry.
To give an example, again on Windows, where would I put such a tool? There’s no standard PATH location for “user-developed utilities” - should I put it in C:\Windows\System32? Surely not. If I put it in my “general stuff” folder, it’s likely to get accidentally deleted in a periodic cleanup. And again, there’s the whole “making an exe is hard, and .cmd files are clunky” problem.
To be clear, I love this idea in principle. I’m just concerned that in practice, most people wouldn’t think of it, or find it particularly accessible[1]. So if it’s to be the solution for non-standard workflows, we need to think about how to promote it (or we need to reduce the barrier for building and deploying standalone utilities in Python, which is a much bigger issue that I don’t think we can tackle here…)
If I, as an experienced developer with familiarity with both Windows and Python, struggle to think of a clean way of implementing this, what hope do the sort of new Python users I worked with in my previous job have? ↩︎
) so I don’t expect to get away with doing nothing, but conversely right now I deliberately do almost nothing to configure my tools, and I get a “good enough” (for me) experience.