Thanks for the feedback!
That’s outside the scope of this PEP. Plus I want to get to the point that people stop wanting to activate virtual environments (it’s already optional).
I’m not aware of any way to make that work.
That also wouldn’t work as only shell scripts can permanently modify the environment. The closest is printing out the shell code to activate (like PDM does), but you still need to run it through your shell appropriately.
The 3.15 feature freeze was a couple of weeks ago, but hopefully this will be in 3.16.
Just to spell this out, I think you’re referring to things like
hatch run path/to/script.pyuv run path/to/script.pyIs that right?
And maybe pip can gain a similar command.
I actually think he’s referring to the even simpler approach of just calling the correct binary explicitly i.e. one of:
.venv/bin/python path/to/script.py.venv/bin/python -m dotted.module.venv/bin/pip (or otherwise declared console-script of any package)All of these will ensure the correct python is used with packages from the virtual environment on sys.path.
The main hurdle is programs that starts subprocess excpecting “python” to be on the path and being the same as the current interpreter (correct way is to use sys.executable).
Those examples are correct for the standard library venv module, but there’s no requirement that tools like virtualenv, hatch, uv or conda[1] offer those exact same paths or behaviours.
You can’t just assume that everyone is on the stdlib module. You may not have realised that’s what you’re assuming, but you are assuming it.
An equivalent but significantly different example, which is why I include it. ↩︎
I understand that the exact paths and locations (especially in relation to working directory or any concept of project-root) is free for tools to put where they want I was under the impression that the concept of the python binary checking for and recognizing a connected pyvenv.cfg (not sure about the details but assumed it was either purely relative to the binary or closest upwards) and if found using that to determine/customising sys.path was purely handed by the site module and common to the major tools.
That is I assumed creation of the directory structure and info in pyvenv.cfg can differ between tools of which the venv module is just one but that the bootstrap part given that when calling just the python binary was the core site modules responsibility.
(basically that any freestanding environment (not a zip-app or a packaged pex for example) was at least represented by some python executable which if invoked without specific other arguments/env-vars is supposed to run supplied modules/scripts in that environment).
I was referring to the former of where I want to get to, but the latter is how it’s already optional.
Steve’s point is you’re assuming all tools create a virtual environment (notice he listed conda which has its own environment type).
So the discussion in API for discovering environments for a project is over! Here is where things stand…
.venv directoryNothing has changed here. Some people have suggested dropping it, but I want the PEP to say that if you don’t know where to put a virtual environment, put it in a .venv directory in a location that makes sense (or by pyproject.toml).
My thoughts on this have changed a bit. Based on people wanting to support multiple environments as well as types of environments other than virtual environments (e.g. conda environments), I would change the proposal for a redirect file to be:
.python-envsBut whether I bother with this change depends on whether people like …
An alternative to the redirect file which gives more flexibility to tools and potential future growth is specifying a CLI command to launch workflow tool implementing what I’m calling the workflow server protocol (or “WSP” for short; for language/parser nerds this has a cute tie-in to Python as “wsp” is the common way to abbreviate “whitespace” in grammar definitions). This would be like LSP, but for Python workflow tools.
Workflow tools would spin up a JSON-RPC server and consuming tools would talk to them that way. The protocol would have endpoints for getting a list of environments, creating an environment, details of an environment, deleting one, etc. We could also go so far as to define an execution endpoint so that we abstract that out so the environment could exist remotely (e.g. over SSH), but still work when just spawning a subprocess.
You would define how to call your workflow tool in pyproject.toml, e.g.:
[workflow]
wsp = {requires = "hatch", cmd = ["hatch", "wsp"]}
There would also be a way to specify code in a repo or locally to use as the WSP server, e.g.:
[workflow]
wsp = {"script" = "./my-script-with-inline-metadata.py"}
The ways to specify Python code to run could be as broad as we choose to go (e.g. directory with a __main__.py file, zipapp, etc.), but I think the bare minimum is inline script metadata (on top of something on PATH which is already covered).
My guess is the PEP would not specify how consuming tools would let users override or provide a default workflow, but I would expect most consuming tools would. That would let projects declare they require a specific tool for some reason, but then let you define a personal workflow tool to use when nothing is defined by the project.
There would be an accompanying CLI tool written with the PEP which did 2 things. One, it would provide a CLI interface to WSP. That way you would not be required to launch a server and keep it running if it didn’t make sense (e.g. scripting something that needed to work with WSP). Two, it could provide a simple fallback workflow so those who just want something without writing their own workflow could easily get such a thing (with a way to opt-out of the fallback so those one-off commands can stay “pure” when desired).
Between the CLI tool and keeping a server running, that covers one-off things like scripts and AI use via the terminal to long-running things like code editors and MCP servers.
All of this would allow for future growth. You could imagine expanding WSP into installing Python, running tests, running tasks, etc. Basically the PEP would lay the groundwork on how to find and launch the workflow tool providing a WSP server and what’s needed to work with environments, and then future PEPs can skip the “how to specify what to run” part and just expand the protocol.
The question becomes is this WSP proposal a good idea and so should replace redirect files, or it’s too much and I should drop the WSP idea? Hopefully people get the basic idea and the broad strokes of the idea as I don’t want to start fleshing out all the details unless there’s consensus that it’s a direction people think is worth pursuing as it isn’t a small thing to do.
Why did you choose the last environment as the default rather than the first? Assuming that each venv is listed in a separate line (I think that’s what you meant by this but I can’t see it explicitly mentioned), I feel like it would be easier for those tools to only parse the file until encountering the first line ending compared to having to parse the whole file (or parse it backwards which seems like, at least slightly, harder thing to do).
You could imagine expanding WSP into installing Python, running tests, running tasks, etc.
Would WSP allow me to do these in any given environment (i.e. one not created by the tool)?
(Preparing a suitable environment would be my responsibility of course. Especially when it comes to native libraries/headers. I’d appreciate getting a list of Python packages to install though.)
If the tool is able to detect and use them, sure. I expect most tools will only want to deal with ones that it created (or recognises), since it may not know how to activate other tools’ environments properly.
The aim here is to not force every tool to standardise on venv (which they already don’t, but the original proposal was to force all users to standardise on venv environments, and now it’s forcing users into a way to specify their tool of choice, and forcing tools into a JSON-RPC API for front-ends to drive them).
Presumably for situations where you don’t have a pyproject.toml (e.g., a non-Python project with some Python utility scripts or whatever) the fallback would be the .venv directory?
I appreciate the simplicity of the redirect file approach. It makes specifying dependencies for various scenarios easy and consistent.
And as someone who stumbles into legacy Python projects a lot, I like idea of WSP as a way to understand how are project gets built. I seem to spend a lot of energy digging through config files for project, tox , pytest, Docker compose, etc. depending on whatever test / deploy automation was most familiar to person-in-charge.
IDK if it justifies the work involved for WSP, but it strikes me as an advantage over redirect files.
It’s easier to add to the end of a file than to the front.
If you’re worrying about how long it would take to split a .python-envs file on newlines then you probably should worry more about the disk usage by all those environments. 
And getting the last thing in a list is easy.
Basically you either make adding the default environment easy or you make it very slightly faster and easier to read the default easy. I went with the former as there’s a bigger chance you will do that in your shell by hand and code that will read the file via code is straightforward.
Up to the tool (i.e. the PEP won’t take a stance).
Correct (and all specs force something).
If you mean the fallback from the CLI tool I mentioned, then that’s my assumption/plan.
I think that’s why the Hatch maintainers have advocated for it.
FYI I want to give people to ask questions to make sure the options are clear, then I can run a poll to see where people’s preferences are.
No, I meant tool (IDE, or similar) behaviour. If I open my IDE on a directory containing a pyproject.toml, it will look up the WSP definition, start the WSP and query it to find the correct environment. If I open my IDE on a directory without a pyproject.toml, I’m assuming that the IDE will look at either .python-envs, or as a last resort use a .venv environment (and I’m asking that the PEP is clear that this is the expected behaviour).
One further question. This proposal doesn’t seem to handle my scenario, where I have particular workflow tools I want to use, for all projects. When the tool protocol was first suggested, the configuration was expected to be in the IDE (or other consuming tool)[1]. By now putting it in pyproject.toml, there’s no way to use the WSP on projects without a pyproject.toml, and there’s no way to specify my preferred WSP just once, for all projects. In particular, if I’m working on a project like pip that has its own pyproject,toml, I can’t modify that to reflect my personal tool preference. Have I missed something?
I’m fine with the WSP idea, but -1 on configuring it via pyproject.toml. If we extend the PEP further to define a mechanism for users to have some sort of global override mechanism for (parts of) pyproject.toml, then I’d be OK with that - but I think that such an extension is a pretty major scope creep, and would realistically be better as a completely separate PEP of its own[2].
That would be up to the tool, but I suspect most would.
Assuming we bother with .python-envs.
It isn’t currently mandated, but it isn’t blocked either. It would be up to the consuming tool to support such a global fallback. For instance, VS Code could let you set a fallback via your settings.json.
The PEP can say consuming tools MAY allow users to override what’s specified in pyproject.toml somehow.
So where would you expect projects to say they require a specific workflow tool then?
Are you suggesting another PEP to specify how tools that work with pyproject.toml data look up something with ways to override certain data at different levels (e.g. per-project, per-machine)? I purposefully didn’t want to go there as that feels like a tool choice, e.g. “pip can’t have pip.ini; everything must be in some pyproject.toml file somewhere”.
The problem I have with this is that pyproject.toml is a project choice, whereas which tool to use is a user choice. Obviously we’re all guessing, but my gut feeling is that tool maintainers wouldn’t feel comfortable allowing overrides of pyproject.toml (it’s a bad idea in many cases, and I’m not sure they’d see WSP as special enough to break that general rule).
I don’t know. But we need a solution for projects that don’t have a pyproject.toml, so why not in the same place that we use for that?
Precisely. So why are we putting tool config like a WSP definition into pyproject.toml? It sounds like we should be saying “tools must provide a way for the user to configure how to launch the WSP”, and nothing beyond that. But if we do that, we end up with [tool.vscode], [tool.vim], [tool.emacs], etc. when tools do want to use pyproject.toml.
Sorry, I know it feels like I’m being obstructive here. But I’ve never been a great fan of pyproject.toml being some sort of “general configuration file” for tools - we didn’t design it for that purpose, and it shows. IMO this is just exposing the weaknesses of that approach, and that’s why I’m (in effect) saying that if we want to start using pyproject.toml for WSP configuration, we should re-think the use of pyproject.toml for tool config, and design it properly. Yes, that means a new PEP, and it is also more work than I imagine you want to do here, but we’re going to have to face the limitations of the current approach at some point.
Just a quick reminder here - as I’ve already said, I don’t really have a very strong use case for this feature. My needs are almost certainly met by nothing more than the .venv naming convention and a way of telling tools to ignore what’s specified in pyproject.toml[1], plus a bit of manual effort. While I can see some value in the WSP approach, and a personal WSP server might let me fine tune my workflow a little, the reality is I’d probably never bother. So if you want to declare my use case as out of scope, and focus solely on the bigger examples like a hatch WSP and on projects that want to mandate a particular workflow for their users, then that’s fine. I’m repeating my requirements, not because I think they are specially important, but because I thought we’d agreed they were worth covering and yet they seem to have been lost somewhere in the process. All I want is to be sure that loss was deliberate, not accidental.
Something I wish we could require, rather than just hoping that tools will include it, but I don’t have the energy to fight that battle right now, especially against the inevitable “standards shouldn’t dictate tool UI” arguments. ↩︎
To an extent. If a project checks in a uv.lock file, for instance, that means using uv is not really a user choice.
As for overriding, we could say only certain things can be overridden, or let the override be a pyworkflow.toml to make it explicit what the file is overloading.
I don’t quite follow. There is no place for that sort of thing. So are you advocating for a 2nd configuration file that goes next to pyproject.toml? I don’t think a 2nd file makes sense, but I think you and I might be disagreeing on whether projects should be allowed to declare they want controbutors to use a specific tool.
Because that’s the file we have to provide configuration for projects.
Exactly. I don’t think projects want to guess what consuming tools you may have, so they want a universal place to put it. But you as a user will know what consuming tool you are using, so you can do that based on what your consuming tool wants you to do.
And yet pretty much every tool that I know of other than flake8 has a [tool] table (perhaps to match some custom TOML configuration file whic his usually just the [tool] table as a file itself). I will admit when I was helping write PEP 518 I was hoping it would eventually lead to a single file for projects to use to configure things.
I’m not seeing the specific concern you have.
I don’t think it’s been lost, we are just not seemingly agreeing on how strict/prescriptive the PEP would have to be to support the use case.
There’s still quite a lot you can do without using a uv-based workflow. Especially if you’re willing to use a one-off uv export to get a requirements file from the uv.lock. Obviously you’d need to follow the project’s uv-based workflow at the point where you want to contribute a PR to submit to the project, but for local work, you can avoid it. Whether that’s a good idea, or simply an example of me being stubborn, is a matter of debate[1].
Precisely. How do I configure a WSP in a (possibly non-Python) project that doesn’t have a pyproject.toml? The PEP doesn’t offer a solution for that at the moment, and unless you’re declaring those use cases as unsupported, it needs to. I don’t yet know what that solution will be, but why wouldn’t it also work for projects that do have a pyproject.toml? I think it’s going too far along the “standards don’t dictate tool UI” route if we leave the means of doing that down to the tool.
This is where I’m finding this discussion really frustrating. In order to have a good view of whether the proposal we’re discussing is going to work for me, I need to know how I’d use it in practice. And too many of the key questions I have aren’t answered, except by a vague “your IDE might let you do this”. Sure it might, but equally it might not - and frankly, in my experience many tools focus mainly on the “easy path”, ignoring users who don’t follow the straightforward approach. To put it another way, I think I have more chance of influencing the PEP process to consider my use case than I do of getting tool maintainers to do so.
I don’t follow that argument, sorry. Projects want to be able to define a tool-agnostic way of saying “this is our preferred workflow”. I can see that. But I use multiple tools (tox, nox, VS Code, vim, …) and I also want a tool-agnostic way to say “this is my preferred workflow”, and “please use my preference rather than the project recommendation”.
My point was that we can leave all of that up to individual tools (which will, if we’re optimistic, provide ways to do all of those things, but they won’t be tool-agnostic), or we can standardise the ways of doing things (which will be tool-agnostic, but requires us to cover the details in the PEP). Doing a halfway solution, where the PEP specifies some things but leaves the rest to tools, makes it extremely likely IMO that tools will simply implement the PEP and no more. You may be more optimistic than me on that final point, but you can’t say the PEP supports a use case by assuming that tools implementing the PEP will support that use case without the PEP requiring them to…
There’s a bunch of conflicting drivers that make pyproject.toml good enough for 90% of use cases. And that’s a major success, don’t get me wrong. But the remaining 10%[2] are hard to achieve without a fairly serious rethink. We may prefer to live with what we have (I’m fine with that) but we shouldn’t ignore its limitations.
I think it’s been lost because it will only be supported if tools implement something that’s not covered by the PEP itself. As I said, if you want to just say “yes, using a tool that complies with the PEP isn’t sufficient to be sure you can opt out of a project’s preferred workflow” then that’s your choice. But if you do, I don’t think it’s fair to say that the opt-out use case is supported by the PEP. That’s what I mean by being deliberate about the choice, rather than having it be accidental. The same goes for other aspects of my use case, like users having a tool-agnostic way to globally set their preferred workflow.
To give a concrete example, the workflow I want to represent is as follows[1]:
.venvs of the project directory/CWD. There are subdirectories for each Python version, and environments under that with arbitrary names.default is the one I want to use by default. If there’s more than one, the one with the latest Python version wins.py -V:x.y -m venv --without-pip .venvs/x.y/<env_name>.I’ll have a Python script (run using py /path/to/script.py) that implements a WSP server for this workflow, located somewhere on my system.
I want this workflow to apply for any tool I use (including the proposed CLI wrapper around the WSP), in every project I work on. Projects can be Python projects of my own with a pyproject.toml, 3rd party Python projects with a pyproject.toml that I can’t change, non-Python projects (for example a Rust project with a few data cleansing scripts and a couple of prototype implementations in Python), my “Scratch” directory with a bunch of temporary scripts, or my .local/scripts directory where I store personal utilities.
In the case of 3rd party projects, I don’t want any project workflow to be applied by default. If I need to follow the project recommendations, I’ll do so just when I explicitly need to. For example, if a project uses hatch, I don’t want my IDE creating environments outside the project directory, which is the default for hatch. Nor do I want to maintain a config for hatch (a tool that I don’t use) overriding this default behaviour. If I use hatch on the project, I’ll either temporarily add a “don’t use central environments” config, or let it use the defaults and manually delete any environments created once I’m finished. Furthermore, project recommendations might assume a setup that I don’t have - for example, assuming that I can run the hatch WSP using hatch wsp, whereas I need to use uvx hatch wsp - so there’s no guarantee a project configuration would work anyway.
Because I want all of this to work for any tool I use, I don’t want to have to set up tool-specific config for each of them. It’s too easy to set things up for the tools I normally use, but then have a rare situation where I need to use a different tool, and it doesn’t follow my normal workflow.
At the moment, I can’t see how to do this just using capabilities that are described in the proposal as it stands. So I’m at the mercy of what tools choose to implement, and I have to configure each tool separately.
As I said, if the answer is that any or all of this is too much to ask the PEP to cover, then that’s OK. Choosing the scope is part of writing a PEP. Just put the capabilities you don’t want to support in the “Rejected alternatives” section, and I’m fine with that.
This is more complex than my actual workflow, so that I can emphasise some particular points, but it’s pretty close to what I do in practice. ↩︎