So the discussion in API for discovering environments for a project is over! Here is where things stand…
.venv directory
Nothing has changed here. Some people have suggested dropping it, but I want the PEP to say that if you don’t know where to put a virtual environment, put it in a .venv directory in a location that makes sense (or by pyproject.toml).
Redirect file
My thoughts on this have changed a bit. Based on people wanting to support multiple environments as well as types of environments other than virtual environments (e.g. conda environments), I would change the proposal for a redirect file to be:
- A file named
.python-envs - UTF-8 encoded to contain paths which each represent an environment
- The last environment listed can be used as a default option if a tool doesn’t want to ask the user which environment to use
- Relative paths to the file location are allowed
- No restriction on the paths; it’s up to the tools to figure out what a path represents and thus whether they support it
- Virtual environments SHOULD be recorded as the directory containing the environment
- Duplicates in the file are fine so that it’s easy to append to the file without much thought (tools can remove duplicates if they want)
- Trailing line ending is still okay
But whether I bother with this change depends on whether people like …
CLI API
An alternative to the redirect file which gives more flexibility to tools and potential future growth is specifying a CLI command to launch workflow tool implementing what I’m calling the workflow server protocol (or “WSP” for short; for language/parser nerds this has a cute tie-in to Python as “wsp” is the common way to abbreviate “whitespace” in grammar definitions). This would be like LSP, but for Python workflow tools.
Workflow tools would spin up a JSON-RPC server and consuming tools would talk to them that way. The protocol would have endpoints for getting a list of environments, creating an environment, details of an environment, deleting one, etc. We could also go so far as to define an execution endpoint so that we abstract that out so the environment could exist remotely (e.g. over SSH), but still work when just spawning a subprocess.
You would define how to call your workflow tool in pyproject.toml, e.g.:
[workflow]
wsp = {requires = "hatch", cmd = ["hatch", "wsp"]}
There would also be a way to specify code in a repo or locally to use as the WSP server, e.g.:
[workflow]
wsp = {"script" = "./my-script-with-inline-metadata.py"}
The ways to specify Python code to run could be as broad as we choose to go (e.g. directory with a __main__.py file, zipapp, etc.), but I think the bare minimum is inline script metadata (on top of something on PATH which is already covered).
My guess is the PEP would not specify how consuming tools would let users override or provide a default workflow, but I would expect most consuming tools would. That would let projects declare they require a specific tool for some reason, but then let you define a personal workflow tool to use when nothing is defined by the project.
There would be an accompanying CLI tool written with the PEP which did 2 things. One, it would provide a CLI interface to WSP. That way you would not be required to launch a server and keep it running if it didn’t make sense (e.g. scripting something that needed to work with WSP). Two, it could provide a simple fallback workflow so those who just want something without writing their own workflow could easily get such a thing (with a way to opt-out of the fallback so those one-off commands can stay “pure” when desired).
Between the CLI tool and keeping a server running, that covers one-off things like scripts and AI use via the terminal to long-running things like code editors and MCP servers.
All of this would allow for future growth. You could imagine expanding WSP into installing Python, running tests, running tasks, etc. Basically the PEP would lay the groundwork on how to find and launch the workflow tool providing a WSP server and what’s needed to work with environments, and then future PEPs can skip the “how to specify what to run” part and just expand the protocol.
The question becomes is this WSP proposal a good idea and so should replace redirect files, or it’s too much and I should drop the WSP idea? Hopefully people get the basic idea and the broad strokes of the idea as I don’t want to start fleshing out all the details unless there’s consensus that it’s a direction people think is worth pursuing as it isn’t a small thing to do.