I might be overthinking this, but I realized after reading PEP 832: virtual environment discovery - #136 by kramkowski that the original “Python API or JSON-RPC” was a bit too restrictive as it assumed having a long-running process was the best outcome. If we changed that to a possible outcome, the options open up quite a bit.
For the following options, I had three rules for myself. One is something had to be able to be written down in pyproject.toml. Two, discovery of support provided by a workflow tool had to be possible as well as listing individual feature support in pyproject.toml. Combined those two options would allow for fine-grained overrides as covered in PEP 832: virtual environment discovery - #139 by ncoghlan . Three, there was some way to represent running a Python process (even if that meant stdin, stdout, and stderr was simply redirected).
With that in mind, I realized some options were:
JSON-RPC
Python API
Listing the full CLI call as defined by the tool (e.g. tool venv --extras <extras>)
Full CLI as defined by a spec (e.g. tool wsp-venv and the spec says how all the CLI arguments work)
JSON via the CLI (e.g. tool wsp-venv <json>)
JSON via stdin (e.g. echo <json> | tool wsp-venv)
These all have drawbacks. JSON-RPC is the most complicated and a bit overkill if the common case is one-off commands, as well as not easily callable by hand on the terminal. The Python API takes a bit more to get running and is also hard to call by hand on the terminal. Listing the full CLI call requires coming up with a way to say where to insert arguments and how to handle optional arguments (e.g. how to do you say what extras are to be installed into an environment, or list no extras), as well as limits on CLI argument lengths. Full CLI as designed by a spec has the CLI legnth limit issue). JSON via the CLI has the same length limit problem as a CLI call. JSON via stdin takes a little bit of work to call by hand and a bit more effort to work with compared to anything being passed via the CLI. And all of the non-server cases have the drawback of overhead from each call if a server scenario is to be the common case.
I also realize the “WSP” acronym might lose the “S” if some command-line approach is taken.
For the “JSON via stdin” approach (which I would favour as a good middle ground between fully specified individual commands and a long lived server process), a traditional workaround for some of the limitations of doing that on an ad hoc basis is to also accept the input via a named file.
I am strongly in the JSON-RPC camp at this point in an AI world. It’s an already established protocol that Agents understand better than CLI commands. We are already in the middle of the shift where the majority of tool interactions are progressively happening through agents and less direct through developers themselves.
I can see the argument for JSON via STDIN using a JSON RPC based schema as that is still close enough that it could be understood by agents and humans alike for ease of making calls.
I still think we should first figure out what capabilities this workflow protocol should expose before nailing down the mechanics of the protocol: if there’s a feature that would benefit from stateful interactions with the tool, suddenly a long-running process becomes much more reasonable.
But also pretty trivial to have a single tool that can load and invoke any implementation of the API (it could even be the same tool that can load any implementation of the API and expose it as any form of RPC we like). That’s really not possible with any of the other approaches.
It’s also the easiest to define, since we have full control of the semantics. It’s very easy to define a CLI format that is simply infeasible on some shells, and that then becomes a specification bug rather than an easily fixed implementation bug.
(JSON-RPC of some variety is also fully controlled, but it puts far more challenging lifetime controls somewhere between the spec and the implementation, so we’ll spend/waste a lot of time making sure the specification is good enough compared to a more direct API spec.)
True, but for some reason it feels weird to me. But I guess any other “big JSON” blob is no worse.
Someone privately went as far as to suggest defining it in terms of MCP (I said no), so you’re not the only one wanting a server solution for AI to interact with as needed.
Yes, that would be one way of trying to support both approaches.
If we’re only talking environments, I’ve also seen:
Create an environment
Delete an environment
If we expand to what WSP could be used for, then task execution has been mentioned. You could even go as far as thinking of anything a majority of workflow tools do (e.g. install Python).
I’m reading this as saying the Python API lets you support either approach since you can easily put it behind a server or run some code once and be done without subprocess overhead (but I could be wrong as I seem to not be inferring what people mean very well today). That might lead to some folks wanting some flag/hint to say “running in a long-running process” to know some caching and such is worth it, but that’s a “later” problem.
Hence why I wouldn’t want to allow for shell execution and only allow for a list of arguments. But I do agree with your point as even with that in place it would still require agreeing on how flags should work and most people use a library for CLI parsing, and so what the greatest common denominator is in that instance would be something to work through.
If we define a Python API, then that flag/hint becomes a personal problem (rather than a specification problem) Specifically, the person who writes their little harness that converts the Python API into a running server.
The only real pushback worth considering here is the people who want[1] a tool with zero Python in it. I get where they’re coming from, but I don’t think we should be actively/officially pursuing non-Python solutions for Python problems. I’d rather end users have to install at least one copy of Python as part of setting up their integration than try to encode our semantics into someone else’s metadata.
If it gets built into the Python install manager, it’ll be written in Python regardless, because I’ve jumped through the hoops to make sure we can do that even if the machine doesn’t have a Python install yet.
Presumably they want to create such a tool, since their users who are actively trying to user Python probably aren’t so concerned. ↩︎
I don’t think the tool needs to have “zero Python in it” for there to be a significant benefit to being able to examine metadata and make decisions without invoking a Python interpreter. I think we’ve demonstrated that there is significant user demand for tools with such qualities, it’s not just a desire of a niche subset of tool authors.
Very true, but admittedly the things being discussed for WSP are about developing or running Python code which inevitably lead to a Python interpreter being installed and run. Now if we were talking about something like an index server or something that stayed entirely in the realm of metadata I think you’re right that there’s not having an expectation of a Python interpreter being installed is reasonable.
Are you saying “zero Python code is good” or “fast tooling is good regardless of the language used”? If it’s the former then I’ve spent too much of my life trying to improve Python . But if it’s the former then that’s a fair assessment.
Let’s talk about potential performance implications here.
For the CLI approaches, you will be spawning a subprocess for every call into WSP no matter what. After that it’s up to the tool to make things fast.
For JSON-RPC, it’s spawning a subprocess and anything people set up for such a server. I don’t see any reason, though, why that can’t be optimized for fast start-up by avoiding doing things that might not be necessary if the process doesn’t stay up as a server (e.g. not setting up file watchers).
For the Python API, it’s a bit different. If the caller is Python, then it could import the WSP Python code and run it, avoiding creating a process. If the caller isn’t Python, though, then there will be a subprocess no matter what. As for the workflow tool implementing WSP, if it’s in Python then that doesn’t change anything. But if it’s in e.g. Rust, then it either would provide an extension module or a Python wrapper that uses a subprocess itself to call the tool. That latter case would be potentially 2 subprocesses, but that’s not a given as the extension module may be an option or the caller itself is in Python.
I think the question is how performance-sensitive will the actions WSP performs are? If we are all thinking that 25-50ms of overhead just to spawn a subprocess is going to be costly, then that might mean JSON-RPC is the best as it’s potentially a single cost for however many calls that will be made and avoids any perf from being written in Python (otherwise the Python API could be run inside a server). If we think 50-100ms overhead is going to be too much (for 2 subprocesses), then the question is whether the risk of a caller into WSP not being in Python, the workflow tool not being in Python, and the workflow tool not wanting to do an extension module and thus using subprocesses itself is enough to kill the Python API idea. Otherwise I don’t think using a Python API is inherently a performance concern (is just might not be the shape we want WSP to take for other reasons).
I won’t have time to comment until the weekend but this is an incredibly bad idea. What is imported may corrupt state which breaks guarantees for the user and is a security risk. Also, that would be impractical for a long-running process because of the memory overhead that comes with each import and whatever the server happens to do over time. Separate long-running processes have similar memory implications but that model doesn’t risk the integrity of everything else like the single-process approach.
I can only really see this being used for one-off actions[1] or interactive actions[2], which means the definitions of “costly” are limited by human perception (typically in the 100’s of ms rather than 10’s of ms).
50-100ms is negligible until you’re trying to launch processes in a tight loop, and if the tradeoff means that users can’t easily write their own API adapters, I don’t think it’s worth it.
Launching large numbers of subprocesses is a special case, and you should get the best performance by hard-coding parts of your environment, rather than trying to use a generic API. The generic API just has to be successful, not ideal.
e.g. “launch the test suite” or “launch the linter” or “launch the app” ↩︎
e.g. triggered by a user typing/clicking a command ↩︎
Specifically, the person who writes their little harness that converts the Python API into a running server.