API for discovering environments for a project

Developer Tools
41

I’m only responding to a few points to help keep discussion focused - not to ignore the rest of the posts, just to avoid repetition or contributing where I don’t have anything of value above what’s already posted.

My suggestion to Brett was to follow PEP 517, but I’m inclined to think this is the better interface in these days of “everything is Rust”. Still, wouldn’t be hard to make a Python API in a single script that translates to this - I don’t think we’re too worried about the “zero Python runtimes on the machine” case, but if we are, adopting an existing stdin/out-based protocol would make sense. As long as the interface doesn’t involve dealing with shell quoting/etc.

Requiring a Python interface also deals with a lot of the “search PATH” and cross-platform issues for more casually installed tools, such as using a setup script that’s in the repository itself. So I still favour that approach and figure we’ll eventually deal with people who want to use Python without getting Python with relocatable builds.

Agree with this list. I think the exec command needs some way to return a Popen-equivalent object so the caller can pipe stdin/out and wait for completion, but that’s the biggest complexity here. We can’t really rely on piping it all back via the interface, so it probably requires more pass-forward arguments to connect up streams.

What we can’t rely on is having the interface return the full command line and let the caller launch it, unfortunately. Too many environments require setup that can’t/shouldn’t just be faked with environment variables. So the direction you’ve proposed is the right one IMHO.

My suggestion was that “no workflow tool defined” implies a specific default workflow tool (to be implemented) that uses venv to create an environment at .venv and can launch it. None of which needs to be overspecified here, as long as people are happy to say “tools that normally use the project’s defined workflow tool may use this one if the tool doesn’t say so, rather than throwing up your hands and making your user figure it out on their own” (and since this would only be in response to the user clicking a “set it up for me button”, I’m quite okay with this as a default).

This is one of the things I think we don’t need to define, we just need to make sure we don’t forbid it. UI tools can offer whatever customisation they like, up to and including seeing the name of a defined workflow tool and using something else entirely (if they’re willing to deal with upset/confused users, but I can think of valid cases for this, mostly involving constrained enterprise environments).

42

The biggest problem with a Python API is deciding which environment the API should be installed into. There’s no obvious one, not least because this is an API for discovering Python environments. I guess we could require every tool consuming this API to host (or create and manage) a Python environment of its own to hold the API, but you’d then need a mechanism for the user to install code into that environment.

I’d like to have a solution to this (because it would be useful in many more contexts than just this one) but I fear it’s not as simple to solve as we’d like.

1 Like
43

Again, I don’t think we have to tell VS Code (for example) what to do here. They can create a private venv using the standard library and put the tool in there, and then provide it with the CWD of the user’s project and it’ll never know the difference. Or they can use pipx, or uvx. Or they can make the user choose a location, or they can try and use the system one (and deal with complaints when it fails).

The key point is, we can’t choose better now than they can choose when they actually implement it. So we only get ourselves into trouble if we try to design their product for them. Let’s just not do that.

44

Normally I’d agree with you. But if we assume @ofek is right, and “write your own wrapper that encapsulates your personal preferred workflow” is to be seen as normal behaviour, it’s worth thinking about implementation choices - not as things that need to be standardised, but as context for how well the proposal will work in practice (and food for the “How to teach this” section).

But if we define a standard that we can’t come up with a reasonable implementation strategy for, then we’re just dumping a problem we don’t have a solution for onto them. I don’t think it’s fair to demand that every proposal comes with a working implementation, but on the other hand, I don’t think it’s fair to make a proposal that you don’t have any decent implementation for.

We don’t design their product for them, but we should give them a starting point.

45

Was there something wrong with all of the five starting points I offered? If we choose one, then we’re effectively restricting them, and the one we’d choose for VS Code is not the same one we’d choose for GitHub Actions (for example), so how deep do we have to go vs. saying “the name maps to a package name on PyPI, how it gets installed is up to you”?

47

Ah, this is where we are misunderstanding each other. I’m trying to understand how the suggestion that @ofek made for my workflow would work. In that situation, the idea is that I’d create my own personal wrapper that implements the discovery API in a way that matches my workflow. Having to publish that wrapper to PyPI is a bigger barrier than I think is reasonable for what is basically just “customising things to support a workflow that isn’t just how one of the big tools does things”.

1 Like
48

Sorry, I didn’t spell out the second option of “the name maps to a location in the current *wave hands*workspace/repository/whatever”, which lets you implement the API directly in your workspace (per-project), and I also didn’t spell out the “tools can totally ignore the package on PyPI if the user tells you to do something else” because I’ve mentioned that in basically every other post. But that is how you would implement your own personal environment manager for all projects regardless of their declared metadata - you literally just ignore the metadata.

There’s still a benefit, in that tools that know how to use the new API can probably use any tool providing that API, whereas today they have to implement support for each tool individually and so will never support your custom one.

But we can’t force third-parties to implement the feature to let their users override metadata in the projects they use, so you won’t see it spelled out in a spec. The best we can do is say that it’s only as binding on the end user as anything else in Python is, which is “not very”.

Tools should be respecting that Python idiom rather than strictly enforcing standards against their users. It’s unfortunate that we’ve put out a number of very strict sounding PEPs recently that don’t seem to allow that leeway, because it trains tools to ignore users in favour of the “official” metadata[1], but we aren’t going to fix that by spelling out every possible preference of every user to make them all official. We need to just encourage the tools we can influence to let their users be the boss and do whatever they need to make things work.

  1. One of my major reasons for disliking static typing. ↩︎

49

I still don’t see how that lets me configure tools to use my workflow in all my projects (unless I specifically choose a different config). I don’t want a workflow API script in every project, just one somewhere on my machine to affect everything where I don’t override it.

(Well, what I actually want is for my personal workflow to be supported without any config, but that’s not realistic once we go beyond the .venv convention for identifying the default virtual environment, i.e., PEP 832.)

1 Like
50

Yes, you want a default setting in your UI tool (IDE or whatever). See every part of my post other than what you quoted, since that’s the bit where I talked about this.

Right, I get that it’s a personal/selfish request for all tools to default to what you do, but that’s why the rest of us are saying that we’re not interested in defining a spec solely for that single workflow. We’d rather define a generic interface that enables interesting tools to be built on either side of it. Saying “if .venv exists you can assume it’s a virtual environment” barely qualifies as interesting for tool building, since any tool that will do that can already go “if .venv/pyvenv.cfg exists then I assume it’s a virtual environment” and you won’t notice the difference. It’s just not worth defining, and probably the only reason Brett even wrote it up is because people wanted to argue over the .venv bit.

51

I think we should be quite worried about requiring Python for environment management and other project workflows as that introduces a bootstrapping issue. In the early days of Hatch I learned how critical of an issue that is from watching users.

Ah, I see! I’d be in favor of that as long as we keep it minimal like build.

The uvx example Paul provided is quite common which made me consider special casing that but I find this view compelling. I change my mind :slight_smile:

Yes, although we should offer project-local override configuration it shouldn’t be required for configuring a more broad override nor for the actual implementation of workflows.

It’s not just a barrier for user customization but also for workflow tools that intend to be published. For example, it doesn’t make sense for maintainers of a tool written in an another language like uv to create a Python package just to manage itself and accept the compatibility hardship that naturally comes from such dual versioning.

1 Like
52

Like many of the other concerns, I mean we’re pushing it to the front-end to solve, rather than expecting the backends to solve it. So if VS Code is going to use this API, it is responsible for solving bootstrapping to get enough Python to use the interface.

Regular humans don’t use the API themselves, so they are never expected to become the bootstrappers. And anyone who bypasses/ignores the API is in exactly the same position as they are today. So while it does put the work somewhere, it’s clear about who is responsible, and the burden is in the right place for user customization/overrides (e.g. VS Code already knows whether the user has a default/preferred/system/bundled Python runtime, while an API wrapper doesn’t have to figure that out).

That’s the wrong analogy - build is the front-end, and I’m talking about a default backend. So it’s “minimal like flit-core”.

I think it totally makes sense to put the “burden” of publishing and versioning on the tool that is being published and versioned.

I’m open to other discoverability mechanisms besides “must be on PyPI”, but I don’t see why we should increase the complexity/decrease the reliability for everyone else for the sake of the handful of tools that would form the backends here.

One reasonable example - uv could say “when a workflow references uv-api-wrapper and you know that uv is already installed, you can find a local copy of the wrapper at <some logic>”. And I’ve previously mentioned specifying a relative path to the config file for in-tree tools. The actual interpretation of the name can vary in ways that are more convenient, provided the default interpretation (i.e. pip install <name>) also works.

Besides, I expect most tools are just going to subprocess.run(...) themselves out of the wrapper anyway (possibly with a download step right before it). If your command line is changing frequently enough to become a versioning issue, your users are going to quit using you.

53

Could you please concretely describe what you think the Python API approach would look like here with an example?

54 
# pyproject.toml, presumably
[workflow]
requires=['my-wrapper-module']
api='my_wrapper_module:WorkflowAPI'

# my_wrapper_module.py

class WorkflowAPI:
    def __init__(self, cwd):
        self.cwd = cwd

    def create_environments(self):
        # our tool defaults to our config file in cwd
        subprocess.check_output(["tool", "create"], self.cwd)

    def list_environments(self):
        # request a list of envs our tool knows about
        return json.loads(subprocess.check_output(["tool", "list", "--json"], self.cwd))

    def run_command(self, env_id, args):
        # Run our tool to calculate what an "activated" environ looks like,
        # because that's all our tool needs. Other tools might do other things,
        # or they might have a suitable 'tool run ...' command
        reqd_env = json.loads(run_and_capture(["tool", "calculate-env", env_id], self.cwd))
        return subprocess.Popen(args, cwd=self.cwd, env=reqd_env)
55

Ah okay, thanks for clarifying! I had imagined based on a previous comment of yours that you were now in favor of the language server-like approach and just wanted the Python API for a more deterministic way of acquiring the tool. Now I understand that you are still advocating for a PEP 517-like approach.

I’m overall against a spec that requires an existing Python distribution in order to install and run something that’s meant to manage Python environments. However, you are right that a human wouldn’t use the API themselves and therefore I am slightly less hesitant. I do still think that the user experience may be degraded in at least the following ways:

  • I assumed that AI agents would be a consumer and, after some time, models will know how to interact with projects out-of-the-box without repo-specific instructions for managing environments, running configured tasks like test, etc. Requiring Python distribution and environment management for using the workflow tool adds either extra context bloat and processing time, or maintenance effort for the agents to have a built-in deterministic fast path. Or, folks would come up with various skills of varying quality that users would be subjected to. It’s likely that the model would be intelligent enough to know that this approach wouldn’t be optimal and they would scan the repo contents for the commands that humans are running, which invariably would use what’s on PATH anyway.
  • Although VS Code has significant maintenance resources, other consumers are unlikely to be as fortunate and managing Python distributions imposes a cost regardless of how great one perceives the cost to be. This has the potential to reduce adoption of the functionality and as a result limits the benefit to users.
    • Not a user issue but I imagine that Debian, Conda, etc. would not like having to go through supporting another standard that may cause bootstrapping issues. Although it wouldn’t be as bad as recursive build dependencies, I could see trouble arising when they start using the workflow tool for testing projects and there are cyclic dependencies for that post-build validation step.

If you still have this opinion, what do you think about the following?

If the requires field is defined then consumers SHOULD add a directory to the front of PATH for all invocations of the workflow tool. The directory MUST contain only the contents of what a satisfactory dependency resolution would install to the scripts directory of a virtual environment.

Here we both compromise:

  • You no longer have a Python API but maintain the ability to dictate versioning requirements and reduce nondeterministic tool execution.
  • I no longer have a spec that’s completely free of a dependence on Python distributions but maintain the CLI-based API and support for users like Paul.
56

I really dislike any assumption about file-system layout here (or assuming that everyone uses/looks like venv).

What you’ve proposed is about the only thing that could work, but the only way to figure it out reliably is to run the Python runtime, at which point all the other concerns also go away and it may as well be a small Python shim that knows how to set things up for the particular workflow tool being used.

And yeah, I forgot about the LSP-like proposal. My earlier comments on that still stand - it’s nice, but could easily be another shim on top of the Python API that provides the LSP translation (essentially a remote procedure call layer) in the exact format desired by the tool using it, just as a Python API could be the layer that wraps up the LSP model so that Python hosts can use it more conveniently.

My preference is for the Python API because it’s the most convenient for us to define and specify and know that it allows implementers the greatest level of flexibility for cross-plat/unknown-plat/etc. support. (And I also still believe that relocatable builds will happen, which reduces the eventual bloat of a bundled runtime to one of the smallest language runtimes currently in use today - it’s about 10MB on Windows.)

57

I don’t think this will ever be useable within the repackager’s world. Repackagers generally will always have to do the same unpacking/reverse engineering down to the most low level commands that the minimalists like me end up doing. Any tool that needs the internet can’t be used. Build isolation blocks distribution-required patches to build backends. Using venvs for testing defeats the purpose of system level testing in the first place since you’re not actually testing with the dependencies you’re deploying with.[1] Anything that tries to detect or install Python installations is self defeated in the same way. Lockfiles or pinned dependencies are unusable in an ecosystem where only one version of each package is available. Any test command+dependencies list that uses linters (pytest-flake8) or formatters (pytest-black), coverage, profiles, benchmarks, heavy fuzz testing or multiple environments needs to be replaced anyway.

So tl;dr, I wouldn’t worry about it.

  1. Although sometimes Fedora do it anyway – I personally thing they’re nuts… ↩︎

58

That’s fair, I didn’t adequately distill the essence of the requirement. How about this?

If the requires field is defined then consumers SHOULD add a directory to the front of PATH for all invocations of the workflow tool. The directory MUST contain all files contained within the scripts subdirectory of the .data directory of all wheels that are selected from a satisfactory dependency resolution.

I care little about what is expedient for us and a lot about what is best for users. The LSP-like approach provides a much snappier experience for all users, allows users like Paul to more easily tailor their workflow to meet their needs, and potentially gives users more options in the long term due to the reduced requirements of implementation. I think that approach provides a better experience until I hear compelling evidence to the contrary.

Surely that can’t be true since the number of supported Python platforms is necessarily less than a CLI written in something like Rust?

1 Like
59

Surely that can’t be true since the number of supported Python platforms is necessarily less than a CLI written in something like Rust?

Did that change recently? Last I recall, there were a lot of platforms with a working C toolchain capable of compiling CPython but no Rust compiler yet.

I guess it all depends on what you mean by “supported” and “platform.”

60

I was thinking mostly in terms of not having to get path/shell quoting or environment rules correct at the “frontend” level, delegating that to the “backend”, which then only has to worry about its own rules (if the frontend was doing it, it has to worry about the rules of any arbitrary backend).

You’re going to have to spell this one out for me, perhaps with an example. I’m really not clear how “implement LSP” is easier for Paul than “implement a class with 3-4 functions” (aka my example).

1 Like
61

I think the point is that I can write the LSP however I want (python, rust, whatever) and just supply the command line needed to run it. But if I write a Python class, I need to make that loadable in a Python interpreter supplied by the tool - and every tool might need me to do that differently, so there could be multiple copies of my Python file scattered around my machine.

1 Like