PEP 832: virtual environment discovery

[brettcannon]

@ofek

I’d like to call attention to what I think is a much better proposal described in my comment above.

Which one? You have at least 2:

@ofek

I view the best course of action that would satisfy everyone to be the construction of an API for environment managers themselves. There was actually a proposal several years ago (pre-uv though) with feedback/buy-in from several maintainers.

and

@ofek

So, one possibility is to introduce a [workflow] table to pyproject.toml. In that table, you can specify a command to use to get the virtual environment path from stdout when the status code is 0.

I think you’re talking about the first one based on other things you said in the post, but I want to be clear first before potentially taking this conversation in another direction.

@ofek

Having knowledge of where an environment resides is an anti-pattern.

I disagree based on one of the key motivators for me bothering with this PEP. When I was the dev manager for the Python extension for VS Code, we got 2 constant complaints: “why can’t you find my environment?”, and “why is start-up slow?”. The former led to the Python environments extension and providing an API for other extensions to use to provide such details.

The perf issue is supposed to be addressed by this PEP. Now you may be asking why knowing the location is important for this? Well, some workflow tools are not exactly fast at listing all of their available environments (and I will not name-and-shame them publicly, so please don’t ask). Tack on users who have environments in the hundreds (and that’s not an exaggeration), and slow workflow tools can lead unacceptable start-up or on-demand listing their workflow tools. For VS Code we had to come up with our own caching, asynchronous refreshing of that cache, having a way to forcibly refresh, etc.

So hopefully that helps explain why I don’t think knowing where an environment is kept is an anti-pattern.

@ofek

What we all actually want, although it’s a bit more design work, is a way to perform actions against environments like listing the installed packages and running commands with its scripts directory first on PATH.

I need to know where it is on disk to read the source for auto-complete after providing users a list of environments to choose from (after going with the default so I don’t have to bother them by asking).

@ofek

I understand that we shouldn’t let the perfect be the enemy of the good but I don’t think the proposal is good enough to warrant the downside of pushing back the timeline for the proper solution.

You also forgot about my patience and time. I’m obviously willing to talk things out, but I am only willing to for so long before I either go ahead with the PEP or walk away entirely.

And to think this was supposed to be the “easy” workflow PEP for me.

@cjames23

I also wonder if we can take it a step further here and say instead of tools being called to write to stdout for the JSON that Ofek proposed, tools automatically write a .workflows-{tool-name} or some other named JSON file that lists everything in the same schema that was proposed to the project directory at the same level as the pyproject.toml

It’s an interesting idea!

@cjames23

Like any PEP tool authors are free to implement or not implement support for it.

Except if the majority don’t bother then we end up like PEP 708 and being rejected 3 years after acceptance due to lack of uptake.

@cjames23

From this perspective and the fact that hatch already has something of a command that is used by IDEs today to discover environments I do not think the costs here are a significant burden for us.

That’s good to hear! That at least suggests to me that the idea isn’t infeasible.

@cburroughs

The term “project” is used in several different ways by existing PEPs. A few examples with emphasis added:

There’s also PEP 621 which added the [project] table. I’m also an author of that PEP and another one you listed, so I’m aware of the lack of hard definition of “project”.

@cburroughs

This PEP uses project_root in a new (and different) sense to mean where a virtual environment marker file is located.

I would invert that and say the environment marker goes where you consider your project root. So the .venv file is not defining where a “project root” is, it’s the other way around.

@cburroughs

I think a functional term (examples: venv_marker_location, search_root, venv_anchor, discovery_dir) would be both clearer for the API this PEP proposes, and minimize confusion with the existing body of PEPs.

I’m not going to make any promises about changing it, but I will think about it (and if anyone else has an opinion they can speak up).

@henryiii

What does supporting “.venv” mean for something like tox, a multi-environment runner?

Based on what @bernatgabor has told me, some tox users have defined a “dev” environment in their tox config for this sort of thing.