PEP 832: virtual environment discovery

[ofek (Ofek Lev)]

If we’re not doing the WSP approach then I’d much prefer that this PEP not handle multiple environments at all and just have it be about .venv. It would be extra confusing for users who checkout a new project that uses multiple environments to then have no environment picker without first manually using the tool directly. It’s a bit better for users of other projects in the sense that a single environment workflow wouldn’t cause the expectation for a choice (but of course introduces the same amount of friction).

I’ll do a PEP for the WSP approach toward the end of summer.

Note: this isn’t a rash decision; I told Brett ~6 weeks ago that I would write such a PEP should this one not propose the workflow server protocol (WSP), a term for which I’m very grateful that he coined

[steve.dower]

@brettcannon

• No restriction on the paths; it’s up to the tools to figure out what a path represents and thus whether they support it

Does this imply that IDEs/(front-end tools) should look inside a path for a pyvenv.cfg (if that’s all they want to support) and ignore any that don’t have it? And back-end tools should be generating something specific so that they can recognise their own environment from path/file-system alone?

[brettcannon]

@ofek

I’d much prefer that this PEP not handle multiple environments at all and just have it be about .venv.

I’m willing to do that, but the way I’m interpreting what people have been saying is .python-envs has more support.

@ofek

I’ll do a PEP for the WSP approach toward the end of summer.

Notice I didn’t say I wouldn’t to a WSP PEP, just not in this PEP. I already started a discussion at Various ways of defining how to call a workflow tool for WSP to continue on in a different space.

@steve.dower

Does this imply that IDEs/(front-end tools) should look inside a path for a pyvenv.cfg (if that’s all they want to support) and ignore any that don’t have it?

Yes (which were covered by PEP 405 – Python Virtual Environments | peps.python.org). Same as conda environments have a HISTORY file (Lucas provided a link to a CEP earlier about that).

@steve.dower

And back-end tools should be generating something specific so that they can recognise their own environment from path/file-system alone?

That’s up to them if they even care. My assumption is any tool that uses .python-envs already has a specific place they put their environments, so they can tell just from the path which environments they created.

[kramkowski (Tomasz Kramkowski)]

@brettcannon

That’s up to them if they even care. My assumption is any tool that uses .python-envs already has a specific place they put their environments, so they can tell just from the path which environments they created.

I alluded to this in my first reply, but for tools that do want to know which environments belong to them, this is not necessarily trivial.

To confirm, we are happy with .venv taking priority, so we won’t necessarily run into this issue ourselves, but this came up internally so I feel it makes sense to mention here.

Environment locations may be configurable. If that configuration changes, references to environments at the old location could get “stuck” in the file.

Moreover, since .python-envs is intended to be shared between tools, it seems like it would need to specify a mechanism for safe concurrent updates. Appending paths could be atomic on some filesystems, but removing or modifying existing entries would normally require an operation which is not inherently atomic.

These are not impossible to solve, but they do add some complexity for tools that intend to maintain entries within a .python-envs.

[brettcannon]

@kramkowski

Environment locations may be configurable. If that configuration changes, references to environments at the old location could get “stuck” in the file.

True, but I still think that’s up to the workflow tool to decide how to manage that. An orphaned environment only comes up if:

1. A user or workflow tool change where they store environments
2. The tool doesn’t bother recording any details to know it’s their environment to maintain
3. No one bothers to clean it up, user or tool
4. The environment is somehow considered invalid while still being listed

And even if all of that passes, the user can just delete the offending environment(s) from the .python-envs file easily. This is no different then the situation happening but the user forgetting the environment exists instead of it being written down in .python-envs.

@kramkowski

Moreover, since .python-envs is intended to be shared between tools, it seems like it would need to specify a mechanism for safe concurrent updates.

I don’t think that’s necessary due to not being a common enough situation, but if people disagree then the PEP can suggest writers use something like the fcntl module for file locking while writing to the file.