Setting up some guidelines around discovering/finding/naming virtual environments

Packaging
21

I only brought it up because @pf_moore mentioned he wanted something like this:

To me personally, it’s not very important, and something that could be quite easily handled with a dedicated discovery/cleanup tool. Compared to the UX benefits, it’s IMO an unnecessarily hard problem to solve well[1] in a more integrated fashion.

  1. including all the edge cases of project/folder/path mutation that happen completely outside of the purview of any specific tool, e.g. because the user just drag-and-drops a project folder somewhere else. ↩︎

22

If I understand you correctly, then, you’re proposing that whilst a “discovery-binary” based approach is the big-picture solution (in your view), the .venv simple-case is one that we could deliver first, and enshrine in a simpler PEP?

i.e., they would form complementary solutions, and e.g. .venv support could be realised in practice using the big-picture environment-discovery binary down the road?

I feel that there are both merits and demerits to this idea:

  • :+1: Smaller scope for initial PEP is more likely to win-over people if it does not prevent other users
  • :+1: Smaller scope PEP is easier to actually PoC implement
  • :-1: Enshrining .venv in its own PEP is a harder requirement than making it a default part of environment discovery[1]

If my reasoning aligns with yours, then why not limit the scope further to a single environment? To be clear, this would not be the end-goal for environment management (and indeed we’d immediately plan the next phase), just the smallest deliverable that helps the most people.[2]

  1. If only binary discovery were the official mechanism for locating venvs, then we might dissuade people from explicitly looking for the .venv location, and instead have them use this more general discovery pattern. This would mean a smaller “surface” to maintain. ↩︎

  2. This is a heavy assertion; that most users only need a one-to-one mapping of environments. ↩︎

23

I rarely use virtual environments stored outside of the project directory, so my needs are limited here (mostly only for cases when I’m using tools that insist on out-of-tree virtual environments). With that said:

  1. A .venv file seems like a good idea. Although existing tools may be confused by it being a file rather than a directory - how about a distinct name, like .venv-location?
  2. A common directory is more important to me than where it is. But note that the Windows “appdata” scheme uses per-application directories, which means I don’t consider that suitable (I don’t want to have to look in all of %APPDATA%\Hatch, %APPDATA%\PDM, %APPDATA%\VSCode…, plus the LOCALAPPDATA equivalents).
  3. I’d rather a file in the virtual environment naming its “owner”. Having to decode a hashed checksum plus partial name is not something I can do easily (if at all). My typical requirement is "what the heck was I using the environment C:\Users\Gustav\AppData\Local\hatch\env\virtual\b--0rwWMov\b for? Do I still have the source directory? Go on - from that filename, can anyone answer that question for me?

There’s lots of other things I’d like to do with shared environments (basically, task-based environments that can be used whenever appropriate for a project) but those are out of scope for this discussion (and don’t seem to be a common pattern people use). For this discussion, being able to programmaitically convert in either direction between the project location and the venv location is sufficient for my needs. Having all tools use in-tree venvs would also be sufficient, though :slightly_smiling_face:

2 Likes
24

Correct. Basically keep the simple case simple, while making the more complicated possible.

I’m honestly gone back and forth on this. One the one hand the .venv name is already ignored by a lot of .gitignore files out there. But you’re right there’s a potential for some tool breakage. My assumption, though, is those tools already have to be resilient in the face of .venv being anything else, not having appropriate permissions, etc., and so it being a file wouldn’t break too much. But maybe the discoverability of the file would be harmed if it shared the .venv name?

25

If it helps, I’d say that tools are much harder to fix than .gitignore files. As much as I like .venv, it’s already used for a different meaning. So let’s choose a new name, e.g.

  • .py-venv
  • .python-venv
  • .venv-link
  • .venv-location
  • .venv-path
  • .project-env

etc.

1 Like
26

I agree, I could be persuaded either way on whether the venv-file should be named .venv or something different. As you say, either is potentially disruptive.

I’m not sure .venv is necessarily in that many gitignore files. Virtualenv adds a .gitignore file inside any virtualenv it creates, and I tend to rely on that making virtual environments “invisible” to git. I suspect many other users might do the same.

:person_shrugging: People are going to have to adapt either way, I guess.

2 Likes
27

As mentioned in the OP, it is in the default one that most people use gitignore/Python.gitignore at 4488915eec0b3a45b5c63ead28f286819c0917de · github/gitignore · GitHub

2 Likes
28

I’ve been using a .venv file for years. It’s been in the .gitignore of almost every project I’ve contributed to[1]. I haven’t encountered any tool weirdness from it being a file.

  1. I don’t delete forks, so click to see N ↩︎

29

Paul’s point still stands: the path is in one super long ignore file that some/many people use, but not everyone.
git is bigger than github’s UI!

2 Likes
30

I mostly just copy pip’s, which didn’t have it for a long time. So older projects of mine won’t either. It really isn’t a big deal either way, though.

31

I first learned about the .venv file idea from a Pipenv bug report. The report contains several “prior arts”. It seems that the file has been used by a small but steady group of people for quite a long time (10+ years), who likely already helped ensured various tools have rudimentary support for it, including Pipenv. So I’d personally not worry too much about the file confusing existing tools; its support from older tools may actually be surprisingly good.

3 Likes
32

People seem to be okay with the file concept, but it’s hard to get a clear signal from the discussion. As such, here’s a poll to get a more direct response.

What do you think of the .venv file suggestion (and you can pretend we chose a file name convention you like)?

For those of you not familiar with this style of voting, +1 means you would champion its use, +0 is you would be fine with it existing, -0 is you would prefer it wasn’t there but you won’t be upset, -1 you will actively argue against the idea.

  • +1
  • +0
  • -0
  • -1

0 voters

33

There are so many conversations happening on this subject, so a preliminary apology if I’ve missed this in the current topic,

One thing I’m not clear on is how ambitious this topic is: are we just proposing to define a named file/directory that associates a venv(s) with a particular filesystem location? Or is it more ambitious to require that our tools use this e.g. pip?

34

Yes.

It would be nice, but I’m not even pushing for that based on how people reacted to PEP 704. At this point I’m just trying to figure out if e.g. VS Code and/or the Python Launcher supported such a .venv file would people view it as a reasonable thing or a horrible thing (.venv as a directory name is honestly already enough of a convention that the question about what to name a single, local virtual environment doesn’t interest me personally).

2 Likes
35

Before I vote, we are kind of hand-waving away support for multiple environments at this stage, right?

1 Like
36

One of the things I like about the .venv file (as opposed to the directory) is that it provides a way for tools to switch between multiple environments and let other tools (like VSCode) know about it.

3 Likes
37

I might be letting my poetry contributions getting the better of me here, but if .venv ends up being multi line, the first column would be path, the second column could be a valid python marker expression to separate out python version and maybe extras installed; that would extend fairly nicely from a single line file with no expression.

38

If you want to use multiple venvs, you can use them with name other than .venv. .venv can be just default venv.

For example,

$ mkdir .venvs/
$ python3.10 -m venv .venvs/py310
$ python3.11 -m venv .venvs/py311
$ ln -s .venvs/py311 .venv  # Use py311 default for now.
$ ln -sf .venvs/py310 .venv  # Change the default to py310.

Of course, some project management tool may automate this process and you don’t need to change .venv symlink/file manually.

1 Like
39

How would having a .venv file/directory interact with an already “activated environment” (assuming we can agree on what that means)? Would the .venv override the active environment, or the other way around? Similarly, what would be the result if you were in a subdirectory of the directory which contained the .venv, would it behave like git and use that? Git has Git security vulnerability announced | The GitHub Blog, which can be both useful and annoying behaviour, so clarifying what the expected behaviour in relation to multiple installs would be useful.

As an aside, if we’re going to be changing behaviour based on the presence of a file, it’s probably better if it’s not hidden (so venv rather than .venv), as otherwise it may be more of a source of confusion than not.

1 Like
40

Assuming I interpret Brett’s intention correctly, nothing. The .venv file would only be read by developer workflow tools, say Pipenv, Poetry, or VS Code’s Python extension). It does not change the behaviour of the command python and other tools that run “inside” the environment (most significant difference from PEPs 582 and 704).

1 Like