Trying to come up with a default directory name for virtual environments

Packaging
1

The question

On Twitter I asked people for suggestions for an environment variable name to store the default/typical name they use for virtual environment directories:

That led to @uranusjr suggesting we should just choose a standard name for virtual environment directories:

Then @pradyunsg agreed:

And so here we are, with me kicking off a more formal discussion of what a standard name for virtual environments should be for tools that just want a default name for users.

My answer

Looking at tools that create virtual environments for people, pipenv uses .venv. Poetry has a setting to create the virtual environment locally, but it isn’t specified where the directory ends up being. If you dig into the code it turns out to be .venv:

So it seems tools have chosen .venv. I’m also a fan as well because:

  • It doesn’t clash with .env files.
  • It’s hidden by default as it’s almost an implementation detail and not part of your workspace.
  • Tools like ripgrep will ignore it.

Other ideas

In my poll on this question the most people voted for venv (although it wasn’t a majority). I think people prefer venv instead of .venv because they don’t like it being hidden while still being obvious what it is. I’ve also heard others suggest __venv__ so they can ignore __*__ in their .gitignore files and get __pycache__ ignoring for free (although that clashes with the __app__ name used by Azure Functions so I have a work bias against that idea).

An enhnacement to the idea

@uranusjr also suggested supporting the idea of a .venv file which pointed to a virtual environment directory:

It was also liked by @pradyunsg:

The idea is basically if .venv is a file instead of a directory is a a one-line file which points to the directory of a virtual environment. I’m personally fine with this idea as it leads to some flexibility as @uranusjr points out:

The reason behind asking this

The entire reason I’m bringing this up is I would love to have the Python Launcher for UNIX automatically select an in-workspace virtual environment even if it isn’t activated. It already picks up an activated virtual environment, but in my typical dev workflow I have a .venv directory with a virtual environment sitting there, so why even have to care about activation when I run py? I obviously want that .venv/bin/python so even have to do anything to select it? It’s the ultimate laziness factor in launching Python; I want two letter to just know what I intend. :grin:

But I’m also a lazy, time-constrained open source developer and don’t want to have to implement any logic to search for an appropriate virtual environment if there are multiple options. :slight_smile: Plus I’m paranoid about a performance hit if I have to search a ton of directories. So I have been trying to think of a way to tell the Python Launcher “use this virtual environment”. But now @pradyunsg and @uranusjr have inspired me to simply simplify things and JFDI by choosing a name and stop worrying about fancy configuration options. Hence this post. :grin:

So does anyone have any arguments against going with .venv as an informal community standard for naming a virtual environment (both a single one or a file pointing to a virtual environment)?

9 Likes
2

Surely searching just one level deep for pyvenv.cfg isn’t unreasonable? Or one level deep for bin/activate? You’ll have to bail out in the presence of multiple environments, but having a fixed name is going to have the same effect anyway.

You’ll want an answer ready for the people who come in complaining about script entry points too. I don’t have an answer they like, but I’ll support whatever ideas you come up with!

(Oh, and please spec it out well enough that it can be copied for the Windows py.exe or even Python itself.)

1 Like
3

4 posts were split to a new topic: What to record in pyvenv.cfg

7

An argument I haven’t seen here yet, not so much against .venv but for __venv__: the dunder convention is used for standardized special names in CPython: __pycache__, __init__.py, __main__.py. It’s good that tools don’t use dunder names for their own thing, but if you’re going for consensus/standardization, maybe dress the part?

6 Likes
8

As we discovered with tox the .venv suggests it’s private. So would not recommend that at all. Some tools use .venv but in that case the intent is that the user will not interract with the venv directly, but rather through the tool (e.g. poetry). I think the venv is what people generally prefer if you have to pick one. I kinda like the __venv__ idea
 though that now means that you need to press two buttons to enter it into a shell to trigger autocomplete; furthermore it might clash with __pycache__ making it even harder to autocomplete it.

2 Likes
9

Unreasonable? No. Do I want to? Also no. :wink: Ambiguity and trying to communicate the search algorithm is always a concern of mine, so I much prefer an “explicit is better than implicit” approach and keep the search algorithm as simple as possible.

What does that have to do with finding a Python executable? If an entry point doesn’t have a -m equivalent then activate the environment. This is meant to solve the world’s problems of discoverability.

1 Like
10

I don’t quite see why it being private makes it a bad thing for the Python Launcher to use if it’s there and the user didn’t specify a specific Python version that they were after? This isn’t about creation or mutating, just utilizing what’s already there.

Sure, and I’m just saying another tool would be the Python Launcher.

Sure, but I still think it’s a bad idea. :smile: For instance having to tell my grep tools, editor, etc. to ignore that directory when doing work. And you’re right that lots of these tools now ignore anything in .gitignore, but that’s making an even bigger assumption about people’s development environment (and I have had users that committed their virtual environment, so I have heard it go both ways of not using git and really using git :wink:).

2 Likes
11

This is the big one for me. I pretty much never want tools that operate on “this project directory” (grep, git, whatever) to look at my local venv. So having it in a hidden directory is exactly right for me.

Even on Windows, where the dot convention doesn’t mean “hidden”, it does separate out the venv, and some of tools (for example, ripgrep) ignore dot directories by default.

So +1 from me for .venv. (It’s what I use manually now).

4 Likes
12

IMHO grep is an advanced tool, probably 98% of the people don’t use it. They prefer using the editor’s built-in tools for this and e.g. PyCharm can automatically detect python environments and ignore them independent of their name.

Wait, maybe then I did not follow correctly. Who will use this default and where? You say the py launcher potentially but for it to make sense to use .venv wouldn’t it need the creator tool to also create it under .venv? Otherwise seems like a very limited functionality for advanced users who know this and start creating venv’s with .venv to take advantage of this feature?

1 Like
13

The concern raised last time was that “everyone” would “always” have to activate to get to their tools, and -m wasn’t viable. Like I said, I’ll support you in ignoring/resolving that complaint, but it was the primary thing that sank this issue last time.

And I think the thing that will sink it this time is Bernat’s concern. If the tools don’t make it transparent, this becomes an advanced scenario - a trick for people who know the right incantation, and a trap for the rest. It’s a step in the right direction, but without removing all the other slightly-different directions we’re left with one more case for new users to have to decide whether they want to use it or not. That doesn’t help the overall image of environments being hard to manage.

Less of a concern is any sort of “up the tree” search. How does this handle environment and src directories being siblings, but you need your CWD to start in src? (Or is this just another choice new users will have to make - src directory or activate-free venv?) But there’s nothing here to make that search any better.

There are also potentially (depending on app) significant differences between running in an activated venv and running the binary directly without activation.

So all up, I have no problem with simply agreeing upon a default directory name for venv or tools that invoke venv or want to assume the presence of a virtual environment.

But I think that any sort of auto-activation is too complicated to be worth it if we’re not also significantly simplifying Python development for the future.

(And I know your py is your project, but py.exe is part of CPython. I’d rather you didn’t muddy the water by putting platform-specific magic into yours if it can’t or shouldn’t go into the upstream one.)

1 Like
14

Yes, hence this discussion. My point is the tool I’m working on isn’t creating anything but if we can agree with this then people like you who do own tools that create environments would follow suit as well.

Hence why I’m talking with Bernat. :smile:

I’m assuming either the Rust version would replace the C version someday or the Rust version would be the place to test things out before upstreaming to the C version.

1 Like
15

Be very clear about that on your project page then. I’ve already seen people calling it an “equivalent”, so you don’t really have the luxury of anonymity.

1 Like
16

And that’s accurate right now if you ignore OS differences. Until something actually occurs that leads to a deviation there’s no need to state otherwise since this is open source and tied to actually having time to make it deviate. :wink:

1 Like
17

I don’t know if more opinions help on this one, but I’m +1 for .venv.

The hidden directory is a valuable point, and the fact that poth pipenv and poetry use it reduces a lot of friction.

Also vscode will find it fine because it looks for virtualenv directories in the project root.

2 Likes
18

Probably not. This seems to have reached a point where somebody (probably Brett) just has to put out a popular tool with the change and suffer the hatred while people adjust their use patterns.

If you’re convinced enough of the long term benefit to people who don’t know how to give you that feedback, it’s easy to suffer the criticism from people who do know. (I’m getting literal hate (e)mail for putting python.exe in Windows, but also 5000 new installs/day, so I can take it.)

The worst thing we can do is create an echo chamber such that someone thinks they can make a change like this without inconveniencing anyone. That lack of empathy is the real problem when you change something that needed to be changed. Not whether you got the change perfect.

6 Likes
19

I’m tempted. :grin:

3 Likes
20

I have projects split between scripts, apps, and notebooks. I use project based names for my virtual environments so that when I create a new notebook it is easy to identify the correct virtual environment that I am using. If I used a single common name, I think I would have a list of all the same names. Is there a work around for that?
image

1 Like
21

I don’t know how that menu works in Jupyter, but I would have a directory for each of my projects (which I assume maps to each of your notebooks), and in that directory have an environment in a directory named .venv. Otherwise I would need an example of your directory structure to understand how have things currently laid out.

1 Like
22

Virtualenvwrapper stores virtualenvs within $WORKON_HOME?
(~/.virtualenvs/ by default)

source virtualenvwrapper.sh
1 Like
23

That menu gets created when you install the environment to the ipython kernel like

ipython kernel install --user --name=env

Maybe there is a way to have unique ipython kernels? It doesn’t seem to come from my file structure, which is pretty simple. Obviously, there are other files, but I think this explains it.

The text version did not render well.
image

1 Like