Trying to come up with a default directory name for virtual environments

Packaging
24

If Jupyter is managing all of this for you then this topic isn’t meant for you. :wink: Otherwise I name my virtual environments .venv because the directory they are contained in makes it obvious what environment they are meant for.

3 Likes
25

.venv (and ~/.virtualenvs/) are suboptimal because:

  • hidden directories require toggling a gui file manager 'Show hidden files`; and then the sort order may also be indeterminate in terms of whether .venv/ is before a/ or before venv/
  • recursive grepping and finding tools may ignore .hidden_dirs by default
  • ls -l doesn’t list a .hidden directory like .venv/ without -a
    • alias lll="ls -altr"

Alternative default directory names:

#!/bin/sh

__WRK="${HOME}/-wrk"                          # cdwrk
__DOTFILES="${HOME}/-dotfiles"                # cdd cddotfiles
PROJECT_HOME="${__WRK}"                       # cdph cdprojecthome
WORKON_HOME="${__WRK}/-ve27"                  # cdwh cdworkonhome

CONDA_ROOT="${__WRK}/-conda27"                # $CONDA_ROOT__py27
CONDA_ENVS_PATH="${__WRK}/-ce27"              # cdce cda cdcondaenvspath

VIRTUAL_ENV_NAME="dotfiles"                   # 'dotfiles'
_APP=$VIRTUAL_ENV_NAME                        # 'dotfiles[/p/a/t/h]'
VIRTUAL_ENV="$WORKON_HOME/$VIRTUAL_ENV_NAME"  # cdv cdvirtualenv
_SRC="${VIRTUAL_ENV}/src"                     # cds cdsrc
_BIN="${VIRTUAL_ENV}/bin"                     # cde cdbin
_ETC="${VIRTUAL_ENV}/etc"                     # cde cdetc
_LOG="${VIRTUAL_ENV}/var/log"                 # cdl cdlog
# ... see: venv.py --print-vars / ...
_WRD="${_SRC}/{_APP}"                         # cdw cdwrd

(set -x; test "$_WRD" == "${HOME}/-wrk/-ve27/dotfiles/src/dotfiles"; \
    || echo "Exception: _WRD = '${_WRD}';" )
1 Like
26

This has only ever been a good thing for me. When I want to find all instances of a word/string in my repo, not searching through 100s to 1000s of MBs of third party files is trivial with grep 'string' * -r

3 Likes
27

This includes _WRD/.git by default but skips everything else in _SRC=VIRTUAL_ENV/src and _LIB and site-packages and $_ETC:

(cd "$_WRD": grep "${@}")
}

_WRD=$_SRC/appname
_WRD=$VIRTUAL_ENV/src/appname

E.g. grind/grind (PyPI:grin3; python regex) (1) skip hidden dirs; and (2) by default specifically dirs named .git,.hg,.svg,etc.

grinwrd, grepwrd, grinw, grepw
grindwrd,
https://github.com/westurner/dotfiles/blob/develop/scripts/_grinwrd.sh

1 Like
28

Are you sure these are cons? Why would you want to browse the environment directory with a GUI file manager, or grep it from the outside? How often do you actually do that?

Having virtualenv hidden means less clutter when using these tools. Less icons in the project directory, ls reporting only the things you actually care to inspect, grep not combing through thousands of modules it shouldn’t touch.

Most contemporary grep substitutes (*cough* ripgrep *cough*) use .gitignore or .ignore, so that directory won’t show up when using them anyway.

29

Why would I want to hide the contents of $VIRTUAL_ENV/src from file browsers that hide .hidden directories by default like Nautilus, Finder, NERDTree, VSCode, and ls?

VIRTUAL_ENV=$WORKON_HOME/appname
_WRD=$WORKON_HOME/appname/src/appname
cdsrc; cd $VIRTUAL_ENV/src; grepX "${@}"
cdwrd; cd $VIRTUAL_ENV/src/appname; grepX "${@}"
30

I can’t speak for Wes, but for me, quite often actually, to the point I dislike existing popular tools all try to use either hidden directories or put the environment in obscure locations (specifically Poetry for the later variant). Recently I’ve been working a lot with Starlette and Websockets, and found it’s actually quicker for me to read the source (at least the class and function declarations) than documentation. And that’s not a one-off occurrence to me.

I fully admit the possibility I am an outlier instead of the norm, but also suspect that’s not the case. Many tools use non-dot directories (node_modules, for example), and it’s IMO safe to say the two opinions are similarly popular, neither is problematic, and it’s really down to personal preference. Looking back at Python, __pycache__ is an even more “useless” directory to users, it’s not at all hidden, and not really anyone is complaining about it (not loudly, at least), so I suspect either camp will be eventually be able to live with a decision the other way. It really is up to someone creating a tool with enough traction to settle the debate.

2 Likes
31

I suspect that, from users perspective, it’s because it comes down to the choice of having either multiple .pyc files clutter your workspace or a single __pycache__ directory. The latter is better, so why complain about it? It’s not necessarily because it’s good for most users to actually see the directory in their browsers, not to mention many tools don’t show it by default. There’s also not much reason to complain if you’re using grep, as binary files don’t generate large output.

To me, a better argument would be “because hiding files is platform specific”, but to that I’d say: everything is platform specific if you consider enough platforms or get low-level enough. It shouldn’t be a basis to reject ideas that improve usage for major parts of the userbase.

I’m not assuming that this one will - the null hypothesis is that it doesn’t matter - but it looks encouraging from the UX side. It hides things that a user shouldn’t need to worry about. People shouldn’t need to inspect the contents of their virtual environments, should they?

IMO the choices made with regard to such standardization should drive users towards more optimal usage patterns without blocking other usage. Using file manager to browse site packages module code is not optimal given current tools available: ctrl+:computer_mouse:left in pycharm, <identifier>?? in IPython, I suspect vim’s gf can be made to do it properly too (I’m actually looking into improving python.vim so that it does it properly).

I’ve never heard of anyone keeping project source solely inside a virtualenv directory, and I’m not sure why would you want to do it. What contents do you have on your mind?

32

Everyone should inspect their virtual environment content sometimes. Dependencies you pulled in are as much code as those you write, and should be read. They probably should not be accessed very often, but actively hiding them from plain sight may be swing too much against best practices.

This is actually a very common place to store code, since pip clones a repository there when you pip install -e git+https://....

1 Like