Venv reasonable defaults: Single activator script, default name for venv, auto install from requirements.txt and auto upgrade-deps

AstroMatt · November 10, 2020, 12:09pm

What’s the problem this feature will solve?
Venv and virtualenv are both great tools. As an educator I prefer venv, because it’s always with Python installation. Therefore this feedback is related to venv.

I am using venv everyday for last couple of years. I’ve been programming in Python since 2009 and recently I turned into teaching software engineering. This turn gave me a new perspective. Pretty often I teach people who first got into programming. Those classes are usually 15 people or more.

At the beginning of a training course I have to teach the following lines:

if you have windows, do:

python3.9 -m venv python-3.9.0
python-3.9.0/Scripts/activate.bat

but if you have Linux or macOS, than

python3.9 -m venv python-3.9.0
python-3.9.0\bin\activate

but if you have different shell than bash (fish, csh, etc. like macOS does).
and then…

Recently this is sent over email, before training course, which is even worse because people are with problems without trainer!

C’mon. Those people are afraid of terminal. Not mentioning:

changing directory (each system has different paths, separators, permissions),
there is a -m venv parameter which should have space before, inside and after,
they have to put some duplicate text (like python3.9 and python-3.9.0) in a one line (why?!),
you have to explain how to check with Python version they have… (python -v and python --version produces completely different output

Sometimes I waste 30 minutes of training course to help people to setup and activate venv!! This ruins the first impression and makes programming inaccessible. I wouldn’t dare, to recummend using venv for a 3h hands-on workshop on a conference (because it will take so much time for explanation). Why, oh why you need to have different paths, activators and no reasonable defaults? In the other hand, if they start without venv, and then install for example jupyter with it’s all dependencies… it’s very messy, and it’s too for fixing it…

This makes using venv hard for Python newcomers, and this is the reason why they prefer Anaconda.
I don’t think this is how we (Python community) should welcome newcomers, by advising to install bloated, 3rd party controlled (with customised settings [such as recursion limit, and whoever knows what else]).

Describe the solution you’d like
Creating and setting up venv should be no-brainer one command, such as:

python -m venv --home --activate --install https://example.com/mycourse/requirements.txt

Let me explain (note, that all of this is backward compatible):

python3.9 -m venv. If you haven’t specified venv directory it should by default create a new one with the name python-3.9.0 (or your current Python version). Simple and backward compatible.
A parameter like python3.9 -m venv --home should place venv inside ~/.virtualenvs directory (of course with default Python version name from example above).
Currently, there is no venv default name. People are using a lot of different approaches:

venv/
.venv/
_venv/

virtualenv/
.virtualenv/
_virtualenv/

py
python
py-3.9
python-3.9
python-3.9.0

.py
.python
.py-3.9
.python-3.9
.python-3.9.0

venv-3.6/
venv-3.7/
venv-3.8/
venv-3.9/
venv-3.10/

venv-3.8.0/
venv-3.8.1/
venv-3.8.2/
venv-3.8.3/

venv-3.9.0/
venv-3.9.1/
venv-3.9.2/
venv-3.9.3/

venv-3.10-alpha1/
venv-3.10-alpha2/
venv-3.10-beta1/
venv-3.10-beta2/
venv-3.10-rc1/

venv-django-2.1
venv-django-2.2
venv-django-2.3
venv-django-3.0
venv-django-3.1
venv-django-3.2
venv-django-4.0a1
venv-django-4.0a2
venv-django-4.0b1
venv-django-4.0b2
venv-django-4.0rc1
venv-django-4.0rc2

venv-py39-dj33
venv-python39-django33
venv-python310alpha1-django40a1

Having standard name convention (and location) would help much with .gitignore file, and prevent from committing those files into repository (it happens more often they you imagine).

--upgrade-deps should be default without need to specify. Each time I’ve got questions like: why there is information about outdated pip after newly created venv. If it knows it’s outdated, why it was not updated automatically. Which indeed is a very good question. And I also don’t understand why. (but this behaviour wouldn’t be backward compatible with current behaviour).
The most important: activation. Current activators live in different paths bin or Scripts which is reflection of Python install directory and I understand why you did that on a developer level. But this should be hidden from end-user during activations! Why activation is not like: python-3.9.0/activate.py (where python-3.9.0 is venv name). This script should recognise OS and Shell and use proper script.
Activate on creation: python3.9 -m venv --activate. This would be handy.
Install requirements on creation (This would solve so many problems):

python3.9 -m venv --install requirements.txt
python3.9 -m venv --install https://example.com/mycourse/requirements.txt

Of course it should be movable. From your OS you should drag-and-drop to some other location without having to recreate it and without breaking it. People does that, and they wonder why everything breaks.
Putting it all together:

python -m venv --home --activate --install https://example.com/mycourse/requirements.txt
note, that python executable should be without version [a bit controversial for veterans, but for newcomers is a must]. And also it’s future proof.

I’ve seen countless python packages documentation with “Installation guides”: if you have windows do this… else do this… This makes venv cumbersome for devs who want to share its software with quick installation guide. This makes using venv hard for Python newcomers, and this is the reason why they prefer Anaconda. And I don’t believe it is a good way to go.

Imagine if you organise a training course or online course and you have to specify installation requirements.
Saying:

Install newest Python from official website
Open terminal (this sucks, but it’s a necessity).
Run python -m venv --home --activate --install https://example.com/mycourse/requirements.txt
That’s it. You’re all set-up and ready to participate.

PS.
The last one is not directly related to venv. IMHO venv should allow other devs, such as Jupyter (or Jupyter-lab) to create a shortcut on a desktop after installation. This shortcut should activate venv and run jupyter. That would be a lifesaver!

PS2. I have created a GitHub issue at virtualenv repository:

bernatgabor · November 10, 2020, 12:27pm

I’ve made my position on these topics known under https://github.com/pypa/virtualenv/issues/2007#issuecomment-724660278. TLDR:

In the spirit of explicit is better than implicit this was reverted in virtualenv.
is trivially solvable via ~/.virtualenv/venv (though questionable if a good idea)
We don’t have a consensus.
Long topic going in detail about it at discuss.python.org/t/how-should-virtualenv-behave-respective-of-seed-packages-e-g-pip-by-default/4146
I’m not sure this is technologically doable. Though would like to be.
I’m not sure this is technologically doable for the same reason as 5.
Doable via virtualenv plugin.
A venv is a reference to host python, containing many absolute paths generated during install (e.g. within console scripts). No one managed to come up with a backward-compatible solution to make them relocatable. This likely would need solution first in the interpreter, then in pip.

uranusjr · November 10, 2020, 1:10pm

Point 5 is technically doable, a la conda activate. conda is implemented as a shell function and can therefore modify the current user session. I know what you’re going to say next, no, I don’t think we should do this

My personal response to proposals trying to improve activation scripts is basically “don’t.” Not just don’t propose them, but don’t do activations at all. Modifying the user’s shell session is always going to go wrong. It’s better for everyone to just launch the Python executable directly—either by wiring the virtual environment concept into the interpreter like __pypackages__, or help people access the virtual environment like the PYTHONHOME proposal a while ago

sinoroc · November 10, 2020, 1:15pm

[I will veer a bit off-topic, others have basically already said what I have in mind… Why would one want to start beginners with installing 3rd party libraries straight out of the gate? There are plenty of things that are doable with Python’s standard library and thus do not require virtual environments. It would make for an easier start into Python, and programming in general.]

AstroMatt · November 10, 2020, 4:21pm

@sinoroc I teach not only Python, but also (and mostly) a Data Science and Machine Learning. Those courses are taken by 75% of my students. NumPy, Pandas, Jupyter (or Jupyter-lab), Matplotlib, Sklearn are the environments to starts with. This is also true to the Python developer survey and Stackoverflow developer survey which states that more or less 50% of Python developer are using this language for ML/DS application.

Although @bernatgabor made clear answers on GitHub (thank you!) I’d like to discuss it a bit further.

I agree. Explicit is better than implicit. I can work with having to specify venv location and a name each time.
Indeed this could be solved by ~/.virtualenv/venv as a parameter.
I think ~/.virtualenv/venv-3.9.0/ would be the best, and I think I will go this way during the next training course and will se what happen. (If there is no consensus, sometimes BDFL has to step up
Like Bernat said on Github: “just teach people using --upgrade-deps” and I’m fine with that.
and 6. Simple solution: in venv main directory create two scripts activate.bat and activate.
User would do venv/activate -> on POSIX it would execute activate (shell script); on Windows will append suffix .bat and run it. Could be also achieved by manipulating PATH and PYTHONHOME variable like pyenv does.
This is MHO most bang for bucks.
Is nice to have, but absolutely can live with current solution. But shouldn’t that be achieved by adjusting PYTHONHOME variable?
I can live with longer version, but still one liner (easy to copy and paste):
python -m venv ~/.virtualenvs/trainingcourse --upgrade-deps --activate --install https://example.com/mycourse/requirements.txt

Although this ~/ will be problematic on Windows. And then maybe flag --default-home would prepend ~/.virtualenvs or %home%\.virtualenvs to the explicitly named venv?!

bernatgabor · November 10, 2020, 4:53pm

Maybe for the python executable yes. For all the console scripts, not at all:

cat venv/bin/pip                                                                                                                                                                                  
#!/path/to/venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])

And then there’s macOs where the system host python location is baked into python executable (if was created via copy), for security reasons, via Mach-O - Wikipedia. Finally, no solution is good that relies on activation, because activation is extra to avoid users have to type venv/bin/python. The contract is that you do not have to activate virtual environments to use it, and they’re completely usable via direct full path invocation.

Getting anything into venv will take a few years at the very least. You could implement most of what you want via virtualenv plugins, and that’s available today. To not need users to install virtualenv you could ship them a zipapp of it: Installation - virtualenv

pf_moore · November 10, 2020, 6:10pm

… and to implement anything else, you could create a small “wrapper” application that invokes virtualenv in the way you want things set up, so people only have to download one script and run it and they will be set up.

brettcannon · November 10, 2020, 10:27pm

Those are slightly contradictory if you don’t assume that python-* is an acceptable entry for .gitignore.

I did a poll on Twitter a ways back about a default name for virtual environments and nothing won a majority. It basically came down to venv versus .venv and whether people wanted the virtual environment to not show up in tools like VS Code and such that hide anything starting with ..

Python itself has no concept of what this would mean since it doesn’t understand what a requirements file is (that’s a pip thing, not a venv thing). As such I would argue that it shouldn’t be in something in the stdlib like venv.

Because it’s really hard to get to work. Shell scripts are executed by the shell to change their state. Trying to change shell state in a subprocess like Python which will persist when that process exits is typically not supported (mostly for security purposes).

This is why I want implicit use of the virtual environment Python in the Python Launcher. Shells are just hard to work with, so coming up with a way to do this without mucking with shell scripts would be preferred.

That doesn’t cover all of the other shells that are supported, e.g. PowerShell, fish, etc.

westurner · November 10, 2020, 10:51pm

PATH=$VIRTUAL_ENV/bin:$PATH

pip install --user -U virtualenvwrapper pip
source ~/.local/bin/virtualenvwrapper.sh

mkvirtualenv my-app_name
echo "$WORKON_HOME"

ls -ald ~/.virtualenvs/my-app_name/bin/*activate
echo "$PATH"

workon my-app_name
echo "$PATH"
echo "$VIRTUAL_ENV"
deactivate

source $WORKON_HOME/my-app_name/bin/activate

echo "$PATH"
echo "$VIRTUAL_ENV"
deactivate

Auto-upgrade deps and run the integration tests for all upgraded components and callers of said APIs before merging that PR branch

steve.dower · November 10, 2020, 11:16pm

Thanks a lot for your post! Very helpful information, and I’m glad you added the detail and experiences that you have.

Others have addressed most of your points, and I will agree that we have considered many of them but they are, in fact, technically infeasible (and the rest are too controversial for now… feedback like yours is how we eventually overcome that).

On this point, there’s nothing preventing an application from configuring Python however they like. A venv is essentially just a convention for setting a search path, and it’s not actually that complicated for someone to make a Jupyter app that includes a copy Python and any packages, as well as creating shortcuts. The Mu editor (https://codewith.mu/) is a perfect example of such an application (it uses Pynsist.)

Ultimately, the challenging part is in installing the desktop shortcut on any OS you care about, rather than anything relating to Python.