Hi all, I would like to propose adding a way to run isolated/virtual environments completely independently of the interpreter. This means, we would be able to run a virtual environment from any interpreter, allowing for control
Before we get into it, I wanted to present the current and past approaches.
In Python 2, we create isolated environments by setting the
PYTHONHOME environment variable.
PYTHONHOME : alternate <prefix> directory (or <prefix>:<exec_prefix>). The default module search path uses <prefix>/lib/pythonX.X.
Setting this changes the modules search path, allowing us to isolate the system defaults and control what exactly goes into
PYTHONHOME not only implcts
platlib (site-packages), but also
platstdlib (standard library). This means that if we want to have a custom environment, we need to copy or symlink the standard library to our environment, making it more difficult to create environments, and making the environments heavier.
Python 3 solved the
PYTHONHOME approach’ issues with PEP 405. PEP 405 introduces a new first-step when calculating the prefix,
pyvenv.cfg is a file that should live alongside, or in the parent directory of, the Python executable and stores the configuration for a virtual environment.
You can read more about it in the PEP, but this is how a normal config looks like:
home = /usr implementation = CPython version_info = 3.8.5.final.0 virtualenv = 20.0.23 include-system-site-packages = false base-prefix = /usr base-exec-prefix = /usr base-executable = /usr/bin/python
PEP 405 also introduces the
venv module, which is capable of creating such environments.
This approach no longer needs to copy/symlink the standard library to the environment, which can be expensive in some cases.
Even though it does not need to copy/symlink the standard library, it now needs to copy/symlink the interpreter.
The introduction of
pyvenv.cfg as the first step of the search breaks the old-style environments.
PYTHONHOME can no longer be relied upon because if the interpreter happens to by from a virtual environment,
pyvenv.cfg is present and takes over.
There are some cases where the interpreter can’t be copied and symlinks are not available, like the Windows Store Python. In the Windows Store for eg. AFAIK this issue is solved by patching
venv to instead of copying or symlinking the interpreter, putting a wrapper in its place that will call the original interpreter. I might be a little fuzzy on the details, so feel free to correct me, but the point is that it needs special handling.
So, I would like to propose introducing an environment variable to ignore
PYTHONHOME isolated environments reliable again, and introducing a few environments variables to configure the
$ python -m sysconfig Platform: "linux-x86_64" Python version: "3.8" Current installation scheme: "posix_prefix" Paths: data = "/usr" include = "/usr/include/python3.8" platinclude = "/usr/include/python3.8" platlib = "/usr/lib/python3.8/site-packages" platstdlib = "/usr/lib/python3.8" purelib = "/usr/lib/python3.8/site-packages" scripts = "/usr/bin" stdlib = "/usr/lib/python3.8" ...
The idea here would be to be able to just take any interpreter and be able to reliably run an isolated environment from it.
A simple example of how it would look like:
$ PYTHONPLATLIB=~/env/myenv PYTHONPURELIB=~/env/myenv python -m sysconfig Platform: "linux-x86_64" Python version: "3.8" Current installation scheme: "posix_prefix" Paths: data = "/usr" include = "/usr/include/python3.8" platinclude = "/usr/include/python3.8" platlib = "~/env/myenv" platstdlib = "/usr/lib/python3.8" purelib = "~/env/myenv" scripts = "/usr/bin" stdlib = "/usr/lib/python3.8" ...
Firstly, it makes creating virtual environments much easier and drops the requirement of
venv for cross-platform usage. Creating/running isolated environments in a cross-platform way would be dead simple. It also makes isolated environments a little bit more lightweight.
This enables us to simply and easily run subprocesses from the current executable in an isolated environment. This is something that would be really helpful in https://github.com/FFY00/python-build.
Introducing this does have a drawback, you can no longer rely on a Python executable to run with a specific environment. This mainly impacts console scripts (I am not sure what else it does impact, hence the thread ), but this can be fixed by just clearing the environment variables, which would actually be pretty easy to implement.
So, any thoughts? Is there anything I am missing? Does anyone have ideas on how to improve this? Please let me know