Hi all, I would like to propose adding a way to run isolated/virtual environments completely independently of the interpreter. This means, we would be able to run a virtual environment from any interpreter, allowing for control
Before we get into it, I wanted to present the current and past approaches.
Python 2
In Python 2, we create isolated environments by setting the PYTHONHOME
environment variable.
PYTHONHOME : alternate <prefix> directory (or <prefix>:<exec_prefix>).
The default module search path uses <prefix>/lib/pythonX.X.
Setting this changes the modules search path, allowing us to isolate the system defaults and control what exactly goes into sys.path
.
Drawbacks
Setting PYTHONHOME
not only implcts purelib
/platlib
(site-packages), but also stdlib
/platstdlib
(standard library). This means that if we want to have a custom environment, we need to copy or symlink the standard library to our environment, making it more difficult to create environments, and making the environments heavier.
Python 3
Python 3 solved the PYTHONHOME
approach’ issues with PEP 405. PEP 405 introduces a new first-step when calculating the prefix, pyvenv.cfg
. pyvenv.cfg
is a file that should live alongside, or in the parent directory of, the Python executable and stores the configuration for a virtual environment.
You can read more about it in the PEP, but this is how a normal config looks like:
home = /usr
implementation = CPython
version_info = 3.8.5.final.0
virtualenv = 20.0.23
include-system-site-packages = false
base-prefix = /usr
base-exec-prefix = /usr
base-executable = /usr/bin/python
PEP 405 also introduces the venv
module, which is capable of creating such environments.
Improvements
This approach no longer needs to copy/symlink the standard library to the environment, which can be expensive in some cases.
Drawbacks
Even though it does not need to copy/symlink the standard library, it now needs to copy/symlink the interpreter.
The introduction of pyvenv.cfg
as the first step of the search breaks the old-style environments. PYTHONHOME
can no longer be relied upon because if the interpreter happens to by from a virtual environment, pyvenv.cfg
is present and takes over.
There are some cases where the interpreter can’t be copied and symlinks are not available, like the Windows Store Python. In the Windows Store for eg. AFAIK this issue is solved by patching venv
to instead of copying or symlinking the interpreter, putting a wrapper in its place that will call the original interpreter. I might be a little fuzzy on the details, so feel free to correct me, but the point is that it needs special handling.
Proposal
So, I would like to propose introducing an environment variable to ignore pyvenv.cfg
, making PYTHONHOME
isolated environments reliable again, and introducing a few environments variables to configure the sysconfig
paths.
$ python -m sysconfig
Platform: "linux-x86_64"
Python version: "3.8"
Current installation scheme: "posix_prefix"
Paths:
data = "/usr"
include = "/usr/include/python3.8"
platinclude = "/usr/include/python3.8"
platlib = "/usr/lib/python3.8/site-packages"
platstdlib = "/usr/lib/python3.8"
purelib = "/usr/lib/python3.8/site-packages"
scripts = "/usr/bin"
stdlib = "/usr/lib/python3.8"
...
The idea here would be to be able to just take any interpreter and be able to reliably run an isolated environment from it.
A simple example of how it would look like:
$ PYTHONPLATLIB=~/env/myenv PYTHONPURELIB=~/env/myenv python -m sysconfig
Platform: "linux-x86_64"
Python version: "3.8"
Current installation scheme: "posix_prefix"
Paths:
data = "/usr"
include = "/usr/include/python3.8"
platinclude = "/usr/include/python3.8"
platlib = "~/env/myenv"
platstdlib = "/usr/lib/python3.8"
purelib = "~/env/myenv"
scripts = "/usr/bin"
stdlib = "/usr/lib/python3.8"
...
Improvements
Firstly, it makes creating virtual environments much easier and drops the requirement of venv
for cross-platform usage. Creating/running isolated environments in a cross-platform way would be dead simple. It also makes isolated environments a little bit more lightweight.
This enables us to simply and easily run subprocesses from the current executable in an isolated environment. This is something that would be really helpful in https://github.com/FFY00/python-build.
Drawbacks
Introducing this does have a drawback, you can no longer rely on a Python executable to run with a specific environment. This mainly impacts console scripts (I am not sure what else it does impact, hence the thread ), but this can be fixed by just clearing the environment variables, which would actually be pretty easy to implement.
So, any thoughts? Is there anything I am missing? Does anyone have ideas on how to improve this? Please let me know