Add programmatic support for rerunning the current program inside a venv

Ideas
1

Python programs often need to be run inside a venv in order to allow the use of packages from pypi.org. But creating and entering a venv before running a script is a barrier, especially if a python program has been written for use by people who are not particularly familiar with python.

So what i’d like is a new function in the standard venvmodule called something like venv.enter(), typically to be called at the start of execution. It would do nothing if we are already in a venv, otherwise it would create/update a venv, install a provided list of packages, then re-run the current program inside the venv.

This would allow python programs that use pypi.org packages, to be run directly without the user needing to know how to create and enter a venv.

Note that this would need to be in the standard library - it cannot be implemented as a package on pypi.org because one cannot generally install packages when not in a venv.

[One can use things like pipx to run scripts inside an automatically-created venv, but this requires installation of pipx first, and the user then has to use pipx run as a prefix.]

I have a module aptest/autovenv.py at main · ArtifexSoftware/aptest · GitHub that does most of what i need, but the implementation is necessarily quite clumsy:

  • It needs to be copied into each project that uses it.
  • To re-run the current program inside the venv, it uses subprocess.run(shell=True) to run a command of the form . <venv>/bin/activate && python {shlex.join(sys.argv)}. This has various problems:
    • shlex is posix-specific and doesn’t work properly on Windows.
    • [There is the mslex package that provides shlex-style fns on Windows, but of course it is on pypi so cannot be installed/used before we are in a venv.]
    • It changes the pid.
    • [One cannot simply exec <venv>/bin/python ... because it doesn’t set PATH, which causes subtle problems if the script creates child processes.]

As i understand things, entering a venv is actually very simple - it just involves adding the venv’s bin directory to PATH, and setting VIRTUAL_ENV.

An implementation inside the standard venv module would know what settings to use for a given venv directory, so it could use os.execve() instead of subprocess.run(shell=1). This would avoid creating a child process and also avoid the need for shlex.join(), thus avoiding all of the limitations of autovenv.py.

2

Running directly from PyPI already exists with pipx, uvx, etc. Running scripts with dependencies exists in those tools as well through PEP 723 - Inline script metadata. Packaging for users that don’t know how to install Python or activate a venv (as well as making phone apps) exists with Briefcase.

2 Likes
3

Maybe i have not have made my reasoning clear here.

Things like pipx or uvx don’t address the underlying problem:

  • They require the user to already be in a venv, or install a system package themselves.
  • They require the user to prefix their command with pipx run or similar.

Things like Briefcase avoids this, but it’s a pretty heavyweight approach, and for example doesn’t allow simple copying of a python script to another machine, or direct use of a script from a git checkout.

For example, with my proposal, one could write a script that uses the requests package like this:

#! /usr/bin/env python3

import venv
venv.enter(packages='requests')

import requests

# Do things using the `requests` package.
...

And it could then be run directly on any machine that has python installed.

For the writer of the script:

  • No need to install and run Briefcase or similar.
  • No need to create/distribute a package or wheel.

For the user:

  • No need to install any system packages.
  • No need to create/enter a venv.
  • No need to install any python packages.
4

This basically already exists with PEP 723:

# /// script
# dependencies = ["requests"]
# ///

import requests

uv run script.py

You can also prefix the script with a shebang #!/usr/bin/env -S uv run --script so the user can just run ./script.py.

So you’re essentially asking for the Python binary to adopt a nontrivial chunk of uv’s/pipx’s functionality.

However, in my opinion:

  • if the user already has Python installed, they likely already know how to also install tools like uv, or possibly have them already installed these days
    • one exception being OS-preinstalled Python, but the script might require a different Python version than the one the user has, so they might have to install a specific version anyway (and extra tooling can also automate that),
  • if the target user doesn’t already have Python installed and isn’t a Python programmer, then indeed forcing them to install uv/pipx for a simple script might be too much, but IMO having them install Python would also be too much; that’s when I’d use something like pyinstaller or Briefcase.
5

This bootstrapping-like issue is well-known and has been for a long time. The Python ecosystem is moving in the direction of supporting such use cases, slowly but surely. We are getting closer but we are still not there yet. If one happens to have a 3rd-party tool like pipx, uv/uvx, or any other tool that supports features like “Inline script metadata” (formerly known as PEP 723) for example, then they are in luck and will likely have a smooth experience. But yes these are still 3rd party tools.

On Windows there is the pymanager/py “Python install manager” tool. From my point of view it is a 1st-party tool and a very good candidate to adopt such features like “inline script metadata”. I do not know if there are concrete plans to make it happen, but as far as I know it is not excluded as an idea. If it happens for Windows, then I guess it would not be long until it happens for other platforms as well. So my gut feeling is that it is where we should put the effort rather than something like the proposal in the original post.

6

This basically already exists with PEP 723:

It really doesn’t. PEP 723’s abstract says:

This PEP specifies a metadata format that can be embedded in single-file Python
scripts to assist launchers, IDEs and other external tools which may need to
interact with such scripts.

My proposal is about running python scripts directly, without “launchers, IDEs or external tools”.

And required packages are specified programmatically, instead of being embedded inside comments as described in PEP 723.

So you’re essentially asking for the Python binary to adopt a nontrivial chunk of uv’s/pipx’s functionality.

Sort of. I’m talking about modifying python’s venv module, not the python binary itself. But of course venv is part of a standard Python install.

I want to do this because it significantly simplifies the writing and running of simple python scripts, in a way that cannot be done with external tools or packages.

7

I don’t think PEP 723 is particularly relevant here.

My proposal is about programmatically rerunning in a venv using just a default Python installation.

The Python code decides whether it needs to rerun inside a venv, and what packages it needs. In some cases either of these could vary according to the OS or other criteria, which is not generally possible with the PEP 723 approach.

The pipx and uvx approaches rely on the user knowing whether to run a particular script with pipx or uvx, but i don’t think we should be relying on the user to make these decisions - it is the Python code itself that knows what is required.

A default-installed Windows’ py or pymanager approach is less restricted, but it still does not have the flexibility that a programmatic solution provides. And to make it a general solution one would need to run all python programs with these tools instead of plain Python. In which case why not put the core functionality into python itself instead?

A basic implementation of the proposal in a standard install’s venv/__init__.py is very simple. It knows about the implementation details so can set os.environ directly and use os.execve() to rerun in the venv, which avoids any shlex-related problems on Windows:

def enter(venv_name=None, packages=None):
    venv_name = venv_name or '.venv'
    packages = packages or list()
    venv_name_abs = os.path.abspath(venv_name)
    if os.path.abspath(sys.prefix) == venv_name_abs:
        # We are already in the venv; install packages and return.
        if packages:
            subprocess.run([sys.executable, '-m', 'pip', 'install', '--upgrade'] + packages, check=1)
    else:
        # We are not in the specified venv so create/update the venv and rerun in it.
        #
        # Create/update venv. (This could use internal venv functions instead of a child process.)
        subprocess.run([sys.executable, '-m', 'venv', venv_name], check=1)
        # Update environment to enter the venv.
        os.environ['PATH'] = f'{venv_name_abs}/bin{os.pathsep}{os.environ["PATH"]}'
        os.environ['VIRTUAL_ENV'] = venv_name_abs
        # Rerun current program in the venv using exec.
        exe = f'{venv_name_abs}/bin/python'
        os.execve(exe, [exe] + sys.argv, os.environ)

As you stated, “this bootstrapping-like issue is well-known and has been for a long time”.

I’m struggling to understand why increasingly complicated approaches involving new command-line tools that have to parse special comments in python code described by PEP 723 , that have taken years to develop, and are still not close to being a general “batteries included” solution to the underlying problem, should be preferred to the trivially simple approach described here.

1 Like
8

I don’t think this needs to be supported at all, and would have preferred the script-runners never have been accepted. In my opinion, script runners make it far too easy for the average user to do the wrong thing, and easier to have confusable security boundaries.

I definitely never want to see someone distributing a script that installs packages into a venv, then replaces itself with execve to run. You’ve got no version conflict detection, are trying to specify a venv by it’s requirements, not it’s application which means a script can poison a venv for all other scripts, and this particular way of doing it makes it more difficult to use various security features like SELinux properly.

You’re also blind upgrading, which is unfortunately something nobody should do not just because of possible breakage, but because of supply chain security issues.

You’re right that the existing tools aren’t good enough, but that’s because the problem is actually hard to do everything right with, not because simpler is better in this case.

2 Likes
9

I understand these concerns, but i think there is a real problem to be solved here, so let’s not give up too easily.

For example we could make venv.enter() create and use a unique venv that existed only for the duration of the program. Pip already maintains a local cache of downloaded wheels so this wouldn’t result in repeated downloads from pypi. This would seem to address your concerns, though at the expense of a startup delay to create the unique venv.

There are various ways of doing this at an OS level, but it could also be done purely in python by making venv.enter() a context manager:

if __name__ == '__main__':
    with venv.Enter():
        main()

If we wanted to allow calling code to have some degree of reuse of venv’s in order to reduce startup delays, we could add an optional venv_name arg to venv.Enter() which the caller could set based on a hash of __file__ or os.path.basename(__file__) etc. [My autovenv.py also allows a venv prefix to which the python version etc are appended, to avoid problems with reusing a venv created by a different python version.]

The point is that once the core support is built-in to Python, deciding whether to specify the venv name, and picking a name that gives an appropriate degree of venv reuse, is just a matter of coding. That’s the benefit of the programmatic approach.

10

From what I know of the tools that have “inline script metadata” features, none of that is true. Or I have greatly misunderstood these claims. As far as I know the “inline script metadata” tools take care of not reusing environments for different scripts, and make sure the dependencies for one script are kept isolated from the dependencies of another script. I think these tools create ephemeral virtual environments that are discarded at the end of the script. But again, I might have misunderstood the message.

1 Like
11

Was referring to the proposed “simple” implementation, not the existing tools the author doesn’t want to use. Those are better, but still too easy for users to overlook proper dependency management IMO.

1 Like
12

Ah. Yes, I see. Then, I think that I agree. Although, I assumed that the code presented in this thread is more akin to a crude proof of concept, and so I did not let the obvious flaws that you mentioned stop me.

I think that in principle, if there was something like OP’s suggestion in the standard library it would be great, but I do not see it happening. This kind of things is very complex to get right. And it would need constant updates and fiddling (more frequent than Python’s release schedule), and if I recall correctly this is part of the reason why pip is not part of the standard library.

Finally let me add that from my point of view there is no problem with continuing the discussion, maybe something will come out of it, maybe something with a completely different scope…

1 Like
13

Here is an example implementation that i think significantly improves on the autovenv.py i mentioned in the first post, and hopefully addresses the issues that have been raised so far.

  • The implementation is 140 lines long (of which about half are comments and logging statements).
    • i couldn’t find any information about how one should make code fragments available, so apologies if this is too long.
  • It is known to work on Linux, Windows and MacOS.
  • It defaults to a new unique venv directory each time, which addresses the concerns about security.
    • This default venv directory is deleted after use, using tempfile.TemporaryDirectory().
    • This means we have to rerun the current program as a child process with subprocess.run() instead of os.execve(), so that tempfile.TemporaryDirectory()'s cleanup can run.
  • One can optionally specify the venv directory path directly, which allows different degrees of venv reuse as required.
    • We also provide some convenience format variables that make it simpler to include things like the python version in the path, which is useful because different python versions cannot use the same venv.
    • In this case we can rerun using os.execve(), although actually we still use subprocess.run() on Windows because of problems with execve and empty args.
  • We use implementation knowledge of the environment variables used by the venv module, to avoid having to enter the venv with . <venv>/bin/activate or <venv>/Scripts/activate. This allows us to rerun inside the venv by passing argv lists to os.execve() or subprocess.run(), so we don’t need shlex on unix and avoid command-line quoting/escaping problems on Windows.
  • We run [sys.executable, '-m', 'venv', venv_path] to create/update venv’s.
    • We could use internal venv functions directly if we were part of the venv module.
'''
Automatic creation/use of a venv.

Example usage:

    import autovenv
    autovenv.enter(packages=['pytest', 'numpy'])
    import numpy
    ...
'''

import os
import platform
import subprocess
import sys
import sysconfig
import tempfile


def _freethreads():
    '''
    Returns true if python is free-threads.
    '''
    Py_GIL_DISABLED = sysconfig.get_config_var('Py_GIL_DISABLED')
    if Py_GIL_DISABLED == 1:
        # Free threads build.
        if not sys._is_gil_enabled():   # pylint:disable=protected-access
            return True


def _bits():
    return int.bit_length(sys.maxsize+1)


def enter(*,
        packages=None,
        venv_path=None,
        verbose=True,
        ):
    '''
    Rerun current python program in a venv.
    Args:
        packages:
            List of packages to install.
        venv_path:
            Path of venv directory. If None (the default) we use a new and
            unique venv directory which is deleted afterwards.
            
            Otherwise we use venv_path.format(**kwargs) where kwargs is a dict
            containing these keys:
                python_version: platform.python_version(),
                freethreads: 't' if python is freethreads else '' .
                wordsize: e.g. 64 or 32.
    '''
    AUTOVENV_VENV_PATH = os.environ.get('AUTOVENV_VENV_PATH')
    if (AUTOVENV_VENV_PATH
            and os.path.realpath(sys.prefix)
                == os.path.realpath(AUTOVENV_VENV_PATH)
            ):
        # We are already in the autovenv venv; install packages and return.
        if verbose:
            print(f'autovenv: Already in autovenv venv, {sys.prefix=}.')
        if packages:
            if isinstance(packages, str):
                packages = [packages]
            if verbose:
                print(f'autovenv: Installing packages: {packages}.')
            subprocess.run(
                    [sys.executable, '-m', 'pip', 'install', '--quiet', '--upgrade']
                        + packages,
                    check=1,
                    )
    else:
        # We are not in the autovenv venv so create/update it and rerun
        # ourselves in it.
        if verbose:
            if sys.prefix == sys.base_prefix:
                print(f'autovenv: Not in a venv.')
            else:
                print(f'autovenv: In non-autovenv venv, {sys.prefix=}.')
        
        def setup(venv_path):
            '''
            Create/update venv, modify os.environ so that any subprocesses
            still run inside the venv, and return path of venv's python.
            '''
            if verbose:
                print(f'autovenv: Using venv: {venv_path}')
            # Create/update venv.
            subprocess.run([sys.executable, '-m', 'venv', venv_path], check=1)
            
            # Update PATH and VIRTUAL_ENV so that any subprocesses still run
            # inside the venv. This uses internal implementation details of
            # the venv module.
            bin_dir = 'Scripts' if platform.system() == 'Windows' else 'bin'
            p = os.path.join(venv_path, bin_dir)
            os.environ['PATH'] = p + os.pathsep + os.environ['PATH']
            os.environ['VIRTUAL_ENV'] = venv_path
            
            # Set AUTOVENV_VENV_PATH so we can distinguish between venv's
            # created by us and venv's created by other means.
            os.environ['AUTOVENV_VENV_PATH'] = venv_path
            
            # Return location of venv's python.
            return f'{venv_path}/{bin_dir}/python'
        
        if venv_path:
            # Expand selected fields.
            kwargs = dict(
                    python_version = platform.python_version(),
                    freethreads = 't' if _freethreads() else '',
                    wordsize = _bits(),
                    )
            venv_path = venv_path.format(**kwargs)
            venv_python = setup(venv_path)
            # Rerun the current python program in the venv.
            if platform.system() == 'Windows':
                # Have seen odd behaviour with os.execve() where empty string
                # args appear to be removed.  So we use a child process
                # instead.
                cp = subprocess.run([venv_python] + sys.argv, env=os.environ)
                sys.exit(cp.returncode)
            else:
                os.execve(venv_python, [venv_python] + sys.argv, os.environ)
        
        else:
            # Use tempfile.TemporaryDirectory() to create venv directory that
            # will be automatically removed after use.
            with tempfile.TemporaryDirectory(prefix='autovenv-') as venv_path:
                if verbose:
                    print(f'autovenv: Using unique venv directory: {venv_path}')
                venv_python = setup(venv_path)
                # Rerun the current program in the venv. We need to use
                # a child process instead of os.execve(), so that our
                # tempfile.TemporaryDirectory gets to delete the venv
                # directory.
                cp = subprocess.run([venv_python] + sys.argv, env=os.environ)
                sys.exit(cp.returncode)
14

Hopefully the latest example implementation is more than just a crude proof of concept. I use it regularly in real build/dev work on multiple platforms.

I’m glad of the positive assessment.

I have no illusions that it would be easy to get this into the standard library. However i don’t think it would be particularly difficult get to a good implementation.

I’m curious about the comparison with pip. Pip has to know all the details about wheel formats and how a python installation is structured. It’s really complicated! And its src/ directory has 130,000 lines of code. Also it has to interact with remote servers which makes testing harder.

Where as autovenv is less than 200 lines of code, has no interaction with remote servers, and is easy to test. So i see no reason why it should be any more difficult to make it work than any other change to the standard library.

Getting a good user interface is always non-trivial. But we could start with the simplest API with no args, which always creates and enters a new venv that is destroyed afterwards. Then later on we could consider adding a venv_path arg, and figure out how this could work usefully for different people.

15

For a start, it relies on pip, which is not guaranteed to be present in the standard library (it’s shipped by default, but I believe some Linux distress make it a separate package). You don’t cover cases like users needing proxy settings to see PyPI, or users having to use a different package index. Basically, you don’t support any of the use cases pip has command line options for - and just allowing the user to pass pip options would include a bunch of options (such as --target) that don’t make sense for this use case.

Standard library functions need to be safe and reliable for all uses. And for this function, that involves handling a huge bunch of environmental issues that the average stdlib function doesn’t have to care about.

2 Likes
16

What Paul said.

Currently there is the discussion about a standardized specification for the location of virtual environments (link below). I am not sure where the discussion stands at right now, I have not followed closely.

But let’s say the discussion ends up deciding that .venv is indeed chosen as the standard location for virtual environment (so far it was only a loose convention but not a standard). It feels to me like there could be room to add code around this future new standard in the venv library. What I have mind is that this new code would only check for the presence of a .venv directory, if there is one the script would be restarted with the virtual environment activated. No need to clean-up so it should be possible to use execve.

By leaving the creation of the virtual environment and the installation of packages out of the proposal and by limiting the proposal to .venv only (or whatever path gets standardized) the scope becomes smaller and maybe more likely to get accepted in the standard library (if this gets accepted one could always build from there with a follow-up proposal?).

Additionally, maybe it could be possible to add a CLI option flag like python --activate-dot-venv-if-any script.py, instead of having dedicated code inside script.py itself.

[Sorry, it is not well written because I do not have that much time, but I hope it makes somewhat sense anyway.]