Run pip with subprocess in venv

Python Help
1

I have a project and use setup.py to manage its dependencies. I don’t use pyproject.toml or sth else as I really need all the flexibility setup.py gives me. I create a venv with:

python -m venv .venv

and activate it. Then, I run pip install -e .. Here is what my setup.py looks like:


def install_dependencies() -> None:
    """Act as a dynamic requirements.txt."""
    subprocess.run(
        [sys.executable, "-c", "import pip; print(pip.__version__)"],
        text=True,
        check=True,
    )


class InstallDependenciesMixin:
    """Mixin to encapsulate shared dependency installation logic."""

    def run(self) -> None:
        """Override default implementation to do equivalent of pip install."""
        install_dependencies()

        super().run()  # type: ignore[misc]  # mixin has no parent to inherit from


class CustomDevelopCommand(InstallDependenciesMixin, develop):
    """Custom command for editable installation (pip install -e .)."""


class CustomInstallCommand(InstallDependenciesMixin, install):
    """Custom command for regular installation (pip install .)."""


setup(
    name="project",
    version="0.0.0",
    description="A Python project with custom installation logic",
    author="Your Name",
    author_email="your.email@example.com",
    license="MIT",
    packages=find_namespace_packages(where=".", include="src.*"),
    cmdclass={
        "develop": CustomDevelopCommand,  # runs with pip install -e
        "install": CustomInstallCommand,  # runs with pip install
    },
)

This does not work. I get an error when subprocess runs pip:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pip'

I have checked that sys.executable correctly points to the python executable inside my venv. I also checked that pip is installed as a module in my venv. I also manually ran the command in subprocess and it worked fine (but when subprocess runs it, it fails). I tried adding shell=True or copying the env in the subprocess.run call. But nothing helped.

Can someone explain to me what is going on and what the correct way to call pip with subprocess in a venv is?

2

As a general rule, it’s a terrible idea to try and use pip from inside a setup.py script. Unless you have to maintain a decade old, highly complex setup.py script, change your approach.

The reason this doesn’t work is because pip by default does “build isolation” by default, following the process outlined in PEP 518. Within this isolated context, pip is not available unless you explicitly specify it as a dependency.

But again: Don’t manually call pip install from within setup.py, it will cause issues.

1 Like