`pip` build isolation without installing build dependencies

Packaging
1

As far as I can tell, pip does two things without --no-build-isolation:

  1. isolates the build environment from any system packages
  2. ensures that build requirements are installed

And --no-build-isolation disables both.

Is it possible to get pip to do (1) but not (2)? In Spack we want to use pip to handle installation as much as possible, especially since many packages in the ecosystem assume pip and only test with a pip install. We just don’t want it to trigger any dependency installs.

Looking through the options and source code, I don’t see a way to do this – they seem to always happen at the same time. Is there a reason for that, or is it something that people would be open to adding?

2

To be clear: by “Spack” you mean this thing which is itself a package manager, and not the PyPI package by that name? And by “use pip to handle installation as much as possible”, you mean “when Spack is used to install something, delegate to Pip to do the heavy lifting”, and not “use Pip for installing Spack”?

3

--no-build-isolation stops pip from doing (1). It also stops (2). If you want isolation, but you still want to manage the installation of the build dependencies for yourself, you should create a new virtual environment, install the build dependencies into it, and then run pip wheel --no-build-isolation in that environment, to create a wheel.

Then, you can run pip install in the original environment to install the wheel you just built.

Of course, you only need to do this for packages that require building. For packages shipped with wheels, pip will download and install the wheel in preference to the source dist, and so won’t need to do any building.

8 Likes
4

Yep, that’s right.

5

Aha.

In fact, I originally wrote out a lot of what Paul said, but I was writing assuming the perspective of someone trying to provide some custom installation process, and I couldn’t see the point of it. Then it occurred to me that it was unlikely that a years-abandoned project was about to receive a new release from someone not listed as a contributor, not to mention the relative popularity…

Anyway, yes, pip install defaults to build isolation for sdist installations - the steps are: 1) isolate, 2) build a wheel, and 3) unpack the wheel in the appropriate directory (i.e. a regular wheel installation of the wheel that was just built). The isolation step means 1a) creating a virtual environment (using venv) and 1b) installing build dependencies in that environment (normally, via a recursive pip install).

pip wheel does 1) and 2) by default, and it accepts the same --no-build-isolation to turn off 1), which you can then do manually. (If you know which build system is being used, you could also use it directly, I suppose.) This part was missing from my original draft, because I didn’t understand why you’d want to omit build dependencies entirely if you’re making something for someone else to install. But yes, if the point is to control that installation by doing it separately, then creating a virtual environment is the only missing step in “build isolation”.

1 Like
6

Thanks. If you’re curious for specifics on why I’m asking, the motivation for the post is this PR in Spack:

Spack is (I think?) unique in that it:

  1. has one prefix per package installation, including python extensions (this is like Nix),
  2. has a potentially impure build environment, in that users can register external packages, including Python, and
  3. will often build packages for the python that’s running Spack (these are mostly for bootstrapping Spack’s dependencies).

So we want build isolation so that things like setuptools can build from source, and so that we can isolate spack from the butchered sysconfig on Ubuntu that inserts a random local directory in most pip installs.

We already use pip to do most of the heavy lifting for Python builds. These are the arguments we invoke it with:

        return [
            # Verbose
            "-vvv",
            # Disable prompting for input
            "--no-input",
            # Disable the cache
            "--no-cache-dir",
            # Don't check to see if pip is up-to-date
            "--disable-pip-version-check",
            # Install packages
            "install",
            # Don't install package dependencies
            "--no-deps",
            # Overwrite existing packages
            "--ignore-installed",
            # Use env vars like PYTHONPATH
            "--no-build-isolation",
            # Don't warn that prefix.bin is not in PATH
            "--no-warn-script-location",
            # Ignore the PyPI package index
            "--no-index",
        ]

We rely on PYTHONPATH because there is one prefix per installation, and we set up the build environment carefully to include dependencies of the thing being built. If the user decides to use their own python and not have Spack build it, the site packages of that python and any distro patching (ala Debian, sigh) can pollute the build environment.

The solution we’re likely going to go with that seems to solve all of these issues (for the price of some nastiness in lower-level Spack packages) is to shim a venv in front of python when it’s used for building python packages. If it’s simply a build dependency (for running scripts), this isn’t needed, but for anything that is going to run pip, cmake, meson, etc. and install python modules, we’re relying on the venv.

Eventually we can hide this behind a virtual python package (currently python in Spack is cpython) and there is some discussion about how that might work if you’re interested.

Anyway, I think the PR is consistent with yours and @pf_moore’s advice. Other attempts, like making a temporary venv just during build (like pip does), had other problems, and we landed on using a tiny persistent venv for isolation after a number of false starts. The prior attempts are linked above as well.

I am not sure how useful this is for the packaging folks here but maybe it’s an interesting use case.

1 Like