# \`pip\` build isolation without installing build dependencies

**URL:** <https://discuss.python.org/t/pip-build-isolation-without-installing-build-dependencies/52145>\
**Category:** Packaging\
**Created:** [April 29, 2024, 4:15am UTC](https://discuss.python.org/t/pip-build-isolation-without-installing-build-dependencies/52145 "2024-04-29T04:15:14Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tgamblin](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tgamblin/32/1211_2.png) [@tgamblin](https://discuss.python.org/u/tgamblin)\
**Post date:** [April 29, 2024, 4:15am UTC](https://discuss.python.org/t/pip-build-isolation-without-installing-build-dependencies/52145/1 "2024-04-29T04:15:14Z")

</div>

As far as I can tell, `pip` does two things _without_ `--no-build-isolation`:

1. isolates the build environment from any system packages
2. ensures that build requirements are installed

And `--no-build-isolation` disables both.

Is it possible to get `pip` to do (1) but not (2)? In [Spack](http://github.com/spack/spack/) we want to use `pip` to handle installation as much as possible, especially since many packages in the ecosystem assume `pip` and only test with a `pip` install. We just don’t want it to trigger any dependency installs.

Looking through the options and source code, I don’t see a way to do this – they seem to always happen at the same time. Is there a reason for that, or is it something that people would be open to adding?

---

<div class="post-metadata">

**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\
**Post date:** [April 29, 2024, 8:00am UTC](https://discuss.python.org/t/pip-build-isolation-without-installing-build-dependencies/52145/2 "2024-04-29T08:00:29Z")

</div>

> [@tgamblin](#):
>
> In Spack we want to use `pip` to handle installation as much as possible, especially since many packages in the ecosystem assume `pip` and only test with a `pip` install. We just don’t want it to trigger any dependency installs.

To be clear: by “Spack” you mean [this thing which is itself a package manager](https://github.com/spack/spack), and not the [PyPI package by that name](https://pypi.org/project/spack/)? And by “use `pip` to handle installation as much as possible”, you mean “when Spack is used to install something, delegate to Pip to do the heavy lifting”, and not “use Pip for installing Spack”?

---

<div class="post-metadata">

**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\
**Post date:** [April 29, 2024, 9:51am UTC](https://discuss.python.org/t/pip-build-isolation-without-installing-build-dependencies/52145/3 "2024-04-29T09:51:27Z")

</div>

> [@tgamblin](#):
>
> As far as I can tell, `pip`’s `--no-build-isolation` argument does two things:
> 
> 1. isolates the build environment from any system packages
> 2. ensures that build requirements are installed
> 
> Is it possible to get `pip` to do (1) but not (2)?

`--no-build-isolation` _stops_ pip from doing (1). It also stops (2). If you want isolation, but you still want to manage the installation of the build dependencies for yourself, you should create a new virtual environment, install the build dependencies into it, and then run `pip wheel --no-build-isolation` in that environment, to create a wheel.

Then, you can run `pip install` in the original environment to install the wheel you just built.

Of course, you only need to do this for packages that require building. For packages shipped with wheels, pip will download and install the wheel in preference to the source dist, and so won’t need to do any building.

---

<div class="post-metadata">

**Author:** ![tgamblin](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tgamblin/32/1211_2.png) [@tgamblin](https://discuss.python.org/u/tgamblin)\
**Post date:** [April 29, 2024, 6:41pm UTC](https://discuss.python.org/t/pip-build-isolation-without-installing-build-dependencies/52145/4 "2024-04-29T18:41:20Z")

</div>

> [@kknechtel](#):
>
> To be clear: by “Spack” you mean [this thing which is itself a package manager](https://github.com/spack/spack), and not the [PyPI package by that name](https://pypi.org/project/spack/)? And by “use `pip` to handle installation as much as possible”, you mean “when Spack is used to install something, delegate to Pip to do the heavy lifting”, and not “use Pip for installing Spack”?

Yep, that’s right.

---

<div class="post-metadata">

**Author:** ![kknechtel](https://avatars.discourse-cdn.com/v4/letter/k/e47c2d/32.png) [@kknechtel](https://discuss.python.org/u/kknechtel)\
**Post date:** [April 30, 2024, 2:20am UTC](https://discuss.python.org/t/pip-build-isolation-without-installing-build-dependencies/52145/5 "2024-04-30T02:20:05Z")

</div>

Aha.

In fact, I originally wrote out a lot of what Paul said, but I was writing assuming the perspective of someone trying to provide some custom installation process, and I couldn’t see the point of it. Then it occurred to me that it was unlikely that a years-abandoned project was about to receive a new release from someone not listed as a contributor, not to mention the relative popularity…

Anyway, yes, `pip install` _defaults to_ build isolation for sdist installations - the steps are: 1) isolate, 2) build a wheel, and 3) unpack the wheel in the appropriate directory (i.e. a regular wheel installation of the wheel that was just built). The isolation step means 1a) creating a virtual environment (using `venv`) and 1b) installing build dependencies in that environment (normally, via a recursive `pip install`).

`pip wheel` does 1) and 2) by default, and it accepts the same `--no-build-isolation` to turn off 1), which you can then do manually. (If you know which build system is being used, you could also use it directly, I suppose.) This part was missing from my original draft, because I didn’t understand why you’d want to omit build dependencies entirely if you’re making something for someone else to install. But yes, if the point is to control that installation by doing it separately, then creating a virtual environment is the only missing step in “build isolation”.

---

<div class="post-metadata">

**Author:** ![tgamblin](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/tgamblin/32/1211_2.png) [@tgamblin](https://discuss.python.org/u/tgamblin)\
**Post date:** [April 30, 2024, 4:45pm UTC](https://discuss.python.org/t/pip-build-isolation-without-installing-build-dependencies/52145/6 "2024-04-30T16:45:02Z")

</div>

Thanks. If you’re curious for specifics on why I’m asking, the motivation for the post is this PR in Spack:

- [spack/spack#40773: “python: always use a venv”](https://github.com/spack/spack/pull/40773)

Spack is (I think?) unique in that it:

1. has one prefix per package installation, including python extensions (this is like Nix),
2. has a potentially impure build environment, in that users can register _external_ packages, including Python, and
3. will often build packages for the python that’s running Spack (these are mostly for bootstrapping Spack’s dependencies).

So we want build isolation so that things like `setuptools` can build from source, and so that we can isolate spack from the butchered `sysconfig` on Ubuntu that inserts a random `local` directory in most `pip` installs.

We already use `pip` to do most of the heavy lifting for Python builds. These are the arguments we invoke it with:

```python
        return [
            # Verbose
            "-vvv",
            # Disable prompting for input
            "--no-input",
            # Disable the cache
            "--no-cache-dir",
            # Don't check to see if pip is up-to-date
            "--disable-pip-version-check",
            # Install packages
            "install",
            # Don't install package dependencies
            "--no-deps",
            # Overwrite existing packages
            "--ignore-installed",
            # Use env vars like PYTHONPATH
            "--no-build-isolation",
            # Don't warn that prefix.bin is not in PATH
            "--no-warn-script-location",
            # Ignore the PyPI package index
            "--no-index",
        ]

```

We rely on PYTHONPATH because there is one prefix per installation, and we set up the build environment carefully to include dependencies of the thing being built. If the user decides to use their own python and _not_ have Spack build it, the site packages of that python and any distro patching (ala Debian, sigh) can pollute the build environment.

The solution we’re likely going to go with that seems to solve all of these issues (for the price of some nastiness in lower-level Spack packages) is to shim a venv in front of `python` when it’s used for building python packages. If it’s simply a build dependency (for running scripts), this isn’t needed, but for anything that is going to run `pip`, `cmake`, `meson`, etc. and install `python` modules, we’re relying on the `venv`.

Eventually we can hide this behind a virtual `python` package (currently `python` in Spack is `cpython`) and there is some discussion about how that might work if you’re interested.

Anyway, I think the PR is consistent with yours and @pf_moore’s advice. Other attempts, like making a temporary `venv` just during build (like `pip` does), had other problems, and we landed on using a tiny persistent `venv` for isolation after a number of false starts. The prior attempts are linked above as well.

I am not sure how useful this is for the packaging folks here but maybe it’s an interesting use case.