Pip not picking up build requirements from pyproject.toml

Packaging
1

I’m trying to force my users to have to install wheel when they try to install my project for local development. However, it appears pip tries to build wheels for packages even when wheel is not available. I don’t know why it does this. So I added wheel to the pyproject.toml as seen below:

[build-system]
requires = ["setuptools>=46.0", "wheel"]
build-backend = "setuptools.build_meta"

This does not install wheel. Instead, it goes ahead without wheel and produces a warning akin to this:

Could not build wheels for <package>, since package 'wheel' is not installed.

I heard that the build environment is isolated from the main environment. What exactly does this mean?

I’ve created a self-contained MVCE to show how to reproduce this warning here:

Simply clicking “run” should show you the warning highlighted in pink. I feel I am misunderstanding the purpose of pyproject.toml as defined in PEP 518. Any guidance appreciated!

1 Like
2

Hi, I don’t think editable install works with PEP 517/518. Editable install is supposed to create symlinks from your local source tree to your Python’s site and no build (thus no built isolation also) is done. That being said, I’m not sure that’s relevant to your error, but I’ve no idea how to clone from repl.it to reproduce. Edit: I look it up and found the download button, but can’t reproduce:

(foo) cnx@debian:/tmp/foo$ pip3 list
Package       Version
------------- -------
pip           20.1
pkg-resources 0.0.0
setuptools    44.0.0
(foo) cnx@debian:/tmp/foo$ pip3 install -e .
Obtaining file:///tmp/foo
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting six
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: six, fakelib
  Running setup.py develop for fakelib
Successfully installed fakelib six-1.14.0
2 Likes
3

I was told that PEP 517/518 still works if a setup.py shim is used which I have already done. Here is the source that says that it works with those PEPs: https://twitter.com/pganssle/status/1241161328137515008

1 Like
4

In order to reproduce please look at the run.sh file contents to know what commands to run in correct order. This is the order of commands:

pip install -e .
pip uninstall wheel -y
pip install -e .

The reason you couldn’t reproduce is because most virualenvs install wheel when creating an env.

1 Like
5

About the reference on Twitter, what they meant is probably (fix me if I’m wrong):

  • Editable needs setup.py is setuptools is the build backend
  • PEP 517/518 is enforced (by pip AFAIK) if the project contains pyproject.toml. If user specify --use-pep517 on a project without the TOML file, the default will be used as described in PEP 518.

I want to emphasize again that the concept of editable and build isolation does not go together, at least for the moment, you can have one or the other but not both.

As for the error, it’s probably pip's mishandling of error message, the more minimal sample is as simple as (recall that you don’t have wheel in a fresh venv):

(foo) cnx@debian:/tmp/foo$ pip install six
Collecting six
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: six
Successfully installed six-1.14.0
(foo) cnx@debian:/tmp/foo$ pip install six
Requirement already satisfied: six in ./lib/python3.8/site-packages (1.14.0)
Could not build wheels for six, since package 'wheel' is not installed.

Feel free to file an issue at pip if it’s not already been, or nudge me to do it if you don’t want to. Edit: my memory is helpful this time, this is already fixed by GH-8180.

1 Like
6

If you don’t include a setup.py editable installs won’t work, but that is true for all backends, not just setuptools (e.g. you need to add a shim setup.py to get pip install -e . to work with flit).

The tweet explicitly states that setup.py is required even if setuptools is not the backend if you want editable installs to work.

I want to emphasize again that the concept of editable and build isolation does not goes together, at least for the moment, you can have one or the other but not both.

I would need a source that states this. It’s very confusing and I have poured through the issues list trying to find a similar issue. It would be a nice reassurance that I shouldn’t be using PEP 518/517. But right now the official recommendation seems to be to start using features of those PEPs as seen here: https://twitter.com/pganssle/status/1152695229105000453

Anyways the issue you linked seems to only change the message and not actually solve the problem. It doesn’t install any packages as defined in a pyproject.toml. There would be double the number of logs every time someone forgets to install wheel.

1 Like
7

Thanks, for the correction, @pganssle said setup.py is needed for editable with flit as the back-end. I’m not sure about other back-ends since editable is not really a well-defined standard (just look up on this discourse and you’ll see many loooong discussions on how to do it properly, but for the moment pip only thinly wraps around setup.py develop AFAIK).

I’m not sure I understand what you mean by the other paragraphs though. Editable install links (either viia file system or via Python import magic that I don’t fully understand) your local source tree to your site, so that when you edit the local source, the installed version is updated as well. On the other hand, wheel creates a portable achieve that is no longer dependent of the source tree (it can be deleted without any problem). Build isolated install does the following:

  • Jump into an isolated environment
  • Install build back-ends (i.e. setuptools and wheel in this case)
  • Build wheels from the source using install back-ends
  • Jump out and Install the wheels

As you can see, it isolated the build back-ends from being installed to one’s working site. It does not matter if one has the back-ends installed on per site or not, a PEP 517 front-end must acquire the front-ends every build (batch). I’m not sure it is possible to somehow unify the editable and isolated build together, but for now it doesn’t happen, simply because editable does not build.

1 Like
8

Turns out this decision is on purpose as per https://www.python.org/dev/peps/pep-0517/#recommendations-for-build-frontends-non-normative

Current discussion to get rid of this “feature” is on https://github.com/pypa/pip/issues/7555.

Credits to @McSinyx. Thanks!

1 Like
9

I think you’re making the common mistake of confusing out-of-tree builds with build isolation. In that thread, they are discussing removing the feature where pip install . first copies the contents of . into a temporary directory before doing the build.

Build isolation is something different, which is covered by your first link — the build-time dependencies are only needed at build time, so installing them in your runtime environment is unnecessary and hurts reproducibility (you don’t need them, for example, to install from a wheel).

With regards to editable installs: Editable installs still use PEP 518, but there is no standard interface between frontends and backends to trigger an editable install. pip install -e builds a PEP 518 build environment and then runs setup.py develop, but eventually someone will create build API hooks for editable installs.

If you have dev dependencies, you should put them in a requirements.txt (or requirements-dev.txt, something of that nature), or put them in an extra, like mypkg[dev]. It’s definitely a good thing that pip install . doesn’t install setuptools and wheel in your build environment for anything that is built with setuptools.

1 Like
10

Also, to be clear, this is not accurate. There is a build step involved in editable installs, and pip will construct an isolated PEP 518 build environment for it. For example, here’s a setup.py that depends on attrs:

import attr
import setuptools
setuptools.setup(name="project")

With this pyproject.toml:

[build-system]
requires = ["setuptools>=46.0", "wheel", "attrs"]
build-backend = "setuptools.build_meta"

Create a project with just those two files, then:

$ virtualenv venv --no-setuptools --no-wheel
$ source venv/bin/activate
$ pip list
Package Version
------- -------
pip     20.1
$ pip install -e .
Obtaining file:///tmp/tmp.TgCmZzr9Do/project
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Installing collected packages: project
  Running setup.py develop for project
Successfully installed project

You can tell that PEP 518 was used (even if PEP 517 wasn’t), because it didn’t error out on the missing import attr. If you turn off build isolation, it will error out on setuptools, since we never installed that, and if you install setuptools it errors out on attr:

$ pip install --no-build-isolation .
Processing /tmp/tmp.TgCmZzr9Do/project
    Preparing wheel metadata ... done
ERROR: Exception:                                          
Traceback (most recent call last):
...
ModuleNotFoundError: No module named 'setuptools'  
$ pip install --no-build-isolation .
Processing /tmp/tmp.TgCmZzr9Do/project
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
...
      File "setup.py", line 1, in <module>
        import attr
    ModuleNotFoundError: No module named 'attr'
...

You can observe the same thing by removing "attrs" from the pyproject.toml and doing a regular editable install.

The difference is that editable installs won’t invoke the PEP 517 backend hook, so if you don’t have a setup.py at all, setup.py develop won’t work.

3 Likes