Pip, pyproject.toml and debug builds

Environment

  • pip version: 20.0.2
  • Python version: 3.8.2
  • Numpy verssion: 1.18.1
  • OS: Microsoft Windows 10

Description
Pip fails to build and install Numpy-1.18.0 when built with debug global option

Details
We have been using Numpy debug modules built and installed in using pip.

python_d.exe -m pip install --no-binary :all: --global-option build --global-option --debug numpy

The builds for versions starting from Numpy-1.18.0 release are failing. The reason for this failure is the introduction of pyproject.toml in this version of Numpy. It contains the following lines:

[build-system]
requires = [
    "setuptools",
    "wheel",
    "Cython>=0.29.14",  # Note: keep in sync with tools/cythonize.py
]

The Python environment already have “setuptools”, “wheel” and “Cython” packages built with appropriate options and installed correctly using pip. But, the command python_d.exe -m pip install --no-binary :all: --global-option build --global-option --debug numpy making pip to download and build them again instead of using the existing packages. Is this correct behavior? Any clarifications would be greatly appreciated.

The global options (build --debug) that are specified for building numpy are not being passed on to the Cython building process. This is causing the creation of numpy debug libraries and Cython release libraries leading to a mix up and failure of numpy build process.

My second question is, is there a way to pass the global options to the dependency build process. I am not sure if this is bug in numpy code, if this can be fixed by altering the numpy’s pyproject.toml file.

Version Information:

  • Python 3.8.2
  • Pip 20.0.2
  • Numpy 1.18.1
  • Microsoft Windows 10

(Quote abbrievated for brevity.)

This is the expected behaviour. pyproject.toml makes pip enter build isolation mode; pip would populate a new environment with the specified packages, separated from the runtime one, for better build reproducibility regardless of pre-existing package installations.

The best solution at the moment is probably supplying --no-use-pep517 to pip can stick to your development workflow prior to the pyproject.toml addition. PEP 517 is still missing a lot of pieces, especially those related to development and debug workflows. I believe there are some work to amend this, but the progress is slow due to resource constraints (see Specification of editable installation - #40 by pganssle).

Cc-ing some people that might be interested in this topic (from the mentioned thread): @pganssle @pradyunsg @techalchemy

1 Like

IMO, the hardest part of designing behaviours to suit those workflows is the difficulty in getting solid information about what is needed. If we could get people who make heavy use of pip in development and debugging situations to write up their requirements (ideally in a general manner, having thought about how to separate essential and what is simply “how we do things at the moment”) then we might be better able to identify what changes are needed in PEP 517, PEP 518, pip and the build backends, in order to effectively support those workflows.

Ideally, of course, would be if someone with a direct need for this functionality were to champion a PEP that defines the needed infrastructure. (It’s really hard to get a design right if you don’t personally have the issue the design is intended to solve).

Thank you for the clarification and for the reference. I got all the required clarifications.

Yes. I am able to build it and install using –no-use-pep517 option.

Thanks again.