What about the build system's own dependencies?

I just now managed to create an install a Python-only wheel for freetype-py, so that it can use the system freetype library, in a dedicated venv.

Initially I had planned to just install into the existing venv where I actually plan to use freetype. But I found that it didn’t work due to Pip’s default build isolation. On the first naive attempt:

$ pip install --no-binary :all: freetype-py
Collecting freetype-py
  Downloading freetype-py-2.4.0.zip (832 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 832.5/832.5 kB 987.2 kB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [115 lines of output]
      Collecting setuptools>=42
        Downloading setuptools-69.1.1.tar.gz (2.2 MB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 824.5 kB/s eta 0:00:00
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting wheel
        Using cached wheel-0.42.0-py3-none-any.whl
      Collecting setuptools_scm>=3.4 (from setuptools_scm[toml]>=3.4)
        Downloading setuptools-scm-8.0.4.tar.gz (74 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.3/74.3 kB 1.3 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting certifi
        Downloading certifi-2024.2.2.tar.gz (164 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.9/164.9 kB 973.1 kB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting cmake
        Downloading cmake-3.28.3.tar.gz (42 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.2/42.2 kB 485.4 kB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting packaging>=20 (from setuptools_scm>=3.4->setuptools_scm[toml]>=3.4)
        Using cached packaging-23.2-py3-none-any.whl
      Collecting typing-extensions (from setuptools_scm>=3.4->setuptools_scm[toml]>=3.4)
        Using cached typing_extensions-4.10.0-py3-none-any.whl
      Building wheels for collected packages: setuptools, setuptools_scm, certifi, cmake
        Building wheel for setuptools (pyproject.toml): started
        Building wheel for setuptools (pyproject.toml): finished with status 'done'
        Created wheel for setuptools: filename=setuptools-69.1.1-py3-none-any.whl size=819326 sha256=933b678a5aaed8fd69cee7378452a31570f481aa098cf02ba079b3b5d56060f5
        Stored in directory: /home/zahlman/.cache/pip/wheels/b7/33/f5/31244f1c091d5056e4733433770be86bd86557cad44d72928e
        Building wheel for setuptools_scm (pyproject.toml): started
        Building wheel for setuptools_scm (pyproject.toml): finished with status 'done'
        Created wheel for setuptools_scm: filename=setuptools_scm-8.0.4-py3-none-any.whl size=42139 sha256=89405989ada4c001287d4cae743e9f3523089c3cb1b984d5ec2231f86a762356
        Stored in directory: /home/zahlman/.cache/pip/wheels/a4/a8/9c/2058c6d2878399349f6d0d3e78c1e6690f702cb16855edbb59
        Building wheel for certifi (pyproject.toml): started
        Building wheel for certifi (pyproject.toml): finished with status 'done'
        Created wheel for certifi: filename=certifi-2024.2.2-py3-none-any.whl size=163774 sha256=23ae3dbe7443007e2145c334d6a43393d74270ddfbaca25bc0beaf857743d957
        Stored in directory: /home/zahlman/.cache/pip/wheels/3d/9e/f5/a182a848fe3b811dc8e1266b8e557599d43a55668f0514942d
        Building wheel for cmake (pyproject.toml): started
        Building wheel for cmake (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error
      
        × Building wheel for cmake (pyproject.toml) did not run successfully.
        │ exit code: 1
        ╰─> [42 lines of output]
            /tmp/pip-build-env-04rqiri5/overlay/lib/python3.11/site-packages/setuptools_scm/git.py:308: UserWarning: git archive did not support describe output
              warnings.warn("git archive did not support describe output")
            /tmp/pip-build-env-04rqiri5/overlay/lib/python3.11/site-packages/setuptools_scm/git.py:327: UserWarning: unprocessed git archival found (no export subst applied)
              warnings.warn("unprocessed git archival found (no export subst applied)")
            Traceback (most recent call last):
              File "/tmp/pip-build-env-04rqiri5/overlay/lib/python3.11/site-packages/skbuild/setuptools_wrap.py", line 645, in setup
                cmkr = cmaker.CMaker(cmake_executable)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/tmp/pip-build-env-04rqiri5/overlay/lib/python3.11/site-packages/skbuild/cmaker.py", line 148, in __init__
                self.cmake_version = get_cmake_version(self.cmake_executable)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
              File "/tmp/pip-build-env-04rqiri5/overlay/lib/python3.11/site-packages/skbuild/cmaker.py", line 105, in get_cmake_version
                raise SKBuildError(msg) from err
      
      
                =============================DEBUG ASSISTANCE=============================
                If you are seeing a compilation error please try the following steps to
                successfully install cmake:
                1) Upgrade to the latest pip and try again. This will fix errors for most
                   users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
                2) If running on Raspberry Pi OS, you can set PIP_ONLY_BINARY=cmake in
                   order to retrieve the latest wheels built by piwheels.
                   c.f. https://github.com/scikit-build/cmake-python-distributions/issues/392#issuecomment-1676284749
                3) If on Linux, with glibc < 2.12, you can set PIP_ONLY_BINARY=cmake in
                   order to retrieve the last manylinux1 compatible wheel.
                4) If on Linux, with glibc < 2.12, you can cap "cmake<3.23" in your
                   requirements in order to retrieve the last manylinux1 compatible wheel.
                5) Open an issue with the debug information that follows at
                   https://github.com/scikit-build/cmake-python-distributions/issues
      
                Python: 3.11.2
                platform: Linux-5.4.0-172-generic-x86_64-with-glibc2.31
                glibc: glibc 2.31
                machine: x86_64
                bits: 64
                pip: n/a
                setuptools: 69.1.1
                scikit-build: 0.17.6
                PEP517_BUILD_BACKEND=setuptools.build_meta
                =============================DEBUG ASSISTANCE=============================
      
            Problem with the CMake installation, aborting build. CMake executable is cmake
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for cmake
      Successfully built setuptools setuptools_scm certifi
      Failed to build cmake
      ERROR: Could not build wheels for cmake, which is required to install pyproject.toml-based projects
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Between the output that CMake offered on the failed build, I decided to start over with a fresh venv and manually install the dependencies there first, including upgrading Pip, and then try the Freetype build again:

$ python3.11 -m venv .local/.freetype-build
$ source .local/.freetype-build/bin/activate 
(.freetype-build) $ pip install --upgrade pip
Requirement already satisfied: pip in ./.local/.freetype-build/lib/python3.11/site-packages (22.3.1)
Collecting pip
  Using cached pip-24.0-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.3.1
    Uninstalling pip-22.3.1:
      Successfully uninstalled pip-22.3.1
Successfully installed pip-24.0
(.freetype-build) $ pip install wheel
Collecting wheel
  Using cached wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB)
Using cached wheel-0.42.0-py3-none-any.whl (65 kB)
Installing collected packages: wheel
Successfully installed wheel-0.42.0
(.freetype-build) $ pip install setuptools_scm[toml]>=3.4
(.freetype-build) $ pip install certifi
Collecting certifi
  Downloading certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Downloading certifi-2024.2.2-py3-none-any.whl (163 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.8/163.8 kB 1.3 MB/s eta 0:00:00
Installing collected packages: certifi
Successfully installed certifi-2024.2.2

So far, so good. And this time, the CMake installation actually works:

(.freetype-build) $ pip install cmake
Collecting cmake
  Downloading cmake-3.28.3-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (6.3 kB)
Downloading cmake-3.28.3-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (26.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.3/26.3 MB 984.5 kB/s eta 0:00:00
Installing collected packages: cmake
Successfully installed cmake-3.28.3

And so does the freetype-py installation:

(.freetype-build) $ pip install --no-build-isolation --no-binary :all: freetype-py
Collecting freetype-py
  Using cached freetype-py-2.4.0.zip (832 kB)
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: freetype-py
  Building wheel for freetype-py (pyproject.toml) ... done
  Created wheel for freetype-py: filename=freetype_py-2.4.0-py3-none-any.whl size=77030 sha256=675a3d89bb603474d95fd51c6586da6c949f90ce5e490ef748c27151f81737d0
  Stored in directory: /home/zahlman/.cache/pip/wheels/fa/cb/7f/fec72063da052f5130dadd111db2d0616236cca8e97c566aab
Successfully built freetype-py
Installing collected packages: freetype-py
Successfully installed freetype-py-2.4.0
(.freetype-build) $ pip list
Package           Version
----------------- --------
certifi           2024.2.2
cmake             3.28.3
freetype-py       2.4.0
packaging         23.2
pip               24.0
setuptools        65.5.0
setuptools-scm    8.0.4
typing_extensions 4.10.0
wheel             0.42.0

Also notable here: the CMake wheel is massive (~26 MB archive expanding to ~65MB on disk), but the tarball is only a few kilobytes.

So what I concluded is: Pip creates a separate venv for build isolation, without bootstrapping itself into that venv (even though venvs created through python -m venv would do so by default). And then because I used :all: for the Pip command, Pip chose to build CMake from source, even though it’s only a build dependency. And then the CMake sdist is actually some kind of bootstrapper that gets the actual source from somewhere else - and requires Pip to do so.

Questions:

  • Is --no-binary :all: actually intended to force build-time dependencies to be built from scratch, or only the ordinary runtime dependencies? (Is it actually feasible to build everything from scratch, transitively?)

  • Any more detailed idea what went wrong? Is it indeed because the venv for build isolation doesn’t include Pip, or else just what?

  • Why don’t Pip’s isolated build environments include Pip?

  • Could the issue be fixed by freetype-py adding pip as a build-time dependency, ahead of cmake, in its pyproject.toml? (Is that at all a good idea?)

  • Why didn’t I get any output at all for the successful setuptools_scm[toml] install in the new venv?

  • The stack trace for the failed attempt mentions a skbuild package. I guess that must be from scikit-build-core. But how did that get installed? And why don’t I see a skbuild folder, or setuptools_wrap.py or cmaker.py, anywhere in that codebase?

1 Like

OK, I solved this one - I forgot to quote setuptools_scm[toml]>=3.4 on the command line, so the output was redirected to a file named =3.4. Oops :slight_smile: Really nice of Linux to tokenize that way…

Yes. It’s intended for people who want to avoid using any binary artefacts. And yes, that makes build-time dependencies quite complex to manage.

We run pip from its installed location in the calling environment for efficiency. Copying pip into every build environment is slow. Most builds don’t need pip as a dependency, and if they do, they should specify that in the build-requires, and not make assumptions (especially now that uv is a viable alternative to pip).

If it needs pip to be present, then yes, it should. This might not work (it’s not a common situation and I can imagine some potential issues) but that would be a pip bug. I’m struggling to imagine why you would need to run pip as part of a build, though - why not just put whatever you are installing in your build-requires in the first place?

2 Likes

Or in whatever metadata calculates get_requires_for_build_wheel, if it can’t be known entirely statically for your project.

Fair enough, I figured it would be something like this.

I wasn’t able to get a successful CMake bulid from scratch in an isolated environment, either. Was hoping someone more specifically familiar with CMake could weigh in on that.

Anyway, I filed an issue with freetype-py to request distributing a source-only wheel. CMake is only listed as a dependency in the first place because the same setup.py is used (via an environment variable) for both builds. It doesn’t even do anything relevant to the wheel I was trying to create, AFAICT.

For this (and for Scikit-Build-Core in particular) you might want to ask @henryiii , FYI.

2 Likes

Is there a reason why you chose
pip install --no-binary :all: freetype-py
instead of
pip install --no-binary freetype-py freetype-py
?

Partly laziness, but partly because writing freetype-py twice in a row seemed wrong. Initially I was confused as to why just pip install --no-binary freetype-py didn’t work at all. So there is probably something that can be improved about the Pip user experience here.

Anyway, freetype-py added logic to make the CMake dependency conditional (on the same environment variable used to decide whether to build the C library).

Actually, I have a more complex situation: my project will have freetype-py as a possible extra. But presumably the same --no-binary freetype-py idea will work in conjunction with installing myproject[freetype] or whatever.

1 Like

Last time I tried (some years ago), building CMake was very easy, so I’m a bit surprised by that. It’s also available as a binary download or package for a very large number of systems, so you should not have to build it in most circumstances. The only potential reason for building it, IME, is if you need a newer CMake version than what’s readily available for your system.

Yes, I had no issues here. However, when I started this, I wasn’t even expecting to need CMake in the first place; I didn’t look in to the build system before trying to install freetype-py and figured Setuptools would handle it.

Perhaps it requires having specific non-Python dependencies set up? Again, I didn’t actually investigate documentation or anything; I just made the simplest possible attempt, and then a couple attempts to install other things first based on the error messages, and then got stuck.

Which, thinking about it, ties in to the stuff people have been saying about the packaging ecosystem generally and all the long threads we’ve had around people reporting frustrating experiences off-site etc. Generally one doesn’t expect to have to do research just in order to install something that’s on PyPI - one imagines that “oh, it’s in the place specifically designed to source stuff that Pip knows how to deal with, so installing it should just work”. But between build dependencies and version incompatibility issues, research often does become necessary - even if the solutions turn out simple.

1 Like

I guess that’s a matter of documentation from the dependent package. Along the same lines as “to build this package from source, you need gcc and GNU autoconf installed”.

Yeah, that’s why it’s very recommended that such packages publish binary wheels. However, some packages do have complicated build dependencies that cannot reasonably be fetched transparently from PyPI (and there’s much worse than just requiring CMake in that regard :slight_smile: ).

It’s quite lightweight, but it still requires a C++ compiler:

More importantly perhaps, CMake has nothing to do with Python, it’s entirely written in C++, so its builds instructions might not be easily translated into something that can be transparently executed by pip install ....

1 Like