No way to pin build dependencies

Packaging
1

Hey all, Cython 3.0.0 just released today, which broke our install of statsmodels. Basically, the statsmodel pyproject.toml defined a build system “requires list” that allowed for cython 3.0.0 to be used (when it’s in fact not compatible).

# more pyproject.toml 
[build-system]
requires = [
    "setuptools",
    "wheel",
    "cython>=0.29.22",
    "oldest-supported-numpy",
    "scipy>=1.3",
]

For normal install requires dependencies, I can (and do) pin exact versions for every requirement that gets installed. So even if a package maintainer forgets to set an upper bound for their install requires package, I’ll be sure that it will be frozen in time with the exact versions installed each time.

However, with the build-system requires packages (which get installed as the package tries to build) don’t seem to respect any currently installed packages or pinned versions in a requirements file. In fact, my understanding is that the build system installs its requirements into a special virtualenv used just for the build.

The problem with this is that I don’t seem to have any control over what the build system does in this virtualenv. If cython releases 3.0.0, and statsmodel’s build system just tries to install the latest version of cython, I have no way to pin what the build system tries to install. I’m completely dependent on every maintainer of every package that I use to follow semantic versioning exactly and to not introduce any bugs in their patch releases. This make immutable builds very difficult, and risky.

Generally speaking, what is the recommended way to pin/freeze dependencies installed by 3rd party packages (which you don’t control) such that they build using the same requirements every time until explicitly updated?

Note: I’m not picking on statsmodel specifically; in fact I think newer versions fixed this issue. However, I’ve been encountering this situation with pip more and more lately, where I can’t seem to pin build requirements and a build requirement version slips. Figuring out how to pin these build dependencies is the crux of my question.

5 Likes
2

My feeling is there that there are a lot of projects scrambling right now to push hot fix releases. Just another example PyYAML. As it is a dependency of pre-commit a lot of CI test setups, especially with 3.12.0-bX are probably failing right now. Thankfully they have uploaded wheels for the major platforms, but building from source is just broken atm.

There are workarounds like --no-build-isolation with a cython < 3.0 pin, but that doesn’t really work in requirement files and automated setups.

3

Thanks Marc for your reply. Yeah, I can imagine a lot of CI pipelines being affected with this with various projects. Maybe --no-build-isolation is the way to go, though I imagine that being more of a work-around.

I’ve had this similar issue a number of times with other packages over the last 6 months or so. I was hoping pip might have a recommended strategy for this, but maybe it’s still a work in progress?

4

FWIW, while this isn’t limited to only build dependencies, it is possible to set PIP_CONSTRAINTS=[path-to-file] and have pip treat those as additional constraints for packages that it selects. This does bubble down to individual build environments, as long as there isn’t something passing a different constraints file via CLI arguments.

https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-c

3 Likes
5

And to give you some idea of how well it scales, in the OpenStack
community (for whom and from whence pip’s constraints option
originated), we’ve been happily using it for a little over 8 years
to pin our transitive dependency set insuring coinstallability
across all our integrated projects. The exact size ebbs and flows
over time, but as of this moment the constraints file we maintain
for that transitive dependency set in our master branches is just in
excess of 600 unique packages.

6

Only with --no-build-isolation though. What to do when one project requires cython>=3.0 and another <3.0? Is the only solution then to split these into separate pip install commands or hope the package maintainer add a build constraint themselves in time?

7

It depends on what you have the stomach for, but the most effective
options are to either help out the maintainers of that dependency
that requires older Cython, or pin back to an older version of the
dependency which requires newer Cython until the other dependency
finally catches up on their own.

1 Like
8

Note that I didn’t propose that as a panacea for the lack of fine-grained control over build dependencies (that’s not something that a user can get with pip today – that’s basically the job of an integrator/redistributor-style model TBH).

9

No, in general.

❯ PIP_CONSTRAINT=/tmp/c.txt pip install -v . 
Using pip 22.3.1 from /Users/pradyunsg/Developer/github/furo/.venv/lib/python3.11/site-packages/pip (python 3.11)
Looking in indexes: http://localhost:3141/root/pypi/+simple/
Processing /Users/pradyunsg/Developer/github/furo
  Running command pip subprocess to install build dependencies
  Looking in indexes: http://localhost:3141/root/pypi/+simple/
  ERROR: Cannot install sphinx-theme-builder>=0.2.0a10 because these package versions have conflicting dependencies.

  The conflict is caused by:
      The user requested sphinx-theme-builder>=0.2.0a10
      The user requested (constraint) sphinx-theme-builder==0.0

  To fix this you could try to:
  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

  ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

  [notice] A new release of pip available: 22.3.1 -> 23.2
  [notice] To update, run: pip install --upgrade pip
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/pradyunsg/Developer/github/furo/.venv/bin/python3.11 /Users/pradyunsg/Developer/github/furo/.venv/lib/python3.11/site-packages/pip/__pip-runner__.py install --ignore-installed --no-user --prefix /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-build-env-ky_ubugv/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i http://localhost:3141/root/pypi/+simple/ -- 'sphinx-theme-builder >= 0.2.0a10'
  cwd: [inherit]
  Installing build dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip available: 22.3.1 -> 23.2
[notice] To update, run: pip install --upgrade pip
10

Oh, sorry. I meant it doesn’t work for overwriting build requirements without --no-build-isolation. Unless I’m missing something else…

11

I’m wondering whether there is value in a community-maintained constraint file that specifies constraints for packages where upstream did not bother with making patch releases. For example, package A depends on package B. B is updated, and an older version of A could use a patch release to add an upper constraint. I imagine it could help version resolution a lot. At the same time, the amount of constraints could be massive. Note in the Nix community we do something like this with poetry2nix.

12

It’s possible I’m misunderstanding the missing patch releases use
case described, but keep in mind that constraints files get supplied
outside the packages you’re installing (pip needs access to them
before it starts retrieving packages), are used for an entire pip
invocation (not independently per-package), and merely narrow pip’s
dependency selection (they can’t force pip to ignore existing
versioned dependencies declared within packages). Constraints are
most useful when you have a complex set of transitive dependencies
and want to hint pip toward a fairly precise known solution to that
set.

In OpenStack, we use constraints as a community-wide “lockfile” so
we can freeze dependency sets when branching in order to stabilize
our testing, and so that we can code review and automatically test
proposed updates to versions of dependencies across many integrated
projects. We have a periodic CI job which basically tries to install
the same set of packages without any constraints applied, proposes a
freeze of the results as a constraints update to be reviewed, and
that kicks off further CI tests to find out how our projects will
behave with the proposed set of versions. Reviewers then fine-tune
the proposed update to eliminate any new problems it highlights, and
the revision is tested again… repeat until everyone’s happy and
that becomes the new constraints list.

2 Likes
13

Thanks all for the great discussion. I didn’t realize that the constraints file applied to the build system as well. I’m going to give that a try. Good to hear that the OpenStack community has had luck with this approach.

I going to try to pip freeze into the constraints file to capture every version of every package (in a working environment), and use the constraints file as basically a lock file.

14

I’m still looking at this, but the constraints file doesn’t seem to be taking effect for the build.

For example, if I run the following with verbose turned on.

echo "statsmodels==0.13.2" >> constraints.txt
echo "Cython==0.29.27" >> constraints.txt 
pip install -v statsmodels==0.13.2 -c constraints.txt

It looks like cython 3.0.0 is somehow making it in there.

Successfully installed cython-3.0.0 numpy-1.14.5 oldest-supported-numpy-2022.11.19 scipy-1.7.3 setuptools-68.0.0 wheel-0.40.0

Same thing happens if I’m less strict about the version and specify cython less than 3.

echo "statsmodels==0.13.2" >> constraints.txt
echo "Cython<3" >> constraints.txt 
pip install -v statsmodels==0.13.2 -c constraints.txt
15

When pip installs the build environment, it invokes a new copy of itself. So using -c won’t work, as that won’t get passed to the subprocess. What you need to do, as @pradyunsg suggested, is use the PIP_CONSTRAINT environment variable, which applies to all pip processes.

Also, you only need to constrain your build tools (Cython), you don’t need to specify your target library in there.

16

Oh, I see. I assumed that they were interchangeable. Thanks for the clarification.

17

Try exporting the equivalent environment variable like in the
earlier example.

https://pip.pypa.io/en/stable/topics/configuration/#environment-variables

18

The hint about using the environment variable (instead of the flag) worked! I appreciate all of the feedback. I’ll be using constraints files more frequently from now on.

19

I’m suggesting using it in the same way as you do with openstack, but with this constraint file automatically applied by integrator tools such as pip. Basically as a “help the users when the package maintainer won’t”. Not judging the package maintainer, but it would be a way forward for users.

20

Right, I’m still not clear on how “the community” maintaining an
arbitrary constraints file on behalf of random users would be
helpful. How does “the community” know what set of packages these
hypothetical users want to install together? Are you suggesting if
something like this had existed, it would have solved the problem
posed in the initial post for this topic?

I can certainly see the maintainers of an application (or suite of
applications) publishing an accompanying constraints file as an
opinionated set of the versions of its transitive dependency set its
users are expected to install in conjunction with it, but that’s
specific to deployment of a particular application/environment.

Also, rereading your initial suggestion, perhaps when you described
“an older version of A could use a patch release to add an upper
constraint” you did not mean a “constraint” file (you wouldn’t
generally include an actual constraints file in a released package
because then pip can’t access it until after the package and its
dependencies have been installed), but rather an upper limit or cap
in a version specifier for its dependencies in the package metadata?
The muddy terminology there could be where you’re losing me.

1 Like