How can build backends avoid breaking users when they make backwards incompatible changes?

Packaging
1

Given PEP 517 lots of projects now have their build backend defined like this:

[build-system]
requires = ["foo"]
build-backend = "foo"

This has lead to the following:

  1. When a new backend gets updated all users who build use the new version that backend

  2. This applies to users dependencies and transitive dependencies for sdists, where they don’t have control over the build-system metadata

  3. In general packages can’t update their old metadata, and many users depend on older packages

  4. Front end tools in large don’t expose warnings or provide, nor plan, robust systems of locking around build backends

In this ecosystem it has meant that any build backend that is heavily relied on that makes a backwards incompatible change can break user workflows on a very large scale, and the tooling available for users to recover isn’t well developed.

So I have some questions for those who feel they have a stake hold in the packaging ecosystem:

  • Should there be a responsibility of build backends never to make packages incompatibilities under the same requires or build-backend name? Is it reasonable to ask build backends they need to publish a new package or create a new build backend entry if they plan to make incompatibilities? And if so how would we encourage users to move to the new names?

  • Is there something front end tools can do here? If a user depends on an old package is there a way for a front end tool to easily guide the user to use an older backend if a newer version of the backend is incompatible?

6 Likes
2

For what it’s worth, the draft documentation for the uv build backend (at Build backend tracking issue · Issue #8779 · astral-sh/uv · GitHub) and uv init --build-backend uv use constraints by default, e.g., requires = ["uv_build>=0.6.9,<0.7"]. I think this could be normalized.

I think we could standardize a way for warnings in backends to be propagated to users via the frontend, but the problem is that the consumers of these packages are not often the ones who can or should resolve the warning. It’d be great to focus on surfacing them to package builders / distributors though.

However, I think setuptools is a clear special-case as far as breaking changes go, since projects without a build system declaration will use it and the version cannot be constrained (without external intervention).

5 Likes
3

I don’t want to speak for backends in general, but uv doesn’t depend on the version of Python that’s being installed into, whereas others do. This approach has been discouraged in the past because if we have build backend requirement foo>=1,<1.1, and in the future Python 3.20 is released and it requires foo v1.2+ to work, now all the pure Python packages that would otherwise work with Python 3.20 don’t because they depend on foo>=1,<1.1.

4 Likes
4

Precisely. Bounds are a two-edged sword: they may prevent a breakage when an incompatible change occurs, but they also may cause a breakage by pinning to an old version that no longer works.

And in the end, with the release rate in setuptools maintaining a specific pin would be untenable, since the breakage is rare, compared to how frequently the major version is bumped. So you’d basically be updating the pin all the time.

Let’s not also forget that all this pinning assumes isolated builds. In any other scenario, this all falls apart, because you’d end up with lots of packages having mismatched pins on the same backend.

5 Likes
5

Sorry I don’t follow the logic here. If the old version no longer works, then surely (all other versions of tools in the system being the same) it never worked? If this is the case, then the package’s build workflow has a bug. Hopefully one that can be fixed by simply changing the bound.

I suppose an issue could be caused by an update with a breaking change in one of the build backends own dependencies. So the point could be that if the build back end version is pinned, then all its deps should also be pinned. This necessitates a lock file, and the whole thing spirals into full machinery for reproducible builds.

2 Likes
6

Packages break over time. Things that used to work stop working. The simplest case — as mentioned in the post I was replying to — is when the package doesn’t work with a new Python version. But other kinds of breakage happen — for example, because the underlying system changes, and bad assumptions made in the past turn out not to work anymore.

What it spirals into is an ecosystem where you can’t install anything anymore, because different packages pin to different version ranges of the same dependencies, and there is no single solution that can satisfy them all.

1 Like
7

Well yeah that’s a classic case of dependency hell. I take your point on board as a cautionary tale, but dependency conflicts will still exist without bounds and pins, and I’d prefer to at least have tools like bounds and pins and lockfiles available to enable possible work arounds, regardless of issues they may cause with unmaintained projects.

1 Like
8

This doesn’t seem fundamentally different from any other compatibility “responsibility”, and it depends entirely on the position of the developers producing it.

Honestly, if I had a magic wand, I’d require Windows-level compatibility for every package in the Python ecosystem. Obviously, that’s never going to happen - some package developers do make a serious attempt at it, but most simply aren’t interested. So we can’t legislate compatibility from those who don’t want to “comply”.

Without being able to “enforce” compatibility on the publishing side, we have to just allow the consumer to manage it. Which we broadly recognise, and it’s why we offer guidance like avoiding upper bounds on dependencies, and tools like pip’s constraints files for users to apply their own (typically upper-bound or exclusion) restrictions.

Provided publishers have the ability to require a minimum viable version of a build backend (which they can), and provided consumers have the ability to further restrict it (which they can), and provided tools are capable of resolving differences between package’s backend dependencies (which they can[1]), there’s no further need to attempt to control backends.

If a backend developer wants to flagrantly ignore any sense of compatibility, that will be noticed and the people who don’t like that will stop using that backend. Those who do like it will continue using it. Consumers who don’t like it will stop using packages that frequently break compatibility, and prefer ones that do it in more controlled manners (but I’m repeating myself, because a build backend is just a package).

So to answer the question in the subject, yes, a build backend should make backwards incompatible changes when necessary. And they should do it as responsibly as they choose to. Just like every other package in our ecosystem.

  1. By building in isolated environments. ↩︎

4 Likes
9

@steve.dower I think this largely misses the point I outlined, which is in the ecosystem backend requires are special, and they are not controllable like install requirements, so breaking changes do have a bigger impact.

If a user has an install time transitive dependency on foo>=1 and in the future foo v3 is released that breaks their application they can add the constraint foo<3 and move on with their day.

However for build backends users may not have some luxury, given the example:

  • bar requires on build backend foo>=1
  • foo v10 breaks bar build process
  • baz requires build backend foo>=10

The user tries to pip install build bar and baz together:

  • pip downloads bar
  • pip downloads build backend foo v10 which breaks bar and the install fails

So the user follows the guidances and uses a pip’s constraints file:

  • User puts foo<10 in pip constraints file
  • pip downloads bar
  • pip downloads build backend foo v9 which succeeds in building bar
  • pip downloads foo
  • pip tried to resolve build backend requirement foo>=10 with constraint foo<10, resolution fails, and install fails

There is currently no single install command available to build these two packages together.

The user could build bar and baz manually themselves, put the wheels in a directory, point pip to that directory, and try the install again, but this assumes the user knows exactly which version of bar and baz they want, they might not, these might be transitive dependencies in a complicated requirement list that requires significant backtracking.

But even if the user does know which versions they want, we’re now asking users to manually build individual packages themselves with per package build constraint files. Which is generally not asked of at all from users for install requirements.

3 Likes
10

This is a tooling limitation. There’s absolutely nothing stopping any tool from supporting per-package constraints within a single command.

There are actually far worse compatibility offences within the current set of tooling that would lead someone to doing per-package builds anyway. If you’re building anything from source, this is just the world that you are entering into - if you want a nice, ready to use distro, go to Anaconda/ActiveState/Red Hat/etc.

If you don’t want to go to them, you are now the system integrator, and it is your responsibility to run as many commands as is necessary to make your system work. Trying to guilt-trip upstream developers into making your life easier doesn’t work.

4 Likes
11

But no such tooling feature currently exists, that’s why I my opening post has the question:

Is there something front end tools can do here?

And why I put in a feature request to help in this situation for uv.

I think the amount of user replies to recent setuptools issues shows that many users, for whatever reason, are using sdist and not prepared for the kind of complexity you layout:

As a pip maintainer I am looking for solutions, both frontend and backend, please don’t throw out subjective accusations. I’m looking for constructive solutions to a problem that has been created here, that is special, is not the same as install requirements, and as the setuptools issue showed there is a lack of current solutions.

6 Likes
12

Are the dates/times on these posts right, or are they insanely active? I’ve never seen a comment get 100 reactions as quickly as these.

But they’re actually a perfect example of the guilt tripping I mentioned (against setuptools, not by setuptools). And as I said, it doesn’t work. Projects will make their own decisions about incompatible changes (for example, I personally would have handled description-file in setuptools and not forced it back onto users, but that’s because I have a different preference for handling compatibility than those maintainers).

“Should …” is a terrible start to a question that’s seeking solutions. You want something like “How can build backends avoid breaking users when they make backwards incompatible changes?”, to which the ultimate answer is “by not making those changes” or “by making them not break”.

But the way it was framed in the first place was very closed, and very much leading towards blaming backend developers. Try and avoid that kind of framing if you want solution-focused discussions.

1 Like
13

Maybe pip could add the ability to supply per-project build constraints? That would fix this issue. Whether we’d implement it depends on how significant this problem is in practice - IMO, not having a solution is no more or less of an “ecosystem issue” than any other feature pip chooses not to implement.

Personally, I think that setuptools is, for whatever reason, going through a phase of making rather too many incompatible changes for my liking. That’s why I’ve stopped using it as my build backend of choice.

I don’t think it’s my place to argue that setuptools is behaving badly - I’ve voted with my feet, and that’s where my interest in the matter ends. But equally, I don’t think it’s pip’s responsibility to mitigate any damage caused by setuptools’ compatibility policy. We can help our users if we feel it’s useful, but that’s as far as it goes.

3 Likes
14

Yes, they are right, the issue got over 200 replies in less than a day.

That’s why I’ve extracted the core problem and brought it as a discussion here and largely avoided linking to the setuptools issue.

You’re going to have to excuse my English literary skills here. I don’t follow your reasoning on my phrasing, but I have changed the title at your request.

I think this is inevitable, build requirements are per package so it’s always been odd that build constraints are universal, but I am concerned about the UX design, I would love to see some good suggestions.

While I personally agree and have done the same, I really did not want this topic to get into a discussion about setuptools, I think there is a core design issue around how build backends are selected. But maybe it’s impossible to talk about build backends and not talk about setuptools.

5 Likes
15

You’ll need to make the case that this is happening frequently across build backends then.

16

In case it’s helpful, and to close this side discussion, the appropriate answers to a “should” question are “yes”, “no”, or “I don’t know, let me go and find out whether it’s yes or no”. Any other response is actually an argument, trying to convince the person who asked it that the question is incorrect (and yes, that’s what my responses were).

I appreciate the retitle.

How can build backends avoid breaking users when they make backwards incompatible changes?

As I facetiously said above, by not making them. But as I also said above in all seriousness, this is not in any way unique to build backends. Any library that changes their interface is going to create issues like this, and their options ultimately are to not make the changes, or to communicate a transition plan very clearly that leads all of their users to be unaffected by the change (which is much much harder than simply not making the change).

However, even though it’s now my framing of the question, I think what we actually want here is:

How can build front ends avoid breaking users when the build backend makes backwards incompatible changes and the package has not indicated that it has any particular needs?

This starts getting us into the space where interesting options exist. Better UX for constraints files is one idea, and a couple of possible directions:

  • a virtual package_name environment marker in constraints
  • a TOML based format with per package tables for build dependencies[1]
  • environment variables per package for build dependency constraints
  • plan a build script that can be modified by the user before running it (e.g. write out each command individually so the user can just run a single script to build/install everything after modifications)
  • date-based build dependency resolution that applies an implicit upper bound of the release date of the package being built

Whether any of these are good ideas or not is very open to discussion, I’m just brainstorming. As a build backend maintainer, I’m happy enough with any of these being applied to packages that use my backend (though I’d likely override the date-based one on my own installs, because I often fix bugs in the backend that improve older versions… but then I almost always get my own wheels, so it’s no big deal either way).

It seems the front ends already accurately report that the backend is responsible for the failure, as those issues show, so there’s probably not much that can be done there.

In some cases, some front ends do report mild incompatibilities in a way that appears like breakage - thinking of PyPI[2] rejecting unnormalised names and directory structures. Front ends could be flexible in what they accept (and strict in what they produce), which ought to reduce the amount of breakage users actually see. (Backends could also be flexible in what they accept and strict in what they produce, which would also help, but my reframed question puts them out of scope right now.)

I think this is just because the constraints file applies to build dependencies largely by accident :wink: The environment variable naturally flowed through to isolated environment creation, though once we realised that this is actually a very useful thing, it’s no doubt been formalised.

It’s certainly possible. Though when the topic is users being surprised that their code that has worked for 10+ years suddenly breaks, well, setuptools is really the only possible culprit. None of the other have been around that long, and don’t have the history.

  1. The overall constraints have to be global, because they have to apply equally to the entire dependency graph. But isolated builds are independent and so can have their own constraints. ↩︎

  2. They process sdists and read build metadata - hence, front end. ↩︎

1 Like
17

FWIW: flit just passed the 10 year milestone this month: flit · PyPI

2 Likes
18

Excellent. Two scapegoats :laughing:

1 Like
19

I think constraints-related UX can be improved, but can’t possibly be the primary/best answer. Constraints will be applied only after the breakage has occurred, at which point a large amount of the damage is done. The “let the end user manage it” was never a good answer for widely used packages.

Build backends should be very conservative to begin with, but they also sometimes do need to make a breaking change. Perhaps part of the answer is having a mechanism to signal that to the frontend so that the frontend actually shows it to the user. The default pip behavior of hiding all build output is quite reasonable, but also quite limiting.

4 Likes
20

Assuming we can only modify/control the front ends (which was in my post), what else would you propose?