Provide a way to signal an sdist isn't meant to be built?

[barry (Barry Warsaw)]

@steve.dower

I can see that, but I can also see that a pyproject.toml is a signal that an sdist is meant to be built.

The ambiguity with that there are a number of who’s the sdist is meant to be built by:

• The package maintainer who is building wheels locally using whatever complex build system is required
• The end user who has none of that build system knowledge or environment and who is almost surely going to fail to build the sdist (which contains the pyproject.toml file), sometimes after a lengthy inexplicable wait, and with error messages that give them no hope of knowing what to do next.
• Downstream redistributors who want to build native platform packages using machinery on top of the package’s build system requirements.

[barry (Barry Warsaw)]

@oscarbenjamin

Can the “pip don’t build me” flag not just go in trove classifiers or something?

That’s an interesting suggestion, because AFAIK, no build backend actually refers to the classifiers in any way during its build process. Which means that package maintainer local builds wouldn’t need anything special to continue to build, no metadata standard needs to change ^[1] and end user tools like pip and uv could consult the classifiers during the installation process for their default behavior (always overridable).

---

1. I’m not sure what the process is to extend the trove classifiers ↩︎

[barry (Barry Warsaw)]

@steve.dower

The critical point is that you can build your own project without a pyproject.toml.[1] It’s not inherently part of the development process - it’s inherently part of the install process

For better or worse, I don’t think package maintainers agree with this at all ^[1]. It might not have been the original intent, but I for one think of pyproject.toml as almost exclusively part of the development process, and actually not part of the installation process at all! Or put another way, as a package maintainer, pyproject.toml is for me, not the consumers of my packages.

---

1. well, at least for one data point! ↩︎

[barry (Barry Warsaw)]

@pf_moore

Too many people use pip as a build tool for me to imagine that this wouldn’t be a huge compatibility break.

Maybe that’s the core problem. pip is all things to all people. I wish it wasn’t but that’s the reality we have to live with.

[notatallshaw (Damian Shaw)]

@barry

Or put another way, as a package maintainer, pyproject.toml is for me, not the consumers of my packages.

FYI, front end tools may not see it that way, e.g. with uv a user can do uv pip install -r pyproject.toml

[barry (Barry Warsaw)]

Could be, although IME, I see -r used with a requirements.txt file of some sort way more often.

[fungi]

Maybe that’s the core problem. pip is all things to all people.
I wish it wasn’t but that’s the reality we have to live with.

To give a concrete example, I’m involved in the maintenance of a
community CI/CD system relied on by some very large and popular
Python-based project upstream developer communities. Some of the
software they develop runs “very close to the hardware” and so
relies on PyPI packaged libraries with C extensions that may be
published only in sdist form or only has wheels for the subset of
systems those libraries’ maintainers had access to build on at the
time they were released. Installing the dependencies for our
projects thus results in repeatedly rebuilding wheels for many of
these things, which is not a great use of limited system resources.

Our workaround to reduce the unnecessary repeated rebuilding of
sdists in every test is a periodic job that, for each and every
platform we use in testing, does a pip install of a list of the
exact package versions all our projects are constraining to in their
tests. The job then analyzes the pip log in order to determine which
wheels were downloaded directly and which were built locally. Then
it copies just the locally-built wheels from pip’s cache to a
central location served to the network as a --extra-index-url in
other jobs so they don’t need to spend time building those
themselves.

We take advantage of the convenience that pip install will fall
back to building sdists when no suitable wheel is available, in
order to make more wheels available to our jobs. In our case, it’s
“just” a matter of making sure that the bulk wheel cache building
job declares all the necessary system dependencies needed to build
wheels from those sdists, which is admittedly something we spend a
fair amount of time keeping on top of.

[kknechtel (Karl Knechtel)]

@pf_moore

Too many people use pip as a build tool for me to imagine that this wouldn’t be a huge compatibility break.

…wouldn’t they be using a different code path (starting at pip wheel)? Presumably the pip install cases could be distinguished elegantly enough…

[kknechtel (Karl Knechtel)]

@barry

Or put another way, as a package maintainer, pyproject.toml is for me, not the consumers of my packages.

Just for the record, I feel the same way, as someone writing pure Python packages. To me, pyproject.toml is the thing I write so that I don’t have to write setup.py.

[oscarbenjamin]

@barry

@oscarbenjamin

Can the “pip don’t build me” flag not just go in trove classifiers or something?

That’s an interesting suggestion, because AFAIK, no build backend actually refers to the classifiers in any way during its build process. Which means that package maintainer local builds wouldn’t need anything special to continue to build, no metadata standard needs to change [1] and end user tools like pip and uv could consult the classifiers during the installation process for their default behavior (always overridable).

Another possibility along these lines is just to add something in pyproject.toml under tool.build:

[tool.build]

dont_build_by_default = true
dont_build_reason = 'Needs a fortran_compiler'
install_instructions = 'https://foo.org/docs/install.html'

I know that tool.build should really be for pypa’s build but it has no need for any configuration in pyproject.toml.

[dstufft (Donald Stufft)]

@barry

That’s an interesting suggestion, because AFAIK, no build backend actually refers to the classifiers in any way during its build process. Which means that package maintainer local builds wouldn’t need anything special to continue to build, no metadata standard needs to change [1] and end user tools like pip and uv could consult the classifiers during the installation process for their default behavior (always overridable).

You’d almost certainly need a PEP to decide what trove classifier should be used for it, and that a trove classifier is the right approach [^1]. I don’t think there’s a way around needing a PEP if you want pip/uv/etc to interpret something the backend provides to indicate that building from sdist shouldn’t be used, since defining how backends and frontends interact with each other is one of the core reasons we have the standards .

The other side of this is that ideally whatever the mechanism is, is something we can expose in the simple API (technically any metadata in the METADATA file is exposed through PEP 658, but that requires a separate http request) so that pip/uv/etc can filter out sdists earlier on in the process.

Another point of concern here is whether we want something that can be turned on for previous sdists or not.

[^1] What trove classifiers are valid isn’t something that we actually define, separately PyPI has a list of classifiers it allows, but in theory you can use any trove classifier you like as long as you’re not trying to use it with PyPI.

[ofek (Ofek Lev)]

@dstufft

Another point of concern here is whether we want something that can be turned on for previous sdists or not.

Sorry, what does this mean?

[oscarbenjamin]

@dstufft

You’d almost certainly need a PEP to decide what trove classifier should be used for it, and that a trove classifier is the right approach

The only real benefit of using a trove classifier is to avoid changing core metadata via the whole PEP process. If a PEP is needed then it is better to add this in pyproject.toml adjacent to where the build backend is defined.

Ideally this would be part of PEP 725 which enumerates the primary reasons for not building the project by default.

A more limited PEP could just define a pyproject.toml entry for adding a dont_build_by_default flag as well as
standardising a URL name for installation/build instructions to go in [project.urls].

[oscarbenjamin]

@oscarbenjamin

A more limited PEP could just define a pyproject.toml entry for adding a dont_build_by_default flag as well as
standardising a URL name for installation/build instructions to go in [project.urls].

It is hard to see what else a PEP could do here besides standardising the names of these few pieces of metadata. I don’t think it should mandate anything like “installers MUST refuse to build if …” because that is really in the realm of tool UI.

[fungi]

@dstufft

Another point of concern here is whether we want something that can be turned on for previous sdists or not.

Sorry, what does this mean?

Basically, it doesn’t help to say “don’t install frobnitz x.y.z from
sdist” when pip will just fall back to installing x.y.z-1 from sdist
as a result.

[pf_moore (Paul Moore)]

@barry

Maybe that’s the core problem. pip is all things to all people. I wish it wasn’t but that’s the reality we have to live with.

Interestingly, in spite of the fact that uv is aiming to implement most (if not all) of pip’s interface, people seem at the moment to view it more as an install tool than they do for pip. It might be possible to use this to subtly change people’s view on the role of pip/uv - or at least to separate the “build tool” and “installer” ideas.

I’m not sure anyone has the motivation to do that, and I’m also not sure we have the developer resource to capitalise on such a change in viewpoint even if they did, but the opportunity exists, nevertheless.

Of course the “one packaging tool to do everything” advocates will want things to go in the opposite direction…

@kknechtel

…wouldn’t they be using a different code path (starting at pip wheel)? Presumably the pip install cases could be distinguished elegantly enough…

You’d like to think so. But in my experience, you’d be wrong, unfortunately

[kknechtel (Karl Knechtel)]

@pf_moore

You’d like to think so. But in my experience, you’d be wrong, unfortunately

Wait, what? What alternative have they even found? (insert “workflow” XKCD here…)

[barry (Barry Warsaw)]

@dstufft

I don’t think there’s a way around needing a PEP if you want pip/uv/etc to interpret something the backend provides to indicate that building from sdist shouldn’t be used

That makes sense of course. I’d lean toward a much narrower, new PEP than trying to incorporate this into PEP 725. That latter is large and complex enough that I’d be afraid this “simple” feature would get lost in the weeds.

@dstufft

The other side of this is that ideally whatever the mechanism is, is something we can expose in the simple API

That also makes sense, but it leads me to a clarifying question. I initially thought that a classifier might make sense over other (new) metadata because classifiers aren’t tied to a specific release version, which would address your other comment:

@dstufft

Another point of concern here is whether we want something that can be turned on for previous sdists or not.

Why did I think that? Because if you just visit a project page on PyPI, you see a list of classifiers which appear to be independent of release version. I think that’s an illusion though, just as with other metadata on the project page, such as the project link, license, etc. I’m now guessing that the information presented here is related to the latest release, but of course any of this could be different for earlier releases.

Which means, a classifier doesn’t actually fix the n-1 problem. Unless the PEP specified that the “don’t-build-from-sdist-by-default” flag should be consulted on the latest release, but that probably won’t work either.

[dstufft (Donald Stufft)]

@barry

Why did I think that? Because if you just visit a project page on PyPI, you see a list of classifiers which appear to be independent of release version. I think that’s an illusion though, just as with other metadata on the project page, such as the project link, license, etc. I’m now guessing that the information presented here is related to the latest release, but of course any of this could be different for earlier releases.

Yea, it’s not super clear, but basically the only metadata that is “project level” is the project name.

[pf_moore (Paul Moore)]

@kknechtel

What alternative have they even found?

Most of the issues I’ve seen reported have been with pip install, installing into development environments.

When I said “build tool”, I wasn’t specifically referring to the build/publish process, but rather to doing builds as part of the development process (so installing the in-progress source into a venv for testing would count). Doing an install to check the code compiles is common.