PEP 735: Dependency Groups in pyproject.toml

[rgommers (Ralf Gommers)]

@sirosen

All of the above is stated rather declaratively, as it is my intent to make this change.
I’m always open to having my mind changed, but this seems like a case in which the author of the proposal needs to make a decision and push for it.

Hi @sirosen, it seems like you are pretty determined to push for this. I’ve expressed some concerns higher up which I won’t repeat here. I’ll just ask if you can please clearly include this topic in the backwards compatibility section of the PEP.

@sirosen

Let’s imagine that a build backend adds support. How will that play out for users?[…]. Setting aside how the PEP will have to explain this situation, the user experience will be quite good.

@sirosen

However, this is already the case any time a build backend adds a feature or fixes a bug, and the entire packaging ecosystem is not coming apart at the seams.

One thing I will point out is that this seems quite incomplete to me. You seem to assume that only build backends use this data, and once they do the work to add support for the new feature and cut a release, then the problem is solved. However, that is not the case. This metadata can be read by a host of tools when it is static. Static metadata is preferred, as expressed in the motivation section for PEP 621:

“To speak specifically to the motivation for static metadata, that has been an overall goal of the packaging ecosystem for some time. As such, making it easy to specify metadata statically is important. […]”

It can get used not only by build backends/frontends, but by tools like dependency analyzers (SBOM tooling, GitHub’s dependency list, Tidelift, libraries.io, etc.), by distro tooling like Grayskull/pyp2spec (see PEP 725 for more detail on that), as well as by custom dev/analysis scripts and tools. When a package starts using {include = ...}, that is all likely to break.

To give one concrete example of that, look at Dependencies · pypa/build · GitHub. GitHub extracts the dependencies from pyproject.toml, as noted by text below each dependency like “Detected automatically on Jan 17, 2024 (pip) · pyproject.toml”. On that same page, it has an “Export SBOM” button. It’s probably not the highest-quality SBOM generator, but anyone who uses it may see dependencies silently go missing, until GitHub adapts to the change (and it took years for them to start supporting the current format, so that adaptation may take a while).

Final thought: you also expressed that you want dependency groups to be extensible in the future. Can you please think about how to obtain that extensibility without the risk of breaking users of dependencies for a second time if such an extension does occur?