Core Metadata for self-referential extras

It’s common for projects to include extras that reference other extras defined in the space project. For example:

[project]
name = "example"
version = "0.1.0"
requires-python = ">=3.13.0"
dependencies = []

[project.optional-dependencies]
all = [
    "example[async]",
]
async = [
    "fastapi",
]

It turns out that some build backends return metadata from the PEP 517 hooks that “flatten” these recursive extras while others do not.

For example, setuptools returns:

Metadata-Version: 2.2
Name: example
Version: 0.1.0
Requires-Python: >=3.13.0
Provides-Extra: all
Requires-Dist: example[async]; extra == "all"
Provides-Extra: async
Requires-Dist: fastapi; extra == "async"

While hatchling returns:

Metadata-Version: 2.4
Name: example
Version: 0.1.0
Requires-Python: >=3.13.0
Provides-Extra: all
Requires-Dist: fastapi; extra == 'all'
Provides-Extra: async
Requires-Dist: fastapi; extra == 'async'

Are these both acceptable? Should one or the other be considered correct?

5 Likes

I didn’t know if all resolvers supported that at the time, pip gained support accidentally. My other consideration was that when auditing package metadata security tools would prefer a flattened dependency list.

1 Like

Extras are badly specified by the standards. I think that both forms have to be considered acceptable, given that they both exist “in the wild”.

If someone wanted to tidy up the behaviour of extras and formalise things, that would be a reasonable thing to do. But given that untangling the existing underspecified behaviour in a backward compatible way is likely impossible, I’d be inclined to say that a better approach would be to design a new mechanism to replace extras, leaving the existing behaviour for backward compatibility only.

2 Likes

For posterity, in case I’m not around, please whoever does this name it features.

10 Likes

Are these both acceptable? Should one or the other be considered correct?

Can you give us some motivation for asking this, is one or the other causing an issue for you?

I’d be inclined to say that a better approach would be to design a new mechanism to replace extras, leaving the existing behaviour for backward compatibility only.

If someone were so motivated to do this, my one request, would be to please please allow a user to specify that the additional requirements MUST be included, i.e. if I specify additional requirement set “bar” from requirement “foo” and “foo” doesn’t have an additional requirement set “bar” either the user receives an error or the tool backtracks on all versions of “foo” to find additional requirement set “bar”.

I think there are cases in which we could transition tools to start doing this now with extras, both preventing user error and enforcing user requests, but it certainly would be easier with a fresh feature.

1 Like

To be clear, I think we could continue incrementally improving extras. I just don’t think we have a hope of standardising them. Which means that questions like Charlie’s will continue coming up, and the answer will continue to be “if it exists, you’re going to have to handle it”…

2 Likes

For sure. In uv, I was attempting to validate the metadata we store in uv.lock against the metadata from the accompanying local source tree. Specifically, I was attempting to validate that the Requires-Dist had not changed. In this case, the relevant source tree had a dynamic version, so the metadata in uv.lock came from invoking the build backend. But when validating, I didn’t need the version, so I could read Requires-Dist from the pyproject.toml directly without going through the PEP 517 hooks. I found that the metadata from the build backend and the metadata I read from the pyproject.toml didn’t match, which led me down this path.

My personal opinion is that the spirit (perhaps even the letter?) of PEP 621 is such that it should be considered spec-incompliant to “normalize” the optional dependencies like this, since it’s not a full-fidelity representation of the declared project metadata. Though I can understand arguments either way.

1 Like

My favourite argument the other way is that it’s very Pythonic to be forgiving in what you accept :slight_smile: The language as a whole leans towards “if it works, it’s correct” and “if the caller/provider wants it to be fast, they should provide the right things”, which in this case I would say means that whoever provided that metadata should’ve normalised it if they wanted to be checked quickly, and if there are complaints, they can still do it.

2 Likes

pyproject.toml specification - Python Packaging User Guide is the spec and it doesn’t say you can’t change it outright (and in the case of optional-dependencies it has to be at least amended to added the extra clause before being written in Requires-Dist). I personally am okay with normalizing as long as the semantic meaning doesn’t change, but the preference isn’t changing since it’s simpler.