PEP 735: Dependency Groups in pyproject.toml

[brettcannon]

@sirosen

I think the question that needs answering here is “what is the behavior of a tool which implements PEP 735 if the data is extended with a future spec that the tool does not implement?”

If it’s backwards-compatible then it ignores it, else it errors out. I don’t think we want to prematurely optimize here.

@h-vetinari

I think it boils down to the fact that pyproject.toml (including project.dependencies) was once meant first and foremost for wheels, but has clearly outgrown that role. We should acknowledge that, instead of twisting ourselves into pretzels to cling to a v1 API which unforeseeable(!) evolution has shown to be inadequate.

That’s going back to Projects that aren't meant to generate a wheel and `pyproject.toml` and the whole reason that conversation was started. We can always revisit that, but people here already complained that it was too long and meandering, suggesting a lack of consensus to begin with.

@sirosen

Just to share a couple of quick thoughts on this (I’ll probably have more to say later), I had an early idea for dependency groups which looked something like this:

[[requirements.groups]]
name = "test"
is_extra = false
deps = ["pytest", "coverage"]
[[requirements.groups]]
name = "mysql"
is_extra = true
deps = ["sqlalchemy"]

But I nixed it because I thought it was so much more complicated to explain

I agree. I don’t want to try and explain that to a beginner who just wants to do the right thing and right down what dependencies the code in some zip file has. I think one of the reasons requirements files have worked is because there isn’t much ceremony to them and they map to what you specify for pip and other installers to install.

@pradyunsg

On the actual proposal: Isn’t the idea here basically the one discussed extensively in Adding a non-metadata installer-only dev-dependencies table to pyproject.toml ?

It’s similar to my strawman, although I proposed a bit of requirement syntax for referring to other dependency groups and a solution to how to write stuff down for PEP 723.

@pradyunsg

The syntax distinction between .[test] vs [test] is extremely subtle, especially since it’s “just a dot”.

I agree it may be easy to overlook.

@pradyunsg

With that, we’d basically bless the extras-based model as a good way to declare such dependencies since you can then do:

[project.optional-dependecies]
test = ["pytest"]

and then being able to use pip like:

pip install --only-deps .[test]

This will avoid needing to introduce a new concept at the standards-level, recognise the fact that this is something that people are already doing and enables the existing workflows without needing a substantial review of what the UX would need to look like for a completely new named-concept.

But this goes back to the “[project] is only for wheels” discussion since that doesn’t work for the web app scenario.

@sirosen

For my own edification, “Draft” means it has been merged but still has Status: Draft? That’s what I take this to mean.

Correct.

@pf_moore

And in particular, I think the point that some people currently use extras for this type of situation needs to be addressed.

As one of those people, it’s honestly to keep my dependencies together and in a consistent format. Using the web app example of run, test, lint, and docs dependency groups, I could have the run requirements in [project.dependencies] and then have a requirements-test.txt that consisted of:

.
pytest

and a requirements-lint.txt that had:

ruff

But that does physically separate stuff a bit and in two different formats. And on top of that it’s pip-specific and not necessarily as discoverable trying to figure out how to run tests or build docs from an sdist as using extras. But the fact they are extras which makes them a public “API” for my wheel is actually an annoyance I’m just willing to live with.

@pf_moore

That’s why I much prefer standardising existing practices, and in particular, why I prefer to strictly scope dependency groups as an[2] attempt to standardise (a subset of) requirement files.

Ditto. Plus requirements files have shown to be flexible and simple enough to meet a lot of needs.

@pf_moore

We’re back to the idea that projects that build into a wheel are in a sense fundamentally different from ones that don’t[1]. And we don’t yet have any form of consensus over how we deal with that.

I just had a thought in response to Pradyun’s point about pip install --only-deps. In that instance we are letting pyproject.toml just record some stuff with some semantic meaning, but we are letting a tool choose how to use that information. Or put another way, with --only-deps, pip is saying it’s willing to ignore that something is meant for a wheel which won’t be installed and effectively skip the whole “this is a wheel” aspect of pyproject.toml. Up until this point, though, we have all operated under the assumption that everything in [project] is for some wheel, while --only-deps suddenly ignores that fact.

What happens if we take this a bit farther? The reason pyproject.toml requires name and version for [project] is because core metadata requires it. But what if we said that build back-ends are the ones that require name and version because they need it for core metadata generation? We can still say that name can’t be dynamic for discoverability purposes, but we could drop the requirement that pyproject.toml consumers must enforce that name always be there. Thus the requirements of the core metadata is enforced by build back-ends and not by the specification of [project].

This then lets us use project.dependencies and project.optional-dependencies as-is since the installer chooses to view that data in isolation from the rest of [project]. Whether everything in optional-dependencies infers dependencies could be left up to the installer and the user since we are giving tools more leeway in choosing how to interpret the information (i.e. some flag which tells the installer to skip project.dependencies when specifying something in project.optional-dependencies). It doesn’t change how you validate those keys in pyproject.toml, it just lets tools choose how to use the information. That then puts us back to being after a way to control what’s visible as an extra when a build back-end interprets the contents of [project] from a wheel-building perspective (i.e. how does the user communicate to the build back-end what is an extra and what’s just a group of dependencies for some other tool to work with?).

Is this basically just back to allowing for more uses of [project] and broadening it outside of wheels? Yes, but at least in my head this is a different angle to view the same outcome.