PEP 735: Dependency Groups in pyproject.toml

Huh? What are you saying this has been “missed” on the basis of?

That’s the whole premise of the proposal there (and here). From the 2nd sentence of the 1st post of the thread I linked to:

1 Like

It was in PEP 518:

Tables not specified in this PEP are reserved for future use by other PEPs.

2 Likes

Strongly agreed. I would be uncomfortable with this moving forward without input and support from those projects. Is it appropriate to @-mention them to pull them into this thread?

For PDM, I don’t think it’s likely to be a challenging topic (I know, I’m jinxing myself here…) – this maps pretty closely to what already exists there.

For Poetry, this spec is strictly less capable than what they have today. That might translate into “easy to implement” or it might translate into “reticent to support it”, since it would mean supporting two different versions of the same idea.

I don’t think that either one is a superset of the other, though I may be mistaken.
Dependency Groups allow you to do an install of a package list without project.dependencies data, which --only-deps can’t do.
--only-deps can install the project.dependencies data without the package itself, which Dependency Groups can’t do.

I’m confident that Dependency Groups address most or almost all of the use-cases for --only-deps, based on issues I’ve read.
The question I have right now is whether or not the outstanding case – installing project.dependencies without the package itself – is significant in its own right. I’ll read through the issues and threads after my workday to try to understand this.

If so, it might be necessary to add a way of expressing that to Dependency Groups, if we want to put pip in a situation to only implement one of the two behaviors.

It already seems like I’ll need to fix the syntax being used anyway. Hopefully these dovetail nicely.

My pessimistic side says that anything new will be criticized as “yet another way of doing the same thing”. So I want to be cautious about how we respond to this pressure – not “disregard it”, just “cautious”.

Including migration guidance for extras-to-Dependency-Groups seems like a good inclusion to me. I’ll work on that soon.

I could make my POC code (still haven’t started on it) something that outputs requirements.txt files. That would make it a useful tool for migrating and maintaining “compatibility shim files”. If it seems like a good idea, I think I can take a crack at it.

There’s an inherent tension here, which is what I’m grappling with.
I want to standardize (a subset of) requirements.txt files into a section in pyproject.toml.
This takes the form of a new standard.

I don’t think we reduce the number of ways of doing things in the short term with any (reasonable) additive improvement. We increase the options, there’s a transitory period during which people adapt, and then later we may be able to remove some of the old ways.

If we want to drop requirements.txt files entirely (:open_mouth: !), we’ll need to address --index-url and --hash at least. I’ve tried to leave space for these (--index-url could go at what is presently namespaced requirements.index-url, I think).

My goal is for this to be a spec which becomes capable enough one day replace all requirements.txt files. requirements.txt will probably never fully “die off”, but it would become the legacy way of doing things. For example, if pip-tools is willing to write back into the Dependency Group data, I could imagine a future in which I have

[tool.pip-tools]
mode = "dependency-groups"
group-map = {"async": "async-locked"}
[requirements.packages]
async = ["aiohttp"]
async-locked = [
  {"spec": "aiohttp==x.y.z", "hash": "abcdefg"},
  {"spec": "aiosignal==p.q.r", "hash": "bcdefgh"}
]

And then I could write my tox config:

[testenv]
# using a tox factor to select the *locked* variant!
dependency_groups =
    async: async-locked
commands = pytest

I know I went a little bit long there on what the future could hold, but that’s what’s in my head as a possible outcome.

A different angle is that this spec would make the Poetry and PDM Dependency Group feature-sets disappear into this new spec. So in that way we could get fewer ways of doing things in the short term. But I think that given how Poetry supports some non-standard and special things (like ^x.y.z version specifiers), it’s hard to see this fully replacing those in short order. For PDM, I think it’s more believable unless PDM has features I don’t know about which make this similarly hard.


In any case, it seems from this feedback that I have some refinements to make and I’ll try to put together some POC bodies of code to add to the PEP soon as well.

1 Like

So run mean dependency for install or runtime?
Run on its own suggests that this is something that is runnable, a script, and as such I find it misleading when it refers to dependencies.

1 Like

I was meaning in terms of “use cases satisfied”, not technical details. For projects that build into wheels (a constraint I stipulated) you can replace a dependency group with an extra that specifies the same requirements, and use pip install --only-deps .[extra]. That’s what I meant by a “strict superset”.

From my recollection, it was the original request, so yes, it’s important. The requirement was to be able to install the dependencies independently of building the project itself (which might need special configuration such as environment variables for the build).

But please don’t just add functionality to dependency groups to do this. If you do that, you’re missing the point that installing “the project dependencies only” for a project that builds into a wheel is more complex than just reading pyproject.toml, and (if the dependencies are dynamic) must of necessity involve the backend.

We’re back to the idea that projects that build into a wheel are in a sense fundamentally different from ones that don’t[1]. And we don’t yet have any form of consensus over how we deal with that.

+1. The key here is understanding what the user is trying to do, and clearly distinguishing the solutions. At the moment, there’s too much focus on implementation, and not enough on intent (IMO). We’re designing a bunch of tools that have a long handle and a heavy bit at the end. Some are hammers, some are spanners, some are pliers, etc. But the user wants to “bang something in”. How do they know that the hammer is the one intended for that use case, and the others, while they will work in a pinch, are actually inappropriate for that task? And given the range of types of hammer, how do we find out more details about what the user actually means when they say “bang something in”? We don’t want to offer them Mjolnir to fix a thumbtack to the wall…


  1. even if it’s only in the sense of “we haven’t really thought about the other types much, yet” ↩︎

2 Likes

Hello, PEP editor here! Thanks Pradyun for sharing those process notes!

It’s great people are keen on putting in the time and effort to write up PEPs, but this is the second time in a week so I’d like to re-iterate the importance of finding a sponsor or core dev co-author first, then check they’re happy with the draft, before opening a PR in the repo. Only then is a number assigned and review can begin.

This helps set expectations and saves a lot of time for everyone: PEP author’s writing, editors’ reviewing and community discussion, and can avoid putting potential sponsors in a tricky spot.

Full process is defined in PEP 1, see also this discussion for some extra notes, and don’t hesitate to open topics if you’ve any questions, we’re happy to help!

5 Likes

Yes.

This situation wasn’t helped by the fact that GitHub added “draft” pull requests as an option that can’t be disabled. :slight_smile:

2 Likes

[Brett has confirmed he’s sponsoring, so we can use PEP 735, and I’ve renamed this topic. Thanks!]

3 Likes

Several people have already weighed in on this aspect, so why not one more person :wink: ! I found this section of the PEP a bit jarring until I reasoned my way through it:

[requirements.packages]
test = ["pytest", "coverage", "."]
docs = ["sphinx", "sphinx-rtd-theme"]
typing = ["mypy", "types-requests", ".[types]"]
typing-test = ["[typing,test]", "useful-types"]

Initially I thought “this must be a typo, there’s nothing defined for types!” But of course types is defined elsewhere, in project.optional-dependencies.

All of this to say: it would be great if the syntax for these requirement groups was able to replace those other options [1]. If I were managing such a complex setup, I would want to put all my various dependency lists in one section.

While there is a distinction between extras and dependency groups, I’m not sure it’s such a distinction that the two can’t be effectively merged, with the right syntax.

One possibility is to define some default values for dependency groups, which can be overridden when needed. Another possibility is to have a top-level field that defines which groups should be packaged as extras (sort of like __all__ defines what it exported from a module). e.g.

[requirements.packages]
test = ["pytest", "coverage"]  # implicitly includes the package
docs = { dependencies = ["sphinx", "sphinx-rtd-theme"], package = false }  # don't install the package for this one
types = { dependencies = ["typeshed"], extra = true }  # moved from optional-dependencies
typing = ["mypy", "types-requests", "[types]"]
typing-test = ["[typing,test]", "useful-types"]

# or alternatively, a keyword to denote which groups are extras (defaulting to all)
[requirements]
extras = ["types"]

I’m not sure which way the defaults should go, in terms of most common usage. But it feels like this table could consolidate some of this stuff into one place, and tools would be able to install these groups using the same syntax as extras.


  1. entirely optionally, of course, not changing the existing project table ↩︎

2 Likes

I’m open to making this a more-capable replacement for package.optional-dependencies which can declare extras, if there’s popular support.

I’ll note that it aligns pretty well with how PDM handles dependency groups – by default they are extras, and a flag puts them in a separate “dev” section.

The main difference I’d like to introduce is the possibility of a dependency group which doesn’t implicitly include package.dependencies (which PDM does not have).

I think I would prefer for extra declarations – if we go this route – to always be explicit. By default, a dependency group should not be an extra. That way, the parts of your package which are public interfaces (extras) are never “on by default”.

3 Likes

I may well have misunderstood something. The arguments that I saw in that thread seemed to be particular to sdists. It came across that people were objecting to “exposing dependencies to end-users as extras” purely on aesthetic grounds, and that others were therefore not appreciating the value of hiding them.

If it’s backwards-compatible then it ignores it, else it errors out. I don’t think we want to prematurely optimize here.

That’s going back to Projects that aren't meant to generate a wheel and `pyproject.toml` and the whole reason that conversation was started. We can always revisit that, but people here already complained that it was too long and meandering, suggesting a lack of consensus to begin with.

I agree. I don’t want to try and explain that to a beginner who just wants to do the right thing and right down what dependencies the code in some zip file has. I think one of the reasons requirements files have worked is because there isn’t much ceremony to them and they map to what you specify for pip and other installers to install.

It’s similar to my strawman, although I proposed a bit of requirement syntax for referring to other dependency groups and a solution to how to write stuff down for PEP 723.

I agree it may be easy to overlook.

But this goes back to the “[project] is only for wheels” discussion since that doesn’t work for the web app scenario.

Correct.

As one of those people, it’s honestly to keep my dependencies together and in a consistent format. Using the web app example of run, test, lint, and docs dependency groups, I could have the run requirements in [project.dependencies] and then have a requirements-test.txt that consisted of:

.
pytest

and a requirements-lint.txt that had:

ruff

But that does physically separate stuff a bit and in two different formats. And on top of that it’s pip-specific and not necessarily as discoverable trying to figure out how to run tests or build docs from an sdist as using extras. But the fact they are extras which makes them a public “API” for my wheel is actually an annoyance I’m just willing to live with.

Ditto. Plus requirements files have shown to be flexible and simple enough to meet a lot of needs.

I just had a thought in response to Pradyun’s point about pip install --only-deps. In that instance we are letting pyproject.toml just record some stuff with some semantic meaning, but we are letting a tool choose how to use that information. Or put another way, with --only-deps, pip is saying it’s willing to ignore that something is meant for a wheel which won’t be installed and effectively skip the whole “this is a wheel” aspect of pyproject.toml. Up until this point, though, we have all operated under the assumption that everything in [project] is for some wheel, while --only-deps suddenly ignores that fact.

What happens if we take this a bit farther? The reason pyproject.toml requires name and version for [project] is because core metadata requires it. But what if we said that build back-ends are the ones that require name and version because they need it for core metadata generation? We can still say that name can’t be dynamic for discoverability purposes, but we could drop the requirement that pyproject.toml consumers must enforce that name always be there. Thus the requirements of the core metadata is enforced by build back-ends and not by the specification of [project].

This then lets us use project.dependencies and project.optional-dependencies as-is since the installer chooses to view that data in isolation from the rest of [project]. Whether everything in optional-dependencies infers dependencies could be left up to the installer and the user since we are giving tools more leeway in choosing how to interpret the information (i.e. some flag which tells the installer to skip project.dependencies when specifying something in project.optional-dependencies). It doesn’t change how you validate those keys in pyproject.toml, it just lets tools choose how to use the information. That then puts us back to being after a way to control what’s visible as an extra when a build back-end interprets the contents of [project] from a wheel-building perspective (i.e. how does the user communicate to the build back-end what is an extra and what’s just a group of dependencies for some other tool to work with?).

Is this basically just back to allowing for more uses of [project] and broadening it outside of wheels? Yes, but at least in my head this is a different angle to view the same outcome.

9 Likes

Err… yeah, when you put it like that it feels simple. I always forget that project.optional-dependencies is just another table. This PEP could definitely be rewritten as an expansion of [project.optional-dependencies] such that

  • the package itself (if there is one) is not an implicit dependency (or it is but there’s a way to disable that)
  • there’s a way to refer to other keys within the table, i.e. the [extra] syntax that @sirosen proposed
  • there’s a way to not bundle an extra into the distribution

And tools could/should be encouraged to allow only-deps or only-extras installs.

The change for the project table itself (making name and version optional) is a separate PEP, I think. I don’t think it’s necessary for this one to work.

1 Like

It seems to me like I already proposed this, and it was rejected. Or rather, this mindset was behind multiple aspects of what I’ve been suggesting, all of which got rejected in part due to that. People will complain that it’s “another way to do things” if a given new list can go in either of two places. Plus you need to define semantics if the [requirements.packages] and [project.optional-dependencies] definitions for a “requirements group” conflict. Plus you get people wondering why one table supports an extended syntax and the other doesn’t. Plus the wheel-builders end up leaving data behind that should have been recognized as an extra. I understand and appreciate that the idea is to be able to put everything in one section, but it ends up raising more questions about having two sections.

And then there’s the signal raised by PEP 725, that they actively want non-Python dependencies to be listed in a totally separate way.

It seems like you already came to this conclusion from the exchange with @brettcannon but I want to lay it out a bit more explicitly, with an actual design.

To get what you want, I think the idea has to be turned on its head. Instead of making a new table at all, expand the syntax for [project.optional-dependencies] - so that entries can either be an ordinary array as now, or an object that unlocks the new features. And then instead of expanding the PEP 508 syntax, cover the additional options with other keys.

For example, piggybacking off yours:

[project.optional-dependencies]
test = { packages = ["pytest", "coverage"], extra = false }
docs = { packages = ["sphinx", "sphinx-rtd-theme"], current = false, extra = false }
types = ["typeshed"]
typing = { packages = ["mypy", "types-requests"], groups = ["types"], extra = false }
typing-test = { packages = ["useful-types"], groups = ["typing", "test"], extra = false }

where the defaults are { packages = [], current = true, extra = true, groups = [] }, and a plain list simply provides a packages value and leaves the other defaults (the current behaviour).

And, of course, TOML allows writing such objects as separate tables, like:

[project.optional-dependencies.docs]
packages = ["sphinx", "sphinx-rtd-theme"]
current = false
extra = false

(which looks kinda similar to the original [[requirements.groups]] idea, actually.)

Anyway, an approach like this solves the problem of redundancy between [project.optional-dependencies] and a separate requirements-group table. But it means tools will be confronted with tons of data they weren’t prepared to consider valid - because [project.optional-dependencies] wasn’t specified with the kind of forward-compatibility mechanism that this PEP has. So if I was convinced up-thread then surely it applies here :wink:

It would also mean the opposite of these semantics, because the existing table already makes that on-by-default assumption.

One suggestion, if we want to standardize Dependency Group, we need to support local path dependencies because they are quite common in project development.

Both Poetry and PDM have solutions for local path dependencies. On this basis, I also hope to support specifying an editable build in the dependency specifier.

It may be similar to:

[requirements.packages]
test = [{path = "./packages/mypackage", editable = true}]

I’m not sure if this goes beyond the scope of this PEP, as I only see spec introduced as a key in the dependency specifier table. But it would not be much beneficial without supporting these.

From the perspective of the author of PDM here.

8 Likes

I don’t think it’s productive to worry about who suggested what first, and certainly it’s not useful to tie this PEP to anything that came previously. This stuff comes down to the nitty-gritty details of the specification.

My point is simply that there seems to be resistance to the idea generally, and I don’t see why that would change now.

Sorry, I didn’t know what you were referring to “up-thread” (I’m still not sure) so I assumed you were referring to your own proto-PEPs.

I think all these different proposals need to be considered in isolation–bringing up arguments from other proposals muddies the waters and it’s why previous threads became too complex for people to follow.

2 Likes

I for one would be very happy to see the “[project]” table expanded/modified as @brettcannon , @jamestwebber , and @kknechtel are describing.

Hi and welcome! And thanks for mentioning this - I’ve found it to be a consistent pain point when working on isolated / air-gapped networks (that are usually very unlikely to have a local package repo.)

2 Likes

I think I haven’t expressed it in this thread yet so I will restate my position a bit stronger this time: I am a hard -1 on this, Hatch will never support this, and I will continue to actively discourage embedding lock files in pyproject.toml files anywhere that I can in writing. Proponents of this don’t realize how few people want that file to be hundreds of lines long.

As for the rest of the proposal, I don’t yet have a comment because I haven’t finished reading this thread. As always, I appreciate the effort here!

8 Likes