I don’t know if @sirosen has had the time to update the draft. My reading of this post from earlier in the thread is that the include-group object would be allowed in the dependencies and optional-dependencies (i.e. extras) tables.
As mentioned in that post, it’s something that is open to discussion. But my understanding is that this would be in an upcoming revision (whenever that may be).
Yep, that’s my intent! I’ve had a ton of work and personal stuff hit this year (not all bad, but just really busy) and haven’t made it back to this PEP yet, but I definitely will.
The next draft, which I intend to be the final significant revision, will include syntax for referring to dependency groups from the project table. And I’ll really need to work on sections like How to Teach This and talk about the tool impact. The spec change is small, but I recognize that the impact is large.
To answer the basic question of “why dependency groups rather than a single dev dependency group”, I think my own experience as a library and application maintainer is that these already exist. Either they are explicit or implicit, but they’re not a new idea. As others have said, there are a mix of technical and organizational benefits to grouping dependencies.
I also know that I have had the experience of needing conflicting or distinct dependency groups for different purposes – in tox, this is often expressed using factors.
Depriving users of a vocabulary for expressing these things makes their lives more complicated when they have need of them. Dependency Groups make the easy cases a little more complex in exchange for adding no new concepts when users outgrow those usages.
I really like this idea, but I’d like to raise a question. Why creating a new table [dependency-groups] when it could just be [project.dev-dependencies]? My reasoning:
[project.optional-dependencies] does.[project.optional-dependencies] and is clearer (dependency-group) as @charliermarsh pointed out might look like the project’s optional dependencies.dev-depencencies name is clear: users already saw them if they use tools like poetry rye and PDM. Usually, people add optional dependency groups named test or dev or docs already.I’d argue the table should be below project because this is what these dependencies are: the project’s development dependencies (not the build system or something else).
For me the important thing is that this is a general mechanism, and not tied to just being for development dependencies. So I prefer the naming as it is in the PEP. Yes, it can (and will) be used for development dependencies, but if you want it for something else we don’t need to make another standard (or misuse dev-dependencies).
I understand your point! But fail to see other possible use-cases. Is this just to “future proof” dependency groups? (Honest question here.) There’s also the draft for PEP725 that is about “other” dependencies that might be necessary.
You can view this as extras for apps if you want. But I also don’t expect people to come to this with an empty table; the groups themselves will probably make it clearer what stuff is for. And the docs around [dependency-groups] should make it clear it can be used for dev-related purposes, but that’s not it’s exclusive domain.
Sure.
The simplest use-case for dependency groups is just “to group my dependencies” in a complex project. They don’t have to be for development, they don’t have to be optional, they don’t have to be extra. It’s a group of dependencies.
The PEP explicitly calls out why it is not a subtable of [project] as part of the rejected alternatives: Why is the table not named…
Supporting non-package projects is one of the goals here.
You mean you failed to see the Use Cases Appendix, right? 
If this section isn’t prominent enough in the PEP, please say so. Maybe it can be called out earlier (e.g. in the abstract with minimal fuss)
I need to break through the psychological wall between me and serious work this PEP, so there will be some smaller scale edits in the short term. Right now I’m opening a branch to switch include = ... to include-group = ...
I thought personally that reading the motivation section and first part of rationale makes it pretty clear that there are two key things you (@sirosen) are seeking to address:
and as someone interested in the second case, and as someone who followed the previous thread closely, I thought it was fairly obvious that “non-package projects” might be things that aren’t developed per se, and therefore don’t have the concept of dev dependencies.
But maybe it’s only obvious to me/people who care about that second case, so given the comments asking otherwise, maybe it would indeed make sense to link to the use cases appendix earlier on.
Maybe it’s even worth stressing somewhere early on that “development” is not really even a relevant concept for many non-wheel projects, and that all that use cases like the data science one are crying out for is some sanctioned way to specify runtime dependencies or environment/project requirements or any other arbitrary dependency list in pyproject.toml.
Maybe there should even be a “Why not just add a dev-dependencies table?” entry in rejected ideas.
Would it make sense to require unique names between dependency groups and extras/optional-dependencies? It would allow for a more streamlined UX for any tool that has to work with both dependency groups and extras through a single point that could take a name for either kind of dependency grouping and then infer what’s being asked.
Feels like a UX decision to me. I’d vote for the PEP to merely alert the reader to the fact that names may collide, and while they are somewhat interchangeable, the tool should decide whether to prefer one or the other (though we should probably give them clear names here, since I just had three goes at deciding which was “project metadata” or “package metadata” or “dependency metadata” and couldn’t decide).
The PEP could require it, but I wonder if it might be more harmful than helpful. Tools aren’t required to do a build to interact with dependency-groups, so it seems like some tools would notice (and flag it as invalid) while others would not (because they are incapable of doing so). Then the UX across tools starts to feel uneven.
I think I would like any tool which supports both and wants to smooth over the distinction to simply choose a precedence order, and handle name conflicts that way.
We could call out explicitly that a dependency group could match an extra name, but that avoiding this has the benefits that
I have a PR open with some updates to the PEP.
It mostly focuses on allowing Dependency Group Includes from project.dependencies and project.optional-dependencies. I’ve made my best effort to address @rgommers’ feedback above – for the most part, I simply accepted the fact that from the perspective of repackaging tools and analyzers, this could be a breaking change. (It won’t be for some cases of PEP 517-derived data, but it will be for others.) I’m not sure that there’s much more to be said than that. Either you think that’s a deal breaker or you don’t.[1]
The only way I can see for the expansion of project.dependencies to be 100% robust under these kinds of usages is for the data to pass through PEP 517 mechanisms, just like dynamic metadata.
The other way it can be handled is for users to be intentional about being slow to adopt new features.
I understand that those answers may be unsatisfying, but I’m trying to be forthright about the fact that there’s a tradeoff here and I don’t see an easy way around it. I’ve tried to present the situation vis-a-vis these tools in the backwards compatibility and how-to-teach sections.
There’s a small inclusion in the PR regarding Dependency Groups whose names match extras. Since it’s just a PR proposal right now, I’ll reproduce here; this is under Installing Dependency Groups:
Overlapping Install UX with Extras
Tools MAY choose to provide the same interfaces for installing Dependency Groups as they do for installing extras.
Note that this specification does not forbid having an extra whose name matches a Dependency Group. In such cases, tools must define their own semantics for precedence order or disambiguation.
Users SHOULD avoid creating Dependency Groups whose names match extras. Tools SHOULD NOT treat such matching as an error.
That’s all phrased as MAY/SHOULD. Nothing is prescriptive. Either that’s a good thing (flexible, pointing a direction without taking a hard stance) or a bad thing (too weak, not strong enough guidance). I’m very open to having my mind changed on this one; the above is just my initial take on the subject.
One of the reasons for the delay on this update is that it took me a while to accept that I actually have very little to say on this subject. I’ve stared at the problems and my opinion is that we have to accept those downsides if we want to cover the pip install --only-deps use cases. ↩︎
Technically this is true, but we can’t specify what users do at all, so I don’t think it’s appropriate to give them SHOULDs/SHOULD NOTs. They’re not the ones who are responsible for adhering to the spec.
“We would discourage users from creating …” is better language. And if there’s a “how to teach this” section then it can be called out there as a potential gotcha.
Good point; I’ve edited it to
Users are advised to avoid creating Dependency Groups whose names match extras.
Rereading the How to Teach This section, I think a larger addition is in order there. It should state that there is no PEP-defined interface for using Dependency Groups, which is what gives rise to this situation.
And it’s now merged!
I’m ready to submit this PEP, but want to give it a brief pause for final feedback and refinement. More on that below.
[project.dependencies] and [project.optional-dependencies] allow the use of includes
--only-deps-like use case.We’re closing in on the end of this epic, and I see it showing up in other discussions (e.g. lockfiles) as an expected coming spec. After an initial flurry of activity, it has settled into a relatively tightly scoped addition to the ecosystem. Some people will wish that it did more and some will wish that it did less, which is in the nature of compromise.
I hope that it will be accepted in its current form, and think that it is just about the best version of this idea which I could hope to produce. But, of course, I did not produce it on my own, and I’m tremendously indebted to the community for giving thoughtful feedback, correcting me when I was in error and allowing me to drive the work in a particular direction.
The main thing left to do is to submit the PEP. I want to do that today, but after such a long break without activity, and then some final changes, submitting it too quickly does not strike me as respectful of everyone’s effort to get the document to its current stage. I will wait at least two weeks – a little patience is the least I can offer to thank you all for being patient with me – and in that time I can pursue some other efforts related to Dependency Groups, like a small pypi package implementation, and share them here as I make progress.
Hi! Thanks for the update. We’ve been watching this PEP while considering what to do with development dependency groups over in uv. I’ll read through it this week and make sure we can support it as-written and make note of any concerns.
In addition to the reference implementation, I’ve recently created and published a new package, dependency-groups, which provides a library-ized implementation plus a CLI and pre-commit hook for linting: GitHub - sirosen/dependency-groups: A standalone implementation of PEP 735 Dependency Groups
The main difference vs the reference implementation is that this version includes caching of resolved groups, which speeds up some usages.
As a fun, but mostly inconsequential, tidbit, I have a tox environment which converts [dependency-groups] in the project’s pyproject.toml file into requirements.txt files:
To the best of my knowledge, that makes this the first published package to use a [dependency-groups] table.[1]
Although its usage is slightly roundabout! ↩︎
First of all, we (the uv team) are very excited about the addition of dependency groups. We’ve been holding off on exposing development dependency groups (there is just one dev-dependencies section for uv right now) while we wait for a decision on this PEP. This is one of our most requested features, and I think it addresses a valuable need in the ecosystem. I’ve compiled some feedback from @charliermarsh and @konstin. Sorry for anything that’s repetitive; I did my best to look through the existing discussion, but it’s a 2+ hour thread and it feels impossible to capture all the context. Thank you Stephen for keeping track of everything!
I’m a bit worried about the lack of discussion (in the PEP) around locking and requirements.txt files. It’s very common for requirements.txt files to be used for locked versions of dependencies but throughout the PEP they are discussed as though they were requirement declaration files (more commonly, requirements.in). The lack of distinction here isn’t a big deal, but I think the text could be improved in this regard to prevent confusion; especially with regards to teaching the feature. However, you’ll see much of my commentary comes from a lack of focus on locking the declared dependencies.
Because extras are package metadata, they are not usable when a project does not build a distribution (i.e., is not a package).
These can be, and are, statically extracted by various tools including uv — even if the project is not built into a distribution. I see context on this in the thread (1), but it’d be nice to avoid stating this in the PEP.
To support users in this case, this PEP defines and recommends validation behaviors in which tools only examine Dependency Groups which they are using. This allows multiple tools, using different versions of Dependency Groups data, to share a single table in
pyproject.toml.
uv will need to read all the groups for locking.
Use of a Dependency Group like test to test a package requires that the user’s configuration or toolchain also installs
.. For example,
It makes sense to require opt-in here, but two operations feels like a non-starter for us. We need to know if the group is intended to be compatible with the package for locking. I have a strong preference for declaring this statically. I believe this has been discussed quite a bit (e.g., in 2) and it sounds like it will need to be tackled separately? I’m not sure what the expectation is for tools in the meantime as this is a critical part of the workflow and we will be forced to write a bespoke implementation.
Note that this specification does not forbid having an extra whose name matches a Dependency Group. In such cases, tools must define their own semantics for precedence order or disambiguation. Users are advised to avoid creating Dependency Groups whose names match extras. Tools SHOULD NOT treat such matching as an error.
These two statements seem contrary and prone to confusion. Why let this be ambiguous?
I see a couple arguments in the thread:
Tools choosing their own user-experience when a name conflict occurs seems much more problematic than some tools not being able to recognize the conflict at all. If tools choose a different precedence, doesn’t that defeat the purpose of the standard here? It seems safer to be restrictive and have a consistent behavior in the tooling.
I think uv would error (e.g., on uv add), or warn (e.g., on uv sync), if this was encountered.
Eager validation is discouraged for tools which primarily install or resolve Dependency Groups.
As previously noted, uv must do this for locking.
The mutual compatibility of Dependency Groups is not guaranteed. For example, the Data Science example above shows conflicting versions of scikit-learn. Therefore, installing multiple locked dependency groups in tandem may require that tools apply additional constraints or generate additional lockfile data.
I believe that uv, Poetry, and PDM will all encounter this problem. This problem exists today, with project.optional-dependencies, so I don’t consider it to be critical for this PEP — but I think these use-cases are important to consider.
There are a variety of use-cases in which package developers would like to install only the dependencies of a package, without the package itself.
This motivation for adding include syntax to the project table feels like a stretch. Installing the project alongside its dependencies is not something installers must do — it’s just common. Tools are figuring out a consistent way to expose this feature, but, for example, both Poetry and uv let you skip installation of the project. Support for excluding the project feels like an installer feature, not something that should be encoded in the metadata itself.
Using include syntax to solve project installation creates a complicated user experience — each project will need to then move their dependencies from the project.dependency table into a new dependency group. Then, at install time, consumers will need to determine the name of the group that houses the project’s dependencies and just install that. The group name can differ per project so, unlike when implemented at the installer level, the invocation will differ per project. Furthermore, it seems like a loss if the best practice for Python projects is to move dependencies out of the project.dependencies table into a dependency-group section. This example from the discussion (5) is particularly concerning as an ideal outcome:
[project]
dependencies = [{include = "runtime"}]
[optional-dependencies]
foo = [{include = "foo"}]
[dependency-groups]
runtime = ["a", "b"]
foo = ["c", "d"]
This change also feels dangerous from an adoption perspective. If a package uses this syntax, it will not be installable by older versions of package managers and will break static metadata readers. For example, when project metadata is static, uv will read it directly instead of invoking the build backend — but if the new syntax is used it will fail. New versions of uv can support this syntax, but the old versions will always be broken. This is also discussed for things like Dependabot and IDEs above (6).
I worry this change will create more fragmentation in the ecosystem. If there was a clear, strong benefit to this feature — I think the fragmentation could be acceptable but I don’t see enough justification for this additional functionality. From the discussion, it sounds this outcome was driven by not wanting to increase the scope of the PEP to handle declaration of a dependency on the project itself, I’d rather see this addressed directly in another PEP instead of adopting an awkward solution now. This is a core part of the user experience and it deserves some focused attention.
I know you previously said that you don’t have much more to say on the topic of this specific change (7), so I don’t expect you to address these points — all I can do is voice our shared concern.
As a final note, thanks again for all your work on this. I appreciate that you’re trying to improve the experience for users that aren’t building distributions.