I don’t know if @sirosen has had the time to update the draft. My reading of this post from earlier in the thread is that the include-group object would be allowed in the dependencies and optional-dependencies (i.e. extras) tables.
As mentioned in that post, it’s something that is open to discussion. But my understanding is that this would be in an upcoming revision (whenever that may be).
Yep, that’s my intent! I’ve had a ton of work and personal stuff hit this year (not all bad, but just really busy) and haven’t made it back to this PEP yet, but I definitely will.
The next draft, which I intend to be the final significant revision, will include syntax for referring to dependency groups from the project table. And I’ll really need to work on sections like How to Teach This and talk about the tool impact. The spec change is small, but I recognize that the impact is large.
To answer the basic question of “why dependency groups rather than a single dev dependency group”, I think my own experience as a library and application maintainer is that these already exist. Either they are explicit or implicit, but they’re not a new idea. As others have said, there are a mix of technical and organizational benefits to grouping dependencies.
I also know that I have had the experience of needing conflicting or distinct dependency groups for different purposes – in tox, this is often expressed using factors.
Depriving users of a vocabulary for expressing these things makes their lives more complicated when they have need of them. Dependency Groups make the easy cases a little more complex in exchange for adding no new concepts when users outgrow those usages.
I really like this idea, but I’d like to raise a question. Why creating a new table [dependency-groups] when it could just be [project.dev-dependencies]? My reasoning:
[project.optional-dependencies] does.[project.optional-dependencies] and is clearer (dependency-group) as @charliermarsh pointed out might look like the project’s optional dependencies.dev-depencencies name is clear: users already saw them if they use tools like poetry rye and PDM. Usually, people add optional dependency groups named test or dev or docs already.I’d argue the table should be below project because this is what these dependencies are: the project’s development dependencies (not the build system or something else).
For me the important thing is that this is a general mechanism, and not tied to just being for development dependencies. So I prefer the naming as it is in the PEP. Yes, it can (and will) be used for development dependencies, but if you want it for something else we don’t need to make another standard (or misuse dev-dependencies).
I understand your point! But fail to see other possible use-cases. Is this just to “future proof” dependency groups? (Honest question here.) There’s also the draft for PEP725 that is about “other” dependencies that might be necessary.
You can view this as extras for apps if you want. But I also don’t expect people to come to this with an empty table; the groups themselves will probably make it clearer what stuff is for. And the docs around [dependency-groups] should make it clear it can be used for dev-related purposes, but that’s not it’s exclusive domain.
Sure.
The simplest use-case for dependency groups is just “to group my dependencies” in a complex project. They don’t have to be for development, they don’t have to be optional, they don’t have to be extra. It’s a group of dependencies.
The PEP explicitly calls out why it is not a subtable of [project] as part of the rejected alternatives: Why is the table not named…
Supporting non-package projects is one of the goals here.
You mean you failed to see the Use Cases Appendix, right? 
If this section isn’t prominent enough in the PEP, please say so. Maybe it can be called out earlier (e.g. in the abstract with minimal fuss)
I need to break through the psychological wall between me and serious work this PEP, so there will be some smaller scale edits in the short term. Right now I’m opening a branch to switch include = ... to include-group = ...
I thought personally that reading the motivation section and first part of rationale makes it pretty clear that there are two key things you (@sirosen) are seeking to address:
and as someone interested in the second case, and as someone who followed the previous thread closely, I thought it was fairly obvious that “non-package projects” might be things that aren’t developed per se, and therefore don’t have the concept of dev dependencies.
But maybe it’s only obvious to me/people who care about that second case, so given the comments asking otherwise, maybe it would indeed make sense to link to the use cases appendix earlier on.
Maybe it’s even worth stressing somewhere early on that “development” is not really even a relevant concept for many non-wheel projects, and that all that use cases like the data science one are crying out for is some sanctioned way to specify runtime dependencies or environment/project requirements or any other arbitrary dependency list in pyproject.toml.
Maybe there should even be a “Why not just add a dev-dependencies table?” entry in rejected ideas.
Would it make sense to require unique names between dependency groups and extras/optional-dependencies? It would allow for a more streamlined UX for any tool that has to work with both dependency groups and extras through a single point that could take a name for either kind of dependency grouping and then infer what’s being asked.
Feels like a UX decision to me. I’d vote for the PEP to merely alert the reader to the fact that names may collide, and while they are somewhat interchangeable, the tool should decide whether to prefer one or the other (though we should probably give them clear names here, since I just had three goes at deciding which was “project metadata” or “package metadata” or “dependency metadata” and couldn’t decide).
The PEP could require it, but I wonder if it might be more harmful than helpful. Tools aren’t required to do a build to interact with dependency-groups, so it seems like some tools would notice (and flag it as invalid) while others would not (because they are incapable of doing so). Then the UX across tools starts to feel uneven.
I think I would like any tool which supports both and wants to smooth over the distinction to simply choose a precedence order, and handle name conflicts that way.
We could call out explicitly that a dependency group could match an extra name, but that avoiding this has the benefits that
I have a PR open with some updates to the PEP.
It mostly focuses on allowing Dependency Group Includes from project.dependencies and project.optional-dependencies. I’ve made my best effort to address @rgommers’ feedback above – for the most part, I simply accepted the fact that from the perspective of repackaging tools and analyzers, this could be a breaking change. (It won’t be for some cases of PEP 517-derived data, but it will be for others.) I’m not sure that there’s much more to be said than that. Either you think that’s a deal breaker or you don’t.[1]
The only way I can see for the expansion of project.dependencies to be 100% robust under these kinds of usages is for the data to pass through PEP 517 mechanisms, just like dynamic metadata.
The other way it can be handled is for users to be intentional about being slow to adopt new features.
I understand that those answers may be unsatisfying, but I’m trying to be forthright about the fact that there’s a tradeoff here and I don’t see an easy way around it. I’ve tried to present the situation vis-a-vis these tools in the backwards compatibility and how-to-teach sections.
There’s a small inclusion in the PR regarding Dependency Groups whose names match extras. Since it’s just a PR proposal right now, I’ll reproduce here; this is under Installing Dependency Groups:
Overlapping Install UX with Extras
Tools MAY choose to provide the same interfaces for installing Dependency Groups as they do for installing extras.
Note that this specification does not forbid having an extra whose name matches a Dependency Group. In such cases, tools must define their own semantics for precedence order or disambiguation.
Users SHOULD avoid creating Dependency Groups whose names match extras. Tools SHOULD NOT treat such matching as an error.
That’s all phrased as MAY/SHOULD. Nothing is prescriptive. Either that’s a good thing (flexible, pointing a direction without taking a hard stance) or a bad thing (too weak, not strong enough guidance). I’m very open to having my mind changed on this one; the above is just my initial take on the subject.
One of the reasons for the delay on this update is that it took me a while to accept that I actually have very little to say on this subject. I’ve stared at the problems and my opinion is that we have to accept those downsides if we want to cover the pip install --only-deps use cases. ↩︎
Technically this is true, but we can’t specify what users do at all, so I don’t think it’s appropriate to give them SHOULDs/SHOULD NOTs. They’re not the ones who are responsible for adhering to the spec.
“We would discourage users from creating …” is better language. And if there’s a “how to teach this” section then it can be called out there as a potential gotcha.
Good point; I’ve edited it to
Users are advised to avoid creating Dependency Groups whose names match extras.
Rereading the How to Teach This section, I think a larger addition is in order there. It should state that there is no PEP-defined interface for using Dependency Groups, which is what gives rise to this situation.
And it’s now merged!
I’m ready to submit this PEP, but want to give it a brief pause for final feedback and refinement. More on that below.
[project.dependencies] and [project.optional-dependencies] allow the use of includes
--only-deps-like use case.We’re closing in on the end of this epic, and I see it showing up in other discussions (e.g. lockfiles) as an expected coming spec. After an initial flurry of activity, it has settled into a relatively tightly scoped addition to the ecosystem. Some people will wish that it did more and some will wish that it did less, which is in the nature of compromise.
I hope that it will be accepted in its current form, and think that it is just about the best version of this idea which I could hope to produce. But, of course, I did not produce it on my own, and I’m tremendously indebted to the community for giving thoughtful feedback, correcting me when I was in error and allowing me to drive the work in a particular direction.
The main thing left to do is to submit the PEP. I want to do that today, but after such a long break without activity, and then some final changes, submitting it too quickly does not strike me as respectful of everyone’s effort to get the document to its current stage. I will wait at least two weeks – a little patience is the least I can offer to thank you all for being patient with me – and in that time I can pursue some other efforts related to Dependency Groups, like a small pypi package implementation, and share them here as I make progress.