PEP 735: Dependency Groups in pyproject.toml

[sirosen (Stephen Rosen)]

Thanks for your detailed feedback! And thanks for reading the thread – it’s quite long, so I hope not too many people feel the need to read it all, but it does contain useful context.
There’s a lot of ground to cover, so let’s get right to it.

@zanie

I’m a bit worried about the lack of discussion (in the PEP) around locking and requirements.txt files. It’s very common for requirements.txt files to be used for locked versions of dependencies but throughout the PEP they are discussed as though they were requirement declaration files (more commonly, requirements.in).

Would it suffice to note and clarify that this is how the PEP speaks? requirements.in has become much more common in recent years, but I don’t think it qualifies as a district file format.
Dependency Groups may be locking inputs, but not every project pins dependencies.

I think all you’re asking for is to have the PEP clarify this, but please correct me if I’m mistaken.

@zanie

Because extras are package metadata, they are not usable when a project does not build a distribution (i.e., is not a package).

These can be, and are, statically extracted by various tools including uv — even if the project is not built into a distribution. I see context on this in the thread (1), but it’d be nice to avoid stating this in the PEP.

There are two subtleties here which I think are important. One is that extras can only be reliably extracted by being ready to invoke build hooks, since they may be defined dynamically. The other is that they are defined in the [project] table, which means that for projects which do not have that table (because they are not built packages), adding extras requires newly defining themselves as packages.

I’m open to altering the text, but some of the important benefits of Dependency Groups are that they are guaranteed to be static and external from the [project] table. These distinguish them from extras.

@zanie

uv will need to read all the groups for locking.

I think this is fine / expected. I wanted to write things such that an expansion of the spec would have minimal impact on a project, but I’ve become a bit doubtful that’s how it would play out in practice. Even a simple lint tool immediately needs to read all of the data.

I don’t think the spec needs to change in what it requires, or even in recommending that tools refrain from eagerly reading the whole table if they don’t need it, but perhaps it should simply de-emphasize this point. I’ll review this section in the coming days to see about some edits.

@zanie

Use of a Dependency Group like test to test a package requires that the user’s configuration or toolchain also installs .. For example,

It makes sense to require opt-in here, but two operations feels like a non-starter for us. We need to know if the group is intended to be compatible with the package for locking. I have a strong preference for declaring this statically. I believe this has been discussed quite a bit (e.g., in 2) and it sounds like it will need to be tackled separately? I’m not sure what the expectation is for tools in the meantime as this is a critical part of the workflow and we will be forced to write a bespoke implementation.

I understand wanting a better UX than two commands for this case. But isn’t this a matter of defining an environment or workspace, and declaring that environment to require the project + one of its dependency groups? I’m unclear on how the spec is failing you in this case.

If I understand your locking concern correctly, you’re saying that it’s important to know whether or not the Dependency Group should be locked in a way which is compatible with the project dependencies, or if it should be locked independently. Neither seems safe to assume, since a dependency group could be incompatible with [project.dependencies] or could be required to be compatible. I need to think more about this, as it’s not a familiar problem-space to me right now.

@zanie

Note that this specification does not forbid having an extra whose name matches a Dependency Group. In such cases, tools must define their own semantics for precedence order or disambiguation. Users are advised to avoid creating Dependency Groups whose names match extras. Tools SHOULD NOT treat such matching as an error.

These two statements seem contrary and prone to confusion. Why let this be ambiguous?

First off, this is relatively new text and little-discussed, so it’s very likely that it can be improved.

There’s no guarantee that a tool reading [dependency-groups] can read [project.optional-dependencies] (in the dynamic case). Therefore, requiring tools to error or warn doesn’t seem feasible. And there’s no guarantee that Dependency Groups and Extras are mixed into the same usages, since that depends on the tool, so it’s unclear what a spec-mandated precedence order would mean.

I’m more attached to the first sentence in the text than any of the rest of it. Would it be acceptable to simply remove the note on tools defining their own semantics, and flip the SHOULD NOT to a MAY?

Note that this specification does not forbid having an extra whose name matches a Dependency Group. Users are advised to avoid creating Dependency Groups whose names match extras. Tools MAY treat such matching as an error.

I think it has to remain open, but I’m okay with the spec guiding towards implementations treating this as bad usage.

---

I’m basically out of time to reply at the moment, so I’m not able to get into full detail on the decision to allow includes from the [project] tables, but I’ll give a small note to point at something which drove this.

A major concern I had was what the landscape looks like if this feature is omitted today and added in a following spec, “PEP N”. Then the situation for tooling becomes more difficult for users, as some tools “Support PEP 735 Dependency Groups but not PEP N” and other tools “Support PEP 735 Dependency Groups and PEP N”.

If the feature itself is not wanted (I have my opinion, but I’m open to being convinced), then there never would be such a “PEP N” and this concern disappears.

[brettcannon]

@zanie

Support for excluding the project feels like an installer feature, not something that should be encoded in the metadata itself.

I think it depends on who should be in charge of not installing the project itself when it’s not necessary (whether this is because you’re not using src/, you don’t need to build your docs, etc. I don’t think plays into it).

Let’s use building your docs as an example. With PEP 735 as-is, your docs dependency group can leave out your runtime group. So how do I document to build my docs? With PEP 735 as-is I can say “install the “docs” dependency group” and then “run mkdocs” or something. If leaving out the runtime requirements is left up to the installers then you need to either document that as part of installation or hide that detail behind another tool like Nox of just so your contributors don’t need to know that detail.

So I think this begs the question of who are dependency groups for? Are they for the maintainer to help organize dependencies into groups, or is it for contributors to a project? The former suggests what @zanie is wanting is totally fine, but I think the latter case leans into what the PEP currently has.

[zanie (Zanie Blue)]

@sirosen

Would it suffice to note and clarify that this is how the PEP speaks? requirements.in has become much more common in recent years, but I don’t think it qualifies as a district file format.

Dependency Groups may be locking inputs, but not every project pins dependencies.

I think all you’re asking for is to have the PEP clarify this, but please correct me if I’m mistaken.

Yes, I think it would be sufficient to update the PEP to clarify this. Especially in the “How to Teach This” section. In the uv issue tracker we see a lot of confusion about the difference and role of pinned and unpinned requirements. I think it’s worth trying to get ahead of that here.

@sirosen

One is that extras can only be reliably extracted by being ready to invoke build hooks, since they may be defined dynamically.

Yes this makes sense — we do invoke the build backend if we detect dynamic metadata. Most users aren’t defining these dynamically but it’s certainly a relevant use-case.

@sirosen

The other is that they are defined in the [project] table, which means that for projects which do not have that table (because they are not built packages), adding extras requires newly defining themselves as packages.

I’d have to look back at the relevant specification here, but I believe this is ambiguous. Notably, when working on a local project, uv no longer uses the presence of a [project] table alone to determine if a project should be a built package. I understand this is not common, but it greatly improves the experience for users that are not publishing their project as they can define and use optional dependency groups without the additional complexity of building the project.

Maybe it’d be better to focus this section on guarantees? e.g.

Because extras are package metadata, they are not guaranteed to be statically defined and may require a build system to resolve.

Is that clearer to you?

@sirosen

But isn’t this a matter of defining an environment or workspace, and declaring that environment to require the project + one of its dependency groups? I’m unclear on how the spec is failing you in this case.

My concern here is that now there needs to be another abstraction to understand the available packages for a command invocation in a project. There’s already:

1. The project and its build system
2. The project’s dependencies
3. The project’s optional dependencies
4. The dependency groups

And now:

5. The environment, which defines some combination of all of the above.

Notably, this last type will be tool-defined and not a part of the standard. In one design for dependency groups in uv, we were initially considering making this a property of the group, e.g.:

[tool.uv.groups.foo]
dependencies = [...]
include_project = true

Or, in a syntax more similar to the one used in the PEP:

[dependency-groups]
foo = ["bar", { include-project = true }]

In short, I’m wary of introducing more levels of abstraction to accomplish a core use-case instead of empowering the existing abstractions.

@sirosen

If I understand your locking concern correctly, you’re saying that it’s important to know whether or not the Dependency Group should be locked in a way which is compatible with the project dependencies, or if it should be locked independently. Neither seems safe to assume, since a dependency group could be incompatible with [project.dependencies] or could be required to be compatible.

Yes, this is an accurate description of the problem. This is why it’d be useful to know statically if the dependency group is expected to be standalone or compatible with the project. Basically, we must assume that (1) all groups are compatible with each other and the project or (2) that groups are all independent and fully describe the expected environment. (1) is an unfortunate restriction for users and is something we’re working on resolving but the problem is quite hard. (2) is an ideal situation for running commands in locked environments, but isn’t addressed by this PEP.

@sirosen

Would it be acceptable to simply remove the note on tools defining their own semantics, and flip the SHOULD NOT to a MAY?

Yeah I like this version better.

@sirosen

A major concern I had was what the landscape looks like if this feature is omitted today and added in a following spec, “PEP N”. Then the situation for tooling becomes more difficult for users, as some tools “Support PEP 735 Dependency Groups but not PEP N” and other tools “Support PEP 735 Dependency Groups and PEP N”.

I think this is a fair concern. If the “inclusion or exclusion of the project” use-case was a smaller part of the user experience, I’d totally agree it makes sense to do it in a single change. However, from the feedback from our users in uv, I think this is a critical part of the experience and deserves a focused PEP. The change feels like a large increase scope for this PEP, not in syntax but in the effect on users. There are also alternative ways to implement support for this use-case without changing the [project] metadata. I don’t think it’s a given that doing a second PEP will create more adoption problems in ecosystem. However, changing the [project] metadata in this PEP will definitely significanly increase the fragmentation in tooling. Launching dependency groups without changing the project metadata will create an opportunity to understand their role in the ecosystem and inform a clear design for addressing additional use-cases.

@brettcannon

With PEP 735 as-is I can say “install the “docs” dependency group” and then “run mkdocs” or something. If leaving out the runtime requirements is left up to the installers then you need to either document that as part of installation or hide that detail behind another tool

I don’t follow this argument, but maybe I made an assumption that we were on the same page about leaving out the project by default. I think leaving out the “runtime” requirements (as defined by [project] should be the default experience for using a dependency group. So, running the docs is documented as uv run --group docs mkdocs.

The problem comes running a tool that requires a project. For example, I’d hope that running mypy was something like uv run --group type-check mypy. However, since the groups themselves don’t define if the project is included, you’d actually need to do something like uv run --group type-check --include-project mypy or take all your runtime dependencies out of the [project.dependencies] section and put them into a dependency group so you can have the type-check group depend on them. I’d like to avoid requiring a contributor to understand if the project is required and avoid the indirection of moving dependencies out of the [project] table into an arbitrary group. Instead, I’d rather have include-project be defined as part of the type-check group.

I don’t think this PEP leans into supporting either contributors or maintainers more than the other.

[brettcannon]

@zanie

but maybe I made an assumption that we were on the same page about leaving out the project by default.

Ah, while the PEP doesn’t include it I did assume most dependency groups would include a “runtime” group that was used in project.dependencies, e.g.:

@zanie

[project]
dependencies = [{include = "runtime"}]
[optional-dependencies]
foo = [{include = "foo"}]
[dependency-groups]
runtime = ["a", "b"]

@zanie

you’d actually need to do something like uv run --group type-check --include-project mypy or take all your runtime dependencies out of the [project.dependencies] section and put them into a dependency group so you can have the type-check group depend on them.

I was expecting the latter.

@zanie

I’d like to avoid requiring a contributor to understand if the project is required and avoid the indirection of moving dependencies out of the [project] table into an arbitrary group. Instead, I’d rather have include-project be defined as part of the type-check group.

Ah, you want to change the PEP so there’s a specific way to signify a dependency group should include project.dependencies so that it acts as a first-class participant via some custom object ala {include-project=true}. It seems reasonable to me and would greatly negate wanting include-group to apply to project.dependencies which helps with backwards-compatibility. I don’t know if it goes so far as to make include-groups not apply to project-optional-dependencies or makes include-groups not worth having at all, though.

[zanie (Zanie Blue)]

@brettcannon

Ah, you want to change the PEP so there’s a specific way to signify a dependency group should include project.dependencies so that it acts as a first-class participant via some custom object ala {include-project=true}. It seems reasonable to me and would greatly negate wanting include-group to apply to project.dependencies which helps with backwards-compatibility.

Yeah this is basically my stance. Though I think I lean more towards just removing the complicated part from the PEP and addressing project inclusion separately.

@brettcannon

I don’t know if it goes so far as to make include-groups not apply to project-optional-dependencies or makes include-groups not worth having at all, though.

I don’t know if the PEP makes a strong argument for including groups in project.optional-dependencies either. Outside of the project metadata, it seems generally useful to include groups from other groups though I think the PEP would benefit from demonstrating a clearer motivation. For example, the PEP includes the example:

[dependency-groups]
test = ["pytest<8", "coverage"]
typing = ["mypy==1.7.1", "types-requests"]
lint = ["black", "flake8"]
typing-test = [{include-group = "typing"}, "pytest<8"]

But why wouldn’t you just include both groups at runtime, e.g., --group typing --group test? Should we be creating groups that are just unions of other groups? Why use a subset of the test requirements and repeat pytest<8 (that seems like a bad outcome, perhaps it’s a mistake)?

I find the PEP example for data science a bit more convincing:

[dependency-groups]
main = ["numpy", "pandas", "matplotlib"]
scikit = [{include-group = "main"}, "scikit-learn==1.3.2"]
scikit-old = [{include-group = "main"}, "scikit-learn==0.24.2"]

But once you introduce incompatibilities in groups suddenly you can’t use arbitrary groups together and both the resolver and user are in a tough place. Arguably, these should just be defined as separate projects or use PEP 723 metadata for single scripts.

[jamestwebber (James Webber)]

@zanie

ut once you introduce incompatibilities in groups suddenly you can’t use arbitrary groups together and both the resolver and user are in a tough place.

What’s tough about it? You ask for incompatible dependencies and you get an error message, just like you would now.

[ncoghlan (Alyssa Coghlan)]

@zanie

Or, in a syntax more similar to the one used in the PEP:

[dependency-groups]
foo = ["bar", { include-project = true }]

I came back to the thread to make this exact suggestion after re-reading the current PEP text on dependency group inclusion and referencing groups from the project table.

It felt cleaner than reserving a special group name like :project: to refer to the project:

[dependency-groups]
foo = ["bar", { include-group = ":project:" }]

However, I realised there’s a genuine advantage to the second approach, as that syntax also makes all the project’s optional dependencies available as named groups using the established notation for installing extras:

[dependency-groups]
foo = ["bar", { include-group = ":project[baz,narf,zort]:" }]

Without that, extras end up in the same position as runtime dependencies: if you want to refer to the extras groups as dependencies, you have to move the details out of the existing fields, and use the new backwards incompatible syntax to add them back in. You would also need to add another inclusion variant to let groups refer to extras:

[dependency-groups]
foo = ["bar", { include-extra = "baz" }, { include-extra = "narf" }, { include-extra = "zort" }]

In addition to being more verbose, the fact including an extra implies { include-project = true } also becomes less clear with a dedicated syntax for including extras.

[zanie (Zanie Blue)]

@jamestwebber

What’s tough about it? You ask for incompatible dependencies and you get an error message, just like you would now.

A user can’t know what groups are compatible or incompatible without trying them — the incompatibilities could be in transitive dependencies. Some resolvers may not even be able to clearly explain that the incompatibility comes from the interaction of two groups, e.g., if they are not using an algorithm like PubGrub, which will create further confusion for users. To create a lockfile for a project that allows incompatible groups, the resolver would need to perform 2^N (where N = # groups) resolutions to solve for all possible group combinations which could be very expensive. Alternatively, a subset of these resolutions would need to be defined and committed — this introduces another tool-specific abstraction that users must learn.

@ncoghlan

Without that, extras end up in the same position as runtime dependencies: if you want to refer to the extras groups as dependencies, you have to move the details out of the existing fields, and use the new backwards incompatible syntax to add them back in

This is a fair point, but you could just extend the schema of the object instead of introducing special strings:

[dependency-groups]
foo = ["bar", { include-project = true, extras = ["foo", "bar"] }]

Or:

[dependency-groups]
foo = ["bar", { include-project = { extras = ["foo", "bar"] }}]

I think we could go back and forth on the best syntax for such a thing and I’m not sure if this PEP is the place for it — I still feel it would be a stronger proposal with a reduced scope.

[jamestwebber (James Webber)]

@zanie

A user can’t know what groups are compatible or incompatible without trying them — the incompatibilities could be in transitive dependencies.

Or by reading the documentation for the project, perhaps? Why are they trying to install these dependency groups in the first place?

I think the “consenting adults” principle applies here–while it’s possible to create a set of mysteriously-named groups with hard-to-debug conflicts between them, it’s not something people will actually want to do a lot, and when they do create conflicting groups they probably have a reason to do so. It seems fine to allow this.

For what it’s worth, I don’t think there’s any requirement that optional-depdencies are all compatible, and it seems to be working out okay?

[zanie (Zanie Blue)]

@jamestwebber

Or by reading the documentation for the project, perhaps?

Yes, a project can document that groups are incompatible or which groups to use together — but why not make that a part of the group declaration so tools can present users with a great experience? Tooling won’t read your documentation to understand which groups are intended to be used with others.

@jamestwebber

… and when they do create conflicting groups they probably have a reason to do so. It seems fine to allow this.

Do you have any examples? I agree it probably doesn’t make sense to ban doing it, but I’m not sure this PEP describes a clear and strong motivation. I think it may be challenging for all tooling to agree on what it means for groups to be incompatible so it may be infeasible to enforce regardless.

@jamestwebber

For what it’s worth, I don’t think there’s any requirement that optional-depdencies are all compatible, and it seems to be working out okay?

Regarding this specifically, both uv and Poetry require this and it’s a significant problem that (at least) the uv team is trying to solve to unblock users that need this. As I described above, there’s a lot of complexity around locking projects if you allow incompatibilities here.

[pf_moore (Paul Moore)]

@zanie

Regarding this specifically, both uv and Poetry require this and it’s a significant problem that (at least) the uv team is trying to solve to unblock users that need this. As I described above, there’s a lot of complexity around locking projects if you allow incompatibilities here.

That’s a completely separate issue, though. If we want to demand that
optional-dependencies are all compatible (and the same for dependency groups), that’s a spec change that would need its own PEP. I’m not against such a rule, but we shouldn’t just impose it without a discussion. Nor should we impose it for dependency groups but not for extras.

[a-reich]

My 2 cents - agree with Brett’s comment. I think I’d prolly be against this PEP if it required all groups to be compatible, or such a change for optional-dependencies. Maybe I’d change my mind based on a dedicated discussion of that, but so far it’s been more of a side note. It’s also a bit difficult to decide based on impacts to locking, which isn’t a finalized or approved PEP yet.

[zanie (Zanie Blue)]

@pf_moore

That’s a completely separate issue, though. If we want to demand that
optional-dependencies are all compatible (and the same for dependency groups), that’s a spec change that would need its own PEP. I’m not against such a rule, but we shouldn’t just impose it without a discussion. Nor should we impose it for dependency groups but not for extras.

I agree we shouldn’t talk about changing project.optional-dependencies here, but I’m not sure they need to be treated the same (or that discussion about how optional dependencies are handled by tools is a separate issue).

I think I need to be clearer about my stance though. @jamestwebber was challenging the idea that incompatibilities are a problem at all — and I was explaining how they are and the real world effects of that. I’m not suggesting this PEP bans incompatibilities. As a contrasting example, the PEP could declare that only a single group could be used at once — then the incompatibility problem goes away, but of course this comes with other problems that people may deem unacceptable. I can’t say if that’s definitely a good idea, but it may be worth mention in the PEP.

However, I’d like to reiterate that the project-metadata changes are the most concerning to us and I’d like to focus my comments there. The incompatibility problem is relevant for locking, but, as others have said, tooling is forced to solve this already for project.optional-dependencies.

@a-reich

Maybe I’d change my mind based on a dedicated discussion of that, but so far it’s been more of a side note. It’s also a bit difficult to decide based on impacts to locking, which isn’t a finalized or approved PEP yet.

There are still some clear real-world locking use-cases to refer to. Part of my concern is that it’s a side-note. I know it’s a little chicken-and-egg, but I think the there could be more context about how this will affect tooling that locks dependencies.

[sirosen (Stephen Rosen)]

I don’t want to overindex on the locking problems as a driver of discussion, but I have experienced use cases which want several of the reasonable, but distinct, locking semantics for a project vis-a-vis a dependency group. To enumerate some of them briefly:

• a group which must be compatible with the project, e.g., typing = ["mypy", "useful-types"]
• two or more groups which are mutually incompatible, e.g., testing-flask1 = ["pytest", "flask<2"] with testing-flask2 = ["pytest", "flask>=2"]
• a group which is independent from the main project, and should not be used to constrain locking of main project dependencies, e.g., deploy-monitor = ["my-cool-deploy-tool", "pyyaml"]

I don’t think that it’s the place of Dependency Groups to try to declare compatibility requirements, or at least in such a way that it covers all of these cases and whatever others we deem important. I’m wary of making changes to accommodate locking in specific cases, but it’s not off the table.

---

To engage with the question of include-project or similar, I think this exchange is a fairly good summary of the available positions.^[1]

@zanie

@brettcannon

Ah, you want to change the PEP so there’s a specific way to signify a dependency group should include project.dependencies so that it acts as a first-class participant via some custom object ala {include-project=true}. It seems reasonable to me and would greatly negate wanting include-group to apply to project.dependencies which helps with backwards-compatibility.

Yeah this is basically my stance. Though I think I lean more towards just removing the complicated part from the PEP and addressing project inclusion separately.

I’m currently against something like include-project. I think removal of include-group from the [project] table is currently the best available option for a change to the PEP, if a change is made.

One of the PEP targets is tox, nox, and hatch for environment building (and now uv as well, although I haven’t gotten a chance to explore uv much yet). Consider the following config under tox, with an addition of --group in deps to indicate a dependency group:

[testenv]
skip_install = false
deps = --group test
[testenv:lint]
skip_install = true
deps = --group lint

skip_install is a control in tox for installing the project or not. hatch and nox also have skip_install controls, for exactly the same purpose.

I think it’s confusing for users to have tools which are already capable of either installing or omitting the current project, and re-encode that distinction in the standard pyproject.toml metadata. What will tox do if skip_install = true and a dependency group is used which includes the project? It would have to define some behavior – my point is only that it’s not intuitively obvious what that behavior will be.

Additionally, keeping the Dependency Groups data all statically declared in pyproject.toml has been an explicit goal. That improves the situation for tools which want to support use of Dependency Groups but which don’t want to include PEP 517 frontend code.

My final concern about any include-project directive is how that information will be rendered and passed between tools – e.g., how will my environment manager tell pip to install the current project, if it uses pip as its installer? Absent a spec for path-dependencies, it will probably be passed around as "." in practice. Maybe others are comfortable with this, but I am not.

---

If I look at the tradeoffs between allowing include-group in [project.dependencies] and not, with a fresh eye, I see against:

• two tooling maintainers^[2] have expressed concerns about including it
• requires updates to all build backends to add support

and in favor:

• opens up options for explicit separation of dependency build/install from package install (the use-cases which drive the pip install --only-deps thread)
• allows more flexibility in how projects transition from dev/test extras to Dependency Groups
• allows various flexible dependency structures, especially for extras (e.g., two distinct extras may both pull in the same dependency group)

I was more comfortable with the inclusion prior to this round of feedback. Primarily, it seemed like the PEP needed to acknowledge the fact that this is a breaking change for tools which attempt to parse package metadata from source repos (e.g., Dependabot).

I’m still not yet convinced that it should be removed, however.

It seems to me that the issue stems from the fact that uv is reading data from pyproject.toml tables directly today (and presumably intelligently handles dynamic = true), which is a usage I had not anticipated. I assumed that in the presence of a [build-system] table we could safely assume that any installer wanting to install [project.dependencies] would read metadata via the PEP 517 hooks. That same assumption seems to underlie some of the non-standard [project.dependencies] extensions we have seen from hatch, pdm, and potentially others (I’m only aware of these two). Clearly, this is not a safe assumption – I need to think more about what I think!

---

1. Although, special call-out, I thought @ncoghlan’s idea of a :project: and :project[extra]: syntax was really interesting. It avoided the main issue I’ve had with proposing a special syntax at the top-level of a Dependency Groups list, where any new syntax conflicts with potential future syntax for requirements, and made me think carefully about whether or not I’d support such a change. ↩︎

2. I count Ralf’s comments above as moderately against, and now we’re hearing from the uv team as well. ↩︎

[pf_moore (Paul Moore)]

@sirosen

I don’t think that it’s the place of Dependency Groups to try to declare compatibility requirements, or at least in such a way that it covers all of these cases and whatever others we deem important.

I strongly agree with this stance. The PEP should focus on defining the mechanism, not trying to judge between different, equally valid, use cases.

@sirosen

I’m wary of making changes to accommodate locking in specific cases, but it’s not off the table.

Again, I agree. I think there’s still enough uncertainty about how different tools approach locking that we shouldn’t get sucked into that debate. It’s getting a lot closer to consensus over on the locking PEP thread, but that proposal isn’t as close to completion as this one, and I’d prefer that we keep this PEP focused, and leave it to the locking PEP to propose changes or modifications if it turns out they are needed.

If this PEP gets delayed, and the locking PEP gets approved before this one, then my position gets inverted, of course

[zanie (Zanie Blue)]

@sirosen thanks for your thoughtful responses!

@sirosen

I don’t think that it’s the place of Dependency Groups to try to declare compatibility requirements, or at least in such a way that it covers all of these cases and whatever others we deem important. I’m wary of making changes to accommodate locking in specific cases, but it’s not off the table.

My expectation is that the PEP is updated to discuss these use-cases, but I agree it’s probably not a goal to address any of them unless adding the context reveals significant concerns.

(Thanks for enumerating those use-cases!)

@sirosen

I think it’s confusing for users to have tools which are already capable of either installing or omitting the current project, and re-encode that distinction in the standard pyproject.toml metadata. What will tox do if skip_install = true and a dependency group is used which includes the project? It would have to define some behavior – my point is only that it’s not intuitively obvious what that behavior will be.

This is a fair point, but it’s already a goal of this PEP is to take metadata that is currently defined in tool-specific ways and move it into the standard metadata. Generally, I think moving existing tool behavior into the standards is the correct path.

@sirosen

If I look at the tradeoffs between allowing include-group in [project.dependencies] and not, with a fresh eye, I see against: …

I’m not quite sure how to express this, but I see moving information out of [project.dependencies] as negative — it feels like it undermines the existing project dependency specification.

[brettcannon]

@zanie

But why wouldn’t you just include both groups at runtime, e.g., --group typing --group test?

You could, but I think it’s a question of which style of approach you prefer: a single group per concept or a separation of concepts that are composable? After this latest bit of discussion I’m open to either personally. As has been suggested, we could punt on include-group to a future PEP if we want to see what the community thinks first, although if we are strongly convinced the community will want it then doing it now makes the transition easier.

@zanie

you could just extend the schema of the object instead of introducing special strings:

[dependency-groups]
foo = ["bar", { include-project = true, extras = ["foo", "bar"] }]

This is how I thought it could be expressed as well.

@sirosen

Additionally, keeping the Dependency Groups data all statically declared in pyproject.toml has been an explicit goal. That improves the situation for tools which want to support use of Dependency Groups but which don’t want to include PEP 517 frontend code.

I think you could tweak this view two different ways that would accommodate what @zanie is suggesting. One is dependency groups don’t make dynamic metadata worse by adding yet another bit of metadata that’s allowed to be dynamic (this is the “don’t be so hard on yourself” answer). Another is to say you can’t specify that something from project.dependencies or project.optional-dependencies is part of a dependency group if what you’re pulling from is listed as dynamic (this is the “carrot” answer).

I’m personally okay w/ the latter answer to push people even more towards static metadata w/o having to make this PEP twist itself up in any way to help with that goal.

[EpicWink (Laurie O)]

@sirosen

requires updates to all build backends to add support

Not just build backends, but schema validators (aka linters) and dependency inspectors (eg Dependant, like you mentioned).

Moreover, twice now has a feature being added support for in build backends caused me to drop support for older Python versions (with no other reason), so I think I have a bad taste for this.

@sirosen

opens up options for explicit separation of dependency build/install from package install (the use-cases which drive the pip install --only-deps thread)

It seems pretty straightforward for tools to solve this themselves, as a package and it’s dependencies are quite distinct.

For your tox (etc) example, I would expect tox to throw if a group requires the project to be installed but the tox config says to not install the project.

Moreover, if the effect here is for project dependencies to be moved out of project.dependencies and into dependency groups, I feel that’s counter to the spirit of PEP 621, which was to centralise this metadata declaration.

@sirosen

allows various flexible dependency structures, especially for extras (e.g., two distinct extras may both pull in the same dependency group)

This a relationship direction decision, as by the same token, I could say the alternative is for two distinct spendy groups to pull both pull in the same extra.

@sirosen

Additionally, keeping the Dependency Groups data all statically declared in pyproject.toml has been an explicit goal. That improves the situation for tools which want to support use of Dependency Groups but which don’t want to include PEP 517 frontend code.

If the project needs to be installed, then things are dynamic and PEP 517 hooks need to be invoked, no matter we got to that point (eg command-line arguments or include-project = true).

Perhaps a compromise here would be too say that tools consuming dependency groups should support include-project = true, but don’t have to? Other than that, maybe there’s some other way of declaring the relationship between a project, it’s dependencies, and dependency groups.

[ncoghlan (Alyssa Coghlan)]

Considering the potential options for near term resolution, I favour deferral of:

• any changes to the project tables (moving metadata out of standard locations seems undesirable, but it needs more flexible inclusion syntax to avoid it)
• group inclusion syntax that doesn’t have a way to refer to the project itself, as well as the extras published in the optional dependency table
• syntax for expressing whether groups can or should be included in the project’s default lock file, or if they should be locked independently

This does mean tools may need to come up with their own interim solutions for those pieces of the problem, but the shape of those workarounds can subsequently be used to inform the design of less speculative standardised solutions.

I’m not personally worried about how projects express their support for standardised dependency groups, as I expect any future standards updates to have meaningful ways to refer to them (such as, “standard dependency groups are supported from vM.N, but handling dependency group inclusions requires at least vX.Y”)

[sirosen (Stephen Rosen)]

I’m going to update the PEP to revert it back to a state in which it does not touch the [project] table.
There’s now evidence that the impact of that change would be greater than I anticipated. I don’t like going back and forth on this, but I think the uv team’s feedback is compelling enough to warrant this reversion.

Although I believe that allowing includes from the project table is the correct long term direction, a smaller scope for this spec has several benefits.

There’s also some feedback to address aside from this decision, so I’ll need to make some minor tweaks.

@ncoghlan

Considering the potential options for near term resolution, I favour deferral of:

• …
• group inclusion syntax that doesn’t have a way to refer to the project itself, as well as the extras published in the optional dependency table
• …

To clarify, you’re proposing removal of {include-group = ...} subtables entirely from the spec? That’s a bit more radical than what I think is best.
It’s pretty common to have – especially for testing – a core group of dependencies which are extended by several other groups.

Is there some significant reason not to retain this? I don’t think I’ve heard an argument against allowing groups to include one another.