PEP 735: Dependency Groups in pyproject.toml

[sirosen (Stephen Rosen)]

@pf_moore

Are there any use cases where someone would want to use a dependency group that referenced the project dependencies, where they didn’t also want to install the project itself? To put it another way, in your example syntax, what is a real-world example of using bar, which doesn’t also install the project?

In 100% agreement with your footnote, I think this is the same as the --only-deps case. It’s not a case that I’ve experienced myself, so my understanding of this case is weak. I may need to reread some of the threads on this topic.

I think any such need could be solved by having dependency group inclusion interact with [project.dependencies] in either direction, since any desire for --only-deps can be satisfied by making a dependency group which is synonymous with [project.dependencies].

@pf_moore

Is the only reason for not having a syntax to say “this project” in a dependency group (i.e., what was previously suggested as ".") because people might sometimes want to install the project as editable? If so, what’s wrong with a group bar = [{include = "foo"}, "."] and pip install -group bar (normal install) or pip install -e . -group bar (editable install)? I’m not suggesting re-introducing path syntax, but rather simply having an equivalent of "self".

As we discussed the path dependencies, one of the things which I grew uncomfortable with was the realization that specifying . would mean that a build happens. Even if behavior is newly defined for this, it’s not necessarily the same as whatever the project’s preferred build frontend does. If a project has any need to configure the build environment, this could start to break down.

Having thought more about this, I’m not ready to reintroduce the idea that a dependency group can refer to the current package as a package. I could be convinced that it would be okay to do so, but right now I would need convincing.

I’m not sure we have a clear use case we’re satisfying by including . as a dependency. It “feels natural” and we know that users will want environments with a dependency group (like “test”) + the current package. But are we making things significantly better for users by including it?

For a tool like hatch or tox which can already install . in addition to some extended set of dependencies or extras, I don’t think there’s any particular benefit in having a dependency group include .. For example, for tox, we’re really talking about replacing

[testenv]
deps = -r test-requirements.txt
commands = pytest

with

[testenv]
dependendency-groups = test
commands = pytest

In both cases the installation of . is managed separately. I think tox would actually find this harder if test included a reference to ., since . is being requested twice via two different paths: once in the dependency group and once implicitly as part of a tox testenv without skip_install=true.

For direct pip usage, we’re comparing pip install --dependency-group test against pip install . --dependency-group test. (Or maybe two invocations of pip, but still something like that.)
So it’s really similar in simplicity, although you could argue that it’s subtle.

Before we reintroduce . of “self” or “current package”, I want to have a clearer handle on what we’re gaining by including it and what we’re losing by excluding it.

This is consistent with @EpicWink’s comment:

@EpicWink

For that matter, is there a use-case for using dependency-groups and installing the project’s dependencies without the project itself? I get the feeling it was alluded to in this thread, but I can’t find it with a quick search.

I’m not sure that there is a clear use case for it. A lot of this thread assumes that there is such a case, but as far as I know we haven’t clearly articulated cases. I will have one such case detailed below, at the end of this comment.

Right now, I’m reading and trying to get a better understanding of what kinds of includes between [project] and [dependency-groups] are important or useful.

@jamestwebber

Another flavor would be something like {include = "project"}. This fits together with my previous post in that I’m essentially defining the value of {include} as “some table in this file”.

To really formalize it, maybe require a preceding . to denote “one of the tables in the [dependency-group] table”, like it’s a relative path. So a full example would be: …

I appreciate that you’re playing around with syntaxes for this, and I think the “relative path” trick is cute/clever (I mean that in a positive way) but too subtle to be a good interface.

We want it to be relatively obvious at a glance what each “thing” in the [dependency-groups] and [project] is. Currently {include = ...} means a Dependency Group Include. If we need multiple types of includes, maybe include is a bad keyword to use as a bare name, and it should be a vocabulary of things like

{include-group = "foo"}  # Dependency Group
{include-optional-dependencies = "bar"}  # Extra
{include-dependencies = true }  # [project.dependencies] list

@EpicWink

PEP editorial comments:
…

Thanks for these! I’ll get the PEP updated with some fixes.

Use Cases for [project] Includes

I only have one user story which I can articulate clearly, and that’s related specifically to static analysis.
This applies mostly to type checkers, but other analyzers like pylint sometimes have similar requirements.

Basically, type checking a codebase requires that all of the runtime dependencies of that project are present. It also often requires that some or all extras are installed. (Theoretically, there could be conflicting/mutually exclusive extras, which could all also be needed over multiple runs of the analysis, but I’ve never seen this in practice.)

So, for a simple case, imagine a project with some library requirements and one extra:

[project]
dependencies = ["a", "b"]
[optional-dependencies]
foo = ["c", "d"]

In order to type check this project, assuming the use of mypy, the following packages need to be installed:

mypy, a, b, c, d

How should this project write the requirements?

Well, here’s how we can write it in one of the proposed forms from this thread:

[project]
dependencies = [{include = "runtime"}]
[optional-dependencies]
foo = [{include = "foo"}]
[dependency-groups]
runtime = ["a", "b"]
foo = ["c", "d"]
typing = ["mypy", {include = "runtime"}, {include = "foo"}]

Name the group pylint and swap mypy for pylint and you have the same case, but for a different analyzer.

Could this be satisfied by installing .[foo]? Yes, but two things have been lost:

• installing . is wasted work – the analyzer doesn’t need it, but it is being built and installed anyway
• if the typing dependency group cannot express the need for these other dependencies, it is now incomplete

---

There’s another element of the way that a test dependency group would typically interact with extras which I’m thinking about but not yet sure how it impacts things:

test requirements are often needed as part of a matrix of build configurations over multiple extras, which may be dependent on the python runtime version.
For example, the following test configurations may be desired for a package where a toml extra refers to tomli and a yaml extra refers to pyyaml:

dependency_groups	extras	pythons
test	(none)	3.9, 3.10, 3.11, 3.12
test	toml	3.9, 3.10
test	yaml	3.9, 3.10, 3.11, 3.12
test	toml, yaml	3.9, 3.10

etc.

tox, nox, etc already let you build these kinds of matrices. Do we benefit from allowing dependency groups to include extras in these cases, or vice-versa? I tend to think not, but maybe I’m missing a potential interaction here.