PEP 735: Dependency Groups in pyproject.toml

Packaging Standards
237

I have a few updates on this PEP after a bit of a hiatus (personal; very limited non-work dev time).

First, regarding the phrasing around Poetry and PDM, I have a PR up right now on the PEPs repo. I hope it will read nicely for everyone, and I’m always happy to tune it further.

Second, I took some time today to see if I could add pip install --dependency-group foo in a branch. It was pretty easy (sans tests). You can see my prototype here: Comparing pypa:main...sirosen:pep-735-support · pypa/pip · GitHub

I’m still not quite ready to approach various projects to start planning support (assuming this is accepted), as there is one final decision that needed to be made. Drumroll please! The decision:

Supporting Includes in project.dependencies and project.optional-dependencies

The PEP will be expanded to include this. I’m going to start work on it tomorrow and we’ll see if I can get a PR posted soon.
Specifically, the path taken will be to allow [project.dependencies] and [project.optional-dependencies] to include from [dependency-groups].

It will look like this:

[dependency-groups]
types = ["useful-types"]
typecheck = ["pyright", { include-group = "types" } ]
[project]
dependencies = ["httpx", { include-group = "types" } ]

I’m aware that this means that this change introduces new syntax into project.dependencies and that several people are concerned about this. I’d be lying if I said I am not at all nervous. However, I believe this is the correct decision in spite of the issues.[1]

I intend for this to be the last major change, after which I’ll be working on direct outreach/engagement with various tool maintainers and finalizing the PEP for submission.[2]

There have been a couple of comments that we could easily teach users to do things like

pip install .
pip install --dependency-group types  # remember, this is imaginary syntax...

if we don’t include the ability to combine data from [project] and [dependency-groups] in the spec.

It’s precisely the fact that such things are possible which convinces me that inclusion in project is the right choice to make! Users will have a viable alternatives until tool support arrives, when the installers in their toolchain (like pip) support reading directly from [dependency-groups] and the build backends (hatchling, flit-core, etc) do not support include-group directives.

Let’s imagine that a build backend adds support. How will that play out for users? They’ll see it as a new feature which arrives in setuptools>=X.Y or hatchling>=P.Q, etc. Setting aside how the PEP will have to explain this situation, the user experience will be quite good. Users will see a two-phase rollout:

  • Phase 1: pip, tox, nox, and other tools start supporting installation from [dependency-groups]
  • Phase 2: setuptools, hatchling, and other build backends start supporting include-group directives in dependency metadata

The transitional period will mostly be handled by build-system.requires, so this is a case in which the past planning will pay off – you can only (safely) use new features of your build backend when you specify a new enough version.

renaming to include-group

This is a small change included in the above. Pradyun mentioned the potential for ambiguity if {include = ...} appears in project.dependencies earlier, and I think that observation is correct.
Luckily, it’s easy enough to switch to include-group with the same semantics. That will solve the issue and make it clear what’s being included no matter the context for this “object”/table.

We can change it back if this seems super-contentious, but as far as I’m concerned this key is a minor detail.

how to teach this

The above sample pyproject.toml will fail with any build-backend today. That means that users can’t use it, even though it’s defined by the PEP. Which is confusing. I expect a great deal of effort on the “How To Teach This” section.

I’m not shy about the fact that this could be confusing for users. I think it’s very likely that sooner or later someone will have

  • a [dependency-groups] table
  • a [project] table which uses the include syntax
  • unbounded build-system.requires
  • a choice of build backend which does support [dependency-groups] in its latest version and docs

And the result is that the feature only works “some of the time”, or not at all for that user.

However, this is already the case any time a build backend adds a feature or fixes a bug, and the entire packaging ecosystem is not coming apart at the seams.[3] I think this will come down to documentation, ecosystem-wide tool buy-in, and accepting that during the rollout period there will be some cases of users who are confused. I don’t think that justifies not taking the longer term view that this feature of the spec will be largely beneficial.

All of the above is stated rather declaratively, as it is my intent to make this change.
I’m always open to having my mind changed, but this seems like a case in which the author of the proposal needs to make a decision and push for it.

I’m aware that there are downsides. But those look to me right now like some short term pain for a long term benefit.

  1. And if this causes the PEP to get rejected, I’ve made my peace with that. I feel strongly that it’s right. ↩︎

  2. Specifically, now that I’ve cooled my jets after the initial flurry of activity here, I don’t want to submit this without a promising path forward in pip and at least one build backend. I was planning on trying to contribute to setuptools and maybe some others. ↩︎

  3. People who were negatively impacted by the Cython 3 release, I see you! But that sort of case is the exception, not the rule. ↩︎

12 Likes