PEP 735: Dependency Groups in pyproject.toml

Packaging Standards
223

I don’t think the relative path version is good but it’s indeed cute. Just experimenting with possibilities as you noted. Personally I prefer the simplicity of only having include as the word to remember but I can see the argument for multiple different keywords.

Another version is “check for a matching dependency group, and if you don’t find one check for any table”. This would mean that you can’t name a dependency group project and expect everything to work, but maybe that’s a fine restriction because that’s a weird thing to do anyway.

2 Likes
224

As noted in the --only-deps discussion on the pip tracker, you can’t have it both ways. If you’re referring to the current project dependencies, you have to accept the possibility that this will trigger a build.

1 Like
225

I have a real-world use case, but it’s admittedly not blocked if I had to install my project automatically, it’s just unnecessary.

I have a project: GitHub - brettcannon/mousebender: Create reproducible installations for a virtual environment from a lock file . It does not use a src/ layout. It has some dependencies:

It also has some test dependencies (currently specified as an extra, but I will move over to this PEP as soon as I can):

Because of the project layout, I can do py -m pytest from a checkout w/o installing the project if I can get the runtime and test dependencies installed into the environment. And so doing an (editable) install of the project is just a waste of time and literal energy (as long as the editable install does the right thing and I can edit files in-place and still have them be picked up, else it would actually impact my productivity).

226

Thanks. For that, presumably pip install --deps-only . followed by pip install --group test would work equally well. I know --deps-only doesn’t exist (yet), but I tend to see “deps only” installs as orthogonal to dependency groups, so (in my head, at least) having two separate mechanisms seems reasonable.

227

And I’m fine with that as well if that’s the UX we are going to train the community to follow; “install the project as appropriate, then to run the tests you also install the ‘test’ dependency group”. I don’t think the cognitive burden is horrible here w/ two commands if we don’t manage to come up w/ something that works for tying the runtime dependencies into dependency groups to allow for one command.

228

What is meant by the following?

The test group of what? Is it meant to be the test group of the project in the current directory? As far as I know pip never reads anything from the current working directory, does it?

229

Nobody has yet designed a syntax that pip would use to install a dependency group. So the concrete syntax here is just an abbreviation for the idea “get pip to install the requirements in dependency group test”. If this PEP is accepted, pip will implement some syntax for this, but it’s not 100% clear yet what form it will take.

Personally, I see this as similar to reading a requirements file, which is somewhere that pip does read a file in the current directory. So the analogy for me is that we’re getting the requirements from a section of pyproject.toml rather than from a requirements.txt file. Others may have different mental models, of course.

4 Likes
230

A post was split to a new topic: Why doesn’t pip write installed packages to pyproject.toml?

231

The existing Poetry and PDM tools already offer a feature which each calls “Dependency Groups”, but using non-standard data belonging to the poetry and pdm tools.

The existing tools use tool specific data, it is standards compliant, just not to this specification, which isn’t even released yet. I would suggest:

The existing Poetry and PDM tools already offer a feature which each calls “Dependency Groups”, which use tool specific data rather than data proposed by this PEP.

1 Like
232

Tools SHOULD NOT eagerly validate the list contents of all Dependency Groups.

I do not like this. If we’re going to have a standard, it should apply to the file format up front.

233

Note that none of these Dependency Group declarations implicitly install the current package, its dependencies, or any optional dependencies.

I agree with this, it gives a lot of required flexibility. I also think that it should be possible to explicitly reference the current package as a dependency in the group. To take the testing example:

$TOOL install-dependency-group test

could then install everything required for testing, without the user having to do an extra pip step.

Is there a standard packaging name for the current package? Could we use “.” ?

234

I’m not sure I understand what you mean when you say that the poetry-specific and pdm-specific tables in pyproject.toml are “standards compliant”. They don’t violate Python standards (as defined in the packaging PEPs), nor do they define invalid TOML data (violating the TOML standard). But there isn’t, to the best of my knowledge, any standard which governs the contents of the [tool.poetry.group] table, for example.

So the contents are valid, but non-standard. Am I missing what you’re getting at here?

There’s a lot of history in this thread on this topic. I’m very emphatically not suggesting that you have to read all of it – the PEP should summarize it – but I mention it in case you want to go digging and read further. I think this is missing from the Rationale section, so I’ll need to address its absence.

The basic argument here is that by recommending against eager and aggressive validation of the whole table, we allow for the following future:

  • New specs can extend this table without instantly breaking all current usages the moment new data appears.
  • Validators/linters are free to ignore the “SHOULD NOT” and validate all data – after all, they have a good reason to do so. (The openness of “SHOULD NOT” is carefully chosen here.)

I wouldn’t say it’s a closed case, but as the PEP author, this is something I’m all-but-convinced is the right choice. You’ll need to make a good argument if you think this needs to change. At the very least, you’d need to describe how new data or formats could be introduced in the future, and how tools should behave when those new data appear.

This is still an active (dare I say lively?) area of discussion and debate.
It’s been quiet here for ~1 week, as I’ve had limited time to work on this and others have already given useful input.

First, to the use of . as a name for the current package, this is the most common suggestion, but there are a couple of issues. The main thing is that there is no standard which governs pip’s interpretation of filesystem paths, including .. So we don’t have the strength and completeness of an existing standard to specify behavior.

Earlier drafts of this PEP included “path dependencies”. I got overambitious in terms of what should be included here. First they were trimmed back to a smaller set of capabilities, and then dropped. These draft changes reflect a lot of input and wisdom from thread participants, for which I am very grateful. I still believe that Path Dependencies are worth specifying, but I now think that that they will require a dedicated PEP. When a specification for Path Dependencies arrives (here, as the author, I will insist that we operate with this assumption – it is my belief that this will happen, but I don’t know when), they may be picked up and added to Dependency Groups. At that point, we’d get the ability for a group to specify . “for free” (okay, actually, for the cost of someone, maybe even me, working up another PEP and getting it accepted :wink: ).

In short: I think what you’re asking for here is sensible, but we might not get there in a single step.

There are a few pending editorial comments and minor improvements. After I get a PR opened, I may have a small update to drop here. I’m still thinking about the relationship between [dependency-groups] and [project] and am not yet ready to give up on allowing one table to reference the other.

3 Likes
235

With my Poetry slanted hat on:

It basically feels like a bit of a slight to say third party tools use non standard data for dependency groups, when there is no standard to describe dependency groups, and as the third party stuff is all under [tool.poetry/pdm] it is standards compliant to the pyproject.toml PEP. I’m very comfortable with ‘tool specific’ but saying “non-standard” makes it feel like a choice was made to be non-standard.

That is a very reasonable argument, so I’m happy to let this drop.

OK, thank you. I have tried to follow along, but…there’s a lot. At least in Poetry, I imagine we’d be supporting path deps in PEP dependency groups and we’d have to support installing the current project as well, I’m guessing that’ll be by special casing “.” as a project name, but other keys are possible. I would have to have another read to see if poetry would be compliant if we did that, I suspect not.

Poetry could add, say:

[tool.poetry.groups.dev.options]
install-root = True

instead of special casing “.” to install the current project.

Thank you for all the work on the PEP so far.

3 Likes
236

Okay, that clarifies. It’s definitely not intended as a slight!
I want to make sure the text is clear about the fact that this spec is meant to produce a standard for something which is not yet standardized.

Let me take more time to think about a potential rephrasing.

4 Likes
237

I have a few updates on this PEP after a bit of a hiatus (personal; very limited non-work dev time).

First, regarding the phrasing around Poetry and PDM, I have a PR up right now on the PEPs repo. I hope it will read nicely for everyone, and I’m always happy to tune it further.

Second, I took some time today to see if I could add pip install --dependency-group foo in a branch. It was pretty easy (sans tests). You can see my prototype here: Comparing pypa:main...sirosen:pep-735-support · pypa/pip · GitHub

I’m still not quite ready to approach various projects to start planning support (assuming this is accepted), as there is one final decision that needed to be made. Drumroll please! The decision:

Supporting Includes in project.dependencies and project.optional-dependencies

The PEP will be expanded to include this. I’m going to start work on it tomorrow and we’ll see if I can get a PR posted soon.
Specifically, the path taken will be to allow [project.dependencies] and [project.optional-dependencies] to include from [dependency-groups].

It will look like this:

[dependency-groups]
types = ["useful-types"]
typecheck = ["pyright", { include-group = "types" } ]
[project]
dependencies = ["httpx", { include-group = "types" } ]

I’m aware that this means that this change introduces new syntax into project.dependencies and that several people are concerned about this. I’d be lying if I said I am not at all nervous. However, I believe this is the correct decision in spite of the issues.[1]

I intend for this to be the last major change, after which I’ll be working on direct outreach/engagement with various tool maintainers and finalizing the PEP for submission.[2]

There have been a couple of comments that we could easily teach users to do things like

pip install .
pip install --dependency-group types  # remember, this is imaginary syntax...

if we don’t include the ability to combine data from [project] and [dependency-groups] in the spec.

It’s precisely the fact that such things are possible which convinces me that inclusion in project is the right choice to make! Users will have a viable alternatives until tool support arrives, when the installers in their toolchain (like pip) support reading directly from [dependency-groups] and the build backends (hatchling, flit-core, etc) do not support include-group directives.

Let’s imagine that a build backend adds support. How will that play out for users? They’ll see it as a new feature which arrives in setuptools>=X.Y or hatchling>=P.Q, etc. Setting aside how the PEP will have to explain this situation, the user experience will be quite good. Users will see a two-phase rollout:

  • Phase 1: pip, tox, nox, and other tools start supporting installation from [dependency-groups]
  • Phase 2: setuptools, hatchling, and other build backends start supporting include-group directives in dependency metadata

The transitional period will mostly be handled by build-system.requires, so this is a case in which the past planning will pay off – you can only (safely) use new features of your build backend when you specify a new enough version.

renaming to include-group

This is a small change included in the above. Pradyun mentioned the potential for ambiguity if {include = ...} appears in project.dependencies earlier, and I think that observation is correct.
Luckily, it’s easy enough to switch to include-group with the same semantics. That will solve the issue and make it clear what’s being included no matter the context for this “object”/table.

We can change it back if this seems super-contentious, but as far as I’m concerned this key is a minor detail.

how to teach this

The above sample pyproject.toml will fail with any build-backend today. That means that users can’t use it, even though it’s defined by the PEP. Which is confusing. I expect a great deal of effort on the “How To Teach This” section.

I’m not shy about the fact that this could be confusing for users. I think it’s very likely that sooner or later someone will have

  • a [dependency-groups] table
  • a [project] table which uses the include syntax
  • unbounded build-system.requires
  • a choice of build backend which does support [dependency-groups] in its latest version and docs

And the result is that the feature only works “some of the time”, or not at all for that user.

However, this is already the case any time a build backend adds a feature or fixes a bug, and the entire packaging ecosystem is not coming apart at the seams.[3] I think this will come down to documentation, ecosystem-wide tool buy-in, and accepting that during the rollout period there will be some cases of users who are confused. I don’t think that justifies not taking the longer term view that this feature of the spec will be largely beneficial.

All of the above is stated rather declaratively, as it is my intent to make this change.
I’m always open to having my mind changed, but this seems like a case in which the author of the proposal needs to make a decision and push for it.

I’m aware that there are downsides. But those look to me right now like some short term pain for a long term benefit.

  1. And if this causes the PEP to get rejected, I’ve made my peace with that. I feel strongly that it’s right. ↩︎

  2. Specifically, now that I’ve cooled my jets after the initial flurry of activity here, I don’t want to submit this without a promising path forward in pip and at least one build backend. I was planning on trying to contribute to setuptools and maybe some others. ↩︎

  3. People who were negatively impacted by the Cython 3 release, I see you! But that sort of case is the exception, not the rule. ↩︎

12 Likes
238

@sirosen love the updates! They don’t seem at all controversial, can’t wait to see the PEP polished and accepted!

I wanted to point out that the build dependencies should really be treated the same way as runtime dependencies, when it comes to using some features. When we integrate a library into a project and start using a feature from that library, it is always a good idea to set a lower boundary for the dependency specifier of that library, effectively communicating that “we rely on features that are provided by a certain lib’s version or higher”.
It is the same with build backends — if we rely on a feature only a certain version introduces, it must be communicated explicitly. This way, there’s no ambiguity or working “some of the time” — the incompatible versions aren’t getting into the env in the first place.
Let’s embed it into teaching by having the first step of the process which would be specifying the first known setuptools version as the lower bound. It’s not uncommon to see setuptools >= 48 or something like that with a code comment explaining that a feature X first appeared in v48.

I believe this approach should eliminate the concern of inconsistency and make sure that learning dependency groups is straightforward.

P.S. The setuptools example is for introducing inclusion into the runtime deps. For teaching the direct install of the dep groups, the same would apply to workflow tools — ask the students to have a certain minimum version of nox/pip/tox/etc. as a pre-requisite first step and everything else will follow.

1 Like
239

@sirosen there’s one more thing I forgot to ask. It came to my attention that some people want to treat [build-system].requires as a special dependency group. This kinda makes sense from the UX perspective but I know that it’s a can of worms since the PEP 517 deps need to be augmented dynamically, based on what hooks return.

So I thought it might be useful to call out this use case in some “bad ideas” section, right?

See Feature request: add a way to install only build dependencies · Issue #1516 · astral-sh/uv · GitHub for the context.

1 Like
240

Oh, that’s interesting! I agree that it’s a bad idea – if you go down this path then I’m pretty sure PEP 517 compatibility starts to break down.

A future (pretty small) PEP could ask for include-group to be supported in build-system.requires. Until then, this should be omitted.

I’ll jot it down in rejected ideas. The PEP already impacts installers and build backends. Having implications for build frontends too is avoidable scope creep.
IMO scope creep is the main reason to reject it here. If there’s a future PEP just for adding this one use-case, it can have focused discussion with the pip and build maintainers, and succeed or fail on its own merits. (And I think such a proposal is easy to write if this PEP is accepted.)

2 Likes
241

It just occurred to me that it could work in an opposite direction — the backends could read the dependency groups and augment the build deps dynamically through hooks, not breaking any backwards compatibility (as opposed to allowing inclusion in the requires entry of [build-system]). So this wouldn’t even need a new PEP in such a configuration.

242

When reading this, I can’t help but think about PEP 633 that was proposing table notation for dependencies but got rejected.

I know it is not the same thing, the comparison is a bit weak, and the context is also not the same. But, nonetheless I feel like there is some imbalance here. It is some kind of table notation for dependencies.

I think something like the following has been suggested already, right? I lost track.

[project]
dependencies = ["httpx", ".[types]" ]

The assumption being that types is either an extra or a group, and if it is both it should trigger a failure (maybe?).