Idea: Allow optional dependencies to specify other optional dependencies

Packaging Standards
1

Optional dependencies currently look like this: Writing your pyproject.toml - Python Packaging User Guide

[project.optional-dependencies]
gui = ["PyQt5"]
cli = ["rich", "click"]

But a very common practise if for optional dependencies to refer to other optional dependencies, e.g.

[project.optional-dependencies]
gui = ["PyQt5"]
cli = ["rich", "click"]
all = ["PyQt5", "rich", "click"]

While this isn’t too cumbersome in this example, it can get exponentially out of control with lots of dependencies and lots of optional dependencies referring to other dependencies. If you look at a project like apache-airflow they can’t specify their optional dependencies sensibly in their metadata so they dynamically build it with their build system.

My proposal, at a high level, would to be to allow optional dependencies to refer to other optional dependencies, like so:

[project.optional-dependencies]
gui = ["PyQt5"]
cli = ["rich", "click"]
all = ["$gui", "$cli"]

It would be up to the build system (or any tool that peeks directly at pyproject.toml files) to recognize that this is referring to other groups and expand out the group. Circular dependencies would be strictly prohibited, but I don’t think there should be a requirement for the ordering to matter (e.g. in this example all could be at the beginning).

This is a high level idea, the specification would need to be more precise, so not looking to bikeshed. I would like to gauge if this would be an idea that the relevant parties like, think is worthwhile, or it has been discussed before and I missed it.

2

Have you looked at the PEP 735 proposal for dependency groups? I think this will be well-covered if/when that PEP is accepted.

Edit: Link to thread: PEP 735: Dependency Groups in pyproject.toml

2 Likes
3

While sounding similar I don’t think PEP 735 is related.

This idea is about providing a new syntax for optional dependencies, hopefully allowing users to be more succinct and less likely to introduce errors here. The tools that would primarily read this would be backend build tools to turn into Provides-Extra metadata for wheels.

PEP 735, at least as I understand it, is about project management and is unrelated to backend build tools, these groups would not and should not be exposed as Provides-Extra metadata: https://peps.python.org/pep-0735/#package-building

4

This is supported already but undocumented: Document support for self-referential extras · Issue #11296 · pypa/pip · GitHub

Every tool that I know of supports this.

5 Likes
5

Amazing, I was not aware there was a way to get this to work with existing syntax, I have some projects to go simplify.

1 Like
6

I use this extensively, and it works well :+1:. I do wish it was standardised, and I also wish you didn’t need to include the project name (i.e. there was a way to reference “this” project, for example with .[test,doc]).

I also wish there was guidance on common extra names - is it doc or docs, test or tests, etc.

1 Like
7

The singular version, documented here: Core metadata specifications - Python Packaging User Guide

2 Likes
8

It was also discussed extensively before here: Pyproject.toml optional dependencies redundancy aka DRY extras - #6 by uranusjr

1 Like