Aliased or aggregated optional dependencies

Packaging
1

I would like some way to declare aliases or groups of optional dependencies. For example:

[project]
...
dynamic = ["optional-dependencies"]

[tool.optdeps.optional-dependencies]
test = ["pytest"]
doc = ["sphinx"]
style = ["black"]

[tool.optdeps.dependency-aliases]
tests = ["test"]  # Help out misspellers
dev = ["test", "doc", "style"]
all = ["*"]

This would resolve into:

[project.optional-dependencies]
test = ["pytest"]
doc = ["sphinx"]
style = ["black"]
tests = ["pytest"]
dev = ["pytest", "sphinx", "black"]
all = ["pytest", "sphinx", "black"]

It’s a simple thing to do, and it would seem cleanest to write a single tool to do it rather than propose a patch to every backend or propose an addition to PEP-621.

From my understanding, there’s no way to add something to [build-system] that would tip off setuptools or flit or whatever that it should let another tool do some munging on pyproject.toml and call out to my imaginary tool optdeps to get back a filled-in optional-dependencies. So I think I would need to instead write a wrapper that would call these backends and either pass them a modified pyproject.toml or modify the metadata of the built sdist or wheel.

Is there any guidance on writing such wrappers, and would it be general to any backend, or would I need to dig into each tool to figure out how to patch it?

There is a more general problem here, where any project metadata field could potentially be dynamic and a small tool could be devised to resolve it for other backends. For example, setuptools_scm could be allowed to set version for any backend.

Background on aliases with setuptools

In the old setup.py days, I saw some projects provide an all extra that would be built from all of the other extras:

extras_require = {
  "test": ["x", "y", "z"],
  "doc": ["a", "b", "c"],
  ...
}
extras_require["all"] = list(set(req for extra in extras_require.values() for req in extra))

You might also have a dev extra that combined a common set, such as tests, docs, and auto-formatters.

extras_require["dev"] = list(set(
    req
    for extra in ("doc", "test", "style")
    for req in extras_require[extra]```
))

Using setup.cfg allowed you to do this in a declarative way, thanks to ConfigParser:

[options.extras_require]
test =
  x
  y
  z
doc =
  a
  b
  c
all =
  %(test)s
  %(doc)s

I can keep doing this in setuptools, at the cost of keeping around both pyproject.toml and setup.cfg (or setup.py), but there doesn’t seem to be any way to do the equivalent in pyproject.toml. So if I want to try out or migrate to a new build backend, I’m back to a place where convenience extras need to be checked for consistency rather than just built from their components.

1 Like
2

It is already possible for a package to self-reference:

[project]
name ="mypackage"

[project.optional-dependencies]
test = ["pytest"]
doc = ["sphinx"]
style = ["black"]
tests = ["mypackage[test]"]
dev = ["mypackage[test]", "mypackage[doc]", "mypackage[style]"]
all = ["mypackage[dev]"]

Perhaps it would be easier to build upon and improve this syntax by focusing on

  1. Reduce repetiveness in config
  2. Help build back-ends to eagerly-flatten self-referential dependencies

For example, we could allow using a single dot to mean “this exact package” to reduce the syntax to this:

[project.optional-dependencies]
test = ["pytest"]
doc = ["sphinx"]
style = ["black"]
tests = [".[test]"]
dev = [".[test]", ".[doc]", ".[style]"]
all = [".[dev]"]

and introduce glob-like pattern matching to extra names, so we could have perhaps

[project.optional-dependencies]
db-mysql = ["mysqlclient"]
db-postgresql = ["psycopg2"]
db = [".[db-*]", "sqlalchemy"]

The special . (which is an invalid package name) would help signal self-reference, and a back-end can therefore choose to eagerly expand the list of dependencies when it builds the sdist or wheels, instead of when the package is being installed, improving dependency resolution performance for package users.

2 Likes
3

This feels to me like it would very rapidly expand in scope. People would want to be able to specify mypackage[db-*] on the command line. Or use my*[test].

And given that there’s no actual use case provided by the OP (beyond “I would like some way to…”) I don’t think there’s a real justification for either the added complexity, or the maintenance cost of having to repeatedly explain the limitations of the new syntax.

For the occasional project where something like this would be helpful, why not just write a preprocessor to generate the pyproject.toml from a template?

4

This seems completely adequate to my needs:

[project]
name ="mypackage"

[project.optional-dependencies]
extra1 = [...]
...
alias = ["mypackage[extra1,...,extraN]"]

The use case is really to have a procedure I can follow to switch over projects that have historically done this to alternative backends, so I’m not worried if the syntax is a bit clunky.

I agree with @pf_moore that I don’t really want to create a new syntax that other tools need to implement and provide support for. That’s why I was hoping there was a way to write modular mini-backends, but I guess I haven’t found a use case that really needs that feature yet.

Thanks, both!

5

FWIW, it might also make sense as an extension of an existing backend. :slight_smile:

Hatch provides a very clear extension point to implement metadata customisations, such as this one: Metadata Hook Plugins.

2 Likes
6

Note: Document support for self-referential extras · Issue #11296 · pypa/pip · GitHub

2 Likes