I get this concern, but it argues for never introducing new syntax anywhere into the packaging ecosystem. I think that long term that’s more destructive than having features roll out across the ecosystem, and users start using them once sufficient support arrives.
Am I happy about the fact that going down this path will probably result in something failing for someone? No.
But I am willing to accept it, and I don’t think that this should prevent us from choosing the best long-term option.
I’m open to this in theory, but I’ll elaborate on why my perspective is shifting strongly towards the inclusion of Dependency Groups in dependency lists.
In short: I very much want to restrict Dependency Groups to static dependency declarations.
If we allow, for example, {project-include = "dependencies"} as a syntax, then we need to think about what this means in the presence of dynamic dependency data. I would like it to be defined as invalid in this case – otherwise, any tool trying to support Dependency Groups needs to be ready to run a build each time it evaluates a group. You could cache builds to try to blunt the pain, but you need to invalidate that cache correctly.
Under the constraint which I care about – static data only – it’s very hard for me to see this solution as the equal to project metadata including Dependency Groups. It means that combining seemingly valid things on the surface, Dependency Groups + dynamic dependencies, produces a broken package config.
So if our choice is between
A:
[project]
dependencies = [{include = "runtime"}]
[dependency-groups]
runtime = ["requests", "packaging", "click"]
and
B:
[project]
dependencies = ["requests", "packaging", "click"]
[dependency-groups]
runtime = [{project-include = "dependencies"}]
I’d say that (B) is better in the short term, but much worse in the long term because it allows for…
B-prime:
# this config would always be invalid but look sensible!
[project]
dynamic = ["dependencies"]
[dependency-groups]
runtime = [{project-include = "dependencies"}]
If the counterargument here is that “B-prime” should be valid too, I simply don’t agree. It imposes a burden for implementers which I don’t (at least, at present) think is acceptable.
The other position I can see as a way to allow for this is that this should be resolved by having dependency group installation also install .. That basically makes any inclusion rule here a “secret path dependency”, which I don’t consider okay. It would also install a package (the current one) which is not stated in the config. This line of thinking leads towards…
I agree that this is a nice, simple way to express these relationships, along with support for extras.
However, this thread has demonstrated that Path Dependencies are tricky. Not all of the issues are related to editable installs (although I think it’s a big driver of complexity). Some of it is the simple fact that it’s a non-PEP 508 dependency specification, so you need to work out lots of details.
I’m concerned that if the pitch for Dependency Groups integrating with project.dependencies is that “it will be solved when we define local Path Dependencies”, we’re really saying “these two components will be integrated in the distant future”.
So I agree that this is a lovely way to write the relationship, with intuitively pretty good semantics… But I’m -1 on going down that path. Maybe I’m overly pessimistic about this, but I think it punts out good user-facing features into a very distant future.