I’ll be honest, in my case it’s because I’ve had a lot of RL issues to deal with. I’ll try to review the new version soon, but I can’t promise. But don’t feel that you have to wait for my comments.
1 Like
And here I am showing up w/ a comment. 
I anticipate the question of how to signal that a dependency group should include project.dependencies or something from project.optional-dependencies (for the “dev dependencies” case)? I’m not expecting a solution in the PEP for it or that include is wrong, it’s just something I thought of that I expect to come up. It might be worth saying that’s considered out-of-scope for the PEP if we aren’t solving it here.
The PEP also has an open issue regarding supporting a list for include, so we should resolve that. I think it should be either a string or list, but not both for simplicity. If I had to choose it would be for the string as most of my requirements files that have -r only have it once.
What kind of buy-in were you thinking?
1 Like
I’ve read the PEP now, and I only have a couple of minor comments.
First and most importantly, the PEP reads really well and explains its position very clearly. For a first PEP, it’s really well-written - congratulations, @sirosen!
The semantics of dependency group includes is implied but not stated explicitly. I assume the semantics are that the included group is to be inserted directly in place of the include directive, so the ordering of the resulting list of requirements will be as you’d naturally expect. This mostly shouldn’t matter, although there’s no explanation of what the semantics of conflicting requirements should be (for example foo>1.0 and foo<1.0). I think it should be mentioned, even though it’s nothing more than “consuming tools should treat the resulting list of requirements exactly as they would any other case where they are asked to process multiple requirements”.
The reason I thought about the above point is that I considered the possibility that people might think includes override values in the including group (or vice versa). But I think that’s overcomplicating things for no benefit, so I’d prefer just to be explicit that the whole thing is just one merged list that tools process “as normal”.
One other point from the notes on the reference implementation - you say the spec doesn’t forbid cycles in includes, although the reference implementation rejects them. Why not just prohibit them in the spec? I can’t see any valid argument for allowing them. You don’t have to insist tools report the error, just say “dependency group includes MUST NOT include cycles”. Maybe say “tools SHOULD report an error if they detect a cycle” if you like.
I agree with @brettcannon that allowing a list in the include item isn’t worth it. Multiple includes are just as good.
I’d drop the PEP 723 use case. Dependency groups don’t match up to the [run] section as the PEP defines, and trying to link dependency groups to the ongoing discussions on how to modify PEP 723 given that [run] may never be standardised is just going to suck the dependency group proposal into that whole debate, for very little benefit.
In the package building section, you talk about build backends having their own mechanisms for referencing dependency groups from the project metadata. While I think that’s a reasonable separation of concerns for this PEP, I am worried about the implications, as it might encourage a greater use of dynamic package dependencies - when we’re trying to encourage developers to make as much metadata as possible static. Having said that, the backend can tell that the data is static for the sdist, and therefore can mark it as such (once we get metadata 2.2 implemented). So it’s probably not a real issue in practice.
As I say, just minor points. Overall, the PEP looks pretty good to me 
Regarding buy-in from pip, I’d say that assuming the PEP gets accepted, pip would happily take a PR implementing dependency groups. I wouldn’t commit the pip maintainers to delivering the implementation - it would likely not be a high priority compared to some of the other stuff that needs doing - but community submissions would be welcome. I think we’d need to consider how dependency groups affect the --only-deps work, but on a superficial glance, I think the two are sufficiently independent that the only real question is “do dependency groups solve enough of the user issues to remove the need for --only-deps, or do we want both?” In any case, --only-deps is also awaiting someone to submit a PR, so it shouldn’t hold up dependency groups.
2 Likes
This is something which is still on my mind, and it overlaps heavily with Paul’s comment about this pattern encouraging dynamic dependencies.
What I would like is a way of supporting inclusion of project.dependencies, but only when it is a literal list. Excluding the use of dynamic dependencies would make it possible to implement a compliant tool using dependency groups which does not include a build frontend.
It’s beneficial when all of the data is statically defined, but I think I may have inverted things in the current draft. In order to make the new data “static only”, I excluded project dependencies (which could be dynamic). Doing that means that including one in the other requires dynamic dependencies.
If I can come up with a palatable way to express “include the TOML list at project.dependencies” and, relatedly, “include the TOML list at project.optional-dependencies.$name”, would that be interesting to pursue? A key characteristic here is that it’s defined against the contents of the TOML file, not package metadata.
I had left it in because it hadn’t been explicitly addressed in this thread, and it was a question asked during the initial PR review.
But I agree – multiple -r usages is a rarity, and I’d rather keep things simpler by only allowing a single string.
I’ll remove it in an upcoming update.
I was going to open a PR which adds a reference implementation. It’s little more than the current reference code, cleaned up a little and with any enhancements needed (like unrolling that recursion into something iterative, if desired), plus some tests. It could accept the parsed TOML contents.
The output would be a list of Requirement objects.
If that stands a chance at being accepted (contingent on the PEP, of course), I’d be glad to know it. If not, I’d like to understand why / how it breaks the boundaries of what packaging provides.
Yep, I consider these comments of yours a good call out. I’ll take a look at buttoning that up more.
I think this is a subtle but worthwhile refinement. I wanted tools to be free to ignore cycles if they have some compelling reason to try to squash or hide errors (this is not a type of design I like, but some tools bend this way). I also don’t see a reason to allow cycles.
I’ll update to state that cycles are forbidden by the spec, but in terms of implementation requirements, reporting cycles as errors is a “SHOULD”.
I agree; I’ll remove it. I thought we were closer to a possible unified solution between the two than we really are. The latest updates on that topic have clarified this quite a bit.
Color me nervous/excited to try to put this together! I’ll probably start by opening an issue to hash out the interface, which nicely lines up with Pradyun’s comment above.
Maybe it would be better to define a means for including a dependency group in the [project.dependencies] data? That would be 100% static, and easy to understand - the only downside is the need to modify the existing syntax of [project.dependencies] which means there’s a transition/compatibility question to address.
1 Like
I personally would love it, but I think it’s optional as it could we addressed later.
packaging currently has no specific code around pyproject.toml, so I wouldn’t worry about it (i.e. it’s a bigger ask than just this PEP).
So are you thinking you define an e.g. “default” group and then say that represents project.dependencies? That way your “test” group depends on “default”? I can see that working via something like a dependencies = {dependency-group = "default"}.
I assume you wouldn’t use it unless your build back-end supported it, so the transition shouldn’t be too bad.
1 Like
More or less. But I wouldn’t (personally) name it “default”, I’d be more likely to name it something like “runtime” as that’s what it is - the runtime dependencies. So you’d have something like
[project]
name = "foo"
version = "1.0"
dependencies = [{include = "runtime"}]
[dependency-groups]
runtime = ["requests", "packaging", "click"]
(re-using the include syntax from dependency groups as a new option allowed in [project.dependencies]).
Or if you want to expose your “test” dependencies as an extra as well as using dependency groups - maybe as a transition because you used to use extras but you’re now moving to dependency groups.
[project.optional-dependencies]
test = [{include = "test"}]
[dependency-groups]
test = ["pytest"]
Note that I’d allow extras and dependency groups to have the same name, and that include always refers to a dependency group and never an extra.
4 Likes
I like this a lot! Letting the includes only go “one way” makes intuitive sense to me with the examples you showed.
2 Likes
I think it might be reasonable to say group = "test" or include-group = "test" to make this relationship more explicit.
2 Likes
I really like where this is going. Including a group in package dependencies follows naturally, and strikes me as easy to explain/teach.
I see two questions arising from this:
- Should the include key be changed, e.g., to
include-group, for better usage and symmetry in multiple contexts? - Should this PEP include a definition of includes for use in package dependencies?
Changing the key seems fine, and using the same key in all places seems desirable to me. If include-group scans well, I’d go with that. Do others agree?
Changing the scope of the PEP makes me nervous. It seems like a small addition to include groups in package dependencies, and one which would make it easier to talk and think about the feature into the future. A “build backend which supports Dependency Groups” would become very clear and easy to grasp as an idea. I also think it improves the value proposition of Dependency Groups if it can integrate with package metadata.
So I want to include this in the spec, but I’m not rushing to do so. What needs to be defined for this to be included?
- build backends would expand any Dependency Groups when building distributions
- … anything else?
1 Like
Build backends are not the only consumers of the project table. There’s Dependabot, IDEs, dependency management tools (eg pip-tools (though I think this one invokes the backend?)), and others. Are you happy to have those tools fail when users start using the new syntax [1]?
I think it makes more sense to me to have groups include the (existing) runtime dependency specifiers. This is because I attribute the dependency groups as the requirements for tasks or scenarios, when I have a copy of the source (as opposed to runtime deps, which are specified in the wheel and installed metadata).
I most like the elegance of using path = "." (from an earlier version of this PEP, I think), and I think that could be re-introduced in the next PEP (perhaps even without editable support).
especially as there’s no intention to version project.toml ↩︎
2 Likes
I get this concern, but it argues for never introducing new syntax anywhere into the packaging ecosystem. I think that long term that’s more destructive than having features roll out across the ecosystem, and users start using them once sufficient support arrives.
Am I happy about the fact that going down this path will probably result in something failing for someone? No.
But I am willing to accept it, and I don’t think that this should prevent us from choosing the best long-term option.
I’m open to this in theory, but I’ll elaborate on why my perspective is shifting strongly towards the inclusion of Dependency Groups in dependency lists.
In short: I very much want to restrict Dependency Groups to static dependency declarations.
If we allow, for example, {project-include = "dependencies"} as a syntax, then we need to think about what this means in the presence of dynamic dependency data. I would like it to be defined as invalid in this case – otherwise, any tool trying to support Dependency Groups needs to be ready to run a build each time it evaluates a group. You could cache builds to try to blunt the pain, but you need to invalidate that cache correctly.
Under the constraint which I care about – static data only – it’s very hard for me to see this solution as the equal to project metadata including Dependency Groups. It means that combining seemingly valid things on the surface, Dependency Groups + dynamic dependencies, produces a broken package config.
So if our choice is between
A:
[project]
dependencies = [{include = "runtime"}]
[dependency-groups]
runtime = ["requests", "packaging", "click"]
and
B:
[project]
dependencies = ["requests", "packaging", "click"]
[dependency-groups]
runtime = [{project-include = "dependencies"}]
I’d say that (B) is better in the short term, but much worse in the long term because it allows for…
B-prime:
# this config would always be invalid but look sensible!
[project]
dynamic = ["dependencies"]
[dependency-groups]
runtime = [{project-include = "dependencies"}]
If the counterargument here is that “B-prime” should be valid too, I simply don’t agree. It imposes a burden for implementers which I don’t (at least, at present) think is acceptable.
The other position I can see as a way to allow for this is that this should be resolved by having dependency group installation also install .. That basically makes any inclusion rule here a “secret path dependency”, which I don’t consider okay. It would also install a package (the current one) which is not stated in the config. This line of thinking leads towards…
I agree that this is a nice, simple way to express these relationships, along with support for extras.
However, this thread has demonstrated that Path Dependencies are tricky. Not all of the issues are related to editable installs (although I think it’s a big driver of complexity). Some of it is the simple fact that it’s a non-PEP 508 dependency specification, so you need to work out lots of details.
I’m concerned that if the pitch for Dependency Groups integrating with project.dependencies is that “it will be solved when we define local Path Dependencies”, we’re really saying “these two components will be integrated in the distant future”.
So I agree that this is a lovely way to write the relationship, with intuitively pretty good semantics… But I’m -1 on going down that path. Maybe I’m overly pessimistic about this, but I think it punts out good user-facing features into a very distant future.
5 Likes
Is the suggestion that this would happen when producing an sdist as well as a wheel?
Basically, it needs a transition plan, because it is a change to the definition of the semantics of the [project] table. Unfortunately, PEP 621 didn’t specify any versioning for the [project] table, so there’s no clear solution here (this would have been a problem anyway when core metadata changes - so it’s not specific to this PEP).
Because of the versioning issue, it may be too much to introduce into this PEP. Also, it does mean that future proposals to add things like path based dependencies would have to address the fact that you can’t include a group that has those extensions in it, in [project.dependencies].
So yes, I think it might be too much of a can of worms.
Having said that, I’m still far more happy with something like this, than with encouraging backends to do something similar via a custom extension and dynamic dependencies.
And while being able to have an “extended form”[1] that says “include the project runtime dependencies” or “include this extra that the project defines” would be an expedient solution, I don’t really like it as it feels like the information is flowing in the wrong direction, somehow…
We should come up with a term for “dependencies that can go in a dependency group which aren’t PEP 508 requirements”, even though we’re not allowing any such thing right now, just so we can talk about them easily!] that ↩︎
5 posts were split to a new topic: Versioning pyproject.toml
Does this require the build backend to edit the pyproject.toml when creating the sdist?
I’m asking these questions because:
- I thought that having “tools” edit pyproject.toml was frowned upon but maybe it is okay when producing an sdist.
- The backwards compatibility concerns for adding new syntax in pyproject.toml seem significantly less to me if the new syntax is not found in sdists.
Expecting build backends, IDEs, dependabot etc to update to be able to understand some new pyproject.toml syntax in VCS checkouts does not seem like a big deal to me. These are developer tools and the developer can choose whether to use the new syntax and what tools they want to use.
What would be a more significant problem is if PyPI started filling up with sdists that break installation for old versions of pip etc.
I like that you are aiming to keep these dependencies static. To enforce this, you could add a requirement that tools supporting dependency groups must raise an error when a reference to a dynamic dependency is made. That is a straightforward requirement and should be preferred over extending project.dependencies in a way that yields substantial adoption and backwards compatibility issues I think. There is no fundamental problem with keeping this inside the design for dependency groups, so talking about versioning of the [project] table is going down the wrong path.
Other issues with the current proposal beyond what @EpicWink mentioned include:
- It is very well possible that one wants to include optional dependencies or build dependencies in a particular dependency group (e.g., a
devgroup probably needs the build dependencies installed). If you want to be able to refer to another list of dependencies, you should aim to support all such lists that exist today imho.- We’ve learned this lesson before, e.g. for overriding constraints IIRC there’s still an asymmetry in Pip’s UX where runtime deps can be overridden more easily than build deps, and that can be painfully difficult to work around.
- This is easier to do when including it in dependency groups, rather than also updating the semantics of those other existing metadata fields.
- If you change
project.dependencies, it is likely to affect other designs. E.g., PEP 725 is making a symmetric introducing ofexternal.dependencies: pep-0725/#dependencies-optional-dependencies. We shouldn’t have to worry about dependency groups there. - The “layering” of the design seems conceptually wrong the way you want to do it. Build and runtime dependencies are more central to a package, and are (as you point out) user-facing.
- A common way of finding a package’s dependencies is to look at its
pyproject.toml(it’s 2-3 clicks: go to PyPI, click the Source link, open pyproject.toml in your browser). Most other methods, likepipdeptree, require installing the package first. So now that user-facing metadata is moved to a more private location with extra mental overhead.
- A common way of finding a package’s dependencies is to look at its
I don’t see why there is a problem. We can just say that it doesn’t allow for B-prime, by mandating that a project-include key can only reference either an actually existing dependencies key or an explicitly specified optional-dependencies.foo. After all, it already needs to be restricted to [project] entries that actually are lists of dependencies - right?
…But given those restrictions, I feel like this sort of structure would make more sense:
[project]
dependencies = ["requests", "packaging", "click"]
[dependency-groups.runtime]
# All of these would be optional, default values shown:
include-dependencies = True # whether to use `project.dependencies`
# extras = [] -- list of names to use from `project.optional-dependencies`
# dependencies = [] -- list of actual PEP 518 names unique to the group, following normal rules
# include-self = True -- hopefully self-explanatory; maybe beyond scope
with the proviso that include-dependencies = True is an error when project.dependencies is dynamic, and extras can only include names actually present in project.optional-dependencies.
There’s one aspect of the versioning/transition issue that is directly relevant to this PEP.
Given that the PEP explicitly notes that things like path dependencies are deferred for future revisions, how should this PEP advise tools to handle such “future extensions” if they encounter them? If we don’t answer that question we are just repeating the same mistake that we made with the [project] table. And while I’m sympathetic to the view that this is an issue that needs to be solved by versioning pyproject.toml as a whole, we still need to decide what to do before then (even if it’s “extensions are prohibited until versioning is solved”).
1 Like
A technicality in the Specification section:
These keys must match the following regular expression:
[a-z0-9][a-z0-9-]*[a-z0-9]. […] These requirements are chosen so that the normalization rules used for PyPI package names are unnecessary as the names are already normalized.
Name normalization calls for collapsing runs of [.,-_] to dashes. x--x would be valid with the regex in the PEP, but it’s not a normalized name.
Did you mean [a-z0-9]([a-z0-9]|-[a-z0-9])* ?
1 Like