Okay, lesson learned! I’ll be sure to wait next time.
For my own edification, “Draft” means it has been merged but still has Status: Draft? That’s what I take this to mean.
I had forgotten about that thread, thanks for raising it! I just gave it a speed-read to refresh.
It’s very, very similar. To the point that some of the criticisms in that thread are valid criticisms of this idea and I should address them.
But there are some distinguishing factors. At the very least there’s this:
This does not assume that the code is a package – it is rather more purpose-built as a requirements.txt replacer.
I also suspect that some of the folks involved in that thread may have shifted their views slightly in the past calendar year. For example, I saw that @pf_moore suggested that poetry and pdm could work together on an informal standard, which could then be formalized. The fact that a calendar year has passed without any attempt at such collaboration (as far as I know) makes me think it’s no longer wise to wait for such a thing to happen unprompted.
There are some ideas in that thread, like “unpublished extras”, which I should address in rejected ideas.
Okay; I’m not overly attached to this. Clearly the naming is essential, so I’ll need to noodle much more on proper naming and namespacing.
That’s a fair criticism. I’ll need to devise some other syntax. I think @kknechtel suggested .[test] and @test above. I don’t really care, but I don’t want to swoop in and change it without any further discussion. Of these, do any appeal? Good alternatives?
@test
<test>
:test:
I don’t agree, so I definitely need to address this.
To me, four issues with --only-deps jump out.
It seems to be defined to include the project.dependencies data. That is explicitly not included by default here (it is not assumed to exist).
This doesn’t address non-package projects (you need package metadata to have an extra).
Nothing inherent to the pyproject.toml contents communicates that an extra is meant to be used --only-deps. So information is fragmented between extras and configs/scripts which use them.
It’s unclear what the expected presentation of this flag is for tox, nox, and hatch, but it seems difficult to mix with other constructs because it’s a flag to pip potentially bound to a single extra name.
I don’t think that --only-deps is bad, but I don’t think it’s solving quite the same problems. And I wonder a little if it would even be on the roadmap if we had dependency groups.
I’ll write these up with a bit more detail and tie them back to the Motivation section, possibly with some edits there.
At this point, I think I have enough input to take a crack at a second draft here. I’m going to go ahead and try to knock that out shortly.
It is something in that vein, yes. In “the meandering thread”, @brettcannon strawman-proposed something similar, which led to the current effort. It didn’t seem to me like any of us in that discussion recalled your prior proposal (I certainly wasn’t aware of it), or else there was just nobody who made the connection. This proposal is a bit more featureful; it also occurs in the context of an additional year+ of discussion about what people want to accomplish with pyproject.toml (and, perhaps, people changing their minds about that).
Something that seems to have been missed in the thread you’re linking now: if someone tries to use extras to define dependencies for testing, those will also end up in wheel metadata, not just sdists - while the wheel is unlikely to include the actual tests. Meanwhile, the idea here is that tools on the developer’s side could know to install the test dependencies in a separate (perhaps even temporary/isolated) environment. A user who tried to pip install mypackage[tests], on the other hand, would pollute the same environment where mypackage was installed, without gaining the ability to test mypackage.
Conversely, though, if we simply design a standard without buy in from the Poetry and PDM developers, why would we have any confidence that such a standard will actually be adopted by those tools? Or, for that matter, that the standard will address the needs of the users of those tools? The only point in standardising an approach here is if it’s something that users can adopt independently of what tools they use. So how do we get the Poetry and PDM developers interested in this effort? Not just willing to implement whatever gets decided, but contributing their experience and insights to the design.
And actually, @pradyunsg made the point about pip’s --only-deps proposal. That’s actually another case of a tool that is currently working on a solution in this area, and which needs to be considered.
I agree that --only-deps and the dependency group proposal here have significant differences. For what it’s worth, the current proposal for --only-deps, as described by me in this comment, is as follows:
Given an invocation pip install --no-deps <target>, what we do depends on the target.
It seems self-evident to me that if we’re looking at a wheel, we should use the METADATA file.
If we’re using a source tree, then we have no metadata file, but if we have static dependencies in a PEP 621 compliant [project] table, then we can be sure those dependencies will be in the final wheel unchanged, and so we can rely on them.
If we’re using a source tree with dependencies specified as dynamic in pyproject.toml, then we have no dependency data at all (note that dynamic fields MUST NOT also have a static value specified, according to PEP 621 - “Build back-ends MUST raise an error if the metadata specifies a field statically as well as being listed in dynamic”).
If we’re using a source tree with no PEP 621 data, we have nothing.
If we’re using a sdist, which has a PKG-INFO metadata file, and the metadata version is 2.2 or greater, then we should use that. Either Requires-Dist is not in Dynamic, and we know it’s reliable, or Requires-Distis in Dynamic, and we therefore know that pyproject.toml does not provide static dependency data either.
If we’re using a sdist where the previous conditions don’t apply, the best we can do is treat it as a source tree.
In all the cases where we have no dependency data (source tree with dynamic dependencies, or no [project] data) then we have no choice other than to call prepare_metadata_for_build_wheel. It’s potentially costly, but there is no other viable option to get the information. We could raise an error saying --only-deps is unsupported in that case, but I don’t see how that’s helpful.
The key thing here is that --only-deps, as described here, is a strict superset of the dependency group proposal for projects where it applies (i.e., ones that are intended to build a wheel) if you use extras to define dependency groups.
The fundamental point here is that, as a pip maintainer, I don’t want to have to support both--only-deps and dependency groups. So unless dependency groups can address the use cases that are driving the design of --only-deps, I’m not comfortable making a commitment that pip will support dependency groups[1].
And in particular, I think the point that some people currently use extras for this type of situation needs to be addressed. Either we have to continue supporting both approaches (further adding to the criticisms that “packaging has too many ways of doing the same thing”) or we have to address the fact that there’s a transition cost for people currently using extras. Specifically, I think the PEP needs to include in “How to teach this” a very clear explanation of how users should choose between using a dependency group (say, for test requirements) and an extra. And in the section on backward compatibility, the process for migrating from extras to dependency groups needs to be explained (including over the transition period where some, but not all, tools support dependency groups).
A minor point, I prefer dependency-groups (with a hyphen) rather than underscore, because that’s the norm in pyproject.toml. Beyond that I don’t have much opinion on naming except to say that the people who are pointing out it’s important are right
While I agree that there seems to be no appetite for a complete v2-style redesign, and I agree that means we have to focus on incremental changes if we don’t want to just stop changing anything, I don’t think that allows us to ignore the wider impact.
Like it or not, these days there are multiple packaging tools in existence (the “one official tool” model is very much in the “v2 design” area right now). As a result, we must ensure that any standards we create are unifying rather than divisive. The xkcd comic on standards is very relevant here - if our standards don’t reduce the number of ways of doing something, we’re not helping our users at all. That’s why I much prefer standardising existing practices, and in particular, why I prefer to strictly scope dependency groups as an[2] attempt to standardise (a subset of) requirement files.
I should point out that it would be really easy for someone to write a tool that reads dependency groups from pyprojects.toml, constructs a pip command line out of the result, and installs the dependency group that way. So pip not supporting dependency groups need not be a showstopper. But based on my experience, I’d expect users to complain at having to use a wrapper tool for this, so I don’t think it’s a reasonable option in practice. ↩︎
Strongly agreed. I would be uncomfortable with this moving forward without input and support from those projects. Is it appropriate to @-mention them to pull them into this thread?
For PDM, I don’t think it’s likely to be a challenging topic (I know, I’m jinxing myself here…) – this maps pretty closely to what already exists there.
For Poetry, this spec is strictly less capable than what they have today. That might translate into “easy to implement” or it might translate into “reticent to support it”, since it would mean supporting two different versions of the same idea.
I don’t think that either one is a superset of the other, though I may be mistaken.
Dependency Groups allow you to do an install of a package list without project.dependencies data, which --only-deps can’t do. --only-deps can install the project.dependencies data without the package itself, which Dependency Groups can’t do.
I’m confident that Dependency Groups address most or almost all of the use-cases for --only-deps, based on issues I’ve read.
The question I have right now is whether or not the outstanding case – installing project.dependencies without the package itself – is significant in its own right. I’ll read through the issues and threads after my workday to try to understand this.
If so, it might be necessary to add a way of expressing that to Dependency Groups, if we want to put pip in a situation to only implement one of the two behaviors.
It already seems like I’ll need to fix the syntax being used anyway. Hopefully these dovetail nicely.
My pessimistic side says that anything new will be criticized as “yet another way of doing the same thing”. So I want to be cautious about how we respond to this pressure – not “disregard it”, just “cautious”.
Including migration guidance for extras-to-Dependency-Groups seems like a good inclusion to me. I’ll work on that soon.
I could make my POC code (still haven’t started on it) something that outputs requirements.txt files. That would make it a useful tool for migrating and maintaining “compatibility shim files”. If it seems like a good idea, I think I can take a crack at it.
There’s an inherent tension here, which is what I’m grappling with.
I want to standardize (a subset of) requirements.txt files into a section in pyproject.toml.
This takes the form of a new standard.
I don’t think we reduce the number of ways of doing things in the short term with any (reasonable) additive improvement. We increase the options, there’s a transitory period during which people adapt, and then later we may be able to remove some of the old ways.
If we want to drop requirements.txt files entirely ( !), we’ll need to address --index-url and --hash at least. I’ve tried to leave space for these (--index-url could go at what is presently namespaced requirements.index-url, I think).
My goal is for this to be a spec which becomes capable enough one day replace allrequirements.txt files. requirements.txt will probably never fully “die off”, but it would become the legacy way of doing things. For example, if pip-tools is willing to write back into the Dependency Group data, I could imagine a future in which I have
[testenv]
# using a tox factor to select the *locked* variant!
dependency_groups =
async: async-locked
commands = pytest
I know I went a little bit long there on what the future could hold, but that’s what’s in my head as a possible outcome.
A different angle is that this spec would make the Poetry and PDM Dependency Group feature-sets disappear into this new spec. So in that way we could get fewer ways of doing things in the short term. But I think that given how Poetry supports some non-standard and special things (like ^x.y.z version specifiers), it’s hard to see this fully replacing those in short order. For PDM, I think it’s more believable unless PDM has features I don’t know about which make this similarly hard.
In any case, it seems from this feedback that I have some refinements to make and I’ll try to put together some POC bodies of code to add to the PEP soon as well.
So run mean dependency for install or runtime?
Run on its own suggests that this is something that is runnable, a script, and as such I find it misleading when it refers to dependencies.
I was meaning in terms of “use cases satisfied”, not technical details. For projects that build into wheels (a constraint I stipulated) you can replace a dependency group with an extra that specifies the same requirements, and use pip install --only-deps .[extra]. That’s what I meant by a “strict superset”.
From my recollection, it was the original request, so yes, it’s important. The requirement was to be able to install the dependencies independently of building the project itself (which might need special configuration such as environment variables for the build).
But please don’t just add functionality to dependency groups to do this. If you do that, you’re missing the point that installing “the project dependencies only” for a project that builds into a wheel is more complex than just reading pyproject.toml, and (if the dependencies are dynamic) must of necessity involve the backend.
We’re back to the idea that projects that build into a wheel are in a sense fundamentally different from ones that don’t[1]. And we don’t yet have any form of consensus over how we deal with that.
+1. The key here is understanding what the user is trying to do, and clearly distinguishing the solutions. At the moment, there’s too much focus on implementation, and not enough on intent (IMO). We’re designing a bunch of tools that have a long handle and a heavy bit at the end. Some are hammers, some are spanners, some are pliers, etc. But the user wants to “bang something in”. How do they know that the hammer is the one intended for that use case, and the others, while they will work in a pinch, are actually inappropriate for that task? And given the range of types of hammer, how do we find out more details about what the user actually means when they say “bang something in”? We don’t want to offer them Mjolnir to fix a thumbtack to the wall…
even if it’s only in the sense of “we haven’t really thought about the other types much, yet” ↩︎
Hello, PEP editor here! Thanks Pradyun for sharing those process notes!
It’s great people are keen on putting in the time and effort to write up PEPs, but this is the second time in a week so I’d like to re-iterate the importance of finding a sponsor or core dev co-author first, then check they’re happy with the draft, before opening a PR in the repo. Only then is a number assigned and review can begin.
This helps set expectations and saves a lot of time for everyone: PEP author’s writing, editors’ reviewing and community discussion, and can avoid putting potential sponsors in a tricky spot.
Full process is defined in PEP 1, see also this discussion for some extra notes, and don’t hesitate to open topics if you’ve any questions, we’re happy to help!
Several people have already weighed in on this aspect, so why not one more person ! I found this section of the PEP a bit jarring until I reasoned my way through it:
Initially I thought “this must be a typo, there’s nothing defined for types!” But of course types is defined elsewhere, in project.optional-dependencies.
All of this to say: it would be great if the syntax for these requirement groups was able to replace those other options [1]. If I were managing such a complex setup, I would want to put all my various dependency lists in one section.
While there is a distinction between extras and dependency groups, I’m not sure it’s such a distinction that the two can’t be effectively merged, with the right syntax.
One possibility is to define some default values for dependency groups, which can be overridden when needed. Another possibility is to have a top-level field that defines which groups should be packaged as extras (sort of like __all__ defines what it exported from a module). e.g.
[requirements.packages]
test = ["pytest", "coverage"] # implicitly includes the package
docs = { dependencies = ["sphinx", "sphinx-rtd-theme"], package = false } # don't install the package for this one
types = { dependencies = ["typeshed"], extra = true } # moved from optional-dependencies
typing = ["mypy", "types-requests", "[types]"]
typing-test = ["[typing,test]", "useful-types"]
# or alternatively, a keyword to denote which groups are extras (defaulting to all)
[requirements]
extras = ["types"]
I’m not sure which way the defaults should go, in terms of most common usage. But it feels like this table could consolidate some of this stuff into one place, and tools would be able to install these groups using the same syntax as extras.
entirely optionally, of course, not changing the existing project table ↩︎
I’m open to making this a more-capable replacement for package.optional-dependencies which can declare extras, if there’s popular support.
I’ll note that it aligns pretty well with how PDM handles dependency groups – by default they are extras, and a flag puts them in a separate “dev” section.
The main difference I’d like to introduce is the possibility of a dependency group which doesn’t implicitly include package.dependencies (which PDM does not have).
I think I would prefer for extra declarations – if we go this route – to always be explicit. By default, a dependency group should not be an extra. That way, the parts of your package which are public interfaces (extras) are never “on by default”.
I may well have misunderstood something. The arguments that I saw in that thread seemed to be particular to sdists. It came across that people were objecting to “exposing dependencies to end-users as extras” purely on aesthetic grounds, and that others were therefore not appreciating the value of hiding them.
If it’s backwards-compatible then it ignores it, else it errors out. I don’t think we want to prematurely optimize here.
That’s going back to Projects that aren't meant to generate a wheel and `pyproject.toml` and the whole reason that conversation was started. We can always revisit that, but people here already complained that it was too long and meandering, suggesting a lack of consensus to begin with.
I agree. I don’t want to try and explain that to a beginner who just wants to do the right thing and right down what dependencies the code in some zip file has. I think one of the reasons requirements files have worked is because there isn’t much ceremony to them and they map to what you specify for pip and other installers to install.
It’s similar to my strawman, although I proposed a bit of requirement syntax for referring to other dependency groups and a solution to how to write stuff down for PEP 723.
I agree it may be easy to overlook.
But this goes back to the “[project] is only for wheels” discussion since that doesn’t work for the web app scenario.
Correct.
As one of those people, it’s honestly to keep my dependencies together and in a consistent format. Using the web app example of run, test, lint, and docs dependency groups, I could have the run requirements in [project.dependencies] and then have a requirements-test.txt that consisted of:
.
pytest
and a requirements-lint.txt that had:
ruff
But that does physically separate stuff a bit and in two different formats. And on top of that it’s pip-specific and not necessarily as discoverable trying to figure out how to run tests or build docs from an sdist as using extras. But the fact they are extras which makes them a public “API” for my wheel is actually an annoyance I’m just willing to live with.
Ditto. Plus requirements files have shown to be flexible and simple enough to meet a lot of needs.
I just had a thought in response to Pradyun’s point about pip install --only-deps. In that instance we are letting pyproject.toml just record some stuff with some semantic meaning, but we are letting a tool choose how to use that information. Or put another way, with --only-deps, pip is saying it’s willing to ignore that something is meant for a wheel which won’t be installed and effectively skip the whole “this is a wheel” aspect of pyproject.toml. Up until this point, though, we have all operated under the assumption that everything in [project] is for some wheel, while --only-deps suddenly ignores that fact.
What happens if we take this a bit farther? The reason pyproject.toml requires name and version for [project] is because core metadata requires it. But what if we said that build back-ends are the ones that require name and version because they need it for core metadata generation? We can still say that name can’t be dynamic for discoverability purposes, but we could drop the requirement that pyproject.toml consumers must enforce that name always be there. Thus the requirements of the core metadata is enforced by build back-ends and not by the specification of [project].
This then lets us use project.dependencies and project.optional-dependencies as-is since the installer chooses to view that data in isolation from the rest of [project]. Whether everything in optional-dependencies infers dependencies could be left up to the installer and the user since we are giving tools more leeway in choosing how to interpret the information (i.e. some flag which tells the installer to skip project.dependencies when specifying something in project.optional-dependencies). It doesn’t change how you validate those keys in pyproject.toml, it just lets tools choose how to use the information. That then puts us back to being after a way to control what’s visible as an extra when a build back-end interprets the contents of [project] from a wheel-building perspective (i.e. how does the user communicate to the build back-end what is an extra and what’s just a group of dependencies for some other tool to work with?).
Is this basically just back to allowing for more uses of [project] and broadening it outside of wheels? Yes, but at least in my head this is a different angle to view the same outcome.
Err… yeah, when you put it like that it feels simple. I always forget that project.optional-dependencies is just another table. This PEP could definitely be rewritten as an expansion of [project.optional-dependencies] such that
the package itself (if there is one) is not an implicit dependency (or it is but there’s a way to disable that)
there’s a way to refer to other keys within the table, i.e. the [extra] syntax that @sirosen proposed
there’s a way to not bundle an extra into the distribution
And tools could/should be encouraged to allow only-deps or only-extras installs.
The change for the project table itself (making name and version optional) is a separate PEP, I think. I don’t think it’s necessary for this one to work.
It seems to me like I already proposed this, and it was rejected. Or rather, this mindset was behind multiple aspects of what I’ve been suggesting, all of which got rejected in part due to that. People will complain that it’s “another way to do things” if a given new list can go in either of two places. Plus you need to define semantics if the [requirements.packages] and [project.optional-dependencies] definitions for a “requirements group” conflict. Plus you get people wondering why one table supports an extended syntax and the other doesn’t. Plus the wheel-builders end up leaving data behind that should have been recognized as an extra. I understand and appreciate that the idea is to be able to put everything in one section, but it ends up raising more questions about having two sections.
And then there’s the signal raised by PEP 725, that they actively want non-Python dependencies to be listed in a totally separate way.
It seems like you already came to this conclusion from the exchange with @brettcannon but I want to lay it out a bit more explicitly, with an actual design.
To get what you want, I think the idea has to be turned on its head. Instead of making a new table at all, expand the syntax for [project.optional-dependencies] - so that entries can either be an ordinary array as now, or an object that unlocks the new features. And then instead of expanding the PEP 508 syntax, cover the additional options with other keys.
For example, piggybacking off yours:
[project.optional-dependencies]
test = { packages = ["pytest", "coverage"], extra = false }
docs = { packages = ["sphinx", "sphinx-rtd-theme"], current = false, extra = false }
types = ["typeshed"]
typing = { packages = ["mypy", "types-requests"], groups = ["types"], extra = false }
typing-test = { packages = ["useful-types"], groups = ["typing", "test"], extra = false }
where the defaults are { packages = [], current = true, extra = true, groups = [] }, and a plain list simply provides a packages value and leaves the other defaults (the current behaviour).
And, of course, TOML allows writing such objects as separate tables, like:
[project.optional-dependencies.docs]
packages = ["sphinx", "sphinx-rtd-theme"]
current = false
extra = false
(which looks kinda similar to the original [[requirements.groups]] idea, actually.)
Anyway, an approach like this solves the problem of redundancy between [project.optional-dependencies] and a separate requirements-group table. But it means tools will be confronted with tons of data they weren’t prepared to consider valid - because [project.optional-dependencies] wasn’t specified with the kind of forward-compatibility mechanism that this PEP has. So if I was convinced up-thread then surely it applies here
It would also mean the opposite of these semantics, because the existing table already makes that on-by-default assumption.
One suggestion, if we want to standardize Dependency Group, we need to support local path dependencies because they are quite common in project development.
Both Poetry and PDM have solutions for local path dependencies. On this basis, I also hope to support specifying an editable build in the dependency specifier.
It may be similar to:
[requirements.packages]
test = [{path = "./packages/mypackage", editable = true}]
I’m not sure if this goes beyond the scope of this PEP, as I only see spec introduced as a key in the dependency specifier table. But it would not be much beneficial without supporting these.
I don’t think it’s productive to worry about who suggested what first, and certainly it’s not useful to tie this PEP to anything that came previously. This stuff comes down to the nitty-gritty details of the specification.
!), we’ll need to address 