FWIW, I requested the moderators to split out the discussion into a dedicated pre-publish topic related to this PEP.
4 Likes
I tried making a reproducer for how moving functionality to a new default extra is a breaking change, but couldn’t get the branch to work (GitHub - konstin/ndeiabc: see branches for versions, Can't unselect default extras · Issue #2 · wheel-next/pep_771 · GitHub). Still the repository should contain an example where a new version (0.1.0 → 0.1.1) moves the numpy dependency to a new default feature and thereby breaks downstream code. (I’m bringing this up because it’s a pain point in the cargo ecosystem, which has default extras with a similar design, and also to contrast the pros and cons of the “foo[…] deselects all default extras” and “foo[-bar] deselects bar only” options better).
A PEP tries to solve a problem in generic way that applies to all the python (packaging) community, so we need to know instances of the problem, to check if that solves each problems (or which ones we won’t solve with the PEP - we don’t need to solve everything, but we should document what made the cut). Ideally, we’d have a collection of example from all major groups of Python users. This is less of a concern for this PEP (at least for me) because (to me) it’s clear we want and need this, but I would prefer to enumerate some use cases. We don’t need to be exhaustive in any way (and I don’t think any single person would know), but I bet we can get more than enough examples from the community.
Maybe a more concrete thing to include in the PEP: What should package authors to write in their docs when adopting default extras as long as support is non-universal? In my experience, trying to write user level docs is a good test whether a feature works for users.
I think we absolutely should record installed extras (like direct_url.json, it would be such an easy change to implement and upgrade extras so much ), but I also think that pip install commands should not need to know about previously selected extras, i.e. for pip install foo[bar]; pip install foo it would be correct to install bar and the default extra.
2 Likes
@konstin - that all sounds reasonable, thank you for the feedback!
Just as a heads-up for everyone - the plan (suggested by @pradyunsg) currently to ease the burden on the PEP editors is to merge/publish the PEP as-is shortly. I’ll then open a follow-up PR with changes that attempt to address all the points that have been discussed in this pre-publish thread, and allow that to go through a review with the PEP editors (it will make it easier to review just the changes). Once that second PR is merged, we’ll open the new discuss.python.org discussion thread and will link to it in the PEP. Once the new thread is set up I will summarize the changes that were made.
TL;DR: The new discuss thread won’t be opened immediately after the PEP is published but once a follow-up PR with changes is merged.
2 Likes
Great job on the PEP! My apologies for being late to the party here.
Overall I am extremely against this proposal because I think we should introduce a new field which will deprecate extras, hopefully called features, that allows us to ignore backward compatibility and focus on what is best for users. Has this been considered? If not I think this should be in the rejected ideas.
edit: also I’d be willing to write a PEP for that if no one else wants to
1 Like
How exactly do you imagine things working?
Namely:
- What are the semantics of features?
- What’s the rationale for deprecating a field that’s existed for decades, and is in heavy use?
Please don’t write a competing PEP here. 
1 Like
I think there’s a rationale for having a better specified and cleaner form of extras. It may be possible to get to that point from where we are now without a rewrite, although it may be challenging[1]. I’d rather see incremental changes than a “big bang” replacement, but I do think we need a vision of what the end goal would be.
Agreed 100%
“Replace extras” would be a multi-year project, with no guarantee of success, not just a one-off PEP, and it would not be in anyone’s interests to stop working on improvements to extras in the meantime.
both in terms of UI changes and implementation details ↩︎
2 Likes
I just found time to read the PEP. Good work so far! I think it’s clear and presents its case in a straightforward way.
I admit I’m not super enthusiastic about this one because I’ve never had the need myself, but if others are looking for it, I’m glad it’s being addressed.
I might use this feature, but only if it’s very easy to instruct users how to do a minimal install with no extras.
My one point of criticism is that I do not find the rationale for rejecting package[] as “no extras” syntax to be compelling at all. After reading that section I’m still convinced that is the right way to spell this. I’ll go point by point, and I’d like the PEP to either better address this or, if I can convince you, to adopt that way of supporting a “no extras” install.
Another way to specify not to install any extras, including default extras, would be to use
package[]. However, this would break the current assumption in packaging tools thatpackage[]is equivalent topackage,
Existing assumptions are already broken by the introduction of default extras because tools can currently rightly assume that such a thing does not exist. I don’t see how this particular change in semantics is significant. Do we have evidence of tools which normalize and record package names with empty brackets?
It may be that there is current normalization code which converts foo[] to foo in some projects, which will need to be updated. I think that’s a very reasonable part of supporting a new feature.
and may also result in developers overusing
[]by default even when it is not needed.
Perhaps more of a philosophical disagreement, but I don’t think this kind of rationale fits Python. We expect users to make responsible choices with the tools they are given.
The downside risk here is that some people will make bad assumptions and use this syntactic form carelessly. But the upside would be having a clean and unambiguous way of writing “no extras”. To me, that trade-off weighs very heavily in favor of supporting the syntax.
As highlighted in How to teach this, there may also be cases where package maintainers do not actually want to support an installation without any extras, for example in cases where at least one backend or frontend must be installed.
Here I just disagree with this rationale.
If a package provides a pluggable system for multiple possible backends or adapters, I may want to write my own. If I write my own, why do I want any of the built-in ones installed?
Sure, a package may try to obscure its pluggability interfaces, but that’s a design mistake. And, thankfully, Python lets me route around that mistake by patching the underlying library if I absolutely have to.
Restricting the user’s choices strikes me as a bad thing which needs very strong justification.
3 Likes
I’m very pleased to see this proposal, it is very well-organized and clear to me, I have no objections. And I can also clearly see how the tools will support it.
2 Likes
@sirosen thanks for the feedback!
Disallowing [] as a way to explicitly disable any extras came about because some people had philosophical objections to it early in the PEP development, but I agree we should re-evaluate this. If it was technically feasible to do this, it might allow thee different problems to be solved.
First, it would avoid different packages using different names for a ‘no default’ extras (e.g. minimal, no-default, no-defaults, etc) and reduce the burden for people who don’t want to have to scan through pyproject files or docs to figure out what the minimal extras is (this was an issue raised by @bwoodsend). Projects could still define an ‘explicit no default extras’ if they wanted but it wouldn’t be needed.
Second, it would allow people who want to use existing packages as-is and future versions of those packages with no default extras to use package[] because that syntax works right now, so it would provide a consistent way over time to get a minimal install.
Third, this would provide a way to solve the pip freeze issue which has been discussed above by @pf_moore and others. In particular, when doing:
pip freeze > requirements.txt
pip could in principle check the DistInfoDistribution to check whether a package has default extras, and if so, it could automatically add a [] after the package name to explicitly disable any default extras, for instance:
astropy[]==6.0.1
numpy
scipy
(of course you could simply also blindly add [] after all packages, but that doesn’t seem very elegant or necessary).
Then when restoring with pip install -r requirements, this would avoid installing default extras in the case the package had been installed with no default extras originally. And if the package had been installed with default extras, those would be in the package list anyway (pip freeze already does not add e.g. [recommended] after a package even if the package was installed that way, so it wouldn’t be a completely new behavior).
There might be obvious reasons why this is not a good idea, but I am just thinking out loud and welcome opinions on the above!
1 Like
Unfortunately as far as I understand it this would be impossible to achieve without a very major and significant re-engineering of numerous core mechanisms inside PIP. I don’t think this solution is “technically possible” without opening a door that probably should stay closed.
Even if like “the API of using []” I just don’t think it’s implementable without creating massive changes (and consequently risks).
I don’t believe that changing the output of pip freeze is desirable.
pip freeze provides data about an environment, namely what packages are installed at what versions. We know it’s already a problem in some cases that we don’t record what extras were selected at install time and this is just another case of that.
Paul gave the example above of freezing, modifying the output to remove all pins, and feeding that to pip install -r. I’ve never seen that workflow but trust him that it’s a real one. But if you do that, you can add the empty brackets to all package names. If you had specific install options (e.g. index URL) which must be used, pip freeze isn’t responsible for those, and I don’t personally see this as being much different.
At it’s core, I consider pip freeze to be providing information about the installed versions of packages in a format which happens to match the installation format. But it isn’t the same as installation data – take pkg_resources appearing in the output for example.
IMO it’s fine (if unfortunate) if that’s the reason not to include a specific idea. But in that case that should be the PEP’s given rationale.
That said, I don’t recall seeing this discussed in this thread or the pip maintainers voicing concerns about feasibility. So I’m a little concerned that this is a premature assessment of the difficulty. pip can be updated significantly if it’s what’s necessary in order for it to adapt to the next decade of usage.
It seems to me that the change is relatively modest – parsed requirements need to track the difference between package and package[], and any new code which selects default extras needs to be made aware of this fact. But I haven’t tried to implement it, so I trust that I’m missing some key details.
2 Likes
I’d also like to make sure we don’t discard this idea too quickly on technical grounds unless we are sure it is impossible (as opposed to hard). At this stage I would be more interested in knowing if people consider it to be a desirable approach if it were possible.
1 Like
Agreed. I haven’t looked at the coding implications here, so I can’t comment (yet). I believe @jonathandekhtiar wrote the POC implementation for pip, so I assume he’s not uninformed on the matter. But I’d rather find the right solution, and deal with the implementation complexity, than accept a suboptimal approach that we have to live with forever, just because i’s easy to code.
I will agree, though, that I don’t think adding [] is a good solution to the pip freeze issue. It’s a workaround[1]. But the point I was making wasn’t just about fixing the pip freeze output, it was about the fundamental idea, that we’re now introducing a means for extras to remove dependencies as well as adding them, and the implications of that are non-trivial.
There’s an important point here. We provide mechanisms for people to use, we don’t dictate how people use them. The fact that people don’t use --no-deps when installing from pip freeze output isn’t wrong, it’s just one way of using the mechanism. It works right now, because extras add dependencies without removing them. The fact that this proposal introduces a way to remove dependencies by specifying an extra breaks that usage. Maybe the breakage is justified, and maybe we can offer help to users who encounter that breakage, but we can’t claim the proposal is backward compatible and ignore the problem.
Similarly, “freeze, remove pins, pip install” is a use of the freeze mechanism to work around the fact that there’s no pip upgrade-all command. It’s a valid use of the mechanism, and it’s one I’ve suggested for people who want to upgrade everything. I don’t know how commonly it’s used. But again, this proposal will break that recipe. Maybe that’s OK, maybe it’s not. But it’s not OK to suggest that people shouldn’t be doing upgrades that way.
The situation with workflow tools like poetry or hatch is different - those tools provide a workflow and you’re supposed to use the tool to implement that workflow. There is an “approved” way of using the commands the tool provides in that case. But pip (and underlying it, the standards we define) isn’t a workflow tool - we don’t say how to use it, just what it does.
and even if pip doesn’t include it, users can add it manually ↩︎
3 Likes
This all makes sense, thanks!
Just as a heads-up for everyone, I am currently working on updating the PEP to reflect ideas and concerns that have been brought up in this thread – once that version is published I will start the main ‘published PEP’ discussion thread (in the mean time I will likely go quiet here).
2 Likes
Nice work, thanks! I’ve seen various places that default extras would be useful over the years, and it would be great to get it in in some form (even if this does get replaced by a more powerful “features” system eventually). On example would be from tools that try to support ultra-minimal dependencies, but most users will want more dependencies. build, for example, doesn’t have a hard dependency on virtualenv, but using virtualenv reduces chances of users hitting bugs. It similarly depends on colorama, but works without it installed. This is important for boot-strapping; today, you just have to know that some required dependencies aren’t required.
The way this handles removing extras, with authors expected to inject them into all extras, does break down the notion of a “no-dependencies” extra, though. For example, let’s pretend this is the new configuration of build:
dependencies = [
"packaging >= 19.1",
"pyproject_hooks",
# not actually a runtime dependency, only supplied as there is not "recommended dependency" support
'tomli >= 1.1.0; python_version < "3.11"',
]
default-optional-dependencies = ["recommended", "virtualenv"]
[project.optional-dependencies]
minimal = []
recommended = [
'colorama; os_name == "nt"',
'importlib-metadata >= 4.6; python_full_version < "3.10.2"', # Not required, but fixes a stdlib bug
]
uv = [
"uv >= 0.1.18",
"build[recommended]",
]
virtualenv = [
"virtualenv >= 20.0.35",
]
Let’s say you wanted to turn off dependencies (to disable colorama, for example), but also wanted to add the uv dependency. So I would have thought that would look like this: build[minimal,uv]. But the uv extra has to add colorama, since build[uv] would be expected to include colorama.
Not a blocker, but a bit of a downside. I like that build[uv] doesn’t include virtualenv but does include the others. I guess you could add a uv-minimal, but users would then have to specify it instead. Still better than what we have today by quite a bit.
Also, as a package author wanting a way to install a minimal set of dependencies that are required today, since this will remove the recommended dependencies for everyone not using a package installer that supports this proposal, I’d not be able to use it for a long time, otherwise things like “colorama” and “importlib-metadata” would disappear. I would be able to add virtualenv with it, though, since it’s not a hard requirement today.
It’s not fully related, but since this PEP is touching this area, what about explicitly allowing a backend to expand self referential extras when generating the metadata?
If you’d like an draft implementation in pyproject-metadata, I could do that.
4 Likes
Thank you for the feedback!
Yes it’s certainly true that including minimal in the extras list is not an override that guarantees a minimal install. I think in cases like this the easiest solution might be to not include build[recommended] in the uv extra but instead tell users to install build[recommended, uv] if they want both those extras - i.e. if users are already reading the docs to find out about [uv], then in those docs you can mention [recommended, uv] instead so that they also have the option of installing just [uv].
In short, maybe the best approach is to assume that either users read the docs or they don’t - if they don’t you just want installing the package with no extras to do the right thing, and if they read the docs to find out about extras then you can tell them about things like [recommended, uv] (or [colorama, uv] if you defined it that way).
Yes it’s definitely the case that for established packages, making any changes to required dependencies is going to be a breaking change until most users would be using PEP 771-compliant tools - and I’m adding text about this to the PEP.
Would the idea here be to essentially simplify the metadata to make life easier for packaging tools?
1 Like
Yes, that’s the idea. It used to be more important when more tools didn’t support self references, not sure it really matters much anymore.
1 Like
Here’s a scenario that I just thought of while discussion the pip implementation.
Suppose package A has extras X and Y, and X is defined as the default extra. X says to install package B, which has a dependency on A[Y]. Now if I install A[Y], I’m supposed to not install the default extra for A, as I understand it, because an actual extra has been requested. So now we have a contradiction - requesting an install of A should include the default extra, but that default extra will require that the default extra is not installed, because an explicit extra, Y, is requested further down the dependency tree. Conversely, if we don’t include the default extra, then A will be installed without any extras, then B isn’t installed, so why did we not install the default extra?
This seems very problematic - it’s not just a backtracking problem to complete the resolution, the dependency graph in effect has feedback in it (and I know of no resolution algorithm that can handle feedback like this).
Can the PEP be clarified to give an explicit answer for this situation? Otherwise we’ll have tools needing to guess at what the intended behaviour is, and inevitably not guessing the same answer 
4 Likes
I’ve talked to the cargo maintainers since they have been supporting default features for over 10 years, with significant efforts to fix some of the initial shortcomings (cargo features are the equivalent to python extras).
One cargo maintainer pointed me to a series of (postponed) RFCs, all targeted at trying to fix default features:
As well as cargo#3126, and the wish for mutually-exclusive features.
Another cargo maintainer, Ed Page, shared these notes:
If I’m understanding correctly, this is taking the approach of “if you explicitly specify an Extra (feature), you get no defaults”. I know in our discussions of features, someone has been advocating for this. Personally, I would find it confusing and frustrating if you just want to add one extra.
imo its important to look at use cases
- Common users who just want to add/remove individual features (e.g. a
clapuser would not want to have to name all features to remove one or to add one)- Libraries that integrate with others (e.g.
clap_completeshould only specify the features it needs)- Minimalists who want to hand pick everything
- Maximalists who just want everything (rare but they do exist, see Add `--all-features` to `cargo add` · Issue #13936 · rust-lang/cargo · GitHub)
A sketch of a potential ideal solution. I know the existing Cargo features and Python extras constrains some of this
- A common, reserved name for the default set of features that is implicitly added if not already there
- While I said I’m not a fan of setting features removing features, in the file syntax, I wonder about having a
featuresfield default tofeatures = ["default"]and so to add features you have to dofeatures = ["default", 'more"]or to remove defaults you just skip it- A “base” property on features (
defaultwould be abase)
- There must always be at least one
base- Ability to remove features from a
base- This allows evolving features / no-default-features
- Feature metadata
- Documentation
- Deprecation
- Private
- Unstable
Of course, the rust ecosystem has different requirements from the python ecosystem in many regard, but i hope these notes give us a better idea of the problem space and help us transfer knowledge from a production user.
10 Likes