PEP 771: Default Extras for Python Software Packages (Round 3)

Packaging Standards
1

We are happy to share an updated version of PEP 771!

There was a lot of good feedback and many valid points raised in the previous round of discussion, and we have updated the PEP to reflect as much of the discussion as possible and address the concerns that were voiced. Discussions around the idea of default extras started a little over six years ago (original thread), and thanks to everyone’s feedback we feel that the PEP has now converged on a version that is close to final. This is not to say that we expect to have unanimous agreement on it, but we hope the latest version has as broad an appeal as possible.

We plan for this to be the final round of discussion before we submit the PEP for a decision. During this discussion, we can of course still make small tweaks to the PEP if needed, but we would like to avoid any more significant changes or additions. Therefore, it would be valuable for anyone interested to read the whole PEP at this point, particularly if you had concerns previously, since these may now be addressed.

Whether or not you took part in previous rounds, it would be great if you could state where you stand on this latest version, since this will be important when it comes to making a decision. If you still have concerns, please let us know how you think these could be addressed, or whether your objection is more fundamental.

Changes in this version

Below is a brief summary of the main changes. It is not exhaustive, so as mentioned above, we recommend reading the whole PEP to get the full picture:

  • Renamed pyproject.toml key: the default-optional-dependency-keys key is now default-optional-dependencies, as this seemed to be the best compromise based on previous discussion. The specification also explicitly states that a default extra must match an existing extra (and backends must enforce this by erroring if not), and we have now included clarifications about how this key should interact with dynamic.
  • Clarified the package[] semantics: package[] means “I don’t need the defaults”, not “I forbid them”. Default extras are installed if the package appears bare anywhere in the dependency tree, and running pip install package after package[] installs the default extras. Nothing is ever uninstalled.
  • New section on the change in meaning of a bare package name: it acknowledges the change directly and explains why we think it is acceptable.
  • New guidance for downstream packages: how to depend on a package that has default extras, including in PEP 723 scripts and PEP 735 dependency groups, as well as when to use package[] and why not to add it to every dependency defensively (section).
  • More guidance for package authors: what belongs in a default extra, why default extras aren’t a substitute for splitting genuinely independent packages, and a recommendation to have a CI job that tests the minimal installation.
  • Prior art and repackaging: comparisons with Recommends in Debian/Fedora and with Cargo’s default features. The section for repackaging maintainers (conda, Linux distributions) has been expanded.
  • New rejected alternatives: a standardized opt-in extra name such as recommended, a metadata-only Recommended-Extra field that leaves behavior to tools, and a “requires one of several extras” mechanism.
  • Expanded security implications: the PEP now notes that tools which inspect dependency metadata, such as security scanners, dependency update services, and package indexes, will need to take Default-Extra into account to accurately report what a bare install includes. It also explains that default extras introduce no new risks beyond those that already exist with required dependencies.

There have also been a number of clarifications, wording improvements, and other smaller changes.

14 Likes
PEP 771: Default Extras for Python Software Packages (Round 2)
2

I was expecting the “standardised opt-in extra name” to be one that tools are told to assume was specified if no extras were specified (the actual rejected alternative is one that users are expected to type). Basically, simplify down the mechanism to “if the package has an extra named spam, and is installed without any extra specification, install [spam]”[1]

Was this discussed and rejected?

Also, I find that all the reasons for rejecting the use of a separate package (foo and foo-core or some equivalent) apply equally to this proposal, apart from “pip install <source directory> will install all the extras with PEP 771 vs. having foo stored outside the root of the repo”.[2] Is that particular reason really strong enough to reject something that could simply be documented as a pattern rather than requiring changes to everyone’s tools?

  1. Where spam is likely spelled default or all or something, but that’s not the angle I want to bikeshed. ↩︎

  2. Which I think is merely debatable as to whether it’s the right action or not. I think it’s not, and would prefer --no-deps-like behaviour in this case, but I guess my assumption is that if I’m building one package from source, I’m probably building the others from source as well, or at least doing something in a dev environment that’s already set up. ↩︎

2 Likes
3

This is a really great write-up (especially the arguments in Rejected Ideas) and I wholeheartedly agree with the proposal and hope it gets through. It will be of great use to a variety of the packages I maintain which are in a similar situation to astropy as the PEP mentions.

One question I have is about the situation when on extra needs to be an “addition” to the default extra(s). Why was the suggested spelling explicit on relisting the default extras if the bare package name is equivalent (and doesn’t require keeping them in sync)?

[project]
default-optional-dependencies = [
    "recommended"
]

[project.optional-dependencies]
recommended = [
    "package1",
    "package2"
]
additional = [
    "package[recommended]",
    "package4"
]

Rather than

[project]
default-optional-dependencies = [
    "recommended"
]

[project.optional-dependencies]
recommended = [
    "package1",
    "package2"
]
additional = [
    "package",
    "package4"
]
2 Likes
4

Thank you for the comments! Yes the above was also discussed (and rejected) and is mentioned separately in the PEP. Just to clarify:

  • In Adding a special entry in extras_require (which was there in previous revisions of the PEP), we discuss the alternative of using a standard name for extras that tools can assume they should install by default, and we say “In addition, no other string (such as ‘default’) can be used as a special string since all strings that would be a backward-compatible valid extras name may already be used in existing packages.”. This was rejected as a solution in one of the first rounds of discussion.
  • In Standardizing an opt-in extra name (new in this revision) we specifically talk about a suggestion that was made in the last round of discussion to consider having a standard extra name to mean the typical one that users will likely want, but it would remain opt-in, so essentially a convention people would learn to rely on.

Does this make sense? Let me know if you have any thoughts on how we could make the distinction clearer.

2 Likes
5

I’d like to better understand your statement about all the reasons for rejecting the meta-package approach applying to PEP 771 as well - here is a summary of the six main reasons as described in the PEP:

  • Mismatch between package and module name: PEP 771 packages only have one package/module, so there is no mismatch/confusion (compared to meta-packages where package-core might install the package module).
  • Multiple repositories or monorepos: this is not an issue for PEP 771 packages, as packages can still exist as a single package in a single repo.
  • Depending on the minimal package: one of the nice things with PEP 771 is that downstream packages that want minimal dependencies could depend on package[]>version even if version pre-dates PEP 771. For meta-packages, this is only an issue for previously single packages that split into meta-packages, not for packages/meta-packages that were split from the start
  • Uninstallation: with PEP 771 packages, uninstalling the main package works and there is no confusion. In contrast, with meta-packages, where if a user does pip install package then pip uninstall package, they will still end up with the package-core package and associated module.
  • Package distributions: with PEP 771, there is only one package to release (versus two for meta-packages).
  • Synchronizing metadata: with PEP 771, there is no synchronization of metadata needed.

So as far as I can see, these six reasons don’t really apply to PEP 771 packages - however I’d be interested in discussing this further if you think I might be missing something?

1 Like
6

Thanks for the comment! You are right that in principle you could just list the bare package name. However, there are a few reasons why it might be better to be explicit:

  • It is more readable and immediately understandable
  • You might not want to pull in all default extras
  • package[additional] will work well even with old versions of installers that don’t recognize default extras

On the other hand, the benefit of using the bare package notation as you are suggesting is that it will pull in all default dependencies automatically without having to worry about keeping things in sync with the default extras names. I’d be happy to consider adding a sentence or two to this effect in the PEP if you think it would be helpful.

1 Like
7

I don’t see any particular issue here. Installing the package module to get package works in both cases, and in both cases gets the default extras. You have to know in one case that you should’ve written package[] to avoid the extras, or package-core to avoid the extras, but the “obvious” behaviour is the same.

Tools could easily be updated to support this, and as far as I’m aware the few that refuse do are simply on ideological grounds. There’s no technical reason to disallow having a metapackage in the same repository.

Tools also need updating for the new metadata, so “avoids fixing the tools” is not a reason.

True, but also a fairly unusual edge case. It also only works for packages that currently don’t include extra dependencies by default, and doesn’t help those that want to convert existing “required” dependencies into “optional but on by default”. Those ones benefit more from the metapackage approach. So it’s break even here, IMHO.

You end up with the extras either way. Uninstalling with pip is not really worth it unless you’ve mapped out the transitive dependencies manually - tools that do it automatically will handle either approach the same.

Okay, there are two that don’t apply to both :wink: But it’s very easy to build and publish two packages at once, and the metapackage doesn’t need to include platform-specific wheels (unless you want to), so chances are this is “N vs N+1” rather than “1 vs 2”. (Of course I’m assuming that you aren’t using a build tool that’s hostile to having two packages in the same repository.)

Strictly speaking, there isn’t for the metapackage approach either. You only need to match the versions if you want to; otherwise, pip install package remains a convenience for installing the latest version of package-core and required dependencies. Anyone who wants to pin their packages is going to want to pin dependencies anyway, so they’ll do a full lock, which will include package-core.

But again, when you’re using a tool that lets you have two packages in the same build, synchronising metadata is easy. We don’t have to build a more complex model for everyone just for the sake of the people using tools that over-simplify things to the point of making them inflexible.

I’m not trying to argue, necessarily. There’s been more than enough debate on this that I’ve opted out of, and nobody is going to really mind if I stick to the -core model rather than adopting a default extra, since they are both entirely compatible with each other.

But you asked for expanded reasoning, so there it is.

2 Likes
8

I remember there was some analysis done of existing extras names - did that show that there are packages including default as an extra without intending it to be the default?

9

I’m not sure who might have done this originally[1] but I just had a go at looking at this - I used the following BigQuery to find all packages that have a dependency in the metadata that contains extra == 'default':

  SELECT
    name,
    ARRAY_AGG(DISTINCT version ORDER BY version DESC LIMIT 5) AS sample_versions,
    MIN(upload_time) AS first_seen,
    MAX(upload_time) AS last_seen,
    COUNT(DISTINCT version) AS n_versions
  FROM `bigquery-public-data.pypi.distribution_metadata`,
    UNNEST(requires_dist) AS req
  WHERE REGEXP_CONTAINS(req, r'(?i)extra\s*==\s*["\']default["\']')
  GROUP BY name
  ORDER BY last_seen DESC

This returns 126 projects which have at some point had this as an extra – of these, 80 still have it in their latest release, and I think around 60 are independent projects (the remaining 20 are forks or copies of these).

Some of these packages prominently feature [default] in the installation docs, for example networkx and in cases like this, it seems to be what most users would want. It looks like about a third of the projects fall into this category.

For some such as asyncdb, the [default] option is mentioned in passing but not prominently shown in the main installation instructions. For some like pytreeclass the default extra is defined but never mentioned in the docs. There are also several packages where it seems that [default] is not what should be installed by default: for example TopoVelo uses it to list exact pinned versions of the dependencies.

So could we use [default] as the ‘special’ name for default extras instead of introducing a new metadata field? Technically it seems that many (but not all) uses of [default] align with this, but in practice we don’t actually know what the various package maintainers would want - if we did adopt this, we’d be opting them all in, and this would also apply retroactively to any release that used [default] in the past, with no way for any of those releases to opt out. For a small number of packages it would definitely not be what is wanted. So if any authors would not be happy with [default] actually being installed by default, they could not change this for past releases.

I personally think it would be preferable to make this new feature be truly opt-in for maintainers via a new metadata field, and let packages migrate to this for future releases once they have had a chance to digest the implications of the truly default extras. After all, as we have emphasized in the PEP, package authors should consider carefully if it is the right fit for their package, so it would not seem right to just opt in a number of packages. The benefit of the new metadata field is that it also allows multiple extras to be listed, and doesn’t require existing packages to rename whatever extra(s) they want to have included by default (default, recommended, all, etc.).

I won’t repeat all the analysis above for [recommended], but the numbers are a little larger: 137 packages have [recommended] as an extra in their latest release. I haven’t checked the intent for those though.

  1. Maybe you were thinking of Paul’s analysis for PEP 685 (PEP 685: Comparison of extra names for optional distribution dependencies - #17 by pf_moore), which looked at normalization clashes in extra names rather than their intent? ↩︎

2 Likes
10

This is true of all dependencies. If we wanted pip to be able to filter the packages of an environment into explicit install, (indirect) dependency of an explicit install and then remove the orphans[1] then ironically creating another category of dependency but we don’t know if it was actually needed takes us further away from that.

Regardless of how default extras are used, package authors should aim to ensure that their package’s documentation makes it clear how extras are to be used. ‘Best practices’ documentation should mention:

Don’t suppose we could have Actually explain what the dependencies are for in that list? :slightly_smiling_face: astropy[recommended] makes me think of that monty python sketch where a bored aeroplane pilot panics the passengers by randomly telling them there is no reason to panic. “There is NOT an important decision to made here about your dependences!”. But there’s also nothing to help make that decision. What does recommended mean? When can I not use those dependencies? What functionality? Which APIs? If I ignore the recommendation and it turns out I needed them, can I trust it to fail with a self-explanatory ModuleNotFoundError or should I be on the lookout for something more subtle? Sigh, go back to reading the source code for answers…

In cases where the repackaging is done by hand, such as for a number of conda-forge recipes, the introduction of default extras should not have a large impact since manual decisions already have to be made as to which dependencies to include

I really dispute this claim. Every one of these decisions is a square peg in a round hole. A choice between unfixable bloat or a surprise missing dependency. “Do what the original authors were unable to do and pick one.” Every answer is a wrong answer. I hate doing it.

I also disagree with this narrative of average user versus advanced user. Between being incompatible with many of the forms of distributing Python applications, the dead weight code that can’t be used without the extra dependencies, the inevitable increase of packaging level/import time regressions and just the general pain of having to continuously audit where the []s are needed, this isn’t really a split between average and advanced users. It’s between people who don’t care what they install versus people who need to go find a replacement dependency.

  1. which I’d support. Alpine’s APK does it and it’s great ↩︎

3 Likes
11

This limitation – the inability to deselect individual defaults while keeping others – is shared by Cargo’s default features in the Rust ecosystem, where the only option is all-or-nothing via default-features = false. If there is a desire to implement a better solution in the future, we believe this PEP should not preclude this. For example, one could imagine in the future adding the ability for an extra to specify which default extras it disables, and if this is not specified then explicitly specified extras would disable all default extras (consistent with the present PEP).

The PEP does not yet capture the big problem that cargo is struggling with: If there is an option to deselect all extras, moving functionality is a breaking change. In the rust ecosystem, projects often can’t move functionality to features due to this and have to wait for the next breaking release to make these changes. This problem is not forward-fixable. This isn’t captured in the PEP yet, and it doesn’t include references to the cargo discussion and RFCs dealing with the problem.

3 Likes
12

Thanks for bringing this up! I agree that this should be addressed more directly in the PEP. We did mention the general issues in a couple of places (e.g. If upstream later moves functionality that the downstream package uses from its required dependencies into a default extra, a downstream package depending on upstream[] would break) but I agree it would be good to be more explicit about the parallels with Rust/Cargo and provide links to the RFCs. I’ve spent a little time trying to write a new subsection that could fit into the Package authors section. Here is the suggested subsection:

Moving existing functionality into default extras

Making previously required functionality optional is inherently a breaking change for anyone who relied on it without asking for it explicitly, with or without this PEP. Without default extras, moving a dependency into an extra breaks all users who relied on it; with default extras, it only breaks users who have opted out of the defaults, either with package[] or by requesting other extras that do not include the default extra. This PEP therefore reduces the impact of such changes, but it also makes them easier to overlook: since the functionality remains installed by default, such a move can easily appear safe, even though it is still a breaking change for users who have opted out.

The Rust/Cargo ecosystem has encountered the same problem with default features: crates that depend on others with default-features = false break when functionality is moved into a new default feature. For example, warp had to revert such changes before a release (warp#913). Making such moves non-breaking remains an open problem in Cargo (cargo#3126), and several proposals to address it (RFC 3146, RFC 3283 and RFC 3347) have not been accepted.

For Python packages using default extras, the impact can be mitigated in several ways:

  • Where possible, functionality should only be moved from required dependencies into default extras when a package first adopts default extras. Until a package defines default extras, there is no reason for anyone to depend on package[], so there would be no minimal installations to break. Users who request other extras will still be affected, unless those extras are made to include the default extra (see Supporting extras that should not remove default extras).
  • Downstream packages should only use [] sparingly and deliberately (see Impact on downstream packages), so that opting out remains the exception rather than the norm, unlike in Cargo, where default-features = false is common practice for libraries.
  • Where such a move is necessary, it should be treated and communicated as a breaking change, in the same way as moving a required dependency to a non-default extra.

Let me know what you think about the proposed text above! In particular, let me know if I’ve missed any relevant RFCs or other discussions you think should be linked to.

2 Likes
13

I think it’s a no-brainer to use [] on all dependencies, contrary to the advice of the PEP. If one of my packages’ dependencies changes a required dependency to recommended and it breaks stuff then the error is obvious and downstream users can install the missing package themselves as a quick workaround, including retrospectively with old versions of my package. There’s no pip install --exclude option though so an unneeded new recommendation is stuck there until I’ve done a release with the [] added.

But I’m aware that I’m probably in the minority with that view. Something that’s more likely to bother everyone is if a package recommends a dependency that some don’t want, they [] away, then a second dependency gets moved from required to recommended, then things are broken again. The required->recommended as a means to avoid breaking changes can only be used once.

2 Likes
14

This only works if all existing named extra depend on the default extra(s), else the breakage will still occur. This feel too much like a footgun, and give unlike Rust where such errors occur at build time (and so are detectable early), with Python they’ll occur at runtime.

I still think metadata-only is a better option, and the arguments are weak:

  • The argument against interactive prompts effectively states that not including the default extras gives a broken install, in which case why are these dependencies optional?
  • The tool-dependent argument ignores all current tools do not support default-extras, so this PEP is introducing the inconsistency, whereas choosing the metadata-only option does not require changing how the dependency specifier astropy>=7.0 is interpreted, so no inconsistency is introduced. Also, as written a fork of pip (or any other installer) is allowed to always interpret astropy>=7.0 as astropy[]>=7.0, so inconsistency is baked into this PEP.
  • For those requiring that one of a set of packages, it would seem to be better to introduce a proper OR definition to the dependency specifier spec, as that then can be used by other packaging ecosystems, rather than an implicit technique that will break as soon as people start using [].
15

Thanks for writing this up! The references look good.

I don’t think that quite captures it: In the third option of default extras without the option to deactivate all of them at once, instead having to opt-out one-by-one, this is not a breaking change. An example would be the [-foo] syntax; I don’t have opinions on the syntax, but semantically this would imho solve the biggest problem the PEP currently has.

Using the opt-out sparingly isn’t effective, as soon as there are any dependents using this, it’s a breaking change which is prohibitive for major libraries. We should rather encourage libraries to opt-out as to keep the dependency tree lean. (In Rust, we semi-regularly have to ask dependencies to remove default from the dependencies to prune unnecessary transitive dependencies.)

16

I’m very excited about this.

fastapi[standard] is the current best solution but it’s not great. This PEP would be great.

I wish it was already available, as I will need this soon for SQLModel.

I read again the sections of the PEP I’m interested in, still sounds great for what I need.

1 Like
17

I’m curious. For existing projects that depend on fastapi, what would you recommend they do? If you plan on making the current standard into a default extra, presumably they need to start using fastapi[] if they want to avoid unnecessary dependencies? Does that mean that every project depending on fastapi needs a new release to change the dependency? Will you remove the standard dependency, or keep it as a duplicate of the default extras, to support users with older versions of the packaging tools that don’t support default extras?

This has been discussed before, but I have yet to see a clear conclusion being reached. Maybe if we take fastapi as a concrete example, we’ll have a better chance of finding an acceptable recommendation?

Long-term, once default extras are a well-established feature, supported by all tools, things should be relatively straightforward. But it seems to me that the transition period could be problematic, especially if we don’t have clear and explicit guidance for package authors (the PEP doesn’t currently give guidance beyond saying that package authors must “carefully consider” the situation…)

18

presumably they need to start using fastapi[] if they want to avoid unnecessary dependencies?

Yes, I think this would be the case.

Does that mean that every project depending on fastapi needs a new release to change the dependency?

Assuming those projects intend to support future versions of fastapi (hopefully they do, not pinning upper bounds), and they intend to support new dependency solvers, yes, they would need to release a new version.

In the case of FastAPI, as it’s mostly a framework for final users to use in their apps more than a tool to be wrapped by other libraries, I would expect the related friction to be minimal.

App users would/should have no upper bounds in their dependencies, but should have package locks, like uv.lock or pylock.toml, that would pin the fastapi version used.

So whenever they are ready to bump the version they use (probably automated by dependabot and similar tools), they can see the release notes and the changes, they can decide to instead of the plain version bump, to update the requirement to fastapi[].

Will you remove the standard dependency, or keep it as a duplicate of the default extras, to support users with older versions of the packaging tools that don’t support default extras?

If I understand the PEP correctly, the default extras point to the optional extras, so removing it would not work, the new default extras would point to the current standard optional extras.

My biased opinions

Affected users

I would think in the end the main affected people would be the ones building end user applications, more than people building libraries. Developers building end user applications (and their users) would be the ones that could “accidentally” end up with more dependencies included than they expected.

But those end user applications would/should have locks for their dependencies, which is now common practice, I would think. So, they wouldn’t really accidentally get new unexpected dependencies. They would have to accept or execute a lock update, including those potentially unwanted dependencies.

Advanced minimal users

Now, I would consider those users that decide to go with the minimal installation advanced users. I would think they are voluntarily opting out of what is considered the recommended, standard, default way to work with a tool.

If they are doing the work of intentionally selecting minimal dependencies, I would consider them capable enough to understand the new changes when updating locks and dependency formats. And they could adopt the new fastapi[].

Regular non-advanced users

For users that simply didn’t even realize there was a recommended set of dependencies and just plainly added fastapi, not even knowing there was another option, maybe not even knowing there’s a syntax to declare extra dependencies, I would expect them to be less worried about getting the absolute minimal installation, and would rather have the recommended set of packages, if they didn’t manage to see there were options, I would expect them to want the recommended option.

1 Like
19

There is a quite extensive section with multiple subsections with guidance for Package authors which does give a lot of guidance, with particular attention for existing packages. For example we discuss how moving a required dependency to a default extra could be a breaking change, how making existing extras into defaults is safer, best practices for using default extras, considerations for impact on downstream packages, how to document default extras, etc. Could you clarify what kind of guidance you feel is missing from the PEP currently?

20

Most of the guidance takes the form of “you need to consider this carefully”. What I’m suggesting is that rather than simply leaving the questions to individual projects to answer (particularly given that for consumers of projects that add a default extra, they might not even be making a change, instead simply being affected by an upstream change), the PEP offers some concrete suggestions on what to do.

To be clear, I don’t think that any of my projects have dependencies that are likely to add a default extra, and I don’t plan on using them in my own projects, so my interest here is mostly theoretical. But looking at the PEP on the presumption that I was faced with a situation where I needed to care, it feels like the current text would leave me with more questions than answers.

3 Likes