(pre-publish) PEP 771: Default Extras for Python Software Packages

Packaging Standards
1

[BIG UPDATE]

The PEP PR is live: PEP 771: Default Extras for Python Software Packages by astrofrog · Pull Request #4198 · python/peps · GitHub

We have been fairly at hard work with @trobitaille.

We published and opened the PR on Github, we invite you to review it and comment it.

As part of the Wheel-Next open-source initiative, there is a “near-complete” and fully testable implementation of this PEP as a “proof of concept”:

We would really appreciate you for taking a few minutes to test it, and give us your feedback.
Promise, it won’t take any longer than a few pip install ... commands :wink:

Full instructions on Github, short summary here:

# 1. Create a virtualenv 
# WARNING: Installing PEP 771 overwrites a bunch of standard python library.
# Do not install this in your main python environment.
virtualenv .venv
source .venv/bin/activate

# 2. Set the extra index to the Wheel-Next Static Wheel Server: MockHouse
pip config set --site global.extra-index-url https://wheel-next.github.io/mockhouse/pep-771/
>>> Writing to /path/to/venv/pip.conf

# 2. Install the PEP 771 Metapackage that will give you the modified libraries:
# - setuptools
# - pip
# - importlib_metadata
# - validate-pyproject
pip install pep-771

# 3. Let's verify everything is good:
pip --version
>>> pip 25.0.dev0+pep-771 from ...  # <=============== Check you can see `+pep-771`

pip freeze | grep setuptools
>>> setuptools @ git+https://github.com/wheel-next/setuptools.git@...

pip freeze | grep validate-pyproject
>>> validate-pyproject @ git+https://github.com/wheel-next/validate-pyproject.git@...

pip freeze | grep importlib_metadata
>>> importlib_metadata @ git+https://github.com/wheel-next/importlib_metadata.git@...

# -------------------- Then choose one of the followings -------------------- #

# ~~~~~~~~ pep-771-demo-a with the "default extra": flask ~~~~~~~~ #

# will install pep-771-demo-a AND the default `flask` extra
pip install pep-771-demo-a

# You should see `pep-771-demo-a` being installed along of `flask` and its dependencies (flask is the default "optional-dependency".

# ~~~~~~~~ pep-771-demo-a with the "user-specified extra": fastapi ~~~~~~~~ #

pip install pep-771-demo-a[fastapi]

# Will install `pep-771-demo-a` with `fastapi` and dependencies (but no flask in sight)

@pradyunsg @pf_moore @barry @emmatyping @aterrel

8 Likes
2

Nice work! Thanks for seeing this through.

Given that this is proposing adding a new core metadata field, I think the PEP should also require a new Metadata-Version as well?

1 Like
3

Yes, it should.

Also, there’s a rendered version of the PEP here for people like me who don’t want to read it in diff format. I don’t know how long that URL will last for - presumably once the PR gets merged the PEP will be available from the “live” location.

1 Like
4

Having reviewed the PEP, there are a few points I think need to be addressed.

The Default-Extra field can simply be described as multiple-use. There’s no need for the paragraph starting “If multiple default extras are needed…” as that simply explains how multiple-use fields work.

You should also note that Default-Extra must only contain values included in Provides-Extra. The discussion of the key in the [project] metadata table mentions this, but it needs to be made explicit in the core metadata as well.

The “How to teach this” section is more accurately an “Examples” section. In the “How to teach this” section, you should be explaining how to ensure people understand the new functionality introduced by this PEP. Things to cover there might be:

  • How do we ensure people understand that adding an extra to a specifier might result in some packages not getting installed that would have been installed without the extra?
  • How do people discover the right way to install a package “without any extras” (given that the name of the “without any extras” extra is not standardised, and may not even exist)?

It might also be worth adding a “Transition” section that reviews how projects are currently handling this situation, and how they could transition to the new feature. This section could have particular emphasis on how to handle issues around transition - for example, if a project goes from having a “minimal” install by default in version 1.0, to having a fuller-featured default in 2.0, but with a “minimal” extra, then how would theire users get a minimal install without needing to pin the version? The answer here may be “that’s not possible, but we don’t think it’s likely to be a big problem”, but if so it should be stated explicitly in case some project authors want to object :slightly_smiling_face:

I should also say, thanks for keeping the proposal simple and resisting scope creep in the discussions up to now. IMO, it’s looking pretty good.

4 Likes
5

@pf_moore @dustin quick question for both of you. Not directly linked to this PEP.

How helpful is the “pip installable demo” of this PEP is ? I really tried to provide a “smooth and end-2-end testable solution” from “building a wheel” to pip install <pkg>.

Thanks for the feedback, this will be helpful for future work (in other PEPs)

6

I haven’t used it. I’m somewhat uncomfortable with the fact that you’ve created a package that overwrites the user’s copies of pip and setuptools. I’d have preferred the POC as PRs to pip and setuptools. And honestly, if the PEP text doesn’t explain things well enough, then I’d rather see the text improved than a “canned demo” to try to clarify the proposal. After all, it’s the text (and only the text) that will be what defines the new standard.

One thing does bother me in particular - why did you need to modify importlib.metadata? Because if that was necessary, this may need to be a core Python PEP rather than just a packaging standard - which is very unusual.

2 Likes
7

It looks like the only real difference is the addition of the Default-Extra key.

1 Like
8

Well - I kinda have to no ? If I want users to be able to “test this feature” I need to modify quite a few libraries. Namely PIP & Setuptools.

Yes indeed as James points it out. I had to, I needed to add the new “Metadata” key Default-Extra. This PEP does introduce a change a new key in pyproject.toml / setup.py and in the METADATA file. Consequently, the whole “chain of dependencies” needs to be adapted.

I can also provide PRs to PIP, Setuptools, and other vendored components, for now I was aiming for simple to show & test how it works.

The other side of the story is that I wanted to see how the implementation of a “fairly simple feature” would look like and how big/small of a change it is (and well it’s more complicated than I originally anticipated).

I really agree with you here. I’ve put a very conscious effort in keeping this PEP “as simple as can be”. I truly believe this change to be “pretty simple” with very little-to no side-effect. And I don’t believe it truly needs a complete implementation to understand the core idea. Though I thought this was a nice gesture to also “do the work” and do “due diligence” in putting the work to show “how it could be implemented” and how “it could work end-2-end”.
I believe in “an example is worth a thousand words” and that’s the philosophy I tried to apply here.

If you want me to open PRs to all projects, and dependencies (mostly vendored) I can do that. You tell me. I was solely hoping to “illustrate the PEP” with an implementation. If you believe there’s a better way to do that, more than happy to oblige.

9

That’s just weird. Why do they have a (partial) list of keys at all? They don’t include License-File, for example, which is also multiple-use.

If you make PRs against those libraries, and instruct people to install from those PRs, that should be enough. No-one else has ever done things this way. I guess I don’t care that much - I’m not going to try out those instructions myself, so I’m not directly affected. I just don’t like the precedent of an installable package that modifies a different package. It feels like the sort of thing malware would do, and just the existence of it makes me uncomfortable.

And yet, as I said, License-Key isn’t in there, so it’s clearly not essential to add it. I don’t know anything about the internals of importlib.metadata so I don’t understand the implications here. Does your demo actually fail if you don’t patch importlib.metadata?

Regardless, you’ve opened up a bit of a can of worms here. If adding a new core metadata field genuinely does require a change to the stdlib, then that’s a whole new consideration that we’ve ignored until now :slightly_frowning_face:

I appreciate immensely that you’ve put the time into ensuring that the changes needed for pip and setuptools are practical. There will be other changes needed (to the other build backends, to uv, and to packaging, for example) but these make a good start. What I’m disputing is the need for a pre-packaged[1] “demo” of the new feature. And in particular, the approach of delivering the demo as an installable project that modifies other projects[2]. But :person_shrugging:, that’s just my opinion and I’m only giving it because you asked :slightly_smiling_face:

Normally, if a PEP includes a proof of concept[3] it’s in the form of one or two PRs against key projects. In this case, PRs against pip and setuptools seems perfectly sufficient. Don’t worry about covering everything - there’s plenty of time for that once the PEP is approved.

  1. excuse the pun! ↩︎

  2. I bet you’ve left the user’s environment corrupt, as the RECORD file for pip will now have the wrong hashes, for example ↩︎

  3. many don’t, you’re way ahead of the game here! ↩︎

2 Likes
10

I see your point and that’s very reasonable. Which is the reason why I added this:

# WARNING: Installing PEP 771 overwrites a bunch of standard python library.
# Do not install this in your main python environment.

It feels like the sort of thing malware would do, and just the existence of it makes me uncomfortable.

I feel you. Though it’s just an “empty package” declaring “direct dependencies” from github.
And that’s also the reason I haven’t uploaded this package to pypi.org (and even if I wanted - it would have been rejected for external dependencies).

I have “more complex and intricate POCs” for other PEPs coming, if you have 30min at some point, I’d appreciate a quick chat with you to deliver the most useful POC for these PEPs. Let’s keep that off this thread to avoid “polluting this one”, nonetheless, thanks for the feedback, very much appreciated. Ultimately I’m trying to provide what is most useful to the community, the format can be debated and is not exactly what I care the most about. The message is more important than the messenger.

11

Ah! If that’s all it is, I’m much less concerned. I assumed it was worse than that.

Happy to help - DM me and we can sort something out.

1 Like
12

I just checked, there’s nothing in the API that I can see which cares. And if I add an invalid metadata item, it handles it just fine.

So I think importlib.metadata is a red herring here (and the POC probably shouldn’t patch it).

2 Likes
13

I’m gonna try to build the demo wheels without importlib_metadata, if it works, I’ll remove this from the POC.

Will do, maybe around mid-February.

Yes. Like it couldn’t be more “trivial”

METADATA:

Metadata-Version: 2.2
Name: pep-771
Version: 1.0.0
Author-email: Thomas Robitaille <thomas.robitaille@gmail.com>, Jonathan Dekhtiar <jonathan@dekhtiar.com>
Requires-Dist: setuptools@ git+https://github.com/wheel-next/setuptools.git@pep_771
Requires-Dist: pip@ git+https://github.com/wheel-next/pip.git@pep_771
Requires-Dist: validate-pyproject@ git+https://github.com/wheel-next/validate-pyproject.git@pep_771
Requires-Dist: importlib_metadata@ git+https://github.com/wheel-next/importlib_metadata.git@pep_771

You’ll note the dependencies are pinned to a branch pep_771 on purpose, so that I don’t need to “respin pep-771” package, everytime/most-of-the-time I do an adjustment (removing importlib_metadata) would be an exception :wink:

Though I share your overall feeling about the approach and that’s the reason I was very purposefully adding a very clear warning and instruction to not pollute your main environment. And I stayed away from pypi.org (even though it would have been impossible to upload anyway because of external dependency check on both warehouse and pip side).

If you’re into that, GitHub - wheel-next/pep_771 provides a complete devcontainer to allow you to play & have fun with the feature & implementation.

Nonetheless, I quite believe there is value to provide something “pre-polished and pre-chewed” for most people to try in under 2min from start-to-finish.

I will absolutely reach out in February when I’ll work on more complex PEPs that will really require a “good example”.

Let’s close this parenthesis, thanks @pf_moore for your feedback. Truly appreciated, this will be very helpful for the rest of 2025-2026 (and the future work coming as part of Wheel-Next)

1 Like
14

Thank you for writing this up and providing an implementation!

Some real world examples with popular package, such as fastapi-cli[standard], would make the PEP more tangible and ensure we’re solving those cases.

Should the Metadata-Version be increased with this PEP, since we’re introducing a new field? (Like PEP 639 with a similar change). I don’t see any required std change, but the metadata bump is for pypi to update their validators and twine/uv publish to send the right form-data fields.

The syntax for deselecting extras was rejected because the breaking change would be incompatible with older packaging tools. Wouldn’t this apply with the current approach too, because installing a package relying on default extras for e.g. a good getting started experience would have a broken installation with older tooling, except it’s not shown as an error?

An important problem with feature flags is backwards compatibility (on the package level): With the current design, it’s a breaking change to move a functionality from core to a default extra: Say a package A has to_foo and from_bar methods, where the foo extra is required for to_foo. In A 1.0.0, foo is a default extra and the bar packages are mandatory dependencies. In A 1.1.0, the authors want to reduce the dependency footprint, from_bar becomes part of the bar extra, which becomes part of the default extras. A library depending on e.g. A[foo] or A[other] can’t use the previously core from_bar anymore and breaks; Alternatively, A has to do a major version bump without API changes, just to move functionality from always-required to default-features. My reference point here is cargo, with has a default-features = false option which works similar to this proposal (in the PEP, specifying any extras manually is equivalent to default-features = false). default-features = false is a known problem for library because of this. Relatedly, one of the most popular cargo feature requests is adding an option to deselect individual features (Ability to disable individual default features · Issue #3126 · rust-lang/cargo · GitHub).

The devil is in the details so we won’t know until we try implementing it, but i don’t see anything in this PEP that looks incompatible with pubgrub/uv or uv.lock.

2 Likes
15

Thanks @konstin for your feedback ! Are you yourself a contributor to fastapi (if not and you know them, I’d love to get their feedback on our proposal) ?

Probably, now it may be batched with a few other PEPs to avoid bumping the Metadata version all of the time. Afaik pip is currently supporting Metadata 2.2 (with some bits from 2.3 & 2.4) but I don’t think it accepts anything above 2.2. If you look at my POC you’ll see that I increased Metadata version to 2.5 out of “good measure”, though I wouldn’t consider bumping MT version number “part of the PEP”, ultimately it doesn’t matter to me if we do or don’t I just think we should be consistent with previous work. If we think it’s better to accumulate a few PEPs and then bump the version and publish everything at once that works for me.

Well, I think that’s something you can detect in the package at import time.
Imagine if in the __init__.py of fastapi-cli you have some code checking at least “some backend or core dependencies” have been installed. Pretty trivial to do. And it allows you to display a nice “ImportError” mentioning to either update pip or pip install fastapi-cli[standard], unfortunately we can’t really go back in time and backport a feature or a CLI flag.

Honestly, I’m having a little bit of difficulty to understand your example. So my answer might shoot over the fence, though I’ll try.

I think you might not look at it from the right angle / perspective:

We don’t actually do that. We keep everything of the original design. We just offer a “shortcut” with a default value.

Think about that pip is doing something along these lines.

def get_user_cli_extra_reqs_keys(pkg_name:str):
      extras = extract_extras_from_pkgname(pkg_name) 

      if extras:
          return extras

      return metadata.getAll("Default-Extra", [])

So this doesn’t really break anything, it “sort of feed a value” if the user doesn’t give one.
Which means that packages that today declare their dependencies as follows:

Metadata-Version: 2.2
Name: my_package
Version: 1.0.0
Requires-Dist: foo[standard] # `standard` to become the "default for foo"
Requires-Dist: bar[basic]    # `basic` to become the "default for bar"

Well that would change absolutely nothing for my_package, nothing prevents you from “hard feeding” the default value (despite that a default value does exist for new pip/uv/ etc.)

Though indeed if the maintainers of my_package decide to remove standard (for foo) and basic (for bar) in the dependency chain, well now they expose themselves to breakage for users on older versions of pip.

Does this makes sense ? I hope my answer did address your comment, I’m not sure I understood it well.

I’m delighted to read this :slight_smile:

16

Some additional thoughts regarding backwards compatibility: I think adopting this feature will be hard for most maintainers: given that installers may or may not respect the new Default-Extra field, end users are going to get two different sets of dependencies in some installation scenarios depending on whether their installer supports PEP 771 or not when they don’t specify an extra.

I think for some use cases, this is OK, but there are use cases where this would result in surprising behavior for the end user. At what point should a project adopt this new feature, moving subdependencies from Requires-Dist to a Default-Extra and potentially break end users using incompatible installers? (I think this is what @konstin is getting at).

Maintainers might also naively expect installers to universally support the feature when this isn’t the case, and their end users might wind up with missing dependencies if their installer doesn’t support the feature.

I’m wondering if this PEP should do more to support backwards compatibility, although I’m not sure what it should be, since there’s no way to “express” default extras in earlier metadata versions. Should installers outright reject new metadata versions that they don’t explicitly support? I’m not sure if there’s precedent for this or what the current status quo is across the many different installers.

2 Likes
17

That’s a bad idea, IMO. Any PEP that needs a new metadata version should have its own. Otherwise what happens if one PEP is accepted but another (that uses the same metadata version) is delayed?

Pip barely cares about most of the metadata fields, so it doesn’t need explicit support. In particular, pip should happily install a package regardless of what metadata version it uses.

I disagree. And that’s with my “PEP delegate” hat on :slightly_smiling_face: If you change the core metadata, you must specify a new metadata version.

Strong -1 on that - it’s already enough work getting one PEP approved and implemented. Accumulating multiple PEPs jus makes everyone’s life harder.

18

Fair enough that works for me. I had the incorrect perspective on how we should approach it. I’m fine with what you suggest @pf_moore (your whole message not just my tiny quote)

19

It’s going to have to be different for different projects and different dependencies. When I thought about making use of this PEP it occurred to me that it would probably be good to add the relevant extras sections some releases ahead of actually making use of the default extras feature. For example if you want to release foo 1.5 in which there are new default extras but it is possible to opt out with pip install foo[minimal] then it is probably helpful downstream if the minimal extra was also available even as a no-op in versions 1.4, 1.3, …

I’m not sure what is the PEP 440 syntax for saying “I want foo[minimal] if the foo version has a minimal extra but otherwise normal foo is fine.”

20

In an ideal world I agree with you, now I’m a bit stuck on the part “what could we do ?”. If you have any idea so that we can “improve the backward compatibility” support, I’m listening with all my attention and good intents :slight_smile:

I think this PEP will be “on a short timeframe” more useful for packages that need a backend that is not explicitly mentioned in the core dependencies for example package[tkinter] or package[pyqt] but package is invalid today. Well tomorrow with this PEP package could become valid and be equivalent to package[tkinter] for instance.

@pf_moore @dustin now this is a frequent problem in packaging. How do we issue a warning to “old installers” that may not support X, Y and Z. I think the lack of “that capability” is strongly holding us back as a community. Would you be opened to a discussion solely on that ? Maybe this could even be a topic for the packaging summit @ PyCon.

There are different possibilities on how we could approach this. One of which being what @dustin mentioned. Or less “opinionated” instead of “rejecting” we could issue a bold & red warning WARNING: this package comes with Metadata version X.Y and Wheel version A.B, these are not officially supported by your installer, there might be unexpected failures. We strongly advise updating your installer. etc. etc. etc.

And the longer we wait to introduce a change like this, well the longer it takes for this change to get propagated in the user base.

I can quote a number of PEPs / discussions that have been getting similar types of comments, for example: Implementation variants: rehashing and refocusing