PEP 771: Default Extras for Python Software Packages (Round 2)

Packaging Standards
1

Following the discussion on the first published version of PEP 771 in February, we are happy to share with you the updated version of PEP 771: Default Extras for Python Software Packages:

Let’s discuss this updated version here, and no longer use the previous thread. Thank you to everyone who participated in the previous discussion!

The main changes since the last published version based on feedback in the previous thread are as follows:

Standard/universal way of requesting a minimal installation

package[] is now considered to be the primary way to specify that a package without any default extras is being requested (rather than letting each package define its own minimal extras such as package[minimal] or package[nodefaults]).

The main benefit of allowing this is that package[] is already valid syntax and is currently a no-op, so package[] will always mean ‘package with no default extras’ including for packages predating default extras, and for installer tools predating default extras. Therefore, packages who only need a minimal installation of a dependency can safely use package[] regardless of whether it applies to versions before/after the implementation of PEP 771, and will work even for users who have old versions of package installers.

In addition, standardizing this will mean that users will not need to depend on package maintainers implementing this option, and will not have to delve into the documentation for each separate package to figure out what the correct minimal extra is, if it exists.

The prototype implementation of PEP 771 in pip has been updated to support this, for anyone interested in trying it out.

Discussion of the alternative of splitting packages

The alternative approach of splitting packages into two, with a core package with minimal dependencies, and a meta-package that adds recommended dependencies, has now been added as a Rejected Idea in Using a meta package for recommended installations

To summarize our view on this approach: it is technically feasible, but it is a much higher burden for package developers, can lead to confusion for users, and would have the risk of breaking some user workflows for existing packages that want to transition. This is just a very rough summary, so please read the section in the PEP for more details.

Mention packages that have e.g. both backends and frontends

We have added text regarding packages that have e.g. both multiple backends and frontends and might want at least one default for each (and more generally, packages that might have different kinds of defaults). This can be found in the new section Packages with multiple kinds of defaults

Mention accepted PEP 751

PEP 751, which defines a new lockfile format, is now mentioned in Packaging-related tools as a solution to the pip freeze issue mentioned in the same section, because the PEP 751 format explicitly lists packages to be installed without resolving dependencies.

If there is still a general consensus that the pip freeze in its current form should not install default extras, and that recommending the usage of --no-deps is not enough in those situations, then it is an option for pip freeze to include [] after packages that use default extras, or even after all packages (but this is not part of the current PEP, and should perhaps not be in any case as it is a decision for the pip developers to make).

Added text on documenting default extras

We have added a new section, Documenting packages with default extras, layout out best practices for what package maintainers should document in relation to default extras.

Considering default extras as ‘weighty’ as required dependencies

We now mention that default extras should be added as carefully as required dependencies, and give pytest as an example of an ecosystem where default extras will effectively always be installed unless downstream packages were to adapt their dependencies. This new text is in Avoiding the addition of many default dependencies

Other changes

Other changes have been made following feedback in the previous discussion, including changing some of the example packages listed in Motivation, removing the open issue relating to package[] since that is now implemented, and various spelling/grammatical issues.

10 Likes
2

Continuing the default-extras vs default-optional-dependencies-keys topic from the previous thread: the optional-dependencies table is defined in the specification as " For optional-dependencies, it is a table where each key specifies an extra and whose value is an array of strings." (emphasis added).

So even though the term “extra” doesn’t currently appear in the syntax of pyproject.toml, it does appear in the existing specification. The only reason it doesn’t appear in the syntax is because that table is named after the values it holds rather than after the keys.

Since folks do need to know that the published groups of optional dependencies are called “extras” when referring to them in a dependency specifier, I think it’s fine to use the shorter field name in this PEP (given that “the keys in the optional dependencies table” is literally one of the definitions of “What are ‘extras’ in Python packaging terms?”).

3 Likes
3

Thanks @trobitaille for the great work !

A few notes on the pip freeze aspect.

I see 2 options.

  1. pip install -r requirements.txt would by default NOT install any default extra. I don’t like this solution because that will break all behaviors from people who write the requirements.txt by hand as a project dependency file.

  2. pip freeze adds [] to every projects (maybe only to those who include a default extra => can read the .dist-info/METADATA to determine that). That solution is forward / backward compatible. Requires “just a little bit more logic” to detect if a project declares default extra. Changes nothing to existing pre PEP-771 requirements.txt files.

Now the questions IMHO are:

  • Should we care given that PEP 751 and pyproject.toml are better ways to manage that anyway ?

  • Shall it be included in the PEP or left to the tool to decide ? pip freeze is not an explicit standard. More like “something everybody converged to because pip popularized it”. In any case - whatever pip decides to support (regardless if included or not in the PEP) will most likely be replicated in other installers to keep consistency. And for that reason - I believe 2) above is probably the best (requires least number of changes from everybody)

On a different note - I’m really happy to see the “last details” to be sorted out are “what name should be used for the key” @ncoghlan :smiley: Means that we are quite close to a consensus ! :+1:

@konstin the issue with --find-links in the tutorial has been fixed. Do you mind checking again ?

@charliermarsh any feedback/opinion on the pip freeze / pip install -r req.txt aspect ? Not sure you care too much but just in case

4

It seems correct that pip freeze and tools like pip compile would want to output with [], yeah. Or rely on users using --no-deps.

1 Like
5

I think this is something that should be left for pip to decide. In fact, we can’t include it in the standard as requirements files and the pip freeze format aren’t standardised. My personal view is that I don’t want the output of pip freeze littered with [] just in case projects define default extras. The output of pip freeze isn’t just a “dumb lockfile” format, it’s also used for human consumption, and for various scripting use cases. As such, I’d rather it stayed human-readable.

People who want to avoid default extras getting installed should be using --no-deps.

Regarding “tools will follow pip”, I would strongly advise tools to make their own decisions, and not treat pip freeze output as an informal standard. We now have a lockfile standard which should be used for any actual locking needs. There’s simply no reason outside of backward compatibility for using pip freeze output for locking, so if anything, I’d argue that “don’t change anything” is the right answer if you’re taking that view.

However, as I said I think this should be a decision for tools to make on their own, so I’ll defer any further debate to when the discussion comes up on the pip tracker (and I reserve the right to change my mind over there if compelling arguments come up).

2 Likes
6

That perfectly works for me - @trobitaille that works for you too ?

7

Yes, I agree that this should be left to the pip developers to decide and is outside the scope of the PEP.

2 Likes
8

In this case I’d be perfectly happy to change the key to default-extras unless anyone objects – it is a much better option than default-optional-dependencies-keys!

3 Likes
9

It only takes one StackOverflow post, an LLM or someone copy/pasting from another project for whatever documentation holds such warnings to not be read, let alone heeded. We’ve seen from the sheer volume of this is great, it’ll let me do [exactly the thing we don’t want them to do] posts on the previous thread and on the setuptools issue before that just how much of a honeypot this mis-use case is. A bit of documentation isn’t going to fix that.

4 Likes
11

What would you propose that would be better from your perspective ?

12

That’s… not quite the reasoning for that naming choice. https://peps.python.org/pep-0621/#other-names-for-dependencies-optional-dependencies covers why this name was chosen. The underlying motivation here was that the “extras” terminology is unique to Python in a suboptimal way. Notably, I do think avoiding “using terminology that is necessarily foreign to new programmers” is a positive aspect of this that we’ve benefited from.

I’ll also note that there’s significant and meaningful difference between the audience of who read a Python packaging standard’s specification vs the audience who uses the Python packaging standard. The fact that the specification uses certain terminology doesn’t necessarily endorse that we should use the same terminology in the user-facing surfaces.

2 Likes
13

“Optional dependency keys are also known as extras” is a fact of the Python packaging ecosystem.

We can either inform newcomers of that fact up front, or we can keep it as not-so-secret specialist jargon that people have to figure out on their own the first time they encounter it as a term of art.

I don’t think we’re doing newcomers any favours by pretending the shorthand term doesn’t exist, even if we wouldn’t have chosen it if we were designing things from scratch today.

2 Likes
14

I think you can probably guess that my answer to that is to reject the PEP. We can’t control what people do with any given feature so I don’t believe that there’s anything we can do to prevent this one devolving into the minefield of "you may also like"s that the equivalent apt and dnf features get used for.

Maybe if there was there was enough added friction, people will stop seeing it as a “free” way to suggest dependencies without making them mandatory. The only option that springs to mind is to instead be able to make not specifying an extra an error so that pip install foo would fail, requesting that the user explicitly specify pip install foo[] or pip install foo[full], probably with some way to propagate a deprecation warning through installers that this requirement will soon be enforced. That at least ticks the users are unaware that they need to do install foo[extra] box without resorting to guessing the user’s intention and gives no illusion of being a backwards compatible change. This isn’t a serious proposal [1] but it’s the kind of make it less attractive for the use cases we want to avoid [2] direction I’d want to steer in if we don’t want to deteriorate the overall health of Python’s packages.

  1. read as I don’t need people to point it it’s glaringly obvious shortcomings ↩︎

  2. “we” being the narrative of the PEP. My personal view is still that an ambiguous dependency tree is always bad but I’ve made plenty of noise about that already ↩︎

1 Like
15

While I agree that default-optional-dependencies-keys probably isn’t the final name we should use, I worry that default-extras is a step backwards.

I strongly agree with Pradyun’s comment, it’s well said.

While extras occurs in the documentation, the fact that it doesn’t occur in the configuration file itself is problematic. It will definitely be confusing to new users that default-extras controls the optional-dependencies.

I would really like for us to remain consistent here, unless there’s another way we can somehow emphasize the relationship between these fields to assuage these concerns.

3 Likes
16

This is a big concern for me as well. The fans of this proposal have made some good arguments for the benefits of the feature, and I don’t think anyone disagrees that there are places where default extras would make things easier. But I don’t think that the converse - the costs, particularly in terms of confusion over when to use default extras, and when they make things more difficult for end users - has been adequately addressed. It’s much too easy to say “any feature can be abused”, but that doesn’t really solve anything. We need a mechanism that’s easy to use well, but hard to use badly, and I don’t think the current proposal meets that standard. “Just add [] if you don’t want to accidentally get default dependencies” is the sort of “too easy to use badly” aspect I’m thinking of. Also, it’s not actually all that easy to use well in some situations - if you have multiple independent sets of extras, it’s not at all easy to use default extras in a user-friendly way :slightly_frowning_face:

More generally, I note that the “how to teach this” section of the PEP is huge, suggesting that it’s not immediately obvious how to use the default extra feature correctly…

I mostly agree, although the one favour I do think we’re doing them is not exposing them to the lack of any clear consensus on the “right” way to use extras. For more on that, see below.

The PEP uses the “extra” term throughout, including in the core metadata field name (following the precedent of the existing field). It’s pyproject.toml which is the outlier here, in avoiding the term. Having said that, what @pradyunsg said is true - the “extras” terminology has always been problematic[1]. I’m concerned that the PEP is in effect building on an unstable base, by expanding the complexity of the “extras” concept, without doing much to deal with the historical problems with the idea[2].

  1. We had to introduce a whole new dependency group mechanism to resolve confusion over whether dev and test dependencies should be extras, for example :slightly_frowning_face: ↩︎

  2. Although I don’t think it’s necessarily fair to expect this PEP to fix those problems - it’s just that trying to write a successful PEP before the problems get fixed will always be difficult ↩︎

3 Likes
17

We could name the table [project.dont_blame_pypa.default-optional-dependencies]. :joy:

In seriousness, testing on a range of dependency versions + combinations of installed dependencies is a pretty high burden. This is not free for responsible maintainers.

I don’t know that designing a spec around “how it will be misused by novices” is the right approach here. If someone asks an LLM to write them a pyproject.toml and doesn’t look at the contents, they’re likely to get some pretty bad outcomes anyway. And we’re not really worried that major ecosystem components like packaging or attrs would start using this without thinking about and testing the results, right?

The ecosystem harms seem not so bad to me, or at least no worse than our present situation. If misuse really will be an issue, I’m not clear on how it’s worse than misuse of metadata-only packages with a separate $PROJECT-core to simulate the feature – a pattern sufficiently laborious for the provider that it is quite rarely used. Somehow, the fact that we’d be making this difficult pattern easier seems key.

Your experience with this feature in other packaging ecosystems leaves me very interested in your thoughts, but I also wonder if there might be key differences (e.g., overall pace of change for packages, difficulty to create and publish packages, …).

18

I can’t speak for @bwoodsend but that’s not the concern I have (although it’s related). My concern is that the problem of catering for multiple groups of users with different needs is a genuinely hard problem, and default extras don’t solve that problem. At best they give users another tool to use. But does that tool actually help? For example, the astropy example given in the PEP suggests that adding the recommended dependencies as a default extra would be better than the current situation. But it isn’t better - it simply changes the balance. It makes it easier for users who want the “bundle” of astronomical tools that astropy recommends, but it makes a new problem for packages that depend on astropy - they need to update their dependencies (which is actually impossible for already-published versions) to depend on astropy[] rather than just astropy.

Yes, the astropy project can consider those questions, and come up with an answer that they believe to be in the best interests of their user base as a whole. But the problem still isn’t “solved”, they’ve just decided to change who gets the most convenient solution. At the cost of a backwards-incompatible change.

My experience is that in reality, projects won’t put that much thought into the matter anyway. Depending on the philosophy of the project, they’ll either switch to the new approach because “keeping up to date is important” or they’ll stay as they are because “we prioritise backward compatibility”. Or a contributor with a strong opinion will come along with a PR, and it will be accepted with some level of review (how much depending on maintainer availability).

The harms aren’t likely to be too bad, I agree. Most projects shouldn’t need default extras (most projects don’t even need extras!). But I personally don’t think we should be making decisions based on “this probably won’t do too much harm”…

5 Likes
19

default-extras doesn’t bother me too much, although I agree that it would be better if it were consistent. I think people will Just Learn and it’ll be fine and I don’t love having to type really long default-optional-dependency-keys, which in all likelihood I’ll mistype[1].

If we really want consistency, then my only other recommendation is that we drop the -keys suffix and rename it to default-optional-dependencies. That seems good enough too and more consistent.

  1. and as long as my AI overlords JFDI then I guess I’ll be fine ↩︎

4 Likes
20

Does it matter that the users who get the less convenient solution are likely those with a higher level of expertise?

Excepting package version conflicts and other “disasters” from the user perspective, wanting the minimal install already puts you in a narrow class of package consumers. Most users don’t care at all. (Whether or not they should care is an interesting philosophical exercise but not a practical one.)

The question is a bit of a prompt but also genuine. It’s worthwhile to try to align the simplest usage towards the least experienced users, but I’m not sure if that reasoning can justify new features on its own.

1 Like
21

A possibly solution to this is that PEP 771 could say that this is only a direct install feature and does not affect implicit install via dependencies. The original motivation for this PEP was to help naive users get something working when they didn’t understand that they had to pick an extra to get a working environment.

Any dependent of a package with default extras should know exactly which extras they want, and they can even cascade default extras in their own package for naive end users directly installing their dependent. You could then define lock files as indirect installations and avoid [] pollution.

This puts more burden on the tools since they have to implement this rule, but I’d rather put the complication on them rather than the vast number of package consumers.

4 Likes