PEP 771: Default Extras for Python Software Packages (Round 2)

Packaging Standards
102

The current meaning is not totally clear cut, though, because one way to handle ‘recommended’ dependencies is just to list them as regular dependencies. I’m thinking of a package I work on for work where really essential dependencies (numpy, h5py) are mixed with things you could miss out and still use core functionality (xarray, matplotlib). In our case these not-quite-required dependencies are likely to be installed anyway, so the cost of treating them as full dependencies is low.

If projects have a way to express recommended dependencies, some dependencies that are currently hard requirements might also be made optional. E.g. matplotlib currently depends on python-dateutil, presumably to support date labels on axes; it’s not hard to imagine this could be made optional for a minimal matplotlib install (although I’m not saying the matplotlib maintainers would do that).

5 Likes
103

Catching up on this discussion after a few weeks, something that stood out was that we’re making a potentially flawed assumption in thinking that “minimal dependencies” and “recommended dependencies” is a sufficient distinction to be able to make (both on the part of project publishers and on the part of end users).

Instead, we’re kinda using that as a proxy for what I’ll call “installation profiles” (for lack of an existing term).

Some potential installation profiles:

  • web service production deployment server (minimal is good!)
  • local web service development (gimme all my dev and debug tools thanks)
  • novice learning data science (please set me up to run all the examples)
  • production data analysis server (minimal is good!)
  • running a CLI interactively (gimme the dependencies that make that pretty)
  • running a CLI headless (no need to be pretty, it’s just going to a log file)

The “requires at least one of …” alternative dependencies then complicate matters even further.

Thus the discussion ends up going round in circles, as “minimal dependencies” and “recommended dependencies” can’t really be meaningfully defined in the absence of a clear understanding of how the installed project is going to be used.

Extras don’t solve this problem as:

  1. Transitive declarations are controlled by the projects specifying them rather than by the target environment; and
  2. Their names aren’t standardised, so tools can’t readily provide a way to specify minimal installs of even transitively included dependencies

This issue also kinda came up in the pylock.toml discussions, in the sense that the final format included the extras and dependency-groups metadata, so a tool can build its own notion of installation profiles by deciding which extras and dependency groups to install from a lockfile.

So whether the spelling is package-core vs package (status quo) or package[] vspackage (PEP 771), the dependency declaration itself feels like the wrong place to be specifying whether we’re trying to minimise the installed dependency set or maximise the available functionality (for a given use case).

8 Likes
104

I acknowledge the limitations of extras that you’re describing, but I also think they generally work well enough, and I’d like to get a solution to the specific problem - that you can’t separate the minimal install profile and the default - without it getting lost in a larger effort to design something totally new.

In general - and I think this covers all 3 of your examples - the packages where you care about installation profiles are packages that you’re interacting with directly, and therefore probably installed (or specified a dependency on) explicitly. If foo gets installed implicitly as a dependency of bar, then most likely whatever flavour bar specified is going to be OK for the end user.

I’d happily support an effort to standardise the name of a default or minimal extra, but neither is feasible without some changes to standards and/or tools:

  • A default extra doesn’t work without a way to tell tools not to use it
  • A minimal extra is impossible, because extras can only add dependencies, not subtract them
1 Like
105

But what if foo is a dependency of both bar and baz and they specify different flavors?

106

I would assume you take the union of the extras that they specify.

107

I like the idea of thinking of this in terms of installation profiles, but I also think that designing something completely new to solve this is likely going to be a difficult task, and extras might be the best programatic solution to this. I don’t think there’s going to be any sane way to formalize all the types of installation profiles, though even if we did, this could always be by standardizing extras names rather than designing a new system.

The only downside of the way extras are designed currently is that the default installation has to also be the minimal installation if you want to cover all possible installation profiles, but this is incompatible with the ideal goal that the default installation profile should be the most common one. PEP 771 would then unlock the ability to basically use extras as installation profiles and would standardize the syntax for the minimal ones. In future, one could then envisage standardizing at least some extras names for other common types of profiles.

1 Like
108

I’ve tried looking around both the PEP content and this thread (and the previous) and I’ve not seen anything though I may have missed it: how does PEP 771 interact with the compatibility tags of the default extras?

For explicit extras it makes sense that you’d just get an error if you request a package which doesn’t have anything suitable for your platform, but for default extras if the default extra is a native wheel with compatibility tags, and none fits the current system, would the extra:

  • be ignored (either silently or with a warning), so the package gets installed just without the default extra(s)
  • cause an explicit error, and the default extra would have to be removed from the dependency (possibly using environment markers to match platform and extras configuration)
1 Like
109

I like this PEP, thanks!

2 Likes
110

OK, so back on track: this is a PEP about having a set of dependencies that most users want, while allowing dependents to opt out of installing these. We don’t need a PEP to help with opting in, because extras already exist for that, and just having a canonical name for “commonly-opted-into” could be a convention. But I think the tangent about making this about opt-in (the + discussion) existed only because people discovered that we’re conflating the opt-out flag with the ability to specify extras.

I think the solution is simply to allow people to also say “I want the default extras but also these extras”. We could perfectly conserve current behavior:

  • package gives the default extras
  • package[extra] gives the default extras and extra

But also add the ability to opt out (bikeshedding syntax):

  • package[-] gives minimal extras
  • package[-,extra] gives minimal extras and extra

we probably need a grammar change

No worries, it’s not that bad! We did it before. Why? We probably won’t find some trick like [] that works for what I propose, but let’s be honest: that idea is a hack, [] wasn’t intentionally made legal grammar for it to serve some unspecified future purpose. Our problem here is that we’re using bespoke syntax for dependency specification instead of something like Rust’s list-of-dicts. Any change will likely require a syntax change, and that must be OK, otherwise we can never change anything about dependency specification.

(this is not “syntax for deselecting extras” from the PEP, which is about deselecting individual extras instead of saying “I want to start selecting extras with no extras instead of the default set”)

4 Likes
111

Another unrelated thing that could go in the “motivation” section:

We need this feature for licensing reasons – we want most users to get GPL licensed dependencies (i.e. we want them to get them by default so they don’t have a basically unusable package on install) while we also want companies to be able to opt out of the GPL parts.

1 Like
112

It’s really not the spelling that’s the problem with default extras. It’s the hoops people have to jump through to get what they need without the extra baggage, the inherent misconception that opt-out-able dependencies are free and can be added willy-nilly, the way people instinctively want to use them for "you may also like"s[1], how it allows a package to scope creep to infinity instead of having the asymetric dependency tree steer them down the right path of splitting things up into well scoped packages[2], the burden it puts on dependent packages to either preserve the optionality or pick one and become incompatible with anything that picked the other[3], the completely impossible situation it leaves repackagers that have no equivalent concept in.

  1. scroll up, you’ll find them ↩︎

  2. often well beyond the point where the dead weight within the package exceeds that of the optional dependencies ↩︎

  3. it’s like function coloring only at a packaging level ↩︎

4 Likes
113

I don’t think any of these vague concerns should block this. It fulfills a real need, and as long as it doesn’t exist, things that don’t end up in the default extra end up in dependencies, so things will only get slimmer.

Rust’s Cargo has basically the same design and I’m not aware of anyone complaining.

Conda also has to deal with us having extras and them not. They deal.

2 Likes
114

There are people complaining about it in this thread…

115

Ctrl-F “cargo” did only find me this, which only mentions the existence of issues, but doesn’t complain. Neither does the linked post, which does give a lot of context. Most of that context is about the shortcomings of purely additive features (i.e. extras), e.g. the lack of mutual exclusivity. Funnily enough it also has a quote by Ed Page advocating for exactly what I proposed above (plus other changes).

The only complaint request wishe that I found that touches on default features (and not exclusively other aspects of how extras/features work) and isn’t already part of the proposal is the

Ability to remove features from a [default feature]

Which I think means that people should be able to deselect default features individually. I think we can manage that. bikeshed syntax:

  • package[-] gives minimal extras
  • package[-,extra] gives minimal extras and adds extra
  • package[-d1] gives the default extras except for d1
  • package[-d1,extra] gives the default extras except for d1 and adds extra
  • package gives the default extras
  • package[extra] gives the default extras and adds extra

note that there is no ambiguity: -d1 doesn’t mean “I forbid d1”, it just means “I need all default extras except for d1, but if someone else wants it, that’s fine”. If d1 is not a default feature (or doesn’t exist at all), it should probably be handled like a missing extra: warning only, as it might exist in some other version.

1 Like
116

Speaking as PEP delegate[1], I disagree. These “vague concerns”, as you describe them, are real issues that IMO the PEP must address if it’s to fairly represent the community view on the subject.

If you want the stated concerns to be more formally expressed (note - this is just my rewording of what @bwoodsend said, I may have misrepresented what he was trying to say, if so I apologise to him):

  1. There’s a concern that default extras will become bloated because there is no clear understanding of what is reasonable to include in a default extra.
  2. There’s a concern that default extras offer an “easy option” for projects to avoid the work of properly structuring their code into well-scoped packages, encouraging bad designs.
  3. Default extras increase the burden on dependent packages, as they have to add complexity to their own extras to preserve the “full vs minimal” split, or pick one option and introduce incompatibilities with packages that made the other choice.
  4. Default extras make it impossible to repackage projects that use them in other packaging systems that have no equivalent concept.

None of these seem particularly vague or unanswerable to me.

It’s not necessary to solve all of these issues. The PEP authors are allowed to reject concerns like these. But they must justify the rejection, and document their reasons in the PEP.

Your proposal is not what PEP 771 says. So are you planning on writing a competing proposal? Or are you arguing against acceptance of PEP 771? Or are you hoping to persuade the PEP authors to rewrite the PEP with a new proposal?

I’m confused…

  1. Until the packaging council is formed :slightly_smiling_face: ↩︎

5 Likes
117

This one in particular is one which I would like to see the PEP authors attempt to address.

The other problems are much more open ended. For this one, I can formulate a very succinct scenario.

If I maintain an AUR repackaging of a Python project, and that project changes to declare a default extra, what do I do? Does it matter if the new default extra

  • only adds new deps
  • only contains deps which were previously required
  • contains a mix of new and old

?

1 Like
118

Sure! I called them vague since some of them can be further rephrased into “if we introduce a new tool, someone might misuse it”. This is always the case! The question is always: do the benefits outweigh the added complexity?

And having used default features in Rust the answer is an extremely clear “yes” to me.

  • As developer of downstream packages, scripts, and applications, it’s just so much nicer to start developing with a good set of defaults without having to understand all the intricacies of how the internals of a package hang together. Later, when you know what feature set you need, you can slim things down if you care.
  • As a developer of a MIT-licensed package which would like to default to features depending on GPL packages, this is basically the only way forward, especially since we target non-technical scientific users. Splitting into package and package-core would be nothing but a headache for us, since there is no benefit in splitting the code up, so the packages would only differ in dependencies, which would mean we’d have to switch to a monorepo, change our deployment, and so on. It would also confuse users since it would be unclear when to use which package whereas the default extras are very clear in purpose for an user.

I think this is the only concern that points to a concrete thing that will definitely happen, instead of being speculative on what might happen.

First, the “impossible” part is wrong, see my conda argument from before: they repackage PyPI package and have no extras at all! I’m very sure this is OK, as packagers have dealt with this for decades. It’s kind of what being a packager means, to figure out how to map things like this to how your linux distro or whatever does things.

But yeah, this is an actual concern pointing out additional work for people caused by introducing this, so this one should be addressed (e.g. by pointing to how conda repackages PyPI packages and debian repackages rust crates)

I maintain >40 of those. Arch is less granular than e.g. Debian and often even combines multiple Python packages into one package. It’s just a few MB of Python code, no big deal. It’s similarly liberal with dependencies. So usually you would move all dependencies from the default extras that are self-contained Python dependencies into the regular dependencies, but leave out dependencies that pull in bigger ones. See e.g. resvg, which which requires either Cairo or Qt 5 to do anything – it doesn’t pull in either, and requires you to manually co-install one to be functional.

3 Likes
119

We see significant dependency bloat today in mandatory dependencies for many packages. I believe that having a set of optional but default dependencies will, on the whole, improve this, because some mandatory dependencies can become recommended. But perhaps we can improve the situation by trying to establish that common understanding. As I see it:

  • Default extras are for things that are likely to benefit most typical users of your package, especially typical new users installing it ‘by hand’ (since experts are more likely to explore non-default extras). E.g.
    • a compiled package which does something faster, where you can fall back to a slower but good enough alternative without it
    • pretty terminal formatting, where you can fall back to plain output, or where a terminal interface is useful but not the core feature
    • live reload for a static site generator
  • Default extras should not include packages you need only for developer/maintainer tasks, such as building docs or running the test suite.
  • Core functionality must still work if only the mandatory dependencies are installed
    • If you have a single top-level public module, it must be possible to import that. If the main use of the tool is a command, it must be possible to launch the command.
    • This is unavoidably vague where you have several pieces of functionality requiring different dependencies - but the same question exists with extras today.
    • It is recommended to have a CI job which does a minimal install (no default extras) and checks the things you expect to work, so you don’t accidentally rely on a default.

I presume the ‘proper structure’ here is implicitly more, smaller packages. This is a preference disguised as a natural law. I don’t think that breaking packages up into pieces is always an improvement.

Also, the absence of default extras so far does not seem to have pushed the Python ecosystem into merrily splitting everything into little pieces. Twine, which we use for uploading packages to PyPI, has a mandatory dependency on a compiled HTML sanitizer library, because the fairly orthogonal twine check and twine upload commands are part of the same package. I don’t see that adding default extras is going to make a significant difference to this question.

I think for most packages with dependencies, choosing one option is the right thing to do, and I don’t think there’s any incompatibility. If A depends on C with default extras, and B depends on C without, then installing A and B (together or in either order) gives you C with its default extras. B is expressing ‘I don’t need those extras’, not ‘I need them to be absent’.

Our packaging metadata doesn’t have any way to express that you need some other package to be absent. IIRC, Debian does, but it’s a better fit for a relatively small, curated collection.

Clearly it’s not impossible, just more challenging. Repackagers have to decide - either by general policy or case-by-case - whether to translate recommended dependencies to full dependencies, or to something opt-in if available, or not at all. I agree with @flying-sheep that this kind of decision is the core of what repackagers do.

I don’t think we can make a general rule, but yes, if it’s something that was previously a mandatory dependency, that might push you in the direction of keeping it as a dependency in AUR, to keep it working as people are used to.

I’d also consider whether the package is primarily a tool that users will install manually & interact with, like IPython or tox, or a dependency of other packages. If it’s something user visible, that might also favour including dependencies, so users get the standard ‘out of the box’ experience. If it’s primarily a dependency, you can look at how many of its dependents need the recommended thing.

@flying-sheep has hinted that the culture of AUR is fairly pragmatic about these things. Where ecosystems favour more precise control of what’s getting installed, they’re more likely to exclude default extras. (I’m thinking of Spack, although that already has ‘features’ which can be on by default)

5 Likes
120

I’m reading though all of the discussion here, and I’m a little confused (or maybe taken aback?) by some of the points being raised here.

One of the things that is sticking out to me here is that there seems to be a general concern that package authors, when handed some new tool, are going to start acting with disregard towards their users. I don’t think that’s a super helpful way to think about that set of users that we are trying to serve. Certainly we should think about how likely it is that someone may misuse a given feature, and if they do what the ramifications of that would be.

Unfortunately the way I’ve read a lot of the discussion here, is coming across to me, as suggesting that if this feature could be misused, then it is a bad feature, and I think that is an unhelpful framing. Every feature you could ever possibly add could be misused somehow, so that argument, taken to it’s logical conclusion, is that we should have no features at all :slight_smile: .

It seems like one of the big sticking points is this idea of "redefining what pip install some-project means, with the idea behind that currently some-project means the minimal set of dependencies.

I’ve been thinking about that a little bit, and it stands out to me the idea that some-project means “the minimal set of dependencies” is technically true when you think about the standards in the abstract, but it’s not actually true in practical reality.

In practical reality some-project means “the set of dependencies that the package author thought you should get by default”. Sometimes that’s the minimal set of dependencies, if the author only listed the minimal set of dependencies as required dependencies, but sometimes it’s not the minimal set of dependencies if the author listed “stuff that most people would want but aren’t strictly required”.

After all, there’s nothing that requires the non-optional dependencies to actually be non-optional, or to even be used at all. We’ve just given authors the ability to list dependencies to install by default, and assumed they’re going to make reasonable use of that feature for their users.

With this framing in mind [1], I would argue that PEP 771 doesn’t actually break backwards compatibility in a practical sense. pip install foo is still going to install "the set of dependencies that the package author thought you should get by default, but now we’ve given authors an extra tool to allow them to more accurately specify their true minimal set of dependencies-- and for users to explicitly request that.

That actually feels like a strictly better outcome for people who are worried about trying to resolve a minimal set of dependencies in their environment. Both with and without PEP 771 there’s no guarantee you’d actually getting the minimal set of dependencies [2], but with PEP 771 there’s a stronger likelihood that projects that feel their users are best served by having more than the strict minimum set installed installed by default, will actually accurately specify what their minimum set of dependencies are.

I understand the concern that some people may feel empowered to add default extras that pull in more dependencies than expected under the guise of “well they can always turn it off”-- but I think that concern is honestly overblown. I suspect that the set of package authors who feel that some dependency should be installed by default, but are resisting that urge because of pedantic adherence to the intent of required dependencies is likely quite small.

As far as making this opt in, I personally think that isn’t really a worthwhile change. We already have an opt-in way to add extra dependencies, it’s called extras :wink: , and that serves reasonably well for the use case of adding optional dependencies that a package author doesn’t think should be installed by default.

What we’re lacking is a mechanism for optional dependencies that a package author does think should be installed by default, and we’ve got clear evidence that because we’re lacking that mechanism, a non zero number of reputable and important packages are over specifying their “required” dependencies, to work around it.

But to go back to my first thing, I think we should, in general, assume that package authors are going to do what they think is best for their users and not assume they’re going to act with disregard towards them.

  1. Which I would argue is quite obviously true given the number of examples we have where package authors are “abusing” the “required” dependencies to pull in something that is not strictly required, but they think users should get by default. ↩︎

  2. Since there’s no mechanism to require that project authors actually list only their minimal required dependencies. ↩︎

13 Likes
121

Agreed. But importantly, the current round of discussion feels extra weird to me, because it’s discussing a proposal that isn’t actually what’s in the PEP.

While exploring other options is obviously valuable, the recent discussion started up with a completely new proposal, but with no discussion of how it compares with the existing PEP, or why a change was even needed (as far as I can tell, the previous discussion just died down because there were open questions that needed addressing, but not because there was some fatal flaw in the proposed solution).

2 Likes