The current meaning is not totally clear cut, though, because one way to handle ‘recommended’ dependencies is just to list them as regular dependencies. I’m thinking of a package I work on for work where really essential dependencies (numpy, h5py) are mixed with things you could miss out and still use core functionality (xarray, matplotlib). In our case these not-quite-required dependencies are likely to be installed anyway, so the cost of treating them as full dependencies is low.
If projects have a way to express recommended dependencies, some dependencies that are currently hard requirements might also be made optional. E.g. matplotlib currently depends on python-dateutil, presumably to support date labels on axes; it’s not hard to imagine this could be made optional for a minimal matplotlib install (although I’m not saying the matplotlib maintainers would do that).
Catching up on this discussion after a few weeks, something that stood out was that we’re making a potentially flawed assumption in thinking that “minimal dependencies” and “recommended dependencies” is a sufficient distinction to be able to make (both on the part of project publishers and on the part of end users).
Instead, we’re kinda using that as a proxy for what I’ll call “installation profiles” (for lack of an existing term).
Some potential installation profiles:
web service production deployment server (minimal is good!)
local web service development (gimme all my dev and debug tools thanks)
novice learning data science (please set me up to run all the examples)
production data analysis server (minimal is good!)
running a CLI interactively (gimme the dependencies that make that pretty)
running a CLI headless (no need to be pretty, it’s just going to a log file)
The “requires at least one of …” alternative dependencies then complicate matters even further.
Thus the discussion ends up going round in circles, as “minimal dependencies” and “recommended dependencies” can’t really be meaningfully defined in the absence of a clear understanding of how the installed project is going to be used.
Extras don’t solve this problem as:
Transitive declarations are controlled by the projects specifying them rather than by the target environment; and
Their names aren’t standardised, so tools can’t readily provide a way to specify minimal installs of even transitively included dependencies
This issue also kinda came up in the pylock.toml discussions, in the sense that the final format included the extras and dependency-groups metadata, so a tool can build its own notion of installation profiles by deciding which extras and dependency groups to install from a lockfile.
So whether the spelling is package-core vs package (status quo) or package[] vspackage (PEP 771), the dependency declaration itself feels like the wrong place to be specifying whether we’re trying to minimise the installed dependency set or maximise the available functionality (for a given use case).
I acknowledge the limitations of extras that you’re describing, but I also think they generally work well enough, and I’d like to get a solution to the specific problem - that you can’t separate the minimal install profile and the default - without it getting lost in a larger effort to design something totally new.
In general - and I think this covers all 3 of your examples - the packages where you care about installation profiles are packages that you’re interacting with directly, and therefore probably installed (or specified a dependency on) explicitly. If foo gets installed implicitly as a dependency of bar, then most likely whatever flavour bar specified is going to be OK for the end user.
I’d happily support an effort to standardise the name of a default or minimal extra, but neither is feasible without some changes to standards and/or tools:
A default extra doesn’t work without a way to tell tools not to use it
A minimal extra is impossible, because extras can only add dependencies, not subtract them
I like the idea of thinking of this in terms of installation profiles, but I also think that designing something completely new to solve this is likely going to be a difficult task, and extras might be the best programatic solution to this. I don’t think there’s going to be any sane way to formalize all the types of installation profiles, though even if we did, this could always be by standardizing extras names rather than designing a new system.
The only downside of the way extras are designed currently is that the default installation has to also be the minimal installation if you want to cover all possible installation profiles, but this is incompatible with the ideal goal that the default installation profile should be the most common one. PEP 771 would then unlock the ability to basically use extras as installation profiles and would standardize the syntax for the minimal ones. In future, one could then envisage standardizing at least some extras names for other common types of profiles.
I’ve tried looking around both the PEP content and this thread (and the previous) and I’ve not seen anything though I may have missed it: how does PEP 771 interact with the compatibility tags of the default extras?
For explicit extras it makes sense that you’d just get an error if you request a package which doesn’t have anything suitable for your platform, but for default extras if the default extra is a native wheel with compatibility tags, and none fits the current system, would the extra:
be ignored (either silently or with a warning), so the package gets installed just without the default extra(s)
cause an explicit error, and the default extra would have to be removed from the dependency (possibly using environment markers to match platform and extras configuration)