All of these “ooh, I’d use this!” posts are just reaffirming my belief that this feature is going to be overused and exclusively misused, leading to caveat-ridden, bloated and buggy installations to be seen as a features rather than a design problem [1].
For all those who aren’t opposed to this change, please just grab a Docker container for a Linux distribution that has already supports some variation of this, try setting up some applications or developer environments (or anything!) in those containers and really get to experience what you’re asking for:
Take Debian/Ubuntu or Fedora (which have recommended dependencies) with and without the --no-install-recommends/--setopt=install_weak_deps=False flag to apt-get install/dnf install:
- Notice how the bloat factor starts at about double and grows with the depth of the dependency tree. Even though only a small ratio of packages use recommendations, that ratio compounds itself for each layer of the dependency tree giving much higher cumulative bloat ratios than you’d imagine.
- See how tenuous some of the recommendations are. e.g.
fedpkgrecommendspodmanandqemu(+368MB!) even though they’re both at the bottom of the list of recommended virtualisation backends. - See if you can find a shred of used functionality lost by dropping the recommended dependencies (in years of doing exactly this, I never have).
- Look at the delta of minimal vs full package name lists and see if you can figure out why the recommended ones are even there. Notice how the bulk of them come from dependencies of dependencies and that such non-top-level recommendations quickly cease to make any sense.
A post PEP-711 world will be much worse than the above because we have no single global --no-install-recommends opt-out. It’ll also be much worse in Python since Python has so many more ways to store or pass around or track or install lists of packages in which the opt-out can get lost [2]. (And also because I suspect recommendations will be (over)used more heavily by Python packagers based on other people’s comments here and from the ways extras are already overused.)
Or look at Void Linux or OpenSUSE (which support interchangeable dependencies):
- Observe how quickly things break on Void Linux because
xbpsthinks two packages are interchangeable when one’s really not due to lack of large matrix testing. - Rewind to about 6 months ago to when OpenSUSE Tumbleweed’s default [3] build system was incapable of building hello world for over a year because it couldn’t handle the ambiguity of interchangeable dependencies!
This feature will be much worse in Python because we have more tooling that needs to adjust and because the Python community as a whole already massively underestimates the need for [4] top level packaging-like testing (look at how many projects still don’t even run a pip install . without the -e flag in CI then are surprised when they publish broken wheels with missing files).
And interchangeable dependencies take so much more environment-level testing than people realise. Take the Qt example already in the PEP:
- To do this properly, you need to test all backends in isolation plus multiple backends including and excluding the default (4+1+1 test jobs/separate venv environments so far).
- You’ll quickly discover that people want some way [5] to force a preference so now you need to cover the cases default explicitly specified, non-default specified, non-installed variant specified (+3 = 9 test jobs now).
- But you’re not done yet. Not many people know that importing more than one Qt variant into a single session can, on Windows, cause one variant to load the DLLs from the other leading to anything up to and including a segfault so you’ll need to do some
"PyQt6" in sys.modulestrickery to protect users from that [6] and test that too (at least +1 test job). - But you’re still not done! You’ve probably been naively using
try: import PyQt6; except ImportError: try the next onebut sincepip install package[PySide2]makes it possible to circumvent thePyQt6 >= minimum_supported_versionconstraints you (hopefully) put under the default extra [7] so you actually want a whole load ofimportlib.metadata.version("PyQt6")comparison checks and a for loop over available backends to find the one that was actually installed with your package rather than some stale version of a preferred backend that was already there. (+1 test for the for loop and +1 test for the explicitly specified backend being too old. 12 test jobs in total and we haven’t even got round to multiplying things by 5 Python versions and 3 OSs!)
And don’t get me started on what it’s like to triage a bug report with such a package involved. It’s already impossible to get a reproducible example out of a confused user with a matplotlib.backends issue who isn’t fluent in venv. [8] [9] This PEP would make that the norm. 
WayTooLong/Didn’tRead [10]: Every ecosystem I know of that has some form of this feature is much worse off for it and a simple comparison shows clear reasons why this proposed Python variant will leave us even worse off than those other prior arts. So to anyone who has explored other ecosystems’ variations of this feature, please show me how this proposal is supposed to do better than those existing ones that are already detriment their corresponding Linux distributions. And to anyone who hasn’t, I honestly don’t believe you have any idea what you’re asking for. It’s certainly not going to be the well measured world the PEP alludes to (and like I say above, the compounding nature of dependency trees makes the definition of “many default dependencies” much smaller than you’d think).
usually a package that should really be two packages ↩︎
or possibly not even be supported! ↩︎
weirdly not the one they use in production ↩︎
and generally isn’t very good at ↩︎
if you’re really lucky, they might even stop at one way! ↩︎
although you can’t do anything to prevent the user importing a variant after you’ve incorrectly guessed the one they want so you should still expect to get segfaults reported to the issue tracker of any vaguely related collateral package ↩︎
Here I want to write a preemptive apology letter to whichever poor soul has to figure out and implement whatever
pip checkwill even mean in this scenario ↩︎even a pip freeze is inadequate since half their environment is usually conda packages squashing pip packages squashing conda packages ↩︎
well, short of asking them to clone their hard-drive and FedEx it to me ↩︎
Sorry it got so long. I tried to not write this at but this whole proposal has been eating at me for months ever since I saw the original setuptools issue. ↩︎