Thanks @notatallshaw and @pf_moore for having a first read and sharing your initial feedback. Let me focus in this reply on the most important point (vendoring) only.
As discussed in the previous wheel variants discussion thread, “installer authors don’t want to vendor” was our starting assumption. However, the security/vendoring tradeoff was fairly extensively discussed in that thread, and @pf_moore did say the following (from this message):
Speaking personally, I’d be against including the code (in the sense of being responsible for it ourselves) but vendoring is much more plausible.
Your points all apply, but if I’m trading them off against the risks and problems around adding a mechanism whereby pip downloads and installs plugins on demand, based on package metadata, then I’d be willing to consider vendoring.
…
I don’t think that is where we want to be - I think the proposal should require specific designs from installers. I’m not talking about UI (command line flags, defaults, etc.) but I do mean everything else - should selectors be downloaded and invoked dynamically, or should they be static based on a fixed whitelist?
As mentioned in the initial post of this thread, the key takeaway from that whole 200+ post discussion was to change the previous wheel variant design from being build backend-like to “installers must not run them automatically, and should vendor the most popular providers”.
That would not have a similar benefit, because build backend are allowed to run without the user opting in to them running (i.e., pip install pkgname works), while wheel variants are not.
In addition, build backends are (at the very least) an order of magnitude more complex than variant providers. It’s also fine to set extra requirements on vendoring, such as to even be considered for vendoring a provider should be pure Python, self-contained (not have any dependencies), and pass whatever checks (linting, type checkers, security scanner, etc.) are deemed needed by installer maintainers.
Similar for vendoring effort, it’d be fine to say “the provider author has to do the work”, or even add more constraints like whoever does the vendoring work must commit to maintenance effort, be part of some vetted group (PyPA, the PEP authors, WheelNext, a special Pip team, or whatever[1]). It doesn’t have to cost the maintainers much effort.
All that said, what providers get vendored is ultimately up to the maintainers of course. I could imagine pip vendoring only 3-4 plugins (CUDA, x86-64, arm64, and maybe ROCm) while uv vendors more of them, taking a different position on the effort-vs-usability spectrum.
Your footnote is the most relevant here. I really don’t think the risk with providers is any higher, since they’re relatively small pure Python packages[2], which are going to evolve only slowly (mostly adding config values for new hardware, and bug fixes - there aren’t any features to add). And, again, if it looks too risky/complex for a particular provider, maintainers are free to reject it.
This vendoring topic is pretty critical to the whole design. Do my answers and the context of the previous discussion help change your initial takes here?
You could even suggest that if the one-off initial review effort is deemed too high, the provider authors or PEP authors may have to find the funds to pay one Pip maintainer. or a dev that the Pip team personally trusts, to check the work. ↩︎
“small” is always relative; but let’s say collectively small compared to the 250 .py files and ~90,000 LoC currently in pip’s
_vendorfolder in total ↩︎