Note: this is a comment on behalf of several PEP authors who are most active in this review process this week, to summarize our common position on the security/vendoring/UX discussion.
One of the key goals of our PEP is to make pip install torch, uv pip install torch, pip install vllm, etc. “just work” for prominent libraries that take advantage of modern hardware. And yes, that includes other installers too - there’s just no good shorthand for <installer-name> <default-install-command> <package-name>.
UX is critically important (nod to Warsaw’s fourth law, nice @barry) in our opinion. We want the default install experience to be selecting the optimal wheels for the system/environment it’s running on. We know it’s possible after spending a lot of time[1] iterating on design and prototypes, and the tradeoffs seem acceptable after careful consideration. They may turn out to require changes to the PEP (e.g., use of a curated allowlist, a new packaging-like central library, or drop or modify MUST/SHOULD working) to be able to come to a consensus, but we’re convinced it’s the right goal. As PEP authors, we therefore choose a design that achieves that key UX goal[2]. We are not ready to compromise on that goal right now based on feedback received so far from a relatively small number of voices, while from our design work and a lot of experimentation[3] we still believe that the current design is feasible and the best approach presented so far.
@oscarbenjamin thank you for the thoughtful posts. We wanted to specifically address this one point. It’s possible you are right here, but we think it’s too early to lower our ambition level or change the design again. We also think that explicitly aiming for the intended end state rather for some intermediate rollout state is the right thing to do. We can always do what you suggest later, if necessary.
End user opt-in design suggestions
A number of suggestions around letting users opt into using variant wheels have been made[4]. These do not achieve the goal of making default install commands “just work”. Therefore, we will put these suggestions as a single “end user opt-in designs” into the Rejected Alternatives section of the PEP, with as clearly expressed reasoning as we can[5], in the next PEP update. It’s always possible to revisit that decision in the future, however for now that’s our stance. And as PEP authors, that is our choice to make. We’ll take new input after that next update, possibly in a dedicated thread.
Next steps on security/vendoring/UX
We will take some actions that have been suggested already in the thread:
- Reach out to other installer tool authors to get more input (@konstin already did for PDM, Poetry, Hatch and Pip),
- Update the PEP with some diagrams and worked examples to make it easier to understand what is actually proposed
- Update the PEP with sections “expected changes from PyPI and index servers”, “expected changes from install tools”, “expected changes from build backends”.[6]
Reviewing other important parts of the design
We’d like to move on to reviewing other parts of the design. There are a lot of other parts to the design that are important, would greatly benefit from community review, and are orthogonal to the security/vendoring/UX choices.
We’d like to review and get agreement on the general mechanisms of the PEP such as:
- the PEP 517-style variant provider interface
- the priorities and sorting logic for variants
- the Ahead-of-Time (AoT) providers concept
- the expressibility of wheel properties: whether we capture all platform combinations without being too expressive for resolvers
- the
abi_dependencyspecial namespace
Any feedback on those topics would be much appreciated.
We also realize that fast-moving DPO threads aren’t ideal for everyone. We’d welcome feedback sent to us directly, or shared on other channels like the PyPA Discord. We appreciate some comments we already got outside of DPO, like some small mistakes we should fix and a question on filename length.
Hard to be exact, but on the order of 2 person-years worth of effort ↩︎
Arguments for better UX being better for security - or conversely “compromising UX is compromising security” - can also be made, because people will default to doing insecure stuff (like enabling all providers). ↩︎
All done in the open, including prototype implementations, the prototype index and more specific discussions on an array of topics - there’s a lot of research we have done, much of which doesn’t fit in the (already longest) PEP. ↩︎
Some abstract ones, and some concrete like
$ [uv] pip install --enable-providers torch↩︎A number of arguments have already been given, both in this thread and the previous one. ↩︎
TBD where exactly. This PEP is already the longest PEP ever it seems, so we may have to use more appendices or separate docs to keep the content navigable ↩︎