PEP 766: handling multiple indexes (Index Priority)

[atalman (Andrey Talman)]

@charliermarsh

My personal opinion is that I’m somewhat skeptical that this should be a PEP, and, instead, that this problem should be solved in pip or uv or other tools as they see fit. I don’t know that I see significant value in standardizing the behavior, and that standardization will come at a cost (namely, that it removes room for tools to innovate or explore or impose their own design constraints on the problem).

@pf_moore

If I’m absolutely honest, I think the motivation behind trying to make index priority into a PEP is mostly to avoid having those conversations with the individual tool maintainers…

Having a formal PEP will provide good visibility and transparency to users of the resolution algorithm. Given that users can opt in to use or not to use this algorithm, tools such as uv can still innovate and provide their own solutions. However, having clear and accepted PEP will benefit projects like PyTorch to provide the best and most secure experience for the users having standard tooling such as pip.

For development teams maintaining their own index, dealing with a single standard algorithm is preferred rather than supporting multiple non-standard tools and use cases that could potentially contradict each other.

@pf_moore

Ideally, PEP 766 should discuss how it integrates with PEP 708 to provide “defense in depth” for dependency confusion attacks.

Since PEP 766 is informational and does not contain technical implementation details, I propose creating a standards track PEP to go along with PEP 766. We should be able to specify and discuss technical implementation details of PEP 766 in this new standards track PEP. The technical details should include the interaction with following PEPs:

• PEP 708 Extending the Repository API to Mitigate Dependency Confusion Attacks
• PEP 610 Recording the Direct URL Origin of installed distributions (in our case we want to record Index from which wheel came from)

Hence in this case it would make sense to close the PR 13210. Create a new standards track PEP and open a new PR based on the standards track PEP.

[emmatyping (Emma Smith)]

@charliermarsh

My personal opinion is that I’m somewhat skeptical that this should be a PEP, and, instead, that this problem should be solved in pip or uv or other tools as they see fit. I don’t know that I see significant value in standardizing the behavior, and that standardization will come at a cost (namely, that it removes room for tools to innovate or explore or impose their own design constraints on the problem).

The goal of this informational PEP is solely to establish shared verbiage and describe common concepts among potential index priority implementations. This PEP does not define any behavioral requirements. From the PEP abstract:

This PEP aims to describe each of these behaviors, which are referred to as “version priority” and “index priority” respectively, so that community discussions and troubleshooting can share a common vocabulary, and so that tools can implement predictable behavior based on these descriptions.

I think establishing common terms and concepts for index priority even without defining behavioral requirements is beneficial. We’ve received feedback from projects and index maintainers who wish to teach their users about index priority to ensure users get expected packages. The differences in tooling today is a complication that leads to confusion and makes it harder for users to know how to use non-PyPI indices, and impose additional work for projects to document best practices for their packages.

@charliermarsh

Reading through the PEP itself (and some of the discussion above), though, there’s kind of a spectrum of goals. If the PEP is mostly about defining terminology, then it seems less objectionable but also less impactful (e.g., the PEP might decide that tools should give users an ability to define index priority; but in that case, if pip implements that behavior, it could look totally different from uv’s implementation or configuration schema, etc.).

I absolutely agree just defining terms and concepts is less impactful. I think it would be much more valuable to have an additional separate PEP that standardizes behavior and configuration among tools for index priority, but such a PEP would be distinct and should be reviewed independently. I think such a PEP could require a certain behavior or configuration options to be available, but also leave room for innovation. uv today has package pinning but can also fall back to pip’s “unsafe-best-match” behavior, so multiple behaviors being available already exists. I don’t want to derail the conversation on this PEP thread however, so I would suggest a discussion of a “standards track” PEP move to either a new DPO thread or the WheelNext Google Group or Discord channel.

[msarahan (Mike Sarahan)]

@jonathandekhtiar

Absolutely Give me 1-2 weeks to come up with something “clean” - I will probably open a DPO thread to send a community wide update.

For the sake of visibility, @jonathandekhtiar posted this at [WheelNext Summit - Spring 2025] Summary