As usual Phil, your comments are well-timed and insightful. Coincidentally, there were some in-depth discussions at the “WheelNext” summit, which NVIDIA and Meta organized this past week to coincide with the GTC conference. @charliermarsh and @konstin were in attendance, among many others. I have been working with @atalman on some index priority proof of concepts. @atalman put up a demonstration of index priority in pip that matches uv’s behavior. We both agree that changing pip’s behavior is too disruptive, and not likely to succeed. What I came up with instead is a generalization of Poetry’s primary and supplementary package source “Groups”.
I note that self-hosted indexes that implement virtual indexes made up of other indexes have been proposed often in this context, but usually people aren’t enthusiastic about needing to run their own servers. I know that’s not what you’re saying here, I’m only acknowledging a lot of past discussion on the topic.
I probably didn’t express this well enough. I think that maybe you are thinking of conda’s strict channel priority, where the presence of a given package name on a given channel means that the search will not look on other channels. I can see the use case for that. I also think that there is a use case in-between that and treating indexes equally. I see it as
- Maximum lock-down - for example, ONLY use IT-supported packages, except where they don’t have a supported package
- Soft fall-back - for example, prefer IT-supported packages unless they don’t meet your requirements
- Equal priority - combine all indexes at once, choose “optimal” version, regardless of where it comes from
As a user, I think the middle one is “least surprise.” Under-constrained packages will be safely preferred from the IT-approved pool. More advanced users who specify constraints will have their escape hatch if necessary.
I wholeheartedly invite you to channel your inner lumberjack/Elon and take a chainsaw to it. Just stay away from government buildings. I will say that I have learned a lot from my efforts to implement “index groups” and I hope to cut down the PEP because the idea of index groups is simpler than trying to switch entire UI’s at once.
I agree on this point completely. I don’t think that PEP 708 is sufficiently configurable for users, so while I definitely respect it, I think that PEP 766 is not duplicating that work, nor intending to compete with it, although they share significant motivation. In my mind:
- PEP 708 allows package maintainers to express their official servers that are safe, and provides a way for installers to handle the case where non-safe servers get utilized.
- PEP 766 allows users to express trust hierarchy among their sources, but does not delineate safe/not-safe servers
Indeed, each tool has different options, even if the core idea of what an index or find-links
location consists of. We’re also finding it quite hard to come up with a sane UI for this naturally hierarchical data in pip’s existing ini
format configuration. What we broadly agreed on in “meat space” was that:
- It is convenient to have one chunk of index priority configuration that users can carry between environments
- A configuration format that allows expression of multiple “things” and separate expression of orders of those “things” - opentelemetry is one example that comes to mind
I believe that this is so core to pip that any change to this behavior would be non-viable. The index group idea that I have mentioned a couple of times now preserves this behavior, while providing index priority in an orthogonal way.
@charliermarsh asked this same question at the WheelNext summit. My answer was that if we can come up with a common expression of multiple indexes and options, then the ecosystem would benefit greatly from un-splitting some of our mental load. He didn’t outright say “Yes, uv will definitely do this work,” but he did seem agreeable to the idea.
In any case, the idea is NOT to make uv’s behavior a standard, because it is NOT PEP 766’s goal to cause changes in the daily workflow of existing users. It is PEP 766’s goal to provide a new way for users to express trust hierarchies between sources, and to have their installers respect those relationships.
Historically speaking, index priority started out for me as an implementation detail for variant metadata.