I’ve been ruminating on the PEP and the various threads for it, and I think there’s sort of a fundamental question that needs answered outside of the specifics in the PEP itself.
Determining what wheels are compatible with a given machine/environment for a given axis requires some amount of code to execute. This is categorically true, regardless of PEP 817 and “Variants” and as such, exists today on the existing axis of compatibility we have. There’s code that installers implicitly execute to determine whether a machine supports amd64 or aarch64, whether it supports Windows or macOS, whether it supports Manylinux, etc.
From a “nuts and bolts” perspective, there’s not really any inherent difference between running some code to determine CUDA version and running some code to determine glibc version.
So fundamental question is, how does an installer enumerate the axis, know what code it needs, and get that code, and do all that in a way that is performant and doesn’t risk arbitrary code execution.
For our existing axis, the answer to that is basically that the spec defines what the axis are, and leaves it up to the installers to figure out the rest [1], although typically someone invested ends up implementing the code and getting it into the major installers (either directly, or via libraries like packaging).
For the primary use cases that are driving variants, I think that they all could be handled the same way. There could be a PEP to add an axis for Nvidia GPUs, another one for adding an axis for AMD GPUs, another one for CPU versions, etc. Each of them would describe their axis, and then leave it up to the installers to figure out how to support them, though like the existing axis, I would expect someone invested to implement the code and get it into the major installers.
I think that is technically feasible, and if we choose to handle it that way, assuming those PEPs were accepted, something akin to the code that is in the various provider plugins would have to be added to each installer, just like similar code was added for manylinux, and iOS, and CPU, etc.
Which is why, fundamentally, there’s no option for an axis that doesn’t involve code that the installer executes [2].
So, if it’s technically feasible for the use cases that motivate variants to be handled like all of our existing axis we’ve introduced, then why does PEP 817 exist? Why not just do that and use the existing process for each of these new axis?
I think there’s a few answers to that question.
- I think the PEP process itself (and the manual integration of a completed PEP into all the various ecosystem tools) is seen as burdensome, heavyweight, and slow, particularly for things that are going to basically be a copy/paste of “like X, but for Y”. I think that sentiment is pretty hard to disagree with, and it was one of the motivating factors for the perennial manylinux PEP, and new hardware “things” seem to be popping up at least as quickly as the old date based manylinux was supposed to work.
- It was an attempt to remove burden from OSS developers and reduce the friction between them and the corporations [3]. For instance, many of the tooling authors have stated they don’t really know anything about GPUs, but the existing model would require them to explicitly handle them (both in terms of discussing/approving a PEP for a given GPU, but also in reviewing and maintaining the code that supports them).
- A realization that the “does a given wheel work on this machine/environment along some axis” is the underlying question behind the platform tag [4] and the motivating use cases for variants, and they are fundamentally the same (run some code to look at system, generate a list of what works on the machine, compare it to the given wheel), which means that an interface for doing this could be created, which would make the whole process easier and less manual in the future.
I think that those reasons are pretty understandable, and hard to disagree with, or at least hard to dismiss as unreasonable or “out there”.
I also think it’s entirely reasonable to say that you don’t think those reasons are enough justification to change how we’ve historically handled defining our axis!
But I don’t think it’s reasonable to suggest that adding a “expected by default” axis through the variant interface is somehow a drastically different trust relationship than adding that same axis through the previous mechanisms. [5].
This all brings us back to our fundamental question. No matter what axis we want to support (including the existing ones!), installers have to execute some code per axis to handle it, so how do we do that?
An earlier draft of this PEP had the installers auto downloading and then executing arbitrary Python packages during resolution. That would have maximally removed the friction to introducing new axis, and allowed both the installers and their end users to be fully “out of the loop” for managing their axis (as they currently are for build backends), which I think in abstract is really great!. Unfortunately, the cost of doing that was way too high– it effectively enabled arbitrary code execution at resolve time, which is a no go for both security and performance.
So someone has to be “in the loop” to decide on what axis are OK to include, which means either end users need to do that or the packaging community needs to do that [6]. For all of our existing axis, the answer has been “the PEP process defines the axis, and the installers decide what code to trust to implement it”.
Personally. I think that asking end users to decide on what axis to trust for anything remotely common is the wrong approach, both from an UX perspective and a security perspective.
As a thought experiment, imagine if pip only support the any platform tag natively, and any other platform tag required doing something like pip install cpu-selectors or passing a --allow-cpu-selector or having the end user pass around a JSON file or something. Even if you say that win32 only requires the stdlib so it’s foundational, the question still applies for manylinux.
I don’t think anyone would reasonably argue that that UX would be fine for win32 or manylinux, and I don’t think it’s reasonable for any axis that is likely to impact a large number of users if we can at all help it!
Why does the manylinux axis does not require any sort of opt in or gymnastics for end users?
Because, installers have chosen to depend on and trust some code that looks at an existing system and determines it’s manylinux support. There’s nothing inherently different from an installer choosing, on behalf of their users, to trust packaging or choosing to trust foobar-variant-provider.
What is inherently different in the variants proposal, is the notion of arbitrary axis, and I do do not think there is any reasonable mechanism by which anyone other than the end user can decide to trust an an arbitrary piece of code.
That being said, I don’t think that all of the ideas in the PEP require arbitrary axis.
- The idea of standardizing an interface for handling axis seems wholly reasonable to me.
- The idea of moving some of this data out of wheel file names an into metadata seems like a wholly reasonable idea to me [7].
I’m not yet sure how I feel about the specifics of the implementation of these ideas yet though!
Putting all of the questions of the exact shape of defining an axis aside, I think that leaves us here:
How do we decide what axis an installer is expected to handle by default? Is the existing PEP per axis defining the axis and leaving it up to the installers to figure out how to implement it the right approach? Should each installer decide on their own what axis are expected? Should we offload that decision to a central repository (either in the form of a library or a list of axis)?
Should we provide a mechanism for arbitrary axis to be supported (presumably on an opt in basis) or should we only allow axis that some central-ish body (be that the existing PEP process, the installers deciding, or a blessed library/list) have approved?
Should we standardize an interface for axis (whether we allow arbitrary axis or not) to make it easier to add new ones across the ecosystem, or is there too much overhead in doing that and manual integration is fine?
Should the “by default” axis and the “arbitrary axis” (assuming we add them) share a mechanism, or should they be independent from each other?
For me, I think that the ability to have arbitrary axis is pretty useful! If for no other reason than it allows experimentation and iteration of axis and it means that esoteric axis have a method of support that doesn’t require the ecosystem as a whole to care about supporting them.
I also think that sharing the mechanism between arbitrary and “by default” axis provides a reasonable glide path for standardization. Something like manylinux could have been an arbitrary “opt in” axis at first and could have been iterated with real world use to discover where it fell short, then as it gained popularity, it could have transparently been migrated to by a “by default” axis, reducing the risk of “getting it wrong” in a PEP, since many of our PEPs have to define something without much chance to experiment with it first in “real world” situations.
I don’t have a strong opinion on what the process should be for something to become a “by default” axis, but I am sympathetic to the idea that the PEP process is too heavyweight for it [8], and I think it’s probably perfectly fine if we offload that to some blessed library or central list with some basic guidelines for what is acceptable to become a “by default” axis.
And for all of that to work, we kind of need a defined interface that is shared between the “arbitrary” axis and the “by default” axis.
So for me, I think we should have a defined interface (whether it’s variants or not) for “axis”, allowing arbitrary axis to be distributed, but in an opt-in manner and that interface should be our preferred mechanism for new axis [9]. We should pick a mechanism for selecting “by default” (or maybe “standard” is a better word) axis, and any axis on that list should be expected to work out of the box with any compliant installer [10].
Thanks for coming to my TED talk [11]
Some of the specs call out explicit stdlib APIs as how you get them, but some of those APIs have gone away or are more generally used as an example of what that data is. ↩︎
At a conceptual level, I’m lumping schemes that require an external process to execute some code and pass that into the installer in some way, ala JSON files or CLI flags etc as still fundamentally being about the installer (or at least the install/resolve “process”) executing that code. ↩︎
Since corporations are often the organizations that are bringing new hardware into existence. ↩︎
And you could pretty easily argue the same for the Python tag and the ABI tag as well! ↩︎
Remember, the perennial manylinux PEP made no mention of how an installer should interrogate the system to determine what glibc version was present, it was up to each installer to figure out how to do that, and I believe most just implicitly trusted packaging to do that for them. ↩︎
This could mean that the installers handle it, it could mean PyPI handles it, it could mean some selected group of people handles it. Basically just some group of people deciding on the end user’s behalf. ↩︎
If we were starting over from scratch, the variant mechanism could have handled platform tags for us entirely, which could mean that instead of a wheel like foo-1.0-py3-cp38-platform1.platform2.platform3.platform4.whl, it could have just been foo-1.0-py3-cp38-$variant.whl, and then the mapping of $variant to what platforms it supports is handled in metadata.
Arguably you could even use it for the Python tag and the ABI tag too! ↩︎
Although the lack of ability to experiment and get real world experience before hand makes the PEP process ultimately more conservative, for fear of having to live with a mistake “forever”, so it’s possible that the PEP process for a “by default” axis would become more streamlined in the world I envision. ↩︎
And if it works, maybe even try to transition previous axis? ↩︎
And the installer could pick whether they wanted to implement that “by default” by vendoring the "arbitrary” provider, or by having an allow list of “arbitrary” providers that they’ll use in an opt-out fashion, or some other, completely different mechanism if they so desire. ↩︎
Some day I’ll figure out how to say things with fewer words, but not today. ↩︎