PEP 817 - Wheel Variants: Beyond Platform Tags

Packaging
261

I’m not sure if it would make it easier or not. They already have to support running some sort of “plugin” like Python code for build backends (unless they don’t support sdists at all), so that general functionality has to exist already.

I’m not sure if reimplementing every axes is easier or harder than re-using something like that?

I think this is what I mentioned here (emphasis added), but no– I think if there’s any kind of plugin system it has to be opt-in in as the general case, and only the axes that we deem “big” or “important” enough get special treatment to make them available by default somehow (varying options for doing that, including not doing it at all at the ecosystem level and letting installers decide).

Sorry if that wasn’t clear!

I don’t believe so, and I think that’s off the table anyways :slight_smile: Right now I’m trying to figure out if we think providing the ability to support non-mainstream hardware is something we think is important enough to have a mechanism for it.

3 Likes
262
  • Pro: It’s the status quo. It’s familiar, and tried and tested.
  • Pro: Having the axes defined in the spec increases community confidence in what’s being proposed.
  • ???: Existing axes can all be defined relatively easily, and implemented in a general library (typically packaging). The proposed axes seem to be a lot less clearly defined, and involve significantly more complex code (even the core ones like GPU/CPU detection).
  • Con: Plugins which run in the execution context of the tool can interact badly with a tool that wasn’t built with plugins in mind from the start[1]
  • Con: Our only real-life example of a plugin mechanism in packaging (build backend hooks) involves significant complexity, including such aspects as environment isolation, exception handling.
  • Con: Introduces 3rd party code execution into a part of the installation process that is currently statically specified.

It may be that people have different ideas in mind for what a “Plugin system” actually is. With my experience trying to come up with a plugin system for pip, I’m probably over-cautious[2]. But without a concrete proposal as to how a plugin system woudl work, it’s hard to be specific.

Here’s some examples of concerns we’ve had with possible plugin systems for pip in the past:

  1. Plugins cannot change process state, like the CWD. Nor can they rely on the process state.
  2. Plugins must not touch the Python logging system, as pip relies on it and assumes that it is the sole owner of the logging stack.
  3. Plugins which use threads cannot touch global values, even with locks, because there is no assurance that pip will lock access (pip as an application is currently single-threaded).
  4. If plugins depend on non-stdlib libraries, things get complex. How will that interact with pip’s vendored libraries? Having two copies of (say) packaging loaded in the same runtime is potentially a significant problem. And given that the list of libraries that pip vendors is an implementation detail, that means the problem could exist for any library.

Can the plugin system envisaged for PEP 817 work under those constraints?

IMO, the question of whether we should have a plugin-based approach is a cost/benefit question. The benefits are relatively easy to explain (although maybe not as easy to quantify). But the costs are much harder to express while we’re talking in the abstract. Without a concrete plugin design, we’re all working on gut feel for the costs, and that means there’s little chance of an objective decision.

I’ll also note that we did have a clear design for the build backend plugin system, and even with that, we still (IMO) severely underestimated the complexity (i.e., cost) of the approach. How do we avoid doing that again? Of course, it’s also true to say that build backends were a clear success, so we shouldn’t dismiss the possibility that a well-designed and tightly scoped plugin system might be the right answer here as well.

It’s also important to remember that the spec currently isn’t just about hardware (mainstream or otherwise). Use cases like having particular libraries (Python or not) installed have been mentioned. And in theory, plugins could do anything. Arguments that focus on hardware detection sound a lot more acceptable, but then someone comes along and says “yes, but what about a plugin that does (something that’s clearly not just probing a bit of hardware)?” and people get concerned again.

IMO, if the PEP clearly limited what plugins are for, even if that’s not enforceable in practice, people might be less concerned. Social pressure and enforcement by convention are very real, and being able to say “that’s not what plugins are for, see here’s where the spec says so” is more powerful than many people realise.

To go back to build backends again, a build backend can put anything it likes in a wheel. It’s only social convention that ensures that backends build a binary from the source provided.

Like you, I’m exploring the arguments on either side here. And I feel like this particular point is leading me to the conclusion that we have two completely different questions here, a social one and a technical one. My instinct is that I’m against plugins for technical reasons, but it’s hard to have that discussion because the social question is overwhelming the conversation. And the right solution to the social question is just to present the idea of a plugin-based system better.

We never had the sorts of debates we’re having now over build backends. Everyone knew what the build backend mechanism was for, and people were comfortable that it would just be used for that purpose[3]. But this PEP feels like it’s over-emphasising the idea that variant plugins can be used for anything, and while that’s almost certainly just over-enthusiastic rhetoric, it’s colouring people’s reactions in a way that’s not helpful.

Can we put that genie back in the bottle? I don’t know. I hope so, because I’d much rather debate the technical questions. But it’s not going to be me that can redirect the debate here.

  1. This is the key reason why pip has always resisted adding a plugin mechanism - we rely on being fully in control of, and the only user of, things like the logging system, exception handlers, and process state ↩︎

  2. Although it’s hard to see “experience with precisely the application that would be affected by the PEP” as implying excessive caution… ↩︎

  3. At least in the sense that backends that didn’t do so were clearly malicious, and we were fine with dealing with that like with any other malicious code ↩︎

3 Likes
263

Well I would also say build backends were replacing setup.py invocations, so it’s hard to imagine how it would be possible to make something worse than the status quo for that use case :sweat_smile:

This is going in the opposite direction, adding “external code”, so I think it’s fair to be concerned!

5 Likes
264

There seem to be a lot of questions on why we designed the proposal with variants and providers, so let me talk about the history of this PEP.

The initial problem is that we want to support GPUs. It’s one of the biggest pain points in Python packaging right, many of our users mention this to us, and it’s also a pain massively felt on the side of those building GPU wheels. This initiative was started by nvidia folks, who reached out to other interested parties. That’s how I came to work the proposal, seeing that someone was building a solution to our biggest user pain. The initiative now includes people from torch, Nvidia, Intel, AMD, Quantsight, Astral, Linux distros and work from many more groups and projects.

The first approach would be extending the wheel filename with another segment, torch-2.10.0-cp314-cp314-manylinux_2_28_x86_64.whl becomes torch-2.10.0-cp314-cp314-manylinux_2_28_x86_64_cuda126.whl. This has some problems: The big one is that the rules around GPU support are way more complex, and change much more frequently. For example, the wheel is built against specific SM, do we need to track this as torch-2.10.0-cp314-cp314-manylinux_2_28_x86_64_cuda126_sm60_sm70_sm80_sm90.whl? Do we need a new PEP when the next CUDA major version changes its compatibility rules? What about TPUs? It also doesn’t handle CPU levels/extensions, and it requires adding each new GPU vendor separately, which unlike CPUs (which have one compact arch tag generally) all have different rules. There was also some concern about wheel filenames become even longer and unwieldy.

The alternative is to introduce variant label: You say torch-2.10.0-cp314-cp314-manylinux_2_28_x86_64-cuda126.whl and add a metadata file explaining what cuda126 means, which SM it entails and how to match it. This metadata file are the variants.json (both in-wheel and on the index) in the PEP. This file could say e.g. that you need to detect the CUDA and the x86_64 level to select the best wheel.

In the wheel filename, Python implementation, CPU and OS are all available through Python std ABIs (with the exception of manylinux). GPUs on the other hand are not available through std, so we have to define something separately, and use non-std code for it [1].

A variant gets mapped to one or more providers, which explains what you need to detect. Each provider has code for running this detection, and return the result in the shared property format. For the main providers that we want to support out of the box, there will be library [2] that you can call directly to get the properties to provide on out-of-the-box working install experience for GPU enabled packages. The PEP specifies a PEP 517-like interface to install arbitrary providers, so that different providers can be used without redefining an interface each time. But that’s something that’s not crucial, we could also do with just specifying that these providers exist and that they return a list of properties, without putting a universal interface in the PEP or tools.

The goal is that tool authors can use this provider code (through a sharedly maintained library) without having to understand or maintain something as complex as CUDA. In the current version of the PEP, we are very abstract just talking about providers, vendoring and not about governance. I consider this partially my fault, because I pushed for keeping these abstract to not get bogged down in the governance of such a library, or different tools preferring regular dependencies, vendoring or allowlists respectively. We got the feedback from multiple packaging tooling maintainers that they want to have a collection of providers in variantlib [3], so we’ll update the PEP to specify a variantlib with a Python interface to the default providers (I think Michał has an upcoming post about what specific format we’re aiming for, using “variantlib” here as it’s the evident option). variantlib will be a place of shared maintainership between tool authors, library authors and hardware vendors (who all contributed to this PEP and the prototype providers that come with it). To stay with my favourite example, I don’t think most packaging tooling maintainers don’t understand how manylinux/musllinux detection works internally, and they don’t need to: There’s a library that does all the wicked ctypes stuff, and you call that. Personally, I’ve been thinking about pypa/packaging and manylinux as reference examples where we already have community process for doing updates around a core defined by PEPs.

Coming back to the motivation for variants and providers, after adding a variants file and providers, we realized that this solves more problems than just GPUs and maybe CPU extensions. With variants, we can finally encode BLAS and OpenMP libraries (something numpy currently struggles with) and it looks like we can even encode ABI compatibility (which allows projects such as vllm publish wheels). For me, the amount of additional things variants enable sway me strongly towards variants over a fixed label extension.

  1. I don’t think putting GPU detection into std is a viable alternative, assuming the CPython devs want it at all ↩︎

  2. Assuming variantlib will be providing this functionality as a Python library for now, I think Michał has an upcoming post about that question ↩︎

  3. The other suggestion was packaging, either works for me ↩︎

2 Likes
265

Circling back a bit, I’d like to clarify that we are planning to continue maintaining variantlib as the utility library long-term (possibly moving it to pypa/ organization once the PEP is approved) and providing the functions requested by users. That said, some functions rely on installer-level integration too: for example, we don’t think it’s a good idea to reinvent isolated environments in variantlib (especially that it would effectively mean pip → variantlib → pip nested call chain), and in our demo implementation variantlib just provides the dependency list and the API to query providers but the BuildEnvironment from pip is used to set the isolated environment up.

@pf_moore, provided that installer authors won’t be expected to maintain “default on” providers and we’d be maintaining and distributing it, could you tell us which of the following options would be preferred by pip:

  1. An allowlist in a form of a lock file with pinned provider package versions and their dependencies. pip would presumably use that list to restrict which packages can be installed into the isolated environment.
  2. A repository of vendored provider packages along with their vendored dependencies. Presumably you’d either vendor or dynamically download and reuse the whole thing.
  3. Something else?
266

This caught my attention. Could you expand a little more on what the implications are of the fact that rules change much more frequently?

For example, if I publish a wheel today that works on a particular set of GPU configurations, and tomorrow a new GPU configuration comes along, how will “the system” know if my code will work with the new configuration? For a start, what are the backward compatibility policies of GPU manufacturers?

Taking a conservative stance, if we assume that my code doesn’t work with the new configuration, does that mean I need to pin the version of the variant provider that I use, so that I don’t pick up false positives from new architectures I haven’t tested against?

Alternatively, taking a more optimistic view, if I assume that my code will work with future, “similar” configurations, do I just use the tool-provided version of the provider, and tell my users to upgrade their tools to get support for new GPU configurations? What if my assumption turns out to be wrong?

Maybe I’m misunderstanding the whole process, but it seems to me that “variants will potentially change too fast to be handled by a manylinux-style solution” seems to be a recurring theme here, and I’m realising that I don’t have any feel for what that actually means in practice. I’m not asking for stats on how often GPU manufacturers release new models, I’m asking what I, as an open source developer of some code that depends on GPU configuration, would be doing in my build process. Would I be deplying new wheels for new configurations weekly? Or monthly? Would every new configuration necessitate additions to my test matrix? How would I even test against the multitude of GPU configurations that exist? Or is this problem so big that the only people who can do it (and hence the only people who will need wheel variants) are large corporations with significant development resources to throw at the problem?

Sorry if all this sems obvious to the PEP authors, but I’m genuinely confused here.

1 Like
267

With the spec approach, I believe that the process would be roughly:

  1. Performing research to establish how the axis should work. I suppose this will also involve some preliminary testing, involving patched local versions of various tools.
  2. Writing the PEP. Hopefully this will be relatively easy, given the existence of prior PEPs doing roughly the same thing.
  3. Getting the PEP approved.
  4. Getting tools to implement the PEP. Presumably, we’d be relying on packaging providing the initial implementation, and tools like pip will use it from there.
  5. Deploying once everything is ready.

The way I see it, the potential problems with this approach are:

  • Any new axis will take a few months to deploy. Maybe that’s not a bad thing.
  • Every time we’d be hacking around stuff to test, and if we get things wrong, things go bad.
  • Do we actually commit to approve any new PEP regarding a new axis? Or are we assuming that there will be some future arbitrary criteria for deciding whether an axis is “important enough” to be approved?
  • And, do we actually commit to implement all the approved axis PEPs? Or are we eventually going to end up with a situation where every installer supports a selected subset of axes?

The key reason we went for the plugin approach is avoiding precisely these problems. That is:

  1. We provide a plugin system that can be used to start testing a new axis immediately, with relatively little effort, and get things right from the start.
  2. Once a plugin is ready and the authors believe it justified for it to be “default on”, it’s a matter of approving it. Of course, every approved plugin means future maintenance effort to monitor updates.
  3. Even if the plugin is rejected from being “default on”, it still can be used in an opt-in manner.

The way I see it, this makes testing easier, deployment faster and in the end reduces the work imposed on maintainers of packaging (who would have to maintain all the axes detection code) and tools in general (since they just need to implement the PEP once). But more importantly, it prevents us from ending up in a corner where we have already approved a bunch of axes, and new ones keep being requested.

1 Like
268

I’m not sure I have enough information to answer that question yet.

My gut feeling is that I’d want to do something similar to packaging, and just vendor variantlib, keeping it up to date whenever we release a new version of pip. I realise that doesn’t answer your question directly, but I’m still not clear on the relationship between variantlib and providers - for example, I don’t see why variantlib couldn’t itself vendor the “default on” providers rather than pip needing to do so. My understanding is that variantlib manages the interaction between pip and the providers, so how would that happen if pip vendors providers directly (variantlib wouldn’t know to find the NVIDIA provider at pip._vendor.nvidida_provider, for example)?

Also, in the light of my other question about rate of change, if key providers are changing as fast as people seem to be suggesting, how often will new releases of variantlib (or its list of “always on” providers) be expected to happen?

Sorry if this isn’t as explicit as you’d like - I’m still struggling with a lot of the details here, and this post is mostly just initial thoughts.

Also, this caught my eye. Are we now requiring that tools use an isolated environment for calling variant providers? I know it’s been discussed, but I haven’t seen a consensus yet that it’s necessary, and that requirement will be imposed on all tools. Right now, I don’t know whether pip would consider the overheads of an isolated environment to call variant plugins to be too great to justify it, and I don’t want to commit to that approach unless it’s a mandatory feature of the PEP (it isn’t a mandatory feature for build backends, and we offer both isolated and non-isolated builds - that’s quite a lot of complexity, but for build processes we consider it worthwhile).

269

There’s none yet, and I’m trying to establish what the best approach would be. That said, I’d rather keep variantlib separate as an utility library, and have another package/repository specifically focused on providers.

Being an utility library, variantlib can provide the logic to:

  • process the JSON files and extract interesting data from them
  • query provider packages via a Python subprocess (so it’s suitable for isolated environments)

In the demo implementation, pip is responsible for creating the isolated BuildEnvironment and installing packages there. I suppose the most natural way to integrate into that would be to having an allowlist in a pypa/… repository: pip would periodically fetch/update it, and use it to limit what can be installed inside the isolated environment.

If you preferred a vendoring approach, we can adjust to that, and figure out the API then. Either way, the main point is that variantlib is an utility library and we aim to adjust it to provide what installers need, rather than imposing installer architecture from it.

I’d say we recommend it, but this is not something that should be enforced. For example, build backends definitely don’t need another isolated environment. However, if you intend to install providers as Python packages, I don’t see any other solution; after all, you don’t want to start installing packages into the system mid-resolution. If you aren’t installing anything, I don’t think there’s a strict need for an isolated environment; uv definitely won’t be creating an isolated environment for plugins it reimplements in Rust.

In the end, we deliberately reused the PEP 517 approach, so that the logic already present for PEP 517 support can be reused to support variant providers.

270

I’d argue that you should provide both, because you can’t be sure what tools might prefer (pip won’t be the only tool using variantlib).

I can’t say which pip would prefer right now. There are tradeoffs.

  • Installing in each environment is expensive - particularly if providers are large, something we can’t control (we can all hope that the nvidia provider gets trimmed down to the essentials, but we can’t be sure that will happen).
  • Installing in each environment means passing the user’s config (network proxy & credentials, custom index configuration, etc.) to the copy of pip doing that install.
  • Installing in each environment means requiring security-conscious users to add the right providers to their curated local copy of PyPI (assuming they don’t allow direct access to PyPI).
  • Vendoring means either no isolation, or injecting pip into the environment (where it isn’t technically needed)
  • Vendoring requires somehow ensuring that the providers are imported as pip._vendor.xxx rather than as xxx.
  • Vendoring requires working out how we support distributors who devendor pip[1]. We don’t technically support devendoring, but I doubt that having variants not supported in Linux distros that devendor will be a popular option :slightly_frowning_face:

To be honest, neither approach feels particularly comfortable to me…

There’s always sys.path manipulation. I’m not saying it’s a good solution (it’s not!) but it’s an alternative. Isolation of library code from other parts of the running environment is a hard problem in Python, and honestly, I think that designs which avoid the need for it are significantly cleaner and more maintainable than anything that requires it.

That’s fair, but a lot of that logic is in pyproject-hooks, and unless variantlib offers an equivalent for handling variant plugins, tools will have to reimplement it.

  1. Which, to be fair, might be as simple as “it’s no worse than any other aspect of devendoring”. ↩︎

271

Well, we kinda can control that by setting specific policies and working with people who are going to want their plugins on by default.

Isn’t this already implemented though, for PEP 517 support?

272

… maybe? It’s an ongoing maintenance issue, and we still get bug reports where interactions that we haven’t considerd get reported. I’d call it more of a work in progress than a solved problem.

And of course, other tools supporting variants would need to implement all of this complexity, including ways for the user to specify the relevant pip options, as well (just invoking pip install in a subprocess would fail in any situation where proxies or custom indexes are involved, for example).

I’m not saying it’s impossible, but I do think people are underestimating the complexity involved.

273

I’m fine with generalizing the mechanism on hardware so long as it is opt in for anything not part of whatever centralized defaults happen.

I don’t think that’s really what the questions have been about. The questions I’ve had, and how I’ve read the questions others have had is a lot more about why the scope is so large for this, why so much of the behavior is left up to tools when a goal of the pep is uniform tool behavior, and if there are better options by not trying to use the same mechanism for 2 different kinds of behavior libraries want to be able to switch on.

This is actually what kills any chance of me feeling okay about this proposal. We shouldn’t need plugins to express things about python packages, this information can all be encoded statically, by packages “I require one of these” + “if you have this one, use this variant” and “I provide this one of these”

I don’t see a way to run a plugin in the environment being currently modified and that needs to be queried only after dependencies of the package are installed to function which doesn’t open this up to changing the execution scope of install commands. It’s simply not a defensible boundary with how python works, everything from lazy loaded resource, path manipulation, or directly modifying sys.modules is all possible from a plugin, and it’s all too possible for a dependency to have modified something here.

It can also be handled in ways that don’t need dynamic behavior that needs to run a plugin in a mutable environment to function, such as python packages (including potentially variants of that in this case) just having a declarative field for which abis they provide, and which abis they require, and an ordered preference for when another dependency hasn’t already constrained the solution.

Hardware detection on the other hand on the other hand should be simple to run as an isolated process, and possible to even restrict things like the capabilities that process is run with, because they don’t have those interactions. It’s easier to envision the path that ensures tools have clearly defined security boundaries, and any plugin should be suitably happy being installed into a dedicated single environment for use by the installer.

And likewise, hardware isn’t something that can be statically specified as being provided by one of the packages in the environment, but also installing python packages won’t change the available hardware.

Yes, and it’s worth extra highlighting that this was in the build stage. I not only don’t expect build tools to be available in a production environment, but consider them being available a liability.

Wheels separating out these stages was excellent for people trying to create chains of trust and reproducible builds and environments; I think by trying to take the extensible mechanism created for hardware detection, and then have it also care about the environment to shoehorn in a couple more cases, it’s creating outcomes that people aren’t going to be able to agree upon.

274

AFAIK PEP 817 does not require nor suggest running a provider in the Python environment being modified. It’s expected these will be installed in an isolated way such as build backends do. I believe the PEP allows an installer to choose not to isolate it (I could be wrong), but that would be a choice for the installer.

275

PDM author here.

Glad to see this move to support that.

In PDM, the component responsible for finding and matching wheels is unearth, so there is no vendor issue like with pip and uv, and we can directly depend on variantslib/providers.

4 Likes
276

Can you explain how one that is meant to detect things like blas implementation of already installed libraries would work if that’s not required?

277

I was thinking about this same question, but why do we even need to detect installed packages like this? Surely the existing dependency mechanisms can be used to allow a wheel to depend on a particular library (bundled as a Python package, which is what we’re talking about here)?

I can understand plugins that detect whether an OS-managed shared library is installed, but that wouldn’t need to inspect the target Python environment.

I guess I’m still confused about the BLAS use case :slightly_frowning_face:

278

Re BLAS, here are some answers and points that will hopefully help:

  1. Indeed, we do not need to detect installed packages. The BLAS/LAPACK variant provider is an “ahead-of-time provider”, so no code runs at install time. As mentioned in both this section of the PEP and in the README of the provider.
  2. BLAS can be linked/vendored in several ways, as normal for wheels. Copied from what I said higher up: no isolation is broken, one can still vendor a BLAS library the same way, or depend on a BLAS-in-a-wheel package.
    • There’s a third way, linking to system libraries, but that of course comes with caveats depending on OS and whether system location can be relied on. For Accelerate on macOS that is the case, and existing numpy/scipy wheels do link against the OS-provided library. Which is why their macosx_14_0 wheels are so much smaller: they don’t contain a vendored OpenBLAS.
  3. In pep_817_wheel_variants#80 we thought pretty hard about which numpy/scipy/scikit-learn variant wheels to build, we picked the ones we thought we’d most likely wanted to actually ship on PyPI (reasons why are given there). They are “link against MKL in a wheel”.

I do want to emphasize that (a) this does not require running code code at install time, (b) there are variant wheels you can try[1], and (c) the use case is important enough that numpy and scipy are already shipping duplicate sets of wheels for macOS on PyPI (with a hack to make that work[2]), and there’s broader interest from other packages[3].

  1. Install with the variant-enabled uv, or download and unpack to look at metadata, wheel contents, etc. ↩︎

  2. Using different platform tags, e.g., macosx_11_0 has OpenBLAS bundled in, macosx_14_0 links against system Accelerate. We’d love to get rid of the hack, and make those wheels selectable by the user/developer. And add one set of Linux x86-64 MKL wheels probably, as explained above. ↩︎

  3. Well beyond PyTorch, JAX, et al. Example of another popular package that hasn’t been discussed here at all: spaCy now maintains its own BLAS wheels: cython-blis. You can really get 3-4 copies of BLAS in a single environment today, as explained in pypackaging-native - not good for performance and reliability!) ↩︎

279

@pf_moore if i can add to what Ralf explained in great details because it personally took me some time to understand. So let me take a different perspective - something that I would understand more easily myself.

Think about it as “flavors” of the package. Variants allows you to ship all of them with no plugin needed at install time at all.

The trick is that you & everybody else always get the same variant unless you statically ask for something else.

In cheese shop words :grinning_face: unless you ask for something else you get cheddar - though as a French i deeply care about my blue cheese.

This “ahead of time” variant provider is a totally different way to use variants. Instead of “matching the user machine” in any way - it’s providing user options with a default. If you want smthg else you have to ask for it.

There are plenty of cool use cases for this - it just didn’t come “obvious” to me initially. And most importantly it comes at virtually no cost in the spec and security wise (there’s no plugin being involved here at install time and sometimes even at build time). It’s purely static.

280

Thanks, that does help. I appreciate the detail and what follows is in no way trying to dismiss the complexity, but I get two takeaway thoughts from this:

  1. It looks like plugins don’t need to inspect what’s installed in the target environment (or at least, it wouldn’t hurt the BLAS use case which seems to be the key one here). So that confirms that running providers in an isolated environment is OK - great.
  2. I’m not at all clear what the intended workflow for an “ahead-of-time provider” is. They seem to be used to embed static data into the wheel - but how can that data reflect the target system, to allow the installer to decide if the wheel is compatible?
1 Like