PEP 825: Wheel Variants, Package Format (split from PEP 817)

Packaging Standards
101

I would prefer to leave this out of the PEP, for the sake of not increasing the scope further. Cross-installs as they currently exist aren’t written down a spec, and I want to avoid starting this discussion here, it’s outside of what we can do here. We have done a bunch of prototyping, including a static file format that is runnable with the wheel variants uv: [WheelVariants] Allow static metadata to override all provider plugin logic · Issue #42 · wheelnext/pep_817_wheel_variants · GitHub. You’d generate this file on the deployment target and then use it e.g. in a CI run generating a docker container. Our prototype variant providers also have env vars to target specific CUDA versions, which can also be used for deploying to specific environments. For the nvidia provider, you can set NV_VARIANT_PROVIDER_FORCE_CUDA_DRIVER_VERSION and NV_VARIANT_PROVIDER_FORCE_SM_ARCH.

I feel in a catch-22 here: Specifying everything at once is too large and too complex to get agreement (PEP 817), but specifying a minimal subset (PEP 825) lacks the complete story, and we risk being too slow, meaning we can’t make complex changes in the ecosystem. If that remains a problem, I’d like to do another sync chat solely focused on the procedure of PEP acceptance: This is the one thing I can genuinely see risking the whole project.

For the lockfile, we had Pipfile.lock, then poetry.lock, then uv.lock (which learned a lot from the both poetry’s and cargo’s experience), and then finally pylock.toml. As a tool-specific feature, tools could experiment and learn, and all the accumulated knowledge went into pylock.toml, which can also still coexists with the vendor specific formats without having to capture everything. Wheel variants are in a tougher spot: They need coordination between at least the package authors and the package installers, and for a real rollout with registries too, it’s not really feasible to roll them out in a tool specific way (at least not without vendor lock-in and/or an implementation-defined format controlling the ecosystem). As an interoperability standard, we need to agree on it

1 Like
102

This discussion has brought up a lot of features that are critically relied on, but that have never been formally written down, existing either tool-defined or by some form of convention. Some are even so common or so old that it’s assumed they are a standard, when they are not. I’m reluctant to address those formally in the PEP, due to the increase in scope and the lack of clarity without a baseline PEP. We do of course care about not breaking existing workflow, and will address any problems in that regard (As @pf_moore said, breaking pip or packaging is a showstopper). We also provide a lot of experimentation and prototyping on github around how these features could be used. These are not part of PEP, to keep a proper separation between specs and tools, and to make it clear what the PEP prescribes.

103

This idea of a file capturing system details was actually discussed as I think part of the lock file spec at some point by writing down the values for environment markers as JSON that could be fed into Markers - Packaging .

104

I’m not sure I understand your problem here. I don’t think the amount of time needed to get acceptance of the full wheel variants proposal will differ much whether we have a single proposal or multiple ones. Either way, it’s a big project.

And I’m not sure what more I can tell you about PEP acceptance. At some basic level it’s simply a matter of you asking for a decision, and me saying yes or no. I try to be more involved than that, though, and advise PEP authors on how to have the best chance of acceptance. But I have to be careful, as it’s an obvious conflict of interest if I come across as saying “write the proposal this way or I won’t accept it”. Right now, I feel like I’m getting too involved in trying to help get the PEP in shape for acceptance, and that may be counter-productive.

In the case of PEP 825, I’m struggling to come to a decision. The functionality is important, and I think we need some solution to the wheel variant question. And the basic ideas in the PEP are solid. But the PEP itself doesn’t feel precise enough to be a good basis for a complex feature like variants - there are too many areas which feel like they are “to be specified later” or “we assume people won’t do dumb things”.

Yes. And therefore, it seems to me that it’ll take longer to come up with a good solution, because we don’t have practical experience to base the design on.

Which reminds me - do any other ecosystems have prior art in this area? Surely Python can’t be the only language that has to deal with the problems wheel variants are addressing. How do other languages handle the situation?

Maybe a starting point would be to formally list those features, along with what behaviours are required (or preferred) by the variants spec. That would be a lot better than continuing to just talk around them without coming to any conclusion.

The only one I can immediately think of is handling of multiple indexes, and even there, there’s no specific behaviour that’s “critically relied on” - as far as I can tell, all that happens is that when multiple indexes are involved, the PEP gets two or more variants.json files containing ordering data, and needs some way to combine them. That combination could depend on having more information about the indexes than just that they exist (for example, having an ordered list rather than a set allows a rule of “prefer indexes earlier in the list”), but the PEP doesn’t give any combination rule at the moment, so it’s not clear what might be needed.

1 Like
105

In particular, how does Conda handle this? Because Conda must be facing exactly the same questions around selecting binaries based on fine-grained system properties as we are, and even though they typically don’t get involved directly in PyPA packaging discussions, this is very much an area where we should be aiming for a common framework, or at a minimum learning from their experiences.

1 Like
106

Some general notes/inspirations were provided in the “prior art” section of PEP 817: PEP 817 – Wheel Variants: Beyond Platform Tags | peps.python.org .

That said, conda/conda-forge is a significantly different system. Most importantly, it is much more amenable to ad-hoc improvements, whereas with this PEP we’re planning for years ahead.

107

I do not think forcing this to have to be generated on deployment target is the correct answer. If that is the solution to require potentially arbitrary code execution in an open network environment I would turn to a very strong No to any PEP that would recommend that. We (my Job $1)network jail our builds and disallow any sort of install post build to protect from SC attacks being able to have a large blast radius and running something to generate this static metadata would create too significant of a risk imo.

If its env vars for now that’s fine but I would want to lean in on declaring dimensions of a variant as static metadata as a part of the install either with flags to a package manager’s CLI or via pyproject.toml to solve for the use case I talked about.

3 Likes
108

These are just two examples that we explored; Each tool is free to do its own custom configuration, we just can’t define tool-specific things in a spec document, and hatch can totally offer this in tool.hatch.

109

What you can do in the PEP is state “Tools MUST provide a mechanism to…” (without saying what the mechanism is, if you don’t want to dictate a particular UI). If having a mechanism is essential to key use cases, that is a reasonable position to take.

4 Likes
110

To expand on this, variants in conda have so far been encoded “next to” the version, in a similar way as the PEP proposes. In particular, a general conda-package looks as follows:

package             "build string"
vvvvvvvv        vvvvvvvvvvvvvvvvvvvvvvvv
libtorch-2.10.0-cuda129_mkl_hd6d2a1f_303.conda
         ^^^^^^ ^^^^^^^^^^^ ^^^^^^^^ ^^^
        version   variants  dep.hash build-number

The build strings originally existed to distinguish different builds for the same package of the same version[1]. It eventually got (ab)used for encoding variants, because of the naturally desirable property (of wanting an exclusive choice for libfoo between its possible variants) that the regular logic already covered.

As can be seen in the example above, build strings may contain more than one variant (here: the cuda version, and whether the BLAS implementation is MKL or generic), concatenated into one; not only that, but there’s no agreement between packages on the order of components within the build strings. This kind of overload of a single string made it very difficult to properly select packages across potentially multiple axes, and the interface for user to select[2] specific variants was/is pretty horrible too.

As it happens, a CEP (like PEPs, but for conda) just passed that finally formalizes a better solution in this space. This will hopefully roll out in the next couple of months to installers, but will take a while before package specs can rely on this. Even so, the way things work under the hood will not fundamentally change; it’s primarily a mechanism that improves the recipe-author- and user-facing interactions with variants.

  1. e.g. if you build v2.10.0 again but change something in the recipe that modifies the result, you increment the build number (unless made impossible by other constraints, the solver will always prefer the higher build number); if the versions of the dependencies you’re compiling against changes, the hash will change automatically, etc. ↩︎

  2. beyond what could automatically be determined the solver, taking into account the detected system capabilities (e.g. glibc version, CUDA version, etc.); this corresponds to what the plugins from PEP 817 would like to achieve. ↩︎

1 Like
111

This got me thinking some more about PEP 825 and its role as a specification of the data model underlying wheel variants. I see three key ways in which variant data needs to be handled in a data model:

  1. Recording the compatible variant values for a wheel.
  2. Recording the priorities to be used when selecting a wheel for an environment.
  3. Recording the supported variant values for an environment.

Item (3) is important when considering cross-environment installs, as in the use case @cjames23 described above. It’s not so much about the UI for cross installs, but more about having a format with which we can persist the description of an environment for use in such a UI.

I’ll discuss the 3 items in turn.

Wheel variant values

PEP 825 supports this pretty well right now. The description of the wheel filename format, labels, and the variant.json file format cover this very effectively. The one qualification I’d make is that for this part of the spec, the default-priorities part of variant.json is not required. See below under “Variant ordering” for a discussion of this data.

For this set of data, the index-level variants.json files are a pure optimisation. When interpreting the label of a wheel served from an index, you can read the index-level variant.json file and use that to interpret the label, rather than having to read and unpack the wheel itself. There’s no need to consider multiple indexes, as everything here is operating on a single wheel from a single index.

Variant ordering

This is less clear, IMO. The problem is that ordering of variant values is not a wheel-level property. Individual wheels have no concept of “other wheels” - in all existing standardised contexts[1], a wheel can be viewed in isolation, with no need to know where it came from, or what other wheels might exist.

I have no problem with the PEP’s approach of having a default-priorities table to define ordering, or of allowing tools to define a UI to override the provided defaults, but I do have a problem with publishing that data at the wheel level. It simply isn’t wheel-level information. At the most fundamental level, ordering data is a resolution-level data item, but in the absence of a standardised idea of a resolver, the best we can do is to say that it’s a “group of wheels” level item.

The obvious groups of wheels that exist are indexes, and (not standardised) directories full of wheels (pip’s --find-links option). IMO, ordering data should be standardised at that level, and should not be present in individual wheels. That still leaves us with a need to define how we order wheels that come from different groups (indexes or other sources of ordering data), but at least the problem is clear now - we can’t know which of two wheels is “better” unless we have a single ordering relation, so we have to merge the data from the indexes the two wheels come from to obtain that ordering relation.

Additionally, by making ordering data a standalone concept applied to a group of wheels, it becomes possible for installers to support overrides by simply having a --variant-ordering flag that takes a file of ordering data in the standard format. I don’t know if there are use cases for that much configurability, but the fact that it’s possible can’t be a bad thing. It is worth noting that for something like this to work, it would be necessary to be able to store ordering data independently from the mapping between labels and variant properties. I don’t think that’s a bad thing, though - storing them in the same file is (as I’ve noted) an artificial association, not justified by the underlying data model.

Environment information

This one currently isn’t covered by PEP 825 at all, and IMO it needs to be. We should define a format for storing in a file the variant values supported by an environment. That format could be nothing more than the same variant.json format that we use for wheel compatibility (with or without the ordering data, I don’t have a good feel for whether that’s needed), but the key here is that we state explicitly that it’s the standard format for storing an “environment description”.

Having a standard file allows for “offline capture” of an environment, as well as manual creation of a description file in cases where the target environment is so locked down that normal discovery methods aren’t permitted. It also simplifies UI decisions for tools - it’s far easier to support a single --target-env-file option than to need a complicated array of individual flags[2].

Once we have an environment description format, future PEPs describing discovery of variant properties for an environment (e.g., plugins) can be described simply in terms of adding data to that file format, and the specification of the variant selection process can be described in terms of reading the data from that file format. A lot of interdependencies go away if we have a well-defined interoperability data structure.

Disclaimer

I want to be clear on what my intention is when posting this. I am juggling various roles (PEP delegate, pip maintainer, interested community member) when posting here, and I want to be sure people don’t misinterpret what I’m saying.

This should be read as a post by an “interested community member”. It’s a possible redesign of how PEP 825 could present the variant data format(s) that feels significantly cleaner to me, and I want to offer it as a suggestion. As a pip maintainer, I doubt it will significantly affect how pip implements variants.

As PEP delegate, I obviously have a liking for this framing of PEP 825, but I’m not saying “rework the PEP this way or I won’t accept it”. I have to be honest, I do think I’d find it easier to accept a PEP that was written from this perspective, but that’s mostly because I see it as clearer than the current version of the PEP. At a minimum, I’d like the PEP authors to consider whether this approach would work for them.

  1. Resolution is the obvious exception, but as has been repeatedly pointed out, we have no standards around resolution. ↩︎

  2. I’m looking at you, existing platform compatibility options in pip! :slightly_frowning_face: ↩︎

4 Likes
112

We pre-build container images on a CI system that we want to be maximally compatible with hardware our users may have. The current “lowest common denominator” status quo in the ecosystem currently makes this work well.

If variants are pushed out, we’d need some kind of mechanism so that we could uv sync and target a lower CPU instruction set, etc that was more broadly compatible so our pre-built container images can be run on the more hardware, even if our CI machines happen to have very new hardware.

2 Likes
113

Technically speaking, it is. The data at wheel level could be seen as a pass-through format.

The full picture would be:

  1. The data is originally defined at project level[1]. The project defines what variant wheels can be built from it, and therefore defines the relation between them.
  2. As wheels are built, the data is copied from the project into the wheels. It is not significant at individual wheel level. It is merely present here as “user convenience”.
  3. As wheels are published, the data is combined and published at the index-level. It is significant here.

The whole point is, skipping 2. has a significant drawbacks: you introduce unnecessary (and IMHO unjustified) difficulty for package publishers. Per the current design, they just define all the data at project level and it is transparently transferred to the index. If we remove it at wheel level, the project maintainers now have to manually pass them to the index level.

  1. …where pyproject.toml is the obvious location for it, but that’s not significant here. ↩︎

114

I disagree that putting data into the wheel and relying on the index to extract it is better - it introduces a risk that wheels don’t have the same data in them[1] so we have a number of extra potential error situations that consumers have to consider.

But it’s your choice. As I said, I only offer this as a suggestion.

I’m not going to debate your “full picture” - it’s certainly one implementation approach, even if it’s not one I would consider particularly developer friendly. I’ve had enough pushback when I’ve tried to discuss UI matters that I’m simply going to say that you can’t use a suboptimal UI to justify a data design choice.

You still have to either define how to merge ordering data from multiple sources, or explain how you intend to enforce that all sources must have the same data[2].

  1. There’s nothing that ensures consistency in the current packaging and publishing workflow, and in fact there’s an explicit requirement in the case of some data, notably dependencies, for different wheels for the same project to have different metadata. ↩︎

  2. And I won’t accept a claim that it’s not allowed so tools are permitted to just assume that the data is the same in all sources - that’s how we got into the whole mess of index priority debates and dependency confusion :slightly_frowning_face: ↩︎

1 Like
115

I think there’s a misunderstanding between an interoperatibility standard and guidelines to the implementation in a tool.

The PEP is an interoperability standard, we write down the minimum that build systems, resolver, installers, package authors and registries need to support to work together. We don’t write down any requirements for tools beyond what we need all tools to do the same. We do of course build this with ideas for user experience in mind, but we intentionally leave a wide range of tool choices open, we don’t want to limit tool authors, be it major general purpose installer or domain specific tool with niche requirements, to choices beyond what’s necessary.

For example, all tools needs to write and parse wheel filenames including the variant labels the same. They need to find important about wheel metadata in the same location. They need to write and read the same simple api format and interpret it the same. They need to have a shared understanding of what’s compatible on the current platform. They need, to some extend, have a shared understanding of the security model.

There are some features that may or may not be a part of a PEP, such as variant ordering. There could be a PEP where ordering is left to the tools, and we define only compatibility, and maybe an ordering of variant property as helpful util. We considered this, but we found having a reliable experience of tool install foo more important, e.g. pip install torch and uv add torch using the same GPU-wheel by default. There’s also some locations where the PEP gives hints, such as avoiding parser differentials to avoid CVEs [1].

The PEP does not list features that all major tools will support, even if they are obvious. This includes providing overrides to not select the best variant on the current platform. This is similar to how versions work currently: PEP 440 defines compatibility and it defines a highest/best version, but it does not define the various forms of overrides that exist, such as constraints files, preferences from lockfiles, dependency cooldowns and resolver and resolving the lowest/worst version intentionally. Similarly, this PEP does not say anything about overrides. It’s totally possible that we define a shared configuration for the common cases in the future, but it’s too early for locking in tools now.

  1. speaking from experience: uv security advisory: ZIP payload obfuscation ↩︎

2 Likes
116

I disagree my ask was not to lock tools into a specific implementation here but that tools need to simply support an ability to override. Which are vastly different things in my opinion. And not considering it is a regression from today where I can very much through PyTorch urls download the binary wheel I am going to need at deployment target.

3 Likes
117

And this isn’t even standardized now anyway.

Are you thinking this is a user preference or something broader? As in do you expect cloud providers like FastAPI Cloud to publish such a file? If so, it might make sense to capture marker details as well or at least make it possible to do so in a future PEP in the same file. That way we don’t end up with a bunch of files that can act as input to a resolver.

118

I agree that this is a drawback! Where you could previously just pin the right wheel from the right torch index for your known target, you now as a user need to use some override mechanism, which the tool authors have to build and support. I wasn’t trying to say that this isn’t a negative for this proposal (and gaps, drawbacks and implementation complexity need to be part of the evaluation when judging the PEP), but rather explain why this isn’t part of the PEP and how we’re thinking about what is and isn’t in the PEP.

119

I’m simply thinking that having a defined format for storing variant values that can be generated on one system and then used on another, as input into a tool that would otherwise query the target environment. I can imagine various uses for it, but the most obvious one for me is as you say for cloud providers or locked down target environments to provide it as input to a resolver.

Including other aspects of what is needed for a resolver (markers and compatibility tags are the obvious two) would be nice, but I’m sure the PEP authors would say that’s out of scope for this PEP. What’s important to me is that the PEP specifies how variant data is recorded, because that’s what we’re working on right now. We can add marker and tag data to the file later (and with that in mind, I’d want PEP 825 to define a format that can be extended in that way).

3 Likes
120

What I am suggesting here is PEP 825 or a related PEP later needs a tools MUST section with regards to having overrides as a possibility. It doesn’t have to be a locked in spec but telling tools if they are going to be complaint with the PEP they need to support some way of overriding.

Though it is true that no tool has to implement a PEP, I think the majority of tools that are going to be involved in the space of this PEP do try to be PEP compliant. What I am trying to avoid here is if neither uv or pip offer this then hatch will be forced to not support this PEP really or we are forced to implement our own resolver.

3 Likes