And to expand on this a bit, I don’t know if you were to go through PEP 825 – Wheel Variants: Package Format | peps.python.org and look at all failure points the PEP covers what the expectation is. If the answer is always “fall back to non-variants anytime something is off” then that might just need to be flat-out said in the PEP. There’s just multiple, independent software pieces running here (build back-ends, index servers, installers) that are bound to have bugs, but also potential disagreement on how to implement something if there’s a (current) gap in the spec. To me that says “make the spec as tight as possible” and/or “the catch-all answer is …” to help wash away ambiguity.
2 Likes
Thanks for the detailed feedback @ncoghlan and getting the ball rolling again here Alyssa and Brett! Pretty much all of your suggestions look good and reasonable to me - there’s a fair bit to unpack though, so I’ll try to not do that all at once here. I’ll start with the build process part.
Regarding the intended build process, we have had a draft PEP pretty much ready for several months (may need another look and a bit of polishing, but it’s close enough): wheelnext/peps#44. At the conceptual level, it didn’t really change from what was in PEP 817. We were just waiting for this review to conclude. However, given that you identify that as the key gap, we could submit it soon if you think that is helpful.
Yes, that seems perfectly clear - we can say explicitly that publishers are at fault. It seems like stating the obvious, but I would agree that it has gone wrong in the past. Also, just like with regular wheels, I think that there will be publishers who choose not to build with a modern build backend in the recommended way[1], which can increase the chances of differences creeping in.
Beyond saying “publishers are at fault if any consistency requirements in this PEP are not met; installers and other tools are allowed to assume these requirements hold”, are there other governing constraints that you think need to be explicitly mentioned? Or, would a different format make sense here, e.g. marking each item of metadata with “may vary” / “must not vary”, or a single section with a list or table summarizing all metadata that must be consistent?
Formulating that answer sounds good to me. The two question marks though are:
- Should that be in this PEP? The catch-all can be I think. For installer behavior though, any more specifics belong in the PEP on installers.
- Falling back to non-variants is probably the wrong default answer for installers, because we’re almost certainly introducing variant wheels as a user opt-in for an extended period. So my gut feel is that the catch-all should be to get a clear error message (EDIT: I could be wrong of course, this isn’t very critical and could be decided later - it’s more an introduction pathway discussion than anything else). Note that for specific cases like “variants don’t match the local machine”[2] we already have a fallback to non-variants defined indeed.
Especially for packages that already build hardware-specific wheels today; we already have variant-repack to help them with transforming those into variant wheels ↩︎
Which could be due to a metadata inconsistency, or to a bug in a provider or installer ↩︎
3 Likes
I should also have added that, if anyone wants to look at build backends in more detail, the most polished one is currently scikit-build-core, for which Henry recently merged an implementation in-tree behind an experimental flag in feat: add experimental PEP 817 variant support by henryiii · Pull Request #1284 · scikit-build/scikit-build-core · GitHub. We have several other build backends with an implementation at GitHub - wheelnext/pep_817_wheel_variants · GitHub.
1 Like
I think so.
Then I think this PEP can say, “it’s undefined behaviour, but will be addressed in a future PEP” or “installers error out for now, but a future PEP may provide more specific guidance”. Basically if you don’t say it in the PEP then it’s undefined behaviour, but you can at least call out that fact when you know there is undefined behaviour upfront rather than have the reader guess whether you thought about it or not.
Me saying that was meant to act as an example rather than a suggestion. Sorry for not explicitly calling that out.
1 Like
Yeah, I think it’s worth stating explicitly. My experience has been that you can never be too explicit in a PEP - users are surprisingly adept at finding ways to make use of loopholes 
On the same principle of you can never be too explicit, I think it’s worth explicitly listing what “consistency requirements” you mean here. It’s surprisingly easy for someone to assume that what you think is clearly a consistency requirement, is actually just an example of one possible situation…
Also, I’d tend to avoid wording like “at fault” - it makes it feel like a matter of assigning blame. Better (IMO) would be to say something like
Publishers are responsible for ensuring that the following consistency requirements are met. When a user is referencing multiple published sources (e.g., multiple indexes) it is the user’s responsibility to ensure that all publishers are consistent with each other. Tools can simply assume consistency, and no requirements are placed on what tools must do if the data provided is not consistent.
(List the consistency requirements here…)
That’s wordy, but only in the interests of being painfully explicit
Honestly, I think you’re probably best placed as the authors to make judgements on how the information should be presented. Just focus on the basic rule that you should be painfully explicit, and not leave anything to the reader’s imagination, and you should be good. If there’s any particular cases you aren’t sure about, pop a quick post here and I’ll give you my view (or DM me if you prefer, although I’ll respond just as quickly to public posts). And if you’re still having trouble deciding, you can always post a draft of what you’re thinking[1] and I’ll let you know how it looks to me.
But if you want my gut instinct, I’d tend to start with a list of the consistency constraints, all in one place, something like I described above, and see how that looked.
No pressure to be polished, work in progress can be as rough as you like ↩︎
4 Likes
On that front, one possibility that came up while discussing potential paths forward for PEP 825 with Paul is that it could be helpful to repurpose PEP 817 as an overall informational PEP for the variants proposal, along the lines of PEP 630 – Isolating Extension Modules | peps.python.org (which doesn’t make any concrete technical proposals itself, it just provides a shared context for understanding other more detailed proposals)
As an Informational PEP, a revised PEP 817 could focus on:
- describing the problem to be solved and the general concept of how the variant proposal solves it
- listing out the sub-PEPs (existing or planned) that will be necessary to make variants usable at all
- listing out any follow up “nice to have” PEPs that are expected to improve the end user experience, but aren’t something we want on the critical path to acceptance of the core capability (coughpluginscough)
Since such an Informational PEP would primarily be a statement of the authors collective intent rather than approval of any specific technical detail, the barrier to approving it is much lower (accepting it is essentially just an agreement to invest time in reviewing the related Standards track PEPs, and we know we have that level of interest and support)
3 Likes
I’ve been thinking a lot more about this constraint, and I’m still concerned that it prevents valid workflows.
Specifically, imagine a company that builds specialised hardware. They take existing published wheels from PyPI, and patch them to add support for their hardware. At the moment, let’s say that they publish these wheels via a dedicated index, in the same sort of way that PyTorch does. But they want to switch to using wheel variants, because this is precisely the sort of problem wheel variants are intended to handle.
The problem is that they can’t do so. This is exactly the “new axis of variation” scenario that Alyssa called out, but it’s a case that explicitly won’t involve a new release, because the company is not the project owner, they are simply releasing patched versions of existing releases.
For another example of this, Christoph Gohlke used to distribute wheels for Windows which were based around the MKL library. I don’t know all the details of the build process or the compatibility issues, but I recall there was a warning not to mix MKL and non-MKL builds in one environment. Again that’s precisely the sort of situation where variants should help, but if we don’t allow “new axis” builds to be published by 3rd parties, it won’t.
I think the PEP needs to explicitly cover how these use cases would be handled. And if they won’t, then make that very clear, as it effectively means that in the case of variants, we’re taking what’s currently a decentralised wheel publication process[1], and making it into a centralised process, controlled by the project maintainers (which feels weirdly in contrast to a lot of the aspirational goals around flexibility and plugin-based customisability that PEP 817 was emphasising).
At least in theory, if not in practice in most cases ↩︎
2 Likes
One of the things I realised wasn’t quite right in my previous post is that namespace definition updates can cover the “new axis of variation” case as well.
Even without accounting for that, though, the right place to opt in to accepting a new axis of variation that neither the namespace definition publisher nor the project release publisher considered is in the configuration of the locking/installation tool
That gives three potential tiers to the variant selection logic:
- the namespace definition gives the standard variant sort order for properties in that namespace
- the project release optionally defines variant sort order overrides for that release (and that info is mirrored into the wheels for that release)
- clients optionally allow overrides for a given locking or installation operation
Allowing variants of the same release to specify different sort orders opens the door to all kinds of complexity we simply don’t need (for example, consider a case where variant A says order should be ABC, B says BAC, and C says CAB).
Third party variants won’t be published randomly (except as a potential attack vector), so requiring the original project to intentionally allow for them (by leaving gaps in the variant selection matrix), or consumers to intentionally opt in to them (by overriding the selection matrix) is a reasonable UX constraint.
2 Likes
What does this ‘leaving gaps’ mean, in terms of PEP 825 – Wheel Variants: Package Format | peps.python.org ?
Namespace properties effectively define the axes of multidimensional matrices that specify all the anticipated wheel variants that theoretically could exist for a particular project release. The project’s own published variant wheels will cover some subset of that space and third parties will be able to fill in additional variant wheels for project releases in the same way that an index like piwheels.org is able to fill in additional regular wheels.
Okay, so IIUC, the core of Paul’s concern was with third parties being able to publish wheels which vary along an unanticipated axis (== not specified in default-priorities.namespace in the project release).
One way would be to mirror all wheels to a dedicated index but patch variants.json for each wheel to add the new axis. But were you hoping for a solution which didn’t require that, Paul?
So here is a second way: publish just the third-party wheels on a dedicated index, then opt-in to considering that new axis in one’s installation tool. I suppose this could come in the form of opting-in to the index-level …variants.json of the dedicated index taking priority?
Does that cover the scope of the concern, or does it also extend to:
- having an unanticipated axis be considered by default (and potentially take priority over some of the project-specified namespaces) by installation tools, without user opt-in?
- introducing an unanticipated variant axis on the same index as the original project release?