Thanks. I didn’t understand that (and your comment was posted while I was writing my reply to @rgommers).
So does that mean that installers will need new UI options to allow the user to choose in the way you suggest? How does that square with the whole “$tool install torch needs to just work” idea?
If the answer is that AOT plugins are different, then I wonder whether they should be a separate PEP, because they have such different characteristics than detection-based variants.
I personally think it’s a super small addition to the spec. The biggest part is “explaining what this is”
Spec wise … ‘install-time=False’ that’s it…
It’s more “conceptually different” than “technically different”.
Now interface wise we thought about two UI in general to pin variants (I’m not sure it’s in the PEP i need to verify)
‘pip install package==A.B.C#variant_label’
That’s the “variant pinning” essentially. We took the same logic that you can pin a version well you can also pin a variant.
Note: if “#” is not the right character - let’s replace it with the one that will make everybody happy.
If the variant is pinned => don’t use any variant provider and just install this (no compatibility guaranteed) - also a good way to allow security focused people to get a variant with absolutely zero provider involved. You make the decision “by hand”
The other way is the “static variant resolution file” that we are working to fully add to the PEP. We initially left it out of the PEP and it was surfaced as “well no this should totally be included” so we are doing that.
Important: both of these “variant pinning” mechanism should exist regardless if AOT variant providers exist or not - in this PEP or not. So having AOT providers is really adding only “install-time=False” to the spec.
I agree, they’re really not all that different. All variant wheels are some variant of the build from the same source that is not captured by the platform tags. If for a given platform tag you have N variant wheels, then
The check/filter step returns the valid wheels, in a certain priority order. The best match is selected by default, and that selection is overridable via some installer UX (e.g., static file). Everything else is the same: how to build the wheels, the metadata formats, the installer UX, the whole logic in the diagram I shared, etc. The only thing that is different is that the package authors guarantee that all variant wheels will work, hence one step of the install process can be skipped.
Thanks. I’m starting to think there’s a big benefit to splitting the proposal up into smaller, more easily digestible chunks.
There might be scope for standardising how users describe what type of cheese they want - that would avoid every tool having its own variant description format, while still leaving the UI down to the tool. I don’t know if that’s something that has been considered yet, at all. Jonathan mentioned a “variant pinning” syntax, but I’m not sure labels (whose meanings are tied to a specific project/version) are the best approach here. I’m open to being persuaded otherwise, though.None of this is to suggest that the PEP doesn’t cover all of these points already. It’s entirely a suggestion about the best organisation and process for presenting the ideas, and getting them approved. Specifically, I think there’s value in getting the rest of the variant mechanism agreed before tackling the “check/filter” problem. Yes, it might mean needing two or three PEPs rather than the current single one, but that doesn’t necessarily mean more effort - the complexity of the current PEP has its own costs, and I think we’re all experiencing the frustration that comes from those costs.
I think I’m missing part of the equation here then, especially around the ahead of time providers, because if these are entirely declared ahead of time, it seems like these shouldn’t be providers at all, but just a dependency.
Also, If it’s accurate to say that plugins should never need to access the environment being installed into, mandating they must not significantly reduces the security surface to where I’d be able to withdraw any substantial objection. (Note that I wouldn’t withdraw the objection if tools are allowed to run the plugins in an environment being modified by an install command).
I’m not sure how it can be accurate even as described, though. Reading the pep doesn’t get me to that being true. It seems true for the specific choice of blas being ahead of time, rather than detecting if any installed wheels link a specific blas, but nothing in the pep forbids such behavior.
To give the full story, they stem from the fact that install-time variants are opt-in by design[1]. Cases like BLAS could be handled by install-time providers, but these providers would effectively be “dummy” from installer perspective, and a hassle from security standpoint. So we introduced a special variant of providers that work for this case.
Yes, technically we could split them. However, they share large part of logic with install-time providers. Most importantly, AoT providers can use plugins at build time (yes, that’s another fork down the road): say, you can either specify all the possible BLAS implementations as part of your pyproject.toml or you can use a plugin that will be queried at build time to fill that for you. It’s “best of both worlds” solution: you can no security impact at install time (since the data is static then) and you can get consistent properties (since they are controlled by a third party package).
I’m not convinced that splitting things will make them easier to comprehend. Yes, maybe the base part of the PEP (say, covering variants without providers) will be easier. However, the subsequent PEPs building on top of them will have the same level complexity, and on top would have to keep cross-referencing the previous PEPs.
I’m not opposed to splitting things up but before we spend a lot of time doing that, are you really sure this will improve things? Or merely defer the problem until another 300+ post DPO thread?
with possible opt-out exceptions, but that’s beside the point ↩︎
We (the PEP authors) agree that splitting up the PEP will make it easier to move forward, so we’re going to go ahead and work on doing that!
We’re going to figure out exactly where we feel like the lines should be drawn between the different PEPs so that they are easier to understand and so we split up the parts where gaining consensus is simpler (or may have already happened) and where we’re (as a community) still working on coming to that consensus.
I personally can see a few different ways to “slice” up the PEP, and it probably won’t be completely clear which way is best until we actually attempt to do it 
.
One way that comes to mind is splitting out the “by default” behavior so that 817 becomes solely opt in (unless an installer chooses on their own to make it opt out) which punts on a lot of the controversy I think, but would also need us (IMO) to make the PEP itself easier to digest/understand. That helps make sure the final result is still logically consistent and makes sense together and avoids some of the cost of splitting things up too much.
I’d be curious how folks would feel about that?
Like I said though, I don’t think it’ll be completely clear which way is best until we actually try splitting it up and see how it feels!
Once we have them split up, we’ll follow up with a new thread(s).
Thanks all!
I would strongly prefer that the mechanism for recording variant properties within wheels be separated out into its own PEP, leaving everything about how that data is used to other PEPs. I’m pretty sure this would not be the easiest place to split the proposal, but I think that in terms of understanding (and importantly for me, approving) the proposal(s), I’d like the data model to be its own item, and the use of the data to be separate.
I’ll leave any other splits up to the PEP authors, but my gut feeling is that anything involving plugins is likely to be by far the most controversial proposal, and therefore if there’s anything you think could be useful even if the proposl involving plugins gets rejected, then that would be worth keeping separate (to avoid it getting “poisoned” by the controversy over plugins).
@pf_moore quick “PEP technical question” for my own understanding.
How would you manage all the context and all the justifications that this work is important as we split the PEP ?
I think we pretty much all agree that this is important work for the community at large. @warsaw months ago was talking about “making an informational PEP” that specifies nothing except “this is an important problem we care about” (not even mention that we potentially have a solution for it).
Potentially given there seems to be consensus on “this is important work for the community” - if we were to extract “context & rationale” in its own PEP - Could we “submit that part for decision” reasonably quickly ?
That would allow us to say “See PEP ABC for context”.
I’m not exactly certain how useful it actually is - given that this PEP would only state “this is an important problem we care about” - no standard, no specification, no hint of a possible solution.
Is it something we should pursue that can be quickly submitted to decision without too much friction ? Taking out all the context & rationale shouldn’t be too hard. It has the benefit of making the rest significantly shorter as the context is currently 1/3 of the PEP itself.
If yes “it’s useful please do that” :
Honestly (and I think I’ve said this earlier but I’m happy to repeat it), just skip 90% of it. Most people seem pretty comfortable with the idea that this is a problem that needs solving - there’s no need to persuade them.
If you want some words to start from, I’d go for something like:
This PEP proposes a way to record arbitrary compatibility data in wheels, to allow tools to pick the correct wheel to use in situations where the traditional platform tags are insufficient. There are many cases where this is necessary, most notably in the case of scientific and ML libraries, where high performance requires extension code that is carefully tailored to the precise hardware available in the user’s environment. Well known examples of this include:
That’s literally it, IMO. The amount of detail currently in the PEP, while impressive, simply isn’t needed. You’re trying to convince people who already know this is a problem, and even if it isn’t something they encounter themselves, they know it matters to the users who are affected.
By all means keep all of the details in the current PEP saved in a webpage somewhere, and point people who aren’t convinced to that. Or if you must, write them up as an article and link that article in the PEP, along the lines of “For anyone interested in details of projects affected by this issue, and the workarounds currently employed, see <this article>”. But they shouldn’t be part of the PEP itself.
I don’t think the justifications need to go in an informational PEP, or be approved in any way, to be more compelling. I’ve seen basically no pushback in this thread on the idea that this is a problem that should be solved. The pushback has all been on how the PEP proposes to solve it. So stop trying to make the case that everyone already agrees on, and get straight to the point, which is to convince people that the solution you propose is the right way to fix things.
Alright seems like I’m gonna dust-off the vaccuum cleaner this weekend !
Thanks @pf_moore - enjoy your evening & weekend. Much appreciated (and useful) feedback
To be clear, that’s just my personal opinion (although as PEP delegate, it does have a certain amount of authority behind it ). I don’t think anyone is going to object if you do as I say, but you should probably allow some time for anyone who objects, and thinks more detail is needed, to speak up.
Basically, give it some time. Maybe take this weekend off to relax, and jump into it next weekend?
That would be https://wheelnext.dev/
My own suggestion for a potential PEP split would be 2-4 covering the following:
pip install torch do the optimal thing )It might be feasible to keep a couple of the standards track PEPs together, but the advantage of three PEPs is that it keeps each one focused on a specific way of using the variant information.
I’ve probably missed where the plugin interface is defined in PEP 817, but I wonder if having a CLI (which I’m going to name variantcli so I’ve got a name to refer to) that handles calling the plugins, but can also list what plugins you have installed. Installers could run variantcli state –json in a subprocess to get the supported variants (which would allow users to configure variantcli with static values without all installers needing to know how to do this).
In addition to listing plugins, variantcli could do some basic detection of what variant plugins you may want to additionally install (see smxi/inxi: inxi is a full featured CLI system information tool. It is available in most Linux distribution repositories, and does its best to support the BSDs. - Codeberg.org for one way this could be done), give a human readable version of what variants are supported (which would be useful for debugging installation issues), perform a scan of all installed packages and verify that the system-supported variants and the installed packages align (i.e. to spot that someone hasn’t installed the per-machine variants on the shared NFS drive), and any other user-facing tasks around variants.
This would mean installers wouldn’t need to vendor something that is changing rapidly, security conscious users can not install variantcli (or configure it based on their needs), whilst for novices there’s a simple command that can be run to pass on what the state of the system is when asking for help.
Here’s one use-case I can think of where you would need the option to disable auto-detection. I would imagine other than users experimenting on their desktops, most GPU usage is going to look similar to the university cluster example below (change out where it’s run and some of the technology stack, and you’ll have a HPC supercomputer or corporate analytics cluster).
Say I run a university cluster which provides jupyterhub[1] to its users.[2] Some machines have GPUs (and this being a university cluster, there’ll be some model variation based on when these GPU were purchased), some don’t and jobs that will benefit from having a GPU (or need one) shouldn’t waste time on these CPU only nodes. As the admin of the cluster, I’ll have prebuilt OCI images with the users’ software appropriate for the nodes, but naturally I don’t want to have to build said images on each machine individually, this can be done via CI. I know what the specs of each node are (either in a machine readable format or not), so I should be able pass this information to the CI system as part of the build to get the required images.
On a slightly different note, I think the “Backward compatibility” note in the PEP needs expanding, as there are some rather serious backward incompatibilities that aren’t discussed.
The basic problem is that, according to the current spec, variant wheels simply are not valid wheels. Tools that test (by checking the filename) that a file is a wheel will fail. I understand that this is a deliberate choice, so that installers that don’t support variants won’t “see” variant wheels, and so won’t fail because of them, but that’s only one use case. If I download a set of wheels using (a variant enabled) pip, and then run an audit scanner on the downloaded files, that audit scanner (if not updated to support variants) will report errors that there are invalid wheels in the download. Or if I write a script to scan the wheels on PyPI for a given package and report their version numbers, and that script uses the function parse_wheel_filename from the current version of packaging, it will fail on any variant wheels that are present.
I’m fairly sure that various aspects of this type of problem were discussed as part of the Wheel 2.0 discussions, and I think this PEP is currently only reflecting one side of those discussions (the side that felt that a change in the name format was acceptable).
I don’t think this is a showstopper for the PEP, but I do think it’s important that the PEP discusses all aspects of backward compatibility, and at the moment the implications on non-installer tools are left for the reader to work out for themself.
Thank you for all the feedback. We have just pushed a draft of the first split PEP, and started a new discussion for it: PEP 817 split: Wheel Variants, Package Format .