So arguably there’s an inherent mechanism for dealing with dependencies for a specific variant wheel given that dependency metadata is per artifact (and I’d argue that any solver that can’t handle that is at least somewhat broken).
There’s no mechanism for dealing with dependencies that depend on the properties of the system unless you strictly limit yourself to one axis per artifact (and again, that would depend on solvers being able to handle artifacts having different dependencies).
Thank you @sirosen, those were insightful questions. Most of your questions require some in-depth discussion or experimentation, so I opened an issue at pep_xxx_wheel_variants#71 with some initial thoughts and to capture that we have homework to do to ensure there’s a good story here.
You should take that up with uv and Poetry then that by design assume dependency metadata is fully expressed, and identical, across a whole release in each artifact.
Does pip also assume this? Or is this just something that’s currently not agreed on between packaging tools? FWIW I agree with @dstufft and would hope that this isn’t a hard restriction.
No, pip doesn’t. It’s one of the less savoury places where uv gets its speed.
Pip doesn’t, at least yet, do universal resolution, so it has no incentive to do this.
I’m pretty sure it is, it’s not just about performance but doing a universal resolution in a sane way. It’s important to discuss with uv and Poetry maintainers.
This somewhat reminds me of the old anti-pattern of:
install_requires = []
if sys.version_info < ...:
install_requires += [...]
This is somewhat tangential to the discussion of variants but I certainly wouldn’t mind more aggressively pushing against that. I think both wheel tags and variants are fully covered by environment markers, aren’t they?
I talked with the uv maintainers yesterday and it does not sound like it is a hard restriction, but rather one done to make the universal solve happen in a reasonable amount of time and to minimize complexity. Currently their resolver treats versions as the “nodes” in the system, but they could treat the artifact as the “nodes”– that would just require fetching the metadata of a lot more artifacts (~75 per version of numpy for instance), so it would be a lot slower.
It’s not really specifically related to variants, but as it stands we’re kind of in the worst of all worlds. What uv and poetry etc make is a simplifying assumption, which is fine, but the specs do not make that assumption, so the specs have to be designed with that in mind, both in terms of situations they have to handle and in terms of what makes them as logically consistent with each other as possible.
My understanding is that it does keep cropping up for uv that projects don’t have consistent metadata, which gets solved by opening up tickets on those projects and asking them to please not do that. Which is kinda crummy because we have this thing that is fully legal, but if you use it you’re going to get push back due to limitations in some people’s chosen installer.
We should probably figure something out at some point, either to require consistent metadata or to adjust the simple API such that uv and such can handle inconsistent metadata without degraded performance. I don’t personally have a strong opinion on which of those options we choose, and that’s not really a question for this spec anyways.
What I think is a question for this spec, is ensuring that it remains consistent with the overall landscape, which currently is such that you can have per wheel dependencies (and other metadata), so I personally think that variants should also allow that.
I do want to call this out though. In talking with the folks proposing variants, it appears that my reading of the proposal was wrong. My new understanding is that the environment markers are not just evaluating against the variants encoded in the wheel and ignoring the variants the system supports, but rather are evaluated against the intersection of what is in the wheel and what comes from the system.
I think that makes sense, and while it’s still different than how the other environment markers work technically, I think that it matches how they work in spirit, and I think is good enough for me personally.
I believe as part of taking the current draft proposal and turning it into an actual PEP, they’ll be clarifying the wording here to make that more obvious.
And to be clear it’s not just that there will be 75 nodes, requiring 75x more look ups, but it’s also a question about how does the resolver handle this.
When you use an environment marker to fork a universal resolution that just creates two resolutions have to be completed and then merged at the end. If you have 75 different platforms you have to start asking questions like:
Optimizing for the best UX for the common case will lead to edge cases for non-common cases, and being strictly spec compliant will reduce edge cases but be hard to present a good UX for common cases.
That should probably be a separate topic.
Wouldn’t it be trivial to merge those nodes based on whether the have the exact same dependency metadata? That would maintain the current fast path if all nodes actually have the same dependencies, and would split appropriately into the minimal set of 2, 3 or X different combinations of dependencies where necessary.
I’ve been encouraging @charliermarsh and his team to formalize this in a PEP.
Effectively 99.9(insert probably a few 9)% of the wheels do share identical dependencies.
Though if it’s something important enough for a significant percentage of the community (and if we accumulate poetry & uv, it quite definitely is), we should take a stab at standardizing this.
Though agreed with Brett, it’s a discussion for a different thread.
Now to answer on the standard we will be proposing, what is said above is correct. We will introduce “variant selectors” for each level (namespace / feature name / value) and these will be matched against the metadata of the wheel itself.
This allows us to not only break the assumption of identical dependencies accross a release (why break something if we don’t have to) and provide an easy to manage variant specific dependencies without having to dynamically define dependencies which could become a headache depending on the build matrix.
I quite sincerely this approach is the best in-between of all the solutions possibles. Though happy to reconsider if someone has a better idea
Are there any expectations around how installers should handle where either the system changes (e.g. replacing a GPU) or where the build/install system is different to the runtime system (e.g. building OCI or singularity containers in CI)? Dynamic dispatch would reduce both these issues (assuming a fallback option that worked on any system expressible by current wheel tags was installable), and means most users wouldn’t need to know the details of how the dynamic dispatch works.
Doesn’t this assume that there are no additional wheels produced at a later point which are uploaded to support new variants, which will need new metadata?
Exactly the way it currently works. Zero expectation. If you change something related to your OS (today) or upgrade Python or anything, there’s a good chance your install will break and you will have to re-install.
Well it will be exactly the same. Hopefully “hardware” is probably less likely to be a fast moving parameters than software components: sudo apt upgrade -y gpu 
I wish … But no 
And I believe it’s important that we don’t break that assumption. It both be a total nightmare to implement, and a complete overall of installer logic.
So: Nope !
We will have a section in the PEP about containers. Because it’s precisely a usecase where the hardware will rarely match between builders/consumers.
This PEP is designed in large part to stop “dynamic dispatch” being the only possible answer. It has so many problems … One of them you can’t force everybody to do it => so that will never work 
No, the metadata are hardcoded inside the wheel themselves. If you publish a new wheel it doesn’t change anything, can include different metadata.
Check this webpage: MockHouse - A fast development platform for the WheelNext Open-Source Initiative.
It’s a good “visual representation” of what these metadata are.
Let’s say there’s a new amd64 baseline (v5) that comes out after I do my initial release, and one of my (external to the Python ecosystem) libraries adds support (internally, so no ABI changes) for the v5 baseline. Can I not build a new wheel (making no changes to the source package) and it will produce a new variant? And a package depending on my package decides to do a new build on top of my new build as well, and so there must be new metadata there for the selectors?
I’m guessing the order this happens is:
If this happens to one of your dependencies, and you haven’t set a max version for your package[3], I don’t think you do anything–users install your package, and the installers should resolve to whatever the best variant of the dependencies are for them.
I explicitly called out making no changes to the code in my example (it was a separate library, but it could be a complier gaining new optimisations). The point being, you either hardcode the metadata and you can’t reflect changes outside your package which affect your package, or such metadata is autogenerated (which is likely the smarter route, less chances for errors) and then an ecosystem change must end up modifying the metadata. Hence variants do not solve the “metadata must be the same across wheels and correct” problem. PyPI could always require that wheels have the same metadata, but that doesn’t mean that installers can ignore other sources of wheels.
Well right now - nothing stops from you doing that and this proposal won’t prevent you to do that.
Now - I personally think it’s not a great idea to go back to previous releases and change stuff (even if it’s just including a new platform / variant).
But really, as the package maintainer you can do this and nothing is stopping you.
You will probably break uv and poetry, but that’s your decision 
If you think about it - there’s nothing specific to this proposal to the scenario you propose…
You release in January version 1.2.3 of your package with only Linux support.
In July, you decide to go back and add Windows support to the 1.2.3 release and that includes new windows-only dependencies.
The only rule - at least on PyPI.org - is that files are immutable. You can’t go back and update them. As long as you don’t do that - frankly, everything is fair game (though not necessarily a good idea …)
Exact same scenario: supported today, supported after this PEP.
Should you do that: IMHO no - but that’s a different debate.
You can, and if you indeed indicate that this is v5 baseline variant, then indeed it will have the metadata saying so. However, depending on the index implementation, you may not be able to update the *-variants.json file to expose the new wheel.
I don’t really understand what problem do you see here.