Following the suggestions given in the previous PEP 817 thread, we have decided to split PEP 817 into a series of smaller PEPs, with the hope that this will make it easier to comprehend the concept and discuss it. We have opened a pull request for the first PEP in series, focusing on the changes to the wheel file itself.
For more high-level introduction to the topic, please see the previous introductory post. This PEP specifically focuses on the low-level details necessary for variants wheels to work, that is:
adding variant label to the filename
storing variant properties in the file
exposing variants on the index
ordering/selecting variants
introducing variant-conditional dependencies via environment markers
exposing variant wheels in pylock.toml
The PEP keeps variant properties abstract, deferring their governance and determining their compatibility to a subsequent PEP, along with building wheels. We’ve also significantly cut motivation down (the original is kept in PEP 817 for reference). We’ve tried to make the “specification” part easier to comprehend, and removed the duplicate “rationale-overview”, in favor of a more focused “rationale” section.
Compared to the previous iteration of PEP 817, we’ve also corrected the variant ordering algorithm to handle corner cases better.
We’d really appreciate your feedback, both concerning the technical aspects of the proposal, as well as anything that needs to be clarified or explained better.
We haven’t decided on the exact content of next PEPs yet. We feel like this new PEP is at the right level of granularity, but we’d like to learn from the feedback before deciding on the next ones.
Thank you for the work you’ve done splitting out this part of the spec. This PEP is a huge improvement over the previous one. I’ve read it all (although I had to skim the second half, as I didn’t have time for a full read right now) but my first reaction is that this is a very reasonable and well-written proposal.
I’ll need to go through it in more detail, and I’m sure questions will arise in the subsequent discussions, but it looks like a solid PEP at this point.
If multiple related subsequent PEPs are planned, it might make sense to reserve consecutive numbers early, makes it far easier to refer back to them. See PEPs 634, 635, 636.
It is unclear to me if in *.dist-info/variant.json and in the index level {name}-{version}-variants.json I have to, am allowed to, or am not allowed to have a variant listed named null (assuming I have a null variant)?
For the dist version, it says that it should contain the label of the variant wheel it’s in, and for the index version, it says it should contain all variants (which include the null variant I’d think).
In #variant-ordering you specify:
For every namespace, the tool MUST obtain an ordered list of compatible features, and for every feature, a list of compatible values.
I agree with the previous posters that this new version is much clearer! I especially like the variant markers section, it should make it much easier to select appropriate wheels for specific systems.
I’ve got some questions about the variant.json which I think all stem from the requirement that it be frozen, which seems to lack a clear justification (which should probably be added):
What should happen in the presence of multiple indices, especially if the variant.json differs between them (e.g. you have PyPI and piwheels)?
For locally-built wheels, what should the variant.json have (if anything)?
A follow up is how this would interact with private/non-PyPI indices where there are additional builds outside of the set that maintainers have uploaded to PyPI (e.g. for newer hardware/baselines or where PyPI is not yet ready for an architecture for which multiple variants would make sense; I’m thinking here about RISC-V where there seem to be lots of different profiles, some of which may be “weaker” that the riscv64 manylinux baseline).
Why is the ordering per package, rather than a global setting? I would imagine for features like BLAS you want to be able to configure this globally (and block specific variants like mkl) rather than leave this to the package maintainer.
Why lock the hash of the variant.json, rather than including the contents directly within the lock file (which would mean the user of the lock file would not care if the variant.json file on the index changed)?
If a tool supports installing wheels from two registries for the same version of a wheel, it should also select the variants.json from the registry of the installed wheel. Or even simpler: If you merge two registries internally, also merge their variants definitions. There are a lot of implementation questions involved, but they can all be solved on the tool level.
Not for PEP 517 build (for now, to not break tools), but if you explicitly request a variant, you can build one. Building is for a future PEP
Is that different from registry merging?
The are two different levels of selection: The per-package ordering is the default, it applies if you just do a pip install foo and ensure that it installs the same best wheel for everyone, allowing maintainers to express preferences. Globally, you can globally exclude mkl (up to tool author what exactly they want to support here) and this will apply to all tool resolutions.
This isn’t a strong decision, but lockfiles can already be large and this file felt too large. For example, exporting the airflow workspace to pylock.toml results in a 1.8MB lockfile with 10k lines, the size of this is already a bottleneck. On the flipside, this of course means additional network requests on a cold cache.
