Implementation variants: rehashing and refocusing

Packaging
4

I think that the critical question here is whether variants are different distribution names or different wheels/builds for the same distribution: variant distributions or variant builds. My preference is to have variant builds/wheels that all have the same distribution name and are built from the same sdist because in the cases I am familiar with that is literally what is happening.

Just having variant builds at all and being able to specify them explicitly in requirements is already a big topic/change in itself. I think that is what leads some people to point towards variant distributions as the quicker fix but ultimately it would be better to make variant builds work. That just needs a lot of thrashing out though before even considering automatic selection in any detail because it needs changes in many places:

  1. The database of installed distributions (originally PEP 376) would need to represent which variant build is installed (whether it was installed by pip, conda, apt etc).
  2. There needs to be a way to encode the variants in wheel filenames (PEP 427).
  3. There needs to be a way for tools to discover what variants are possible for a given distribution and version combination.
  4. There needs to be a way to encode the variants in PEP 440 requirements so that a user can request a particular variant from pip or another wheel or requirements.txt can require a particular variant.
  5. There needs to be a way to build particular variants from sdists for the case when installation falls back on sdists (PEP 517).
  6. There also needs to be a way for maintainers/distributors to build and encode particular variants which is separate from the case where e.g. pip attempts the build. Tooling like cibuildwheel etc would need to be changed to build the variants.

Lastly to add to the test cases for proof-of-concept: there needs to be a transition plan for any proposal. The question is what happens when e.g. an old version of pip encounters a distribution that is shipping new build variants on PyPI that it doesn’t understand. That is not just a hypothetical or short-term transitional problem because old versions of pip are everywhere and will continue to be so and will also continue to be used by often inexperienced users.

4 Likes