@tgamblin I understand exactly what you’re asking and why, and could write an essay-length response, but I’m afraid that it’s going to raise even more questions from others who aren’t as deep into this stuff as you are. So I’ll keep it to some essential points here; we may want to jump on a call to talk through some of this and how this will relate to Spack possibly consuming wheels in the future, and if and how that may need to be addressed in design and/or PEP, or summarized for the audience on this thread.
Re BLAS: please have a look at the first figure on BLAS, LAPACK and OpenMP - pypackaging-native . The “wheels vendor a BLAS library” isn’t assumed/required to change with wheel variants (it could, but unvendoring is an orthogonal topic/choice to which BLAS library is used at build time). To do that, we’d indeed either need a BLAS package which acts as a mutex somehow that guarantees uniqueness (like conda-forge
), or direct support in the resolver for achieving the same (like Spack’s +blas
), rather than only a variant property. In this design/PEP, we definitely cannot say or prescribe anything for resolvers in this regard. And a package can be written, but there’s no mutex concept nor an OR
operator in the grammar for dependencies to express something like blas_impl_mkl OR blas_impl_openblas OR blas_impl_accelerate
).
Also note that we actually do have OpenBLAS wheels (scipy-openblas32
/scipy-openblas64
packages) on PyPI. But we’re not using them as runtime dependencies, yet at least, only as a distribution mechanism to consume as a build-dependency. That is an orthogonal topic to wheel variants. I’d really rather not dive deeper into that here - that gets us back to Enforcing consistent metadata for packages .
ABI consistency is a broader example of the need for uniqueness, but otherwise similar to BLAS. We don’t tackle that, and my take is that it’s inherently unsolvable since that’s just not how PyPI and wheels work, see PyPI's author-led social model and its limitations - pypackaging-native . Spack, Conda, Linux distros, etc. all solve this problem in very different ways but do all end up with a consistent set of packages leaning on shared libraries. PyPI/wheels do not, they “solve” it by hiding the ABI and vendoring the shared libraries. Trying to change that requires things like mutable metadata, being able to rebuild packages in a centralized fashion, and other such topics that are each individually very hard and/or already vetoed by this community multiple times in the past (often for good, PyPI-specific, reasons).