That is what happens in other distros although there you also get virtual packages and alternatives etc so I think we would still need some version of variants somehow. I don’t see enforcing a single BLAS library being achievable in PyPI-land because of the general lack of overall authority as you say.
I think that what would work though is providing the mechanisms that enable projects to do this so that they can converge on making something that works organically. Currently it just isn’t really possible to share binaries of underlying native libraries across different PyPI wheels.
Take a very limited scope and consider python-flint. Its main dependency is Flint which is a C library. If we could I would split it into two packages where one (say libflint) contains the C library and the other (python-flint) has all the actual code i.e. the Cython wrappers and the public interface for users. It would be useful for python-flint to be able to do this even if no other project ever depended on libflint:
- From python-flint’s perspective the Flint library becomes an installable binary.
- Making a dev setup for python-flint would be much easier.
- Most python-flint contributors would never need to build Flint themselves.
- Building python-flint would be about 10x faster.
- CI would take about 5 minutes rather than 45 minutes.
- python-flint wheels would be about 20x smaller.
- The libflint wheels would still be large but would not be specific to the Python version so there would fewer of them and they would only need new releases for new Flint releases so less total storage in PyPI.
- If we did have CPU variant builds then it would only need to be for libflint and not for python-flint.
- Smaller/fewer wheels means less total space on PyPI so we can have more regular releases and support more platforms.
We (as in python-flint) would be in control of both packages so we could do what we need to keep them compatible with an exact version pin. (Exact pins are part of the problem in this approach, particularly for BLAS.)
Even though the benefits are clear we don’t do it. It takes us into the realm of packaging non-Python libraries as Python distributions which has always been something that was discouraged (or disallowed on PyPI?). Note also that doing this would necessarily translate into wheels having different dependency constraints from sdists (cf the consistent metadata thread).
It is also something for which the basic machinery isn’t there: how do I load libflint.so at runtime if it comes from a different package? How do I build against dynamic libraries that can only be found via sys.path and how do I find the headers? How do I prevent users from ending up with incompatible binaries (e.g. if they build libflint themselves)?
Unbundling in the BLAS case is more complicated. More libraries want BLAS and I don’t know how the ABI constraints would translate into version constraints among the wheels for the different projects. There would be a strong incentive to solve this but I don’t know enough to say exactly how it could pan out. The risk is lots of projects making tight dependency constraints on that BLAS library and then resolves becoming impossible.