Over the past year, I have been tracking various packages in the PyTorch, JAX, and other GPU communities, aiming to address the challenges discussed here: What to do about GPUs and the built distributions that support them. These communities have consistently emphasized the need to keep pypi.org as the central distribution platform.
In response, we started developing some experiments under Wheel Next, which serves as a neutral space, not tied to any specific community, and adheres to the same open-source principles as Python.
To those who argue this isn’t an issue, I can point to several examples of file size approval processes that took over a month:
In collaboration with the PSF, we’ve sought to secure more funding for ecosystem support, but the existing limits are already affecting how the community packages the software. For instance, some groups use nightly labels on conda, which is not something supported on pypi.org. I also see more and more is an breaking up of large projects into lots of smaller ones. Additionally, PyTorch bundles many vendored packages, making it harder to build optional dependencies.
The concept of external wheels, or the “wheel-stub pattern,” allows these communities to participate in the ecosystem without burdening users with the complexity of identifying which indexes are trusted. While people often cite poor download speeds as bad user experience, it’s much worse when a package is unavailable due to platform limits. A service that marks when the last package was downloadable and updates the metadata in Wheelhouse could improve the UI for external wheels, helping users better understand what’s available.
The community wants to remain within the pip ecosystem. They don’t want to switch tools just to access these libraries, so this pep helps us keep true to that goal.