Idea: selector packages

[msarahan (Mike Sarahan)]

Thanks for that reference, Jeremy! I had missed it back then. There was so much great knowledge in there from @virtuald @steve.dower @njs and the rest. My ultimate conclusion from that thread was summed up well by @barry-scott and @rgommers

@rgommers

I do have to caution that this is only a reasonable idea if you control all the separate pieces.

@barry-scott

At the end of the day you will end up following in the footsteps of the conda, Fedora, Debian et al.

It’s not that conda, Fedora, or Debian have better tools. It’s that conda, Fedora, Debian, et al. are coherent, organized groups with consistent build practices. There’s a lot of compatibility “metadata” tied up in consistent build practices. Have you ever tried to mix more than a couple of PPA’s or similar that have package overlap?

Can NVIDIA get away with the approach of pushing CUDA into a separate wheel, and then using RPATH or similar to make RAPIDS libraries work? Probably, since NVIDIA controls all the parts. What does it mean for other projects that might want to utilize CUDA?

My motivation ultimately is how to loosen the need for tight control of package systems and allow more interchangeable parts. I worked on the Conda and Conda-forge ecosystems for a while, and slowly developed tools for improving compatibility. The notion of “run_exports” (originally named “pin_downstream”, originating in WIP Build customization by msarahan · Pull Request #1585 · conda/conda-build · GitHub) was especially helpful. It basically does the dependency injection described by @njs:

@njs

The big thing I’d suggest adding is metadata for “if you build your wheel against this version of this package, then here are the install-requirements that you should add to your final wheel’s metadata, to make sure that an appropriate version of this package is available”. So e.g. building against ntcore 1.2.7 might suggest a runtime requirement of ntcore == 1.2.*, >= 1.2.7, or ntcore == 1.*, >=1.2.7, or even ntcore == 1.2.7 – whichever one matches ntcore’s ABI compatibility guarantees, which the ntcore distributors will understand better than their users.

Even with all of that, it was very easy and common to have accidental breakage with package updates, especially when both Anaconda defaults channel and conda-forge were involved. My conclusion from all of this is that we were/are not capturing compatibility data adequately. I see selector packages, PEP 725, et al. as tools to either capture or utilize “fingerprint” metadata that first and foremost will prevent using incompatible things together. Things like selectors are ways to make decisions based on metadata. If we can understand and feed all of the different compatibility facets into the solver (pip or otherwise), I believe we can improve the shared library situation in a safe and reliable way.

Worst case scenario is that you just end up not being able to share libraries broadly, but at least you’ll understand why they aren’t compatible.

EDIT: I was remiss in not mentioning ABI Laboratory, by Andrey Ponomarenko, which was truly essential in figuring out what compatibility bounds should be for each package.