Trusted publishing of other peoples wheels?

[h-vetinari]

@hansgeunsmeyer

One of the reasons I got enamored with Rust recently is that Rust seems to have gotten this right (or at least a lot more right than the Python dev community has) with cargo and rustc.

Some more background why this is so much harder for python.

@steve.dower

What’s missing is persuading users to use anything other than the default, which is incredibly difficult. Among other reasons, as soon as you are perceived to lag even one version of one package, users will declare your solution “unusable”, no matter what other benefits you offer.

If we do this, it needs to become the default IMO, i.e. get the original maintainers invested/involved in the process, because it’ll be the main delivery path to their users. I’m aware that this a very tall order… Speaking of:

@thejcannon

I’m guessing the immediate next step would be to informally chat with someone(s) from PyPI on their appetite.

One thing that makes this endeavour a couple orders of magnitude more difficult still, is that build farms are expensive, both in terms of resources and maintenance. Such costs are the death of many a good idea in FOSS, because if you can’t find someone to pay for it, it’s just not happening.

For example, conda-forge could not exist without Microsoft (and other providers) offering free CI resources, and without Anaconda footing the bill for Petabytes worth of storage and traffic. Completely aside from that, there’s an army of bots & volunteers, plus a substantial core team (some of which are paid at least in part for their work) to keep things running.

@steve.dower

[…] my point is that these already exist and people don’t like using [distributions]. Also worth noting that virtually every distribution has figured out that they can’t just use the existing infrastructure to distribute their builds, which is why none of them do it.

As a counter-example, CRAN has made it such that the default publication path needs to go through a build farm. I just doubt that it scales to Python, because Python is much more of a glue language than R, and so needs to build a much wider variety of stuff across its ecosystem. Having a build farm that does all of {C, C++, Fortran, Rust, CUDA, Java, JavaScript, …} sanely, much less across all relevant platforms, is a mind-boggling amount of work^[1].

---

1. and would be essentially reinventing conda & conda-forge ↩︎