Use of PyPI as a generic storage platform for binaries

Packaging
41

To a channel: yes; to conda-forge: no. Which I think is fine. If you want to share something quickly and hassle-free, you have no barrier for sharing binaries that people can easily install. If you want to be part of a curated[1] channel, you have to go through some extra steps.

I already see some key contributors behind pixi liked your comment, I’m sure they can explain more. The pixi team has pretty big ambitions that it’s executing on with breakneck pace (including local builds, debugging, publishing workflows and even a GUI); IMNSHO it is the strongest contender towards – as you phrased it – “finding a cross-platform solution that is simple enough for any user.”

  1. I’m using this term generously; the initial review is the only “real” curation, the rest is mostly done by custom bots & linters – or, if you manage to mess up something so badly that it draws the attention of the core team. :smile: I’m being tongue-in-cheek of course, the vast majority of people don’t try to sneak in changes under the radar, but rather ask for help and try to help each other out. ↩︎

5 Likes
42

OK. The problem with “publishing binaries to PyPI” is that it’s not intended for that purpose. But you could easily publish to a different index. It’s just that no such index exists, so you’d have to set one up on your own.

With conda channels, the problem with publishing to conda-forge is that it requires approval. You can publish to a different channel, but the same question applies - does that mean you need to set up your own channel, or is there a suitable one available?

My opinion is still the same - this whole idea of “abuse Python’s publishing infrastructure to avoid having to set up a publishing solution of your own” is not something I agree with, regardless of whether the infrastructure is PyPI or conda. But I do think there’s merit in acknowledging that the problem is real, and considering what (non-abusive) solutions might exist.

For example, the folks at Astral might want to get into this space, and create a new public index that is intended for publishing binaries. Add it to uv’s defaults and it provides a supported solution for the “publish as a Python package” approach. Or someone might want to set up a “free for binary publishing” conda channel to make pixi a solution in this area.

2 Likes
43

Anyone can create an account at https://anaconda.org[1] and start publishing packages to the channel corresponding to that account, for free. I don’t think the bar can be reasonably made much lower than that. It’s in a uniform location, with a simple pattern:

https://anaconda.org/<account>/<package>

that you can install using $tool install <package> -c https://anaconda.org/<account>. Here’s my channel[2] for example.

  1. It’s not dependent on anaconda though, and that same channel could be hosted elsewhere. It’s just that anaconda provides hosting to the community for this purpose. ↩︎

  2. which I don’t actually use except to do work for conda-forge, e.g. to work around an unsolved network issue that prevents some packages on special-ops CI queues from getting uploaded. ↩︎

4 Likes
44

Hi Pixi maintainer here,

As previously mentioned Pixi is designed to solve the complex, cross-platform, multi language installation of binary packages.

pixi is to conda what uv is to pip. A virtual environment and workspace based package manager reimplementation in Rust. Creating workflows similar to poetryand npm.

The conda-ecosystem has a lot of similarities to PyPI. But the most notable difference right now is the main open-source distribution. Where PyPI is hosting an index that anyone can push to, conda-forge is not allowing this without an approval first. There is also the compute costs linked to that. conda-forge hosts all the builds in their own compute cluster.

I’m personally not a conda-forge maintainer, but I could see a future where this would becomes more distributed, mostly because of the issues @pf_moore presents with easily publishing your own work, but also because the reliance on hosted compute has a scaling problem. The current improvements in trusted publishing, sigstore attestations and improved authentication might help the conda-forge ecosystem to change how it decides who/what is approved to push to conda-forge.:man_shrugging:

In the conda-ecosystem, there are multiple options for the channels, just like the pypi indexs; local storage, s3 buckets, artifactory just work. But there are also more focused hosting tools available. Like our own prefix.dev/channels or anaconda’s channels. (conda channel == pypi index)

That said, this is related to the conda-ecosystem. I think Pixi offers a good alternative solution to solve the binary distribution problem. But this doesn’t answer the question whether PyPI should be used to do that. Obviously I hope that packages get published to conda-forge as this makes it available outside of the Python ecosystem.

9 Likes
45

This confuses installing with building and distribution.

It’s easy install a binary like uv, it’s not easy to build a binary like uv, or become so popular that users are willing to look up the URL for distribution.

There are nuances to every package manager that make what you’re saying a vast over generalization.

For example, some package managers only support source code distributions, some package managers only support specific platforms, some package managers don’t do real dependency resolving, some package managers are gated by an approval process, some package managers have legal implications etc.

With regards to that last one it is why some users are wary of conda. If you happened to use the wrong conda distribution you ended up in a meeting with Anaconda’s sales and legal team (I have first hand experience). The reasoning that they used to switch from free to not free that caught many users off guard could easily be used on any other channel that Anaconda hosts.

46

Not at all. If it’s easy to install a “binary like uv”, then that implies it’s easy to install any of the binaries the OP was talking about, because they are all “binaries like uv” - specifically, single-file executables available for all of the common platforms. Which is my point - you can’t claim “it’s easy to install uv” while still claiming “it’s hard to install sqlite-scanner[1]”.

  1. The binary used as an example in the OP ↩︎

1 Like
47

Broadly, I think it’d be really valuable for the

It’s “easy to install uv” in the sense that you can either:

  1. Manually select a version, download an archive for your platform, and unpack it
  2. Run a curl | sh script which does complex platform detection

Neither of those play well with specifying your dependencies in a project and having them be resolve for consistency using robust platform detection. Without publishing the binaries to a package registry, users have to write installers for everything they need and lose out on the benefits of package management tooling like lock files, hash verification, etc.

3 Likes
48

Agreed. Having a Python project depend (somehow) on a binary executable is one of those “grey area” cases where there is potentially some value in distributing binaries via PyPI.

And there’s no reason that any other 3rd party binary couldn’t offer the same installation mechanism. Even the creation of the shell script needed for the curl | sh case could be automated using a tool like the go-to-wheel tool @simonw created to handle the “publish on PyPI” case.

Shipping binaries via PyPI is a neat trick, with some interesting (albeit relatively niche) use cases. Suggesting it’s a general distribution mechanism for arbitrary tools, though, is not reasonable.

1 Like
49

I’m not sure it’s super useful to litigate whether PyPI & related tooling is easier or harder than conda/pixi & related tooling for distributing arbitrary binaries, particularly when I suspect the answer is that the one you’re more familiar with (or the one whose tooling you have already) is the one that’s easier for you :sweat_smile: , unless the thing you’re trying to distribute requires some more advanced feature.

I think it’s reasonable to say that PyPI & related tooling has done a pretty good job at making a system that can distribute arbitrary binaries [1], and that makes it attractive for some people to want to reuse it for use cases it’s not really intended to handle. I don’t think anyone would suggest that it can handle every use case that one might want to distribute arbitrary binaries for (e.g. it’d never replace apt), but I also think that’s largely immaterial.

I think the PyPI side of this is vague but there’s reasonable guidelines? If a tool falls into those guidelines and thinks distributing on PyPI will make their user’s lives easier, then they should! If they think distributing through conda or curl | sh or apt or any other system, then they should do that!

  1. With limitations! Which every system has limitations, and PyPI & related tooling certainly has more limitations than others (like conda & friends) do with regards to arbitrary binaries, but if those limitations don’t apply to you, then they don’t really matter! ↩︎

5 Likes
50

I don’t think that’s quite true. uv[1] is easier to install than the average binary. uv[1:1] has the resources to maintain a fancy install script that handles most of the complexity, along with multiple distribution channels. Most projects simply don’t have the time or expertise to do that.

From my own experience, a while back I built a Rust CLI and published binaries to GitHub Releases. Once I finished, it hit me: How do I actually distribute this to non-developer users? I’ve never really had to think about this with Python projects, especially since tools like pipx and uv tool exist.

I don’t have the expertise or resources to maintain multiple install scripts or distribution paths. So if you want to install my binary, it’s definitely harder than installing uv. I didn’t put it on PyPI because it’s not Python-related, but at the time I really wished there was something equivalent.

  1. I’m sure there are more, and I encounter them every now and then, but they aren’t the majority, and I’m most familiar with uv. ↩︎ ↩︎

1 Like
51

If so, then why do you think that pip, and increasingly uv, are the go-to-first tools that people are reaching for and promoting? I’m not trolling, I’m trying to dig deeper into motivation and user experience.

From my personal perspective, I’m tending more these days to just install uv using my OS package manager[1], and then just go from there, and I’m generally recommending that workflow as the lowest friction way to go. uv run in the shebang for standalone scripts, and my project manager[2] uses uv under the hood by default, so it’s pretty much one tool for a consistent experience.

I used to use pipx for things my OS package manager didn’t have, but TBH, that’s also not a great experience because if I upgrade my Pythons, it’s more work to keep those pipx installed things up-to-date.

  1. homebrew for macOS ↩︎

  2. hatch in my case, which I also install with homebrew ↩︎

52

I think conda and the conda ecosystem has had some bad PR due to UI slowness and legal moves from anaconda inc.

Pixi is working against that, doing a lot of similar things to what uv is doing and radically improving conda. But it’s a newish project and has to work against conda’s baggage.

All that said, if you don’t normally run into issues around non-Python dependencies, uv is probably all you need. If you do, you’re probably already using or looking at pixi.

2 Likes
53

Except that apt in particular has its own problems, not the least of which being stuck on the version of your app tied to the distro version you’re on. It’s one of the reasons why I’ve greatly preferred to develop on macOS + homebrew over the last few years. Yeah, I know there are ways around that, but few are as low friction as brew upgrade.

1 Like
54

At the end of the day, all of these systems have their own problems :wink:, maybe some day one won’t and we can all just pack up and use that instead! I was mostly just commenting that a vim vs emacs war [1] with PyPI & Friends [2] vs Conda & Friends probably isn’t super useful.

At the risk of participating in that though…

I suspect the real answer here is something along the lines of the PyPI & co tools are “good enough” for a large number of people, and they come by default with Python [3] (which might mean they come by default with your OS), and there’s a lot of network effect helping to boost them.

I think it’s also pretty common for someone using the conda ecosystems tools to still ultimately end up using pip or uv in addition to those tools [4], either directly or indirectly via the “conda” tools calling into the PyPI tools. There’s a lot more “stuff” available on PyPI, so if you need something from PyPI that isn’t in conda land yet, you end up using it anyways– so shipping on PyPI means you’re available to conda users too, but the same isn’t true in reverse [5].

In other words, I highly doubt the vast majority of people sat down and really considered what package management ecosystem they should use; PyPI & co were the “default” and it worked well enough for a large number of people they never had a reason to use something else, and network effects compounded that.

I suspect a larger number of conda users probably did sit down and consider that, but likely because they ran into issues where PyPI & co did not work well enough, so they had to make a choice somehow.

I say this as someone whose basically never used any of the tools from the conda ecosystem more than once or twice to reproduce some issue, and just operating on what I know of that ecosystem to know that their solution isn’t that different from PyPI [6] in a way that is going to affect something as simple as a Rust or Go binary (or any thing that ends up producing a single file executable).

If some other ecosystem of tools besides the PyPI centric ones that made different decisions had been “good enough” and been “the default”, then I think they’d be the ones with the network effects instead.

  1. VS Code being obviously the one true choice :wink: ↩︎

  2. We really need a word to describe the set of tools that are based around PyPI/pip/uv/etc. I used to say PyPA tooling but that’s not even really accurate anymore! ↩︎

  3. Well uv doesn’t, but pip does, and migrating from pip to uv is an incremental move to your existing workflows, vs migrating to a different package manager. ↩︎

  4. Forgive me if I’m wrong, it’s been a long time since I last looked! ↩︎

  5. This remains, in my opinion, the one true super power of the “PyPI & co” tooling. It doesn’t care what kind of environment you’re in, as long as there’s Python it’ll work. AFAIK Conda and other alternatives all want to own the whole “stack”, so you need to use their environments and you can’t (for instance) install Conda things into a Python you got from Homebrew. Owning the whole stack has other benefits though! So it gives them some of their super powers. ↩︎

  6. And to be honest, most package managers! ↩︎

1 Like
55

PyPI/pip/uv/etc

I usually say the “wheel ecosystem” although I’m not sure that’s really any better.

2 Likes
56

Thanks everyone for this thoughtful discussion. Speaking as PSF Board Chair, I want to acknowledge the importance of this topic.

The PSF has a clear mission, and PyPI exists to serve that mission. As Ee rightly pointed out, this is not just colloquial but a real fiduciary and legal obligation. The PSF’s sponsors generously support PyPI as infrastructure for the Python ecosystem, and we owe it to them and to the community to be thoughtful stewards of that trust.

At the same time, I want to acknowledge that this is genuinely hard to draw a bright line around. There is a well-established history of packages on PyPI that wrap non-Python tools to serve Python development workflows. Build tools like cmake, compilers like ziglang, and linters like cpplint are real, practical examples. Nobody seriously disputes their value to the Python community. The question is what happens when that pattern is generalized into a distribution mechanism for software that has no meaningful connection to Python, and where mature, purpose-built distribution channels already exist in the software’s native ecosystem.

I share the concerns raised about sustainability. PyPI’s operational costs (storage, bandwidth, support, and the human effort of maintaining the software) are covered by the PSF and its sponsors. Encouraging the use of PyPI as a general-purpose CDN for arbitrary binaries, particularly from ecosystems that already have well-funded distribution infrastructure of their own, is not something we should take lightly. The Go module ecosystem, for example, has approximately 2.5x more packages than PyPI and is backed by significant resources.

Donald’s framing is a good starting point: if there is a clear, practical benefit to Python users and developers, that has historically been and should remain acceptable. If someone is simply looking for a convenient place to host binaries unrelated to Python, that falls outside what PyPI is for.

We recognize that the existing Acceptable Use Policy and Terms of Service could be clearer on this point. We are actively looking at this at the Board and organizational level, and our goal is to arrive at a well-considered policy that provides clearer guidance to both publishers and PyPI administrators. Given the range of opinions here and the potential impact on existing packages, we want to get community input as part of that process.

In the meantime, I would encourage publishers to consider whether PyPI is genuinely the right distribution channel for their software, or whether their users and the broader ecosystem would be better served by publishing through the tools and registries designed for their language. The conda OSS ecosystem (including tools like pixi) and language-native registries are purpose-built for cross-language and binary distribution and deserve serious consideration.

14 Likes
57

Hi everyone - this whole thing is my fault! I only just heard about this thread.

If community consensus is that this is a bad pattern I’ll note that on the go-to-wheel repo and other places I’ve talked about this (I don’t know if taking the project down is called for, but I can also archive the GitHub repository). If this results in an official PyPI policy I’ll link to that prominently as well.

Donald Stufft

If there’s a benefit to the Python community to having something distributed on PyPI, then that’s OK. If there’s not really a benefit to the Python community and someone is just using PyPI as an alternative to Homebrew, then that’s something that would probably be discouraged.

This is the key question for me. I’ll try to make the case for why I think this is a benefit to the Python community in most cases.

I first noticed this pattern of distributing binaries via PyPI back in 2022 - initially with ziglang and then with the revelation that the enormously popular playwright quietly bundles an entire node.js executable in its own package!

The more time I’ve spent with this pattern the more I’ve grown to like it.

I want to be able to distribute software systems I’ve written in Python for other people to use. Sometimes that means I need them to have extra tools available, like ffmpeg (did you know PyPI hosts static-ffmpeg these days?) or Deno (I built an experimental Python library depending on that a few weeks ago).

Being able to pull those in as regular dependencies solves so many problems. I can release a script that turns a YouTube livestream into a GIF and drop static-ffmpeg in as inline script metadata and everything just works!

~5 years ago the general consensus I saw from the wider technology community was that Python packaging was a big mess. Today I don’t think that reputation holds at all: I think Python’s packaging story has never been stronger, and a great demonstration of that is that PyPI now increasingly acts as a distribution platform of choice outside our own community.

Brénainn Woodsend

We’re not going to solve cross-language cross-platform package management with this.

I think we’ve come surprisingly close!

I was looking at what it would take to use NPM to distribute a Go binary the other day and it was gnarly - you pretty much need a package that identifies your platform on first run and downloads the necessary binary on demand.

My current intuition is that the pattern is a net positive for Python overall.

  • It helps promote the Python ecosystem - anyone who gets pip or uv up and running just took their first step to joining the Python community.

  • It demonstrates how exceptional Python’s packaging has become.

  • It makes binary tools available as Python dependencies. I see this as the strongest argument in favor - it unlocks an entire world of things that can be turned into an executable.

  • The per-project overhead looks modest to me - half a dozen additional wheels per release likely well under the 100MB limit (my largest Go binary wheel is 11.9MB).

  • On the moderation overhead side it’s not clear to me if this changes the challenge PyPI already faces: we already have policies for handling malicious or illegal Python packages. The big risk would be if this dramatically increases the volume of packages being uploaded to PyPI - the Homebrew alternative Donald mentioned, only worse because Homebrew has an approval step for first publishing and PyPI does not.

fungi

The people maintaining Python package wrappers are often not the same people maintaining the upstream non-Python project, so if they disappear and stop updating the wrapper packages then users who relied those are left in the lurch and may not realize they’re running outdated copies

That was one of the reasons I designed go-to-wheel - it’s intended to run in CI such that any release of the Go package can be automatically shipped to PyPI as well, as seen in this GitHub Actions workflow. I should add that to the documentation!

My overall position on this is that I hope we resolve that this pattern is an appropriate use of PyPI that supports the Python community and upholds the mission of the PSF - but if consensus falls the other way I’ll do what I can to undo the damage and make it clear in the right places that this isn’t an appropriate way to distribute software.

10 Likes
58

Playwright on PyPI is primarily a Python library though, not a Node.js runtime, just as how PyTorch is a Python library and not a CUDA driver.

So if you want to use this as your precedent, then your precedent dictates that go-to-wheel should be including a Python API that completely disguises the fact that there’s also native code present. If that makes sense, great! I suspect it doesn’t make sense, and certainly doesn’t align with the promotional material around it.

4 Likes
59

I don’t personally think a Python API is required, but it does make it a lot more clear that a given thing is being shipped for Python reasons and not for “Homebrew but PyPI” reasons.

In the interim while the AUP is being sorted, I’d probably at least add a note to go-to-wheel that you should generally use this for providing go tooling to Python, and not as a generic ship stuff via PyPI thing.

4 Likes
60

Sure, you can use the CMake or certifi or pybind11 packages as other precedents that don’t lead to “you must have a Python API”.

But if you want to say “Playwright does it” then it follows that you mean it provides a Python API.