The 10+ year view on Python packaging (what's yours?)

[BrenBarn (Brendan Barnwell)]

@sinoroc

[I have not read @BrenBarn’s post, too long, sorry. But I hope it is fair to answer the question in the title anyway.]

Sure. Most of the post is just me trying to bring together what a few different people (including me) thought about the topic, so more is welcome.

@sinoroc

One item on my wish list is interoperability between (packaging) ecosystems. For me this means interoperability between for example the (packaging) ecosystems of Python, Node.js, Rust, as well as system packaging ecosystems such as apt, yum, brew and so on.

When I see initiatives like PyBI, PEP 725 , purl , and others, I can’t help but think what if I could npm install python Django or pip install nodejs reactjs? Wouldn’t it be great if pip could install gcc in order to build C extensions?

Conda can install nodejs, R, etc. It looks like there’s even a conda package for gcc although I haven’t really used that.

However, at least right now as I understand it this isn’t really “interoperability”, it’s more just people repackaging things for conda. It’s not that conda can use npm to get npm packages, it’s just you can use conda to install node and then use that npm to install npm packages. It seems to me that having true interop, where different language package managers could actually install one another’s packages, would be quite a heavy lift. Of course, this thread is about dreaming, so yeah, it would be cool.

@sinoroc

I am tired of having things installed with pipx, others with conda, some with apt (I have managed to avoid brew and nix so far) and losing track of what should be updated when and where.

That is indeed a pain point with the current situation. It sounds like what you’re describing is a situation where each package “doesn’t care” which manager (conda/apt/etc.) installed it, and the different managers themselves would be largely equivalent. That would be pretty awesome although again, a bit of a reach. I feel like to at least get parity among Python-oriented tools would be a start.

Also, I tend to be a little uncertain about a situation where there are multiple tools that have total interop. If (in the imagined future) every tool can do everything the others can do, why are there multiple tools instead of one? And if different ones can do different things, how does interop work between a tool that can do X and one that can’t? It seems that usually in this situation what winds up happening is that some information gets lost in the transfer of information between tools. I could imagine different tools that just provide different UX layers over the same base functionality though (sort of like different email clients or web browsers).