2024 Python package manager overview

A colleague of mine (does not seem to have a handle here) pointed out the following article, which I think is a pretty nice – even if opinionated – high-level overview over the current state of affairs in python packaging land (at least from the POV of user-facing tools):

The one nitpick I have is that I’d wish the article underscored a bit more is how pixi is explicitly trying to bridge the gap between the world of PyPI-based packages and conda packages, which is (in my biased opinion) one of the coolest things to happen in python packaging in a long time.

PS. I’ve found the author on github, but doesn’t seem to be on DPO either – in any case, thanks!

5 Likes

Curiously enough, one of my friends, who I also don’t know to have a handle here, showed me the same article :slight_smile:

Personally I wish more were written about how we got to this point and why the problems aren’t trivial (including ones that are neatly, almost accidentally sidestepped by others).

I wrote this thing a while ago, but it’s in French: L’installation et la distribution de paquets Python (1/4) - LinuxFr.org / L'installation et la distribution de paquets Python (2/4) - LinuxFr.org.

1 Like

There has been some. Let’s link them! The more we link them, the more likely people will find them and read them.

The ones I recall:

5 Likes

I thought I’d read Pradyun’s post before, but now I’m not so sure. I’ll probably want to dig into it (again).

I’m not a huge fan of the threaded-short-message style of discourse (Twitter, Mastodon etc.) generally, and would much rather have had those points as a blog article. But Stargirl’s main points are well taken. I guess it’s worth directly linking the previous PDO thread that got mentioned in the replies there, as well as the other blog post that came up a few times (since in particular, pointing out “how everyone else is doing it” doesn’t explain why - or if - we can’t follow suit, but is a very useful starting point for analysis):

I just (re-read?) it and it’s a great article. Not to take away one iota of the great points Pradyun is making, but I think it underplays the perspective of the myriad of personas that are involved here, which is one of the fundamental reasons why IMHO, “fixing packaging” is so difficult. The article does touch on this under the On formal UX analysis section.

I say this knowing full well that Pradyun totally does understand the complexities of the user cases and varying needs of all those personas.

Agreed[1]!

Quoting from Stargirl’s thread:

You gotta deal with the Python Core team and the steering council. They have consistently abdicated the details of packaging to the community. They aren’t, at this time, very interesting in taking over packaging and telling the community how to manage their dependencies.

There are decades of history for why and how we got in the current situation. Past active disinterest is certainly one component of that.

While I can’t speak for my SC colleagues, or for the SC officially, I can say that I am personally of the opinion that we’re way past the point where this laissez-faire approach is serving us well. I’m not sure I can say what the shape of a more hands-on attitude toward packaging looks like, but I think we need to recognize that this is causing considerable pain for our users, and it’s our collective responsibility to commit to a compassionate response to making this better, even though it may take another decade of sustained effort to solve.


  1. on both counts ↩︎

12 Likes

@kknechtel I thought you meant to share resources that are written by the makers of Python packaging ecosystem as a counterpoint to the usual rants that keep popping up again and again.

I would not recommend re-sharing Chris Warrick’s article, it has too many inaccuracies in my opinion. This kind of articles articles keep getting “viral”, everyone has seen at least a couple of them. It feels to me like they attract too much negativity and are often inaccurate which does not help motivate the volunteers actually doing the work.

3 Likes

I did, originally. But now I see value in gathering all of these sorts of think-pieces.

From what I could make of Chris’ post, it isn’t inaccurate (aside from perhaps a biased summary of prior PDO discussion) - it just presents opinions that are less than sympathetic, and fails to anticipate how people might respond if they got what they think they want. (I have more thoughts inspired by what’s said there about venvs in particular; but they belong in a separate thread.)


A friend gave me another blog link I like in this context:

https://blog.williammanley.net/2022/02/23/pip-and-cargo-are-not-the-same.html

1 Like

My requisite response to that XKCD comic.

4 Likes

There is a condensed, UX representation of that comic at 4:25 :slightly_smiling_face: https://youtu.be/BgxklT94W0I?si=EGqYv7hvziFIfMWi&t=265

This is why I am adamant about not expanding the capabilities of pip and always push back against ideas to ship a full project manager with every Python distribution. Users should only be downloading a tool like Hatch and this tool will manage everything for you with a clean UX. Although Python has to support many more gnarly use cases compared to Cargo or NPM, there is no reason why everything should not be exposed through a centralized interface that lives independent of a Python installation.

3 Likes

While I was writing about CPython at the time, Considering Python's Target Audience | Curious Efficiency also ties into @theacodes’s Mastodon thread about the diversity of use cases that Python’s packaging ecosystem needs to cover.

Specifically, the listed Python use cases frequently also have different needs and expectations around what they want to get out of a package management tool (ranging from “I don’t want to deal with any dependencies that aren’t already preinstalled” all the way up to “I want an opinionated workflow management tool that tells me how to structure my entire development project rather than just installing the packages it is told to install where it is told to install them”).

3 Likes