A colleague of mine (does not seem to have a handle here) pointed out the following article, which I think is a pretty nice – even if opinionated – high-level overview over the current state of affairs in python packaging land (at least from the POV of user-facing tools):
The one nitpick I have is that I’d wish the article underscored a bit more is how pixi is explicitly trying to bridge the gap between the world of PyPI-based packages and conda packages, which is (in my biased opinion) one of the coolest things to happen in python packaging in a long time.
PS. I’ve found the author on github, but doesn’t seem to be on DPO either – in any case, thanks!
Curiously enough, one of my friends, who I also don’t know to have a handle here, showed me the same article
Personally I wish more were written about how we got to this point and why the problems aren’t trivial (including ones that are neatly, almost accidentally sidestepped by others).
I thought I’d read Pradyun’s post before, but now I’m not so sure. I’ll probably want to dig into it (again).
I’m not a huge fan of the threaded-short-message style of discourse (Twitter, Mastodon etc.) generally, and would much rather have had those points as a blog article. But Stargirl’s main points are well taken. I guess it’s worth directly linking the previous PDO thread that got mentioned in the replies there, as well as the other blog post that came up a few times (since in particular, pointing out “how everyone else is doing it” doesn’t explain why - or if - we can’t follow suit, but is a very useful starting point for analysis):
I just (re-read?) it and it’s a great article. Not to take away one iota of the great points Pradyun is making, but I think it underplays the perspective of the myriad of personas that are involved here, which is one of the fundamental reasons why IMHO, “fixing packaging” is so difficult. The article does touch on this under the On formal UX analysis section.
I say this knowing full well that Pradyun totally does understand the complexities of the user cases and varying needs of all those personas.
You gotta deal with the Python Core team and the steering council. They have consistently abdicated the details of packaging to the community. They aren’t, at this time, very interesting in taking over packaging and telling the community how to manage their dependencies.
There are decades of history for why and how we got in the current situation. Past active disinterest is certainly one component of that.
While I can’t speak for my SC colleagues, or for the SC officially, I can say that I am personally of the opinion that we’re way past the point where this laissez-faire approach is serving us well. I’m not sure I can say what the shape of a more hands-on attitude toward packaging looks like, but I think we need to recognize that this is causing considerable pain for our users, and it’s our collective responsibility to commit to a compassionate response to making this better, even though it may take another decade of sustained effort to solve.
@kknechtel I thought you meant to share resources that are written by the makers of Python packaging ecosystem as a counterpoint to the usual rants that keep popping up again and again.
I would not recommend re-sharing Chris Warrick’s article, it has too many inaccuracies in my opinion. This kind of articles articles keep getting “viral”, everyone has seen at least a couple of them. It feels to me like they attract too much negativity and are often inaccurate which does not help motivate the volunteers actually doing the work.
I did, originally. But now I see value in gathering all of these sorts of think-pieces.
From what I could make of Chris’ post, it isn’t inaccurate (aside from perhaps a biased summary of prior PDO discussion) - it just presents opinions that are less than sympathetic, and fails to anticipate how people might respond if they got what they think they want. (I have more thoughts inspired by what’s said there about venvs in particular; but they belong in a separate thread.)
A friend gave me another blog link I like in this context:
This is why I am adamant about not expanding the capabilities of pip and always push back against ideas to ship a full project manager with every Python distribution. Users should only be downloading a tool like Hatch and this tool will manage everything for you with a clean UX. Although Python has to support many more gnarly use cases compared to Cargo or NPM, there is no reason why everything should not be exposed through a centralized interface that lives independent of a Python installation.
Specifically, the listed Python use cases frequently also have different needs and expectations around what they want to get out of a package management tool (ranging from “I don’t want to deal with any dependencies that aren’t already preinstalled” all the way up to “I want an opinionated workflow management tool that tells me how to structure my entire development project rather than just installing the packages it is told to install where it is told to install them”).
Very interesting. Some discussion about these tools is, it seems, at cross purposes because there are multiple base functions of which each tool performs a subset:
environment manager
package manager
project manager
Conda is an environment and package manager. pip is a package manager; venv is an environment manager. Poetry seems to be all three, but with the proviso that the project part is required (i.e., it cannot manage environments/packages independent of a defined project). Articles like this talk about each tool without making clear that they differ in which of these tasks they even attempt.
I still think it makes most sense to start with an environment manager, because packages and projects can be layered on top of that, but not vice versa.
Also I get a kick out of this remark:
The core tradeoff with conda is that it attempts to do package metadata the “right” way […] This can be onerous to adapt in a company that already has many internal python packages.
This looks to me like the all-too-common tragedy of “this tool’s main/only disadvantage is that everyone isn’t using it already”.
Not quite. The Python Packaging Ecosystem | Curious Efficiency goes into more detail, but the gist is that because conda is a general purpose cross-platform component management tool, it intrinsically doesn’t play nice with each platform’s native component management, instead introducing an extra independent layer to the software stack. That can be a good or bad thing depending on what you’re doing, but it’s highly unlikely to be a neutral thing (hence the divergence of opinions on conda’s utility).
I’ve seen your article and I get what you’re saying, but I still don’t really agree. I can’t say I know every possible use case, but I think the number of cases where you really need to “bring your own runtime” is much, much smaller than most people think it is. Most of the cases where people think they need that are just cases where they started out that way but would be better off if they hadn’t.
There’s a very easy trap for software application developers to fall into: believing that all software development is standalone application development. That assumption simply isn’t true (software development is so diverse that there’s no one use case that makes up a majority of deployment scenarios).
Folks deploying to cloud platforms should typically use the runtimes that the cloud provider offers. Operating system components should run on the runtimes built into the native packaging system rather than third party ones. Larger applications embedding Python should build their own Python or use a distro intended specifically for the purpose (such as python-build-standalone). Embedded processors should run MicroPython rather than a runtime aimed at desktop/server/mobile operating systems.
conda is genuinely valuable for the use cases it covers [1], but that still leaves a vast array of legitimate use cases where it isn’t the right answer (and trying to use it anyway will mean you end up spending up more time try to make it work differently from the way it normally works than it would have taken to build something more suited to the use case from lower level building blocks).
as long as you don’t have any non-ASCII characters in your username, or work around the problem by installing it to an ASCII-only path ↩︎
Some specific suggestions/updates (since you asked):
distutils/setuptools used to be able to upload (python setup.py upload), but this was removed (or at least deprecated), thereby decoupling uploads from building.
I think setuptools fully supports using pyproject.toml now (and dropping support for metadata in setup.py/setup.cfg would be a major breaking change, so I don’t expect that to happen any time soon)
I’m pretty sure wheel is now completely merged into setuptools, and the package is no longer needed (but that should be checked).
However, what I think is the biggest issue is that what you are listing is fairly arbitrary (and hence misses a significant part of the picture):
venv is in the standard library (I suspect you meant virtualenv, which is still a bit odd given it’s a library, and most of the other projects use it underneath)
everything is mixed together, with build-backends and build-frontends conflated (I note that pymeson, scikit-build or pyscons are not listed, nor is build).
There’s no indication of which projects depend on which: virtualenv is used as a library by most of the tools, as is calling out to pip.
There’s also the wider issue of related-to-packaging tools: tox/nox, beeware, cx-Freeze, pyinstaller, and even things like docker, flatpak and more general purpose package managers (of which conda is only one).
It also ignores why some solutions are not appropriate for some users, i.e. there’s no context provided, historical or otherwise.
This ignores the fact that the python packaging landscape is so confusing that it is very easy for any user to get into a state like the one described in the XKCD comic. Blaming the user is not a great look, when the tools available make it difficult to build a working system across multiple projects.
