Python Packaging Strategy Discussion - Part 1

(I’m a maintainer of pip).

I agree wholeheartedly with this. And I’d go further and say that for such a tool to be successful, it will be at least as important to decide what workflows we won’t support, as what we will. The lesson we have learned from pip is that if you try to support everything that people come up with, you’ll end up in a mess. And of course, the big problem here will be that many of the respondents wanting a unified tool will have been making an implicit assumption that their preferred workflow will be supported…

Again, I agree. I do not think pip is a good base for such a tool. It’s too low level, and it tries to support everyone.

+1 again. Historical complexity can be removed, but do we have the stomach (or the resources) for deliberately breaking how a lot of users work? As I said, I doubt any of the survey respondents were asking for a unified tool that didn’t support their workflow…

Inherent complexity is a different matter. It would be a mistake to assume Python is a “special snowflake” and has nothing to learn from other languages and ecosystems, but we do need to be careful. For example, any lessons we learn from cargo need to consider that Rust has no deployment complexity - you build a binary and ship it[1]. I don’t know npm as well, but I’m sure there are aspects of the Javascript lifecycle that don’t match Python. For example, does Javascript have the concept of compiled extension libraries?

I’m frankly not optimistic about doing something like this. It’s a worthwhile goal, and I completely agree it would be a substantial improvement for Python users. But it could suck up a huge amount of resource, which might be better spent on smaller, more incremental improvements in other areas. Even as a funded project, with its own hired resources, it would consume a big chunk of attention from the packaging community (assuming we wanted at least some say in what the successor to all the tools we’ve built would look like :wink:).

One alternative that I think we should consider is continuing to work on the goal of splitting out the various “components” of packaging into reusable libraries[2]. Projects like installer, build, packaging and resolvelib are good examples of this. Using those libraries, and more like them, it will be a lot easier to build new workflow tools, any one of which has the potential to become the unified solution people want. It’s definitely going to result in more complexity on the way to something simpler, and I’m pretty sure it’s not what the survey respondents were imagining, but I feel that it might be a better trade off for the long term.

  1. That’s an over-simplification, of course. You can build applications with Rust with dependencies on runtime DLLs, etc. But it’s not common, and the ecosystem doesn’t really offer any support for deploying applications built like that. So I guess it’s a “workflow that the tool chooses to discourage”. ↩︎

  2. In particular, I’d like to see an ecosystem where “pip is the only viable approach to installing packages” is no longer the self-evident truth that it is today. ↩︎