After letting things stew in the back of my mind for a while, I think that this set of packaging discussions may have started in the wrong place.
I’m not fully caught up on both threads currently, but as best I can tell the discussion started with a focus on tools and functionality—and I don’t think this is the right starting point.
Where I think it needs to start is on people and their needs.
If we don’t have a good picture of who all is out there, and what they need to do with (a) distribution of Python itself and (b) Python package creation, distribution and installation, how can we correctly understand the scope of the challenges, and then make sensible decisions about how many and what kind of tools need to exist, with what functionality? (And yes, I’m in ‘Camp One Tool to Rule Them All Is Flatly Impossible™.’)
As I see it, the solution has to involve[1]:
- Developing an understanding of the landscape of myriad Python distribution and packaging situations and user needs
- Identifying where those situations/needs have mutually-exclusive constraints
- Defining a relatively small number of large swaths of the landscape, each of which can be served by single tools (existing tooling meeting real-world needs can help us here)
- Developing (or continuing to develop) those tools to meet those scoped needs
- Communicating which is the right tool for its corresponding application spaces
- And then, as bandwidth and energy allow, filling in the chinks that are left over
In particular, I think it’s key to recognize that (1) is an extremely high-dimensional problem, spanning different:
- Types of user roles (distributor, packager, installer, …)
- Levels of expertise in different areas, including varying expertise within a single user
- Platforms (OS, PC/mobile/browser, …)
- Project code types (pure Python, C extensions, Rust extensions, …)
- Project types (library, tool, application (further divided to web/CLI/GUI/…), data analysis/science/ML)
- Runtimes (CPython, PyPy, Micropython, …)
- plus more…
Even representing this full landscape seems like a grand challenge in its own right:
But, I don’t see how we effectively move to (2) and beyond[2], without at least a reasonably complete picture of this elephant…of the full sweep of who is building and using Python distributions and packages, for whatever reasons and with whatever goals[3].
Without tackling the first hard problem[4], we can’t effectively tackle the second.
It seems to me that most of the discussions to date have been centered on (4) of this list. Occasionally, conversation has touched on some elements of (1), but in a fragmented and incomplete way—certainly not enough to set the stage effectively for all the attention we’ve paid so far to (4). ↩︎
I think the un- or indirectly-addressed considerations of (2) have led to frustration when one party asserts something about a tool, which another party immediately sees as incompatible with their needs. ↩︎
@rgommers’ pypackaging-native seems to me one example of work toward both (1) and (2), at least for a slice of the ecosystem. ↩︎
Which is arguably the hard-er problem! ↩︎