Python Packaging Strategy Discussion - Part 1

Packaging
137

OK, here’s my answers.

  • Unification of PyPI/conda models: NO.
  • Unification of the consumer-facing tooling → NO, with a caveat. I don’t think we should try to force maintainers to work on a single tool, but if competition between tools results in users choosing a clear winner, I think we should accept that.
  • Unification of the publisher-facing tooling → NO. I assume this means things like build backends.
  • Unification of the workflow setups/tooling → PARTIALLY. I very definitely don’t think that (like cargo) we should mandate that every time anyone uses Python, they should create a directory, containing a src subdirectory and a pyproject.toml. The workflow of writing a simple script (with dependencies) in a scratch directory full of “other stuff” is an entirely reasonable workflow that we should support. Having said that, I support unified workflows for the tasks of “write a Python package”, and “write a Python application” (althought I think the latter is something we’ve traditionally ignored in favour of “write a package with a script entry point”).
  • Unification/Consistency in the deployment processes → NO. Although I’m not 100% sure what this entails. It shouldn’t be user-facing, though, which is why I say “no”.
  • Unification/Consistency in “Python” installation/management experience → NO. Although I think we should accept that this is not under our control, and like it or not, the main Python website is where people go for advice on where to get Python from. So we should work with the guidance given there, not fight against it.
  • Unification of the interface of tools → YES (but see below).

I’m not sure I understand the difference between “consumer-facing tooling” and “workflow setups/tooling” though. For the purposes of the above, I’ve taken the former as meaning the actual software, and the latter as meaning the processes. So we can have hatch and PDM, but they should manage the same project layout, expect tests and documentation to be laid out in the same way, etc.

As regards “interface”, there are two aspects - low level details such as the names of options, configuration files, etc., and higher level concerns like what subcommands a tool supports. For example, I’d love to see a shared configuration mechanism, so that users can set their network details and preferred index once. And I’d like a common set of workflow commands (things like “run an interpreter in the project environment”, “run the tests”, “build the docs”, “build the project artifacts”, “publish the project”, …) But I don’t want this to be an excuse to argue endlessly over whether an option should be called --config or -C. And I definitely don’t want it to override questions of backward compatibility for individual tools (which should very much be the tool maintainer’s choice).

Regarding the other discussions in the thread, I support better integration with, and support for, other system distributors/integrators. But I strongly disagree with any suggestion that PyPI and wheels should no longer be considered the primary route by which (most) users get Python packages[1]. Having said that, I think such support needs to be a two-way street, and if “other system integrators” want to be supported, they need to engage with the community as a whole and get involved with this process - otherwise, we should accept that what support and integration we provide will, of necessity, be limited (e.g., I don’t think we should try to write a “how to use apt to install Python packages” page in the Python packaging documentation, but we could link to a Debian “how to use apt for Python users” page if they provided a suitable link that was written for the same audience that we are addressing).

I also don’t think it’s at all clear from what I’ve heard of the survey results, what the users are asking for in terms of the above questions. And I think that’s a far more important question than what we think[2].

  1. Maybe it’s a case of “worse is better”, but I strongly believe that without PyPI and wheels, Python would never have achieved the popularity it has. ↩︎

  2. Although if the users are asking for (for example) “cargo for Python”, then my response is “great, I hope someone writes it for them”. Just because the users want it, doesn’t mean that’s where I’ll personally devote my open source (volunteer) time. ↩︎

4 Likes