Python packaging documentation feedback and discussion

[BrenBarn (Brendan Barnwell)]

I’m going to pull in some quotes from another thread because I think they’re relevant here:

@steve.dower

If you’re planning to produce a Python package, you need to choose which backend to use.

This aspect is no different from how all development works. About the only difference is that other ecosystems dictated a single option from the start and never allowed the range of diversity we have in Python. So people asking for one option are literally asking us to crush that range of options. Fundamentally, we don’t like that approach to “community development”, which is why we’re resistant.

That comment was made in reference to whether PyPA should make “official” recommendations, but I think there are overlapping considerations between that and this thread. For me personally (and I think for others looking for more opinionated references), the point of an “official” recommendation is it would be so official it would go into the docs.

@lwasser

Pure python packages they can use any back end. Those users don’t need to know too much about back ends if they don’t want to know about back ends

I agree with this, but to me the question is, if those users can use any back end, then in documentation that is tutorial/howto style, why is it necessary to tell them anything but “use this one, click here for information about others if you’re interested”? For instance in the PyPA guide the very first sentence is:

This tutorial walks you through how to package a simple Python project.

If that is the goal of that tutorial, that tutorial only needs to tell people one recommended way to do that task. We don’t need to tell people how to do other tasks in that particular tutorial. (I think we do need docs explaining how to choose a build backend for more complex tasks, but we don’t need to burden the entry-level tutorial with that.)

@steve.dower

For basic, pure-Python apps that just zip up everything in the repo, yeah it’s possible to standardise things. I think I’ve written maybe 2 of those in the last five years

But let’s remember you’re an expert! It’s true we need documentation that can help people become experts, but I think what’s more urgently needed is documentation for non-experts.

There’s one thing that I strongly believe, but that I think is getting a bit lost in the discussion of various build backends: documentation for Python can and should privilege the use case of creating a Python package. For me, that also includes pure-Python packages that have dependencies on non-pure-Python packages. As long as the dependencies can be satisfied by auto-getting pre-built binaries^[1], that to me falls into the category of “simple thing that we should have an end-to-end walkthrough for”.

Although many people in this discussion are intimately familiar with the complexity required to build non-Python dependencies, I think that does not represent the broader userspace. There are orders of magnitude more packages that depend on, say, Numpy or Qt while being pure Python themselves, than there are packages like Numpy or Qt that directly build non-Python parts as part of their build process. I think we should recognize this and have a tutorial that clearly states its limitations (i.e., it does not handle direct non-Python deps), and that tutorial can give explicit examples using a single backend.

From the other thread again:

@steve.dower

Listing the known backends that all fundamentally work the same is better - or possibly a feature matrix to make comparisons easier.

I may be misunderstanding this, but let me ask again: if there are multiple backends that all the work the same. . . why do we need them all?

---

1. whether those come from PyPI or anaconda or conda-forge ↩︎