What is the PyPA?

[astrojuanlu (Juan Luis Cano Rodríguez)]

A Twitter thread sparked after flit was moved under the PyPA GitHub organization prompted me to ask the question I have been asking myself for many years…

It’s easier to say what PyPA is not. For example, a project being part of PyPA does not mean it’s official:

@takluyver

If PyPA did adopt Flit, I’d like to avoid any suggestion that it’s the ‘official’ or ‘standard’ tool for creating & publishing packages.

It also does not mean that the project is preferred by PyPA:

@pf_moore

I think it would be good to have multiple backends under the PyPA umbrella, precisely to avoid giving any impression that there’s a “PyPA preferred” option.

In addition, it does not make a project necessarily easier to maintain:

@pf_moore

I think it would be good to have multiple backends under the PyPA umbrella, precisely to avoid giving any impression that there’s a “PyPA preferred” option.

because:

@pf_moore

There’s no requirement or implication that just by flit becoming a PyPA project, that all PyPA members will automatically be maintainers of flit (or that they’d necessarily want to be).

On the other hand, as per PEP 609,

there is no formal relationship between the Packaging-WG and PyPA.

Speaking of which, the “Goals” section of PEP 609 is somewhat autoreferential. The list of goals of the PyPA includes:

• Provide support for existing projects under the PyPA
• Foster the creation and acceptance of standards for PyPA projects
• Guide decisions which affect multiple PyPA projects
• Determine which projects should be under the guidance of the PyPA
• Enforce adherence to a Code of Conduct across all projects

which… look like circular definitions to me.

It has also been mentioned repeatedly, here and on GitHub, that the PyPA name is a “joke” and that it does not bear true “authority”.

Despite of that, the full spelling can be read in the minutes of the PSF Board, which has this point in the January 2021 minutes:

RESOLVED, that the Python Software Foundation provide fiscal sponsorship for the “direct project” the Python Packaging Authority (PyPA). The PSF will collect targeted donations and reimburse properly receipted expenses on their behalf up to the level of the donations made to this project.

The only hint I found about what the PyPA is are this sentence from PEP 609:

The Python Packaging Authority (PyPA) is a collaborative community that maintains and advances many of the relevant projects in Python packaging.

and this from the Packaging guide:

This is a list of currently active interoperability specifications maintained by the Python Packaging Authority.

However, the former seems to contradict everything I said above. And about the latter, as per the list of open issues and some of the discussions, the Packaging guide suffers from general lack of maintenance and offers somewhat outdated advice, so I am not even sure about how much trust we can put on that sentence.

I get that packaging is a difficult topic in Python. In part because the plethora of use cases is just too wide (that’s a good thing, and also a curse), in part because of how it has historically developed, in part because folks love to moan in The Orange Site™ about how difficult it is without really understanding anything about it. I understand that PyPA members sometimes get defensive, probably because they are somewhat tired. You do thankless work and the packaging ecosystem has objectively and measurably improved a lot over the course of the past years and I am very thankful for it.

However, circling back to the original question, given that a project joining the PyPA doesn’t have any implications at all regarding its officiality or level of maintenance, that the word “packaging” does not appear a single time in the “Goals” section of the PyPA Governance PEP, and that some (a few? most?) PyPA members have stated repeatedly that the PyPA should not be taken as an official place to look for packaging recommendations…

What is the PyPA? Or, in less clickbaity terms: What can the Python community at large expect from the PyPA?

[takluyver (Thomas Kluyver)]

From my perspective, PyPA is two main things:

• A body to hammer out & maintain interoperability specifications (or ‘standards’), like the wheel format, or PEP 517, so that different tools can work together reliably in the packaging space.
• A home for packaging related software projects that aim to work with these specifications (not imposing that any particular project has to comply with a particular standard, but as a more general expectation).
  • That ‘home’ can help with continuity if e.g. I were to be hit by a bus (PEP 609 refers to this under the ‘provide support’ goal) and with code of conduct issues (because there are other people with some kind of authority who aren’t directly involved with the project).
  • To my mind, putting a project under the PyPA organisation also has symbolic significance beyond the relatively limited practical difference it makes. I don’t want to give the impression that Flit is the official, blessed tool for making packages, but I would like it to be seen as an accepted option which you can expect to keep working.

The ‘authority’ part certainly started out as a joke, but I think it’s gradually becoming less true. Both the details of current specifications and the procedures for adding and updating them are on PyPA websites. You don’t get much more authoritative than that without an army.

(You might argue that @pf_moore and @dstufft are the actual authorities, as the standing PEP-delegates who can approve or reject packaging PEPs. But my understanding is that a PEP delegate is meant to act as an arbiter, drawing a line under a discussion, not unilaterally, and I trust them to do so.)

[mbussonn (Matthias Bussonnier)]

I know it might not be much, but I think I expect that projects under /pypa have a higher quality than things that can be found in a random user namespace. At least I think I hold myself to higher standard if I contribute back to projects from organizations in general.

I think that’s already a sufficient signal for people not closely involved with a given project to know whether this is worth looking into.

[dustin (Dustin Ingram)]

I think @takluyver pretty much got it right, especially the part about how the PyPA has evolved significantly recently, and does produce standards, which does make it somewhat of an authority, even though it really only can ever have authority over its own projects.

One thing I’ll add is that the fact that the PyPA is a fiscal sponsoree of the PSF means that it can a) accept funding, either from the PSF, things like GitHub Sponsors and Tidelift, or external organizations, and b) distribute funding to its member projects by paying for infrastructure including CI providers, or by hiring developers or paying maintainers. (The latter hasn’t happened yet, but there’s not really anything preventing it from happening at this point).

Obviously this is something an individual project can do as well, but having an entity like the PyPA be an umbrella like this makes it easier to acquire this funding and help ensure it will be fairly distributed, since it’s backed by the PSF.

Finally, like @mbussonn said, there is somewhat of a bar for a project to become a “PyPA project”, in that it requires existing PyPA contributors to agree that the project has some degree of usefulness / quality / etc. Becoming a PyPA project doesn’t effectively change the project itself in any way, but for users that already trust existing PyPA projects, this is definitely a differentiator for determining whether they should use it or not.

[bernatgabor (Bernát Gábor)]

@takluyver

You might argue that @pf_moore and @dstufft are the actual authorities, as the standing PEP-delegates who can approve or reject packaging PEPs. But my understanding is that a PEP delegate is meant to act as an arbiter, drawing a line under a discussion, not unilaterally, and I trust them to do so.

They act as arbitrators, kinda, but not entirely. On contentious points, their POV will be the authoritative one (though hopefully, they use the arguments presented by other people to see both sides before making a decision - however, it’s up to their personal POV if some trade-off is acceptable/worse than others - e.g. see PEP 660 vs 662 decision).

[jezdez (Jannis Leidel)]

That joke

As somewhat known to some but lost in a decade since, the joke was originally made to crudely point out the apparent lack of interest in figuring out a better packaging and installation story for some Python users who were not part of the Python core team at the time. In my recollection (which is likely flawed), a lack of authority was seen, so it had to be developed, to be a project home for people interested in Python packaging.

It’s not without irony that a) the joke wasn’t that great though and had to be explained numerous times and often resulted in only little excitement from the person asking, given the lack of a punch line, but more importantly b) the “authority” over time actually gained the structure and experience and people, that it so naively lacked when it started. Obviously that’s all because of the tremendous work everyone has done, in- AND outside this group, to evolve what was there and help Python users install and manage software easier.

In hindsight it’s maybe not the joke that needs reflection, but instead all the progress that has been made in its name since then, and that the way it was achieved was not ideal, non-obvious and often boringly niche. But if that’s not what a great Open Source backstory is, then I don’t know what is.

So here’s to the PyPA, may it continue to struggle to find its purpose in the future!

Now about that distant sister project…