Proposal to create a Python Packaging Council

Packaging
1

As the recent Packaging strategy discussions have revealed, there is an appetite for a council or authority that would set a vision, settle technical disputes, and act as an authority on decisions for the packaging community (or at least for the PyPA). This council would define the vision for the Packaging community and projects, and work in tandem with the Python Steering Council to ensure a coherent strategy across the core Python and Packaging ecosystem. Such a council would be the final authority on approving packaging PEPs and working with the PSF to identify funding for major packaging projects development.

Based on the feedback from the users and Packaging maintainers, it is clear that everyone wants to see change in terms of improving user experience, community interaction and seeing PEPs actually get implemented. A council would be able to facilitate these improvements. Such a council might also be able to drive change in a more timely manner. Needless to say, all decisions and their motivations from such a council would be transparent and shared with the community.

The purpose of this post is to gather your thoughts for this proposal. Mainly,

  1. Is there an interest in creating such a council?
  2. Considering PEP 13 as a proven model for the Python community, what adaptations might be necessary to establish a similar PEP more suitable for Python Packaging governance?
  3. Who would be eligible to run for the council? Who would be eligible to vote in electing members to the council? What is the ideal size of the council?
  4. What powers would this council have? Do we want it to have powers beyond approving/declining PEPs?

Please share any other questions/sticky points we need to discuss. The main outcome of this post will be understanding whether a majority of community members are for or against creating such a council. Our goal is to use this feedback to develop a more concrete proposal, to be raised as a subject at the Packaging summit at PyCon US 2023.

14 Likes
2

I think so, yes.

Probably not a huge amount, but I think it would need some modification. Do we need as many SC members as core Python (you cover this in the next point)? Do we have someone who will manage the voting process? Is an annual turnaround the right period? Many packaging changes are multi-year projects, so more continuity might be better.

That will be difficult. While we don’t want to exclude people, allowing completely free access is likely to be impractical. The only equivalent we have to the core devs is the PyPA members, and given that part of the problem is the “open” nature of the PyPA, combined with the fact that some key players haven’t applied to be PyPA members, I don’t think that would be ideal. But I do think that “having a commitment to at least some packaging project(s)” is an important criterion. I think this is going to be the trickiest question to address.

My feeling is that we should start - before discussing a council - by formalising the structure, membership, goals and scope of the PyPA. Otherwise we risk creating a council that is ineffective by design.

It would absolutely need powers beyond PEP approval. Speaking from experience, being able to approve PEPs is nowhere near enough authority to influence the sort of questions that triggered this idea. I’d go so far as to say that if the council only had PEP approval powers, I don’t think it’s worth forming it. (Unless people are so fed up with me and Donald as standing delegates that they just want us removed from the role - but if that’s all that is behind this, just say and we can step down in favour of someone else with no need for a council :slightly_smiling_face:)

6 Likes
3

That was the goal of PEP 609, which is still active. Do you mean there are gaps in that PEP we need to address? Or do we need to deprecate that PEP and create a new formalization entirely?

1 Like
4

I’m thinking specifically of @pradyunsg’s blog post, specifically the sections ”not a PyPA project” and On the Python Packaging Authority, which IMO very clearly describe the problems with the current model.

I have no particular view on how we address these issues, but I think they need addressing before we form a packaging council. Of course part of the solution might be to give the council much wider powers (to declare certain projects or ideas “out of scope”, for example). So the two are intertwined somewhat.

6 Likes
5

I don’t have any particularly strong views on whether a formally governed council is better or worse than the current arrangement (well, apart from believing that it should’ve been under the scope of the Steering Council from the start and then delegated out from there, but that’s a much harder thing to change).

I would, however, like to propose that we see whether it makes sense to move the pre-built distributions of Python under the same umbrella. Right now, this would include the builds from python.org (excluding the source tarballs, obviously), the Docker containers from @barry (which I can’t find right now?), potential future builds for WASM, and potentially responsibility for guiding 3rd party distributors on “what a Python install looks like”.

Right now, these responsibilities are presumed to be under the core dev team, but are not actually listed in PEP 13. Neither is packaging listed, which is why it was determined to be out of scope for the SC, so there’s a possible case for distributions being out of scope as well. Of course, historically they’ve very much been in scope, particularly from the POV of responsibility for security issues (CVEs affecting the installers were treated as affecting the interpreter). So I expect quite a lively discussion :slight_smile:

However, given that the first-party distributions have a much greater impact on packaging and environment management than they do on the core language or interpreter, I think we could see benefits all around by having them be closer to the tools that they get used with.

6 Likes
6

One thing that concerns me a lot about this is that it’s already a lot of work to have informed opinions on the current scope of packaging (ask me how I know :wink:) Adding “how Python gets distributed” into that is going to make that problem even worse.

If we make the PyPA’s (and by implication, the council’s) scope too broad, we’re setting ourselves up for failure. It’s possible to add things later, but much harder to declare things out of scope that were previously in scope. So let’s aim for a scope that’s “wide enough, but no wider” for now.

3 Likes
7

They are on GitLab. I’m all for moving these to a more official governance and wouldn’t even mind if they move to GitHub (it’s just that I won’t do that work :stuck_out_tongue_winking_eye:).

3 Likes
8

I think this is a great idea though I agree with Paul that it needs to be well defined.

Also, while I understand the scope would increase and there are lots of unknowns, I am also in favor of owning the official distributions. This would require lots of documentation that may or may not already exist, and also preferably in person meetings to handoff current duties/knowledge.

5 Likes
9

I don’t have a fully formed opinion on this, but there’s certainly interest. :slight_smile:

Yes. The Python Steering Council has the following powers:

  • Accept or reject PEPs
  • Enforce or update the project’s code of conduct
  • Work with the PSF to manage any project assets
  • Delegate parts of their authority to other subcommittees or processes
  • Remove someone from the core team against their will (non-delegatable, 4:1 vote)
  • Veto over adding new members to core team

I expect that the Packaging Council will have at least these powers. I expect we’d also give them authority over packaging.python.org, and the ability to expel projects from the PyPA, as well as veto over adding projects.

I think we should adopt that as much of that as feasible (I don’t want a debate over voting methods, for example). That’s assuming the PSF would be willing to help us with conducting votes of course, and that we’d setup some checks and balances for managing the voter list (we currently have discrepency around “externally contributors” added repos like pipenv and other inconsistencies like that).

All good questions. My answers:

  • Eligible for council: Anyone nominated by a voter.
  • Eligible for voting: I don’t have a fully-formed opinion on this, and don’t have a good answer. :confused:
  • Ideal size: 5 – we’re larger than CPython as a voting corpus (under PEP 609) already.

PEP 609 (as it stands) fits well for a “we’ll all be building a set of tools that interoperate and don’t step on each other’s toes” environment, and that’s where we were when we wrote it. I hope it’s not controversial to say that we’re pushing against the limits of that open tent model.

I would consider this, if we do go down this route, a case of having outgrown that model. Whether we update it in place vs write a separate PEP is process details IMO – I think we can defer that discussion until after the main questions that OP lists have clearer opinions/answers for them.

I think the funding angle here is worth emphasising – having a specified group of decision makers (whom we collectively are willing to trust as liaisons for handling decision making) makes it much easier for funders to look at the group and go “yea, we’ll fund you”.

That’s a piece in why pip and PyPI got the substantial funding that they’ve gotten (in addition to the work of many in the PSF and beyond); the individual projects had clear buy in and clear decisions made by the relevant decision makers for the individual project. None of this comes for “free”, of course, and folks are gonna have to put in the work to make things happen.

An additional cost is that we’d need to have existing maintainers be willing to give up some amount of control/authority they have on the individual projects to the broader group (pragmatically, modelling after PEP 13 means we’d also give the council power to expel individuals from projects and the PyPA). I think that’s reasonable.

Having a council as a single point of contact for the packaging ecosystem (at large) would theoretically also make it easier for us to coordinate things like rolling out new metadata versions, or PEPs that affect multiple projects, or manage longer-term funded initiatives like a DiR etc.

7 Likes
10

I’m sure we can get the PSF to handle this for us (e.g. Ee has always done it for the SC elections).

Actually, the SC decided to continue with Guido’s historical preference to leave packaging to folks deemed more of experts at it than the core dev team. Since packaging was always separate, building packaging knowledge on the Python core dev team was never prioritized, so suddenly shifting didn’t make sense to the SC. This is why there were PEPs about PyPA governance that the SC ended up approving.

1 Like
11

I would go slightly further than that. I wouldn’t consider such a council to be worth it unless it has authority over all packaging docs at python.org — i.e., the council needs to be able to put top-level pages directly under docs.python.org (like the “installing” and “distributing” pages that are there now), and control the content of those pages.

2 Likes
12

I think that is unnecessary. Ultimately, any documentation change will originate as a PR from a community member. It gets merged by a maintainer of the documentation project (the core devs for docs.python.org, the PUG maintainers for packaging.python.org). Neither the SC nor the packaging council has any involvement - except as final appeal if there is a dispute. And I can’t imagine a case where the SC and the packaging council wouldn’t work together in that situation if needed, so forcing a hierarchy seems unnecessary.

3 Likes
13

Yup, obvious improvements should go in without an authority signing off on them.
The Packaging Council should decide how things should be, and updating the docs to reflect their decision should be an obvious improvement.

Slightly OT: If you have content/improvements for these docs now and see any obstacles to merging them, feel free to ping me directly. I volunteer for removing barriers to docs contributions.

6 Likes
14

I concur that a council would be beneficial to improve PyPA’s decision-making process, given its many member projects and the perceived (correct me if I’m wrong) issues of consensus finding and coordination around user experience, standards, and communication.

A PEP 13 derived election process seems sensible to me to reduce procedural bike-shedding, and would interact with the current PEP 609 draft in a few ways, e.g.:

It is expected (but not required) that the Python Steering Council would delegate authority to sponsor and/or approve/reject PEPs related to packaging interoperability specifications, to individuals within the PyPA community.

This should delegate the authority to the new Python Packaging Council instead of “individuals within the PyPA community”. Oh and I guess it would also be good to just rename the PEP to “Python Packaging Governance” since it wouldn’t be only about the current PyPA projects alone anymore.

I would rather see all publications of Python currently maintained by the community under one governance process, including the packaging docs, docker images, installers etc than having a wild-growth of hard to align subprojects. My thoughts are mostly with the Python users, that don’t know how trustworthy all these projects are and have repeatedly voiced concern about it.

Finally, forgive me if it’s a tiny bit off-topic, maybe it would be worth renaming PyPA to Python Packaging Organization while we’re at it, to get rid of the misnomer and recognize the evolution of PyPA’s governance model.

7 Likes
15

In principle, I agree (and the point about addressing user concerns is important) but I have serious concerns about making the scope so big that the council can’t be effective. That’s something that we can discuss separately, though. And depending on how we set things up, it may even be something the council themselves can adjust once they are in office - although that may be procedurally difficult to limit appropriately.

I’d refer you back to my post here - how many of those items should be under the packaging council’s authority? I don’t think there’s a self-evident answer to that question.

That sounds like a very good idea - although it’s another change for a user community that’s signalling they are tired of changes. And of course, with a formal council “Authority” isn’t as inaccurate as it currently is…

16

Something that came up in a chat with @CAM-Gerlach – if we’re touching the membership model, one thing that might be worth exploring is moving away from having a GitHub organisation being the source of truth for membership of projects. We regularly hit issues due to shared GitHub Actions queues and admin bottlenecks, which would be mitigated if we were to allow projects to be PyPA projects without being under the pypa organisation directly or if we split the org into smaller sub-orgs.

Not fully sure what the shape of this would look like, but this is a part of why warehouse moved out into a separate GitHub org and it’s also a blocker to having heavily-parallelized CIs on individual projects as well.

1 Like
17

FWIW, my initial reaction to the thread was precisely: “wait, doesn’t PyPA stand for ‘Python Packaging Authority’? Now you’re telling me it doesn’t actually have authority?”

18

To me that sounds like a non-issue and we can just request funding to become an enterprise organization which has 500 concurrent runners.

1 Like
19

The “authority” part started out as an in-joke; only that this wasn’t easily recognisable as a joke to outsiders[1], and its origin got even more obscured over time, so that the vast majority of users who encounter PyPA assume that the content is what it says on the tin. Alas…

I don’t think users are tired of changing things that need changing (in fact the survey and the unification thread contained lots of calls for changing the status quo). Beyond that, PyPA has barely any user-facing impact, e.g. no average user would have to change any CLI invocation / files / dependencies if PyPA changed its name[2]. That’s not to say that change for the sake of change is worth it, just that I don’t see a negative impact of changing the name on users.

Fair enough, but then the scope for the council should equally be everything under that “Authority” (i.e. all things packaging).

I still like the renaming idea more though. I think that undoing the joke aspect makes the situation even harder to communicate[3], than starting over with a new name, scope, goal, etc.

  1. unlike, say, “cheeseshop” ↩︎

  2. even for those pulling from https://www.github/pypa/xyz, since github forwards links after renaming orgs ↩︎

  3. I’m sure we don’t want to say “until 2023, PyPA['s name] was just a joke” – but even more accurate (and much more circuitous) explanations risk leaving people with the feeling of having been fooled. ↩︎

5 Likes
20

We’re retreading well-trodden paths in the conversation right now, so I’ll post a relevant link to something I wrote 3 months ago.

The naming is a bike-sheddy topic. I request folks to not derail this topic into a naming discussion for now, and instead respond to the questions put forward in the first post. If you want to argue about naming, go read Remove the "Authority" from Packaging in its entirety and respond there.

1 Like