Proposal to create a Python Packaging Council

I’m sure we can get the PSF to handle this for us (e.g. Ee has always done it for the SC elections).

Actually, the SC decided to continue with Guido’s historical preference to leave packaging to folks deemed more of experts at it than the core dev team. Since packaging was always separate, building packaging knowledge on the Python core dev team was never prioritized, so suddenly shifting didn’t make sense to the SC. This is why there were PEPs about PyPA governance that the SC ended up approving.

1 Like

I would go slightly further than that. I wouldn’t consider such a council to be worth it unless it has authority over all packaging docs at python.org — i.e., the council needs to be able to put top-level pages directly under docs.python.org (like the “installing” and “distributing” pages that are there now), and control the content of those pages.

2 Likes

I think that is unnecessary. Ultimately, any documentation change will originate as a PR from a community member. It gets merged by a maintainer of the documentation project (the core devs for docs.python.org, the PUG maintainers for packaging.python.org). Neither the SC nor the packaging council has any involvement - except as final appeal if there is a dispute. And I can’t imagine a case where the SC and the packaging council wouldn’t work together in that situation if needed, so forcing a hierarchy seems unnecessary.

3 Likes

Yup, obvious improvements should go in without an authority signing off on them.
The Packaging Council should decide how things should be, and updating the docs to reflect their decision should be an obvious improvement.

Slightly OT: If you have content/improvements for these docs now and see any obstacles to merging them, feel free to ping me directly. I volunteer for removing barriers to docs contributions.

6 Likes

I concur that a council would be beneficial to improve PyPA’s decision-making process, given its many member projects and the perceived (correct me if I’m wrong) issues of consensus finding and coordination around user experience, standards, and communication.

A PEP 13 derived election process seems sensible to me to reduce procedural bike-shedding, and would interact with the current PEP 609 draft in a few ways, e.g.:

It is expected (but not required) that the Python Steering Council would delegate authority to sponsor and/or approve/reject PEPs related to packaging interoperability specifications, to individuals within the PyPA community.

This should delegate the authority to the new Python Packaging Council instead of “individuals within the PyPA community”. Oh and I guess it would also be good to just rename the PEP to “Python Packaging Governance” since it wouldn’t be only about the current PyPA projects alone anymore.

I would rather see all publications of Python currently maintained by the community under one governance process, including the packaging docs, docker images, installers etc than having a wild-growth of hard to align subprojects. My thoughts are mostly with the Python users, that don’t know how trustworthy all these projects are and have repeatedly voiced concern about it.

Finally, forgive me if it’s a tiny bit off-topic, maybe it would be worth renaming PyPA to Python Packaging Organization while we’re at it, to get rid of the misnomer and recognize the evolution of PyPA’s governance model.

7 Likes

In principle, I agree (and the point about addressing user concerns is important) but I have serious concerns about making the scope so big that the council can’t be effective. That’s something that we can discuss separately, though. And depending on how we set things up, it may even be something the council themselves can adjust once they are in office - although that may be procedurally difficult to limit appropriately.

I’d refer you back to my post here - how many of those items should be under the packaging council’s authority? I don’t think there’s a self-evident answer to that question.

That sounds like a very good idea - although it’s another change for a user community that’s signalling they are tired of changes. And of course, with a formal council “Authority” isn’t as inaccurate as it currently is…

Something that came up in a chat with @CAM-Gerlach – if we’re touching the membership model, one thing that might be worth exploring is moving away from having a GitHub organisation being the source of truth for membership of projects. We regularly hit issues due to shared GitHub Actions queues and admin bottlenecks, which would be mitigated if we were to allow projects to be PyPA projects without being under the pypa organisation directly or if we split the org into smaller sub-orgs.

Not fully sure what the shape of this would look like, but this is a part of why warehouse moved out into a separate GitHub org and it’s also a blocker to having heavily-parallelized CIs on individual projects as well.

1 Like

FWIW, my initial reaction to the thread was precisely: “wait, doesn’t PyPA stand for ‘Python Packaging Authority’? Now you’re telling me it doesn’t actually have authority?”

To me that sounds like a non-issue and we can just request funding to become an enterprise organization which has 500 concurrent runners.

1 Like

The “authority” part started out as an in-joke; only that this wasn’t easily recognisable as a joke to outsiders[1], and its origin got even more obscured over time, so that the vast majority of users who encounter PyPA assume that the content is what it says on the tin. Alas…

I don’t think users are tired of changing things that need changing (in fact the survey and the unification thread contained lots of calls for changing the status quo). Beyond that, PyPA has barely any user-facing impact, e.g. no average user would have to change any CLI invocation / files / dependencies if PyPA changed its name[2]. That’s not to say that change for the sake of change is worth it, just that I don’t see a negative impact of changing the name on users.

Fair enough, but then the scope for the council should equally be everything under that “Authority” (i.e. all things packaging).

I still like the renaming idea more though. I think that undoing the joke aspect makes the situation even harder to communicate[3], than starting over with a new name, scope, goal, etc.


  1. unlike, say, “cheeseshop” ↩︎

  2. even for those pulling from https://www.github/pypa/xyz, since github forwards links after renaming orgs ↩︎

  3. I’m sure we don’t want to say “until 2023, PyPA['s name] was just a joke” – but even more accurate (and much more circuitous) explanations risk leaving people with the feeling of having been fooled. ↩︎

5 Likes

We’re retreading well-trodden paths in the conversation right now, so I’ll post a relevant link to something I wrote 3 months ago.

The naming is a bike-sheddy topic. I request folks to not derail this topic into a naming discussion for now, and instead respond to the questions put forward in the first post. If you want to argue about naming, go read Remove the "Authority" from Packaging in its entirety and respond there.

1 Like

I assume you’re looking at https://docs.github.com/en/enterprise-cloud@latest/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits?

I’ll point out that “Enterprise” is 21 USD/user/month. That works out to, for the current membership in the PyPA, around 25k/year. If we have 25k USD/year, I really don’t think handing that over to GitHub for slightly faster CI times is the best use of such funding.

And, such a payment model is also incompatible with a “any project can add however many maintainers whenever” model (which is what we have today).

Perhaps we can continue this discussion elsewhere but I think it would be valid to reduce the number of users in the organization and instead give people specific permissions per repository. Adding collaborators to public repositories does not count as a user for billing purposes.

1 Like

What would be the election cycle? Right now, the entire SC is elected every year, and while that does line up well the the annual release process, I still have a nagging discomfort with the possibility of a full turn over in any election cycle. In order to change that though, terms would have to be extended (you can’t have continuity if all terms are only one year).

I’m not following, do you want to derive the responsibilities of the council from a question of what to document for end users? I think there are more topics that would benefit from an elected council that aren’t end-user problems, but are more about “infrastructure” tasks like maintaining the installers, reviewing PEPs etc. Wondering now if I wrongly used “publications” in my post, I meant everything that is published for end users, including packages, docs, installers etc.

To cater to a constantly evolving packaging ecosystem, my suggestion would be to acknowledge the full extent of what we consider as “packaging” at the moment, when defining the responsibilities of the council, but also allow additional topics to be added via a defined process, such as additional PEPs.

1 Like

Sorry, I wasn’t clear enough. Those topics are all things that in the past have been suggested as things the PyPA should be making pronouncements (of one form or another) about. To me, that says that people expect the packaging council to have a very wide authority - far wider than I think is either reasonable or practical.

My main point though was simply that I don’t think we have a consensus (or even a good shared understanding!) of what constitutes “packaging”. And I think we need to get that addressed first. For example, your original comment was

I would consider Linux distro packages and conda as “publications of Python maintained by the community”. Are you suggesting that they should follow the same governance process that we’re putting the packaging council in place to manage? Would they agree to that?

Maybe I’m underestimating the community’s willingness to work under a single authority. But to me, this feels like new territory, which is why I’m suggesting that we start with a relatively small scope, and expand when we’ve proved the model works.

1 Like

Yeah, generally naming discussions are wasteful, it was meant as a scoping point related to @smm’s fourth question. I disagree that the thread you linked would be useful to comment on, given its premise and frankly the pretty eye-rolling linked GitHub issue about the virtues of wheels and conda and pip. Not going to wade into this :stuck_out_tongue:

FWIW, what I do care about is looking at packaging and distribution problems holistically, instead of the various niches we currently hold on to. It would be really great to empower a group of people to be councilors for all Python users, no matter the used distribution.

Sections to update:

  • mandate: spelling out the wide scope of “packaging” and “distribution”
  • roles:
    • “packaging team” instead of “core team” (initial members: PyPA members)
    • difference to PyPA, packaging WG, Python core team
  • relation to 3rd party distributions of Python (goal: partnerships/collaboration)
  • not sure about the yearly election cycle, would it be useful to stagger it?

Eligible to run for the council: anyone
Eligible to vote: packaging team
Ideal size of the council: 5

  • maintenance of installers
  • authors of a packaging roadmap
  • advisory role related to docs
  • contact for security team
2 Likes

Yeah, I’d propose that because the alternative we know (a lack of a governance process like that) led to lots of heartache over the years. The coordination essentially still needed to happen to cater to the demand, just on a per-project level, and was often handled by contributors that had no interest in cross-organizational coordination, who just wanted to work on the software.

I think these distributors would have an incentive to agree to a new governance process like this and become partners in the process, given the increased predictability and transparency, and ability of participation. I think that’s currently missing.

1 Like

I like this - it’s a much more concrete way of approaching this (far better than where I was going, which was frankly a bad case of overthinking the problem :slightly_frowning_face:)

There is one key factor that might need sorting out, though, which is PyPA membership. I’m not sure whether the current fairly relaxed approach to how membership is gained is sufficient. Ideally, it’s fine, if we assume everyone’s acting in good faith. But if we get a situation where the council needs to take action to expel someone from the PyPA, things could get tricky.

Why just installers? What about build backends, Warehouse, and other PyPA software?

Also, if we assume PEP 13 as the base, the council should also have the authority to:

  • Enforce the PyPA CoC
  • Act as liaison with the PSF around funding and any other areas of shared interest
  • Eject PyPA members if necessary
  • Act as liaison with the SC

That’s a good question, I don’t have an answer for

Right, sorry, “maintenance of installers” was in addition to PyPA maintained software, I tacitly assumed PyPA projects would be in the scope of the council, but you’re right, this should be explicitly stated.