Draft update to Python Packaging Governance

Packaging Coordination
1

Based on discussions to create a Python Packaging Council, I have drafted a proposal to formalize the creation of a Python Packaging Council. I welcome comments from any Packaging stakeholder. This includes but is not limited to PyPA/non-PyPA tools.

Please share constructive feedback as this is my first time drafting a PEP.

PEP: 9999
Title: Python Packaging Governance
Author: Shamika Mohanan
Sponsor:
Status: Draft
Type: Process
Topic: Governance
Content-Type: text/x-rst
Created: 

Abstract
========
This PEP describes the Python Packaging Governance for 
all tools that are included as part of the PyPA. The Packaging 
Council will govern all PyPA tools, represent the community 
that supports these tools and act as a representative of PyPA 
tools. From here on, this Packaging council will be called the 
Python Packaging Authority (PyPA). This PEP will supersede 
[PEP 609](https://peps.python.org/pep-0609/) 
and will cover all aspects of Python Packaging Governance.

Rationale
=========
One issue facing Python Packaging is that there is a lack of a 
comprehensive vision and strategy for Python Packaging tools. While 
`PEP 609 <https://peps.python.org/pep-0609/>`_
formalized the governance of tools included under the PyPA 
umbrella, this governance does not extend to setting long-term goals 
for PyPA tools. In order to ensure PyPA tools remain relevant in the 
future and cater to user requirements, it is important to set a vision 
and strategy for Python Packaging and PyPA tools. It is hoped that 
the council will focus on the Packaging ecosystem, define the 
vision and work with the community to develop the strategy. 
Attempts at defining the vision and strategy have failed so far.

The main motivation for this PEP is to create a body with the necessary 
powers to define the short-term and long-term goals for Python Packaging 
and to support the community in setting the strategy to achieve the goals. This 
governing body will represent PyPA tools when working with any other tool or group. 

As such, the mission of the initial council will be-

* Define the short-term and long-term vision and strategy for Python Packaging
* Approve PEPs that support the Packaging strategy
* Identify solutions to few (1-2) problems identified in the 
[Python Packaging survey](https://drive.google.com/file/d/1U5d5SiXLVkzDpS0i1dJIA4Hu5Qg704T9/view) and marshall resources to implement the solutions

We can expect the initial council to pick up a few problems 
outside of this list that they feel is important to the community. 
With each council, this mission will change depending on 
pressing issues and how the vision and strategy has changed.

The mandate and powers of the PyPA have been drawn up such that they 
are not too restrictive. Certain issues such as authority over maintenance 
and improvement of installers, build backends, Warehouse, and other PyPA 
software are not in the scope of the initial council. This could be added 
the future if all stakeholders agree to this change. It is expected that 
the mandate of powers of the council will grow as long as it can show that 
it is working in the best interest of the Packaging community. This 
council will be accountable to the Python Steering Council.

Specification
=============

Python Packaging Authority
--------------------------

Composition
~~~~~~~~~~~

The packaging council will consist of 5 members.

Mandate
~~~~~~~

The packaging council will-

* Set vision and strategy for Python Packaging and PyPA tools (defined below)
* Support, encourage and facilitate the maintenance and improvement 
  of the quality of all PyPA tools
* Represent PyPA tools as a group when working with non-PyPA tools, 
  PSF, Python Steering Council and third-party entities such as operating 
  system package managers. The council will start discussions and engage 
  with stakeholders and the Packaging community and represent the PyPA interests.
* Arbitrate on any packaging related issue brought to the council. This 
  should be done as the final resort.
* Improve diversity of the Packaging community. The council should work 
  with the PSF to introduce programmes aimed at improving diversity of contributors.

Powers
~~~~~~

Packaging Governance
++++++++++++++++++++

* Accept or reject PEPs within the scope delegated by the Python Steering Council
* Delegate parts of their authority to other subcommittees or processes

Packaging Assets
++++++++++++++++

* Authority over `packaging.python.org <https://packaging.python.org/en/latest/>`_
* Maintain advisory role for any issue related to packaging docs
* Authority over PyPA GitHub org

Stakeholder Management
++++++++++++++++++++++

* Act as a liaison with any packaging-related Python or non Python 
  technical group when specific tool representation is not appropriate. 
  For example, any group representing supply chain security. 
* Act as a liaison with the PSF and the Packaging Working Group to obtain 
  funding and any other areas of shared interest
* Act as liaison with the Python Steering Council

Packaging ecosystem and community
+++++++++++++++++++++++++++++++++

* Maintain ownership of PyPA tools. The council will not interfere 
with a project's day-to-day operations but rather govern 
the community that supports these tools. The PyPA can require
projects to implement accepted PEP standards. In such a situation, 
the PyPA should liaise with project maintainers and the PSF to 
secure resources to implement accepted PEP standards when necessary.
* Recommend projects for adoption into PyPA by vote. External projects can self-nominate to 
  be included under the PyPA umbrella.
* Expel PyPA project/individual

Scope: It is expected that the Packaging council will consider the Python Packaging 
ecosystem holistically and improve all aspects of Python Packaging rather than just 
one project. This includes any tool and workflow related to the PyPA and PyPI. The 
council should work towards improving user experience of all packaging tools under 
its purview and this might include deprecating certain tools and developing new tools 
depending on requirements. While it does not have any authority over non-PyPA tools, 
it is expected that the council makes decisions that benefit the entire Python Packaging 
ecosystem. This scope includes all aspects of packaging and distribution. It is expected 
the council will work on long-term goals such as improving interoperability with non-PyPA 
packaging tools and improving packaging UX.

To use its powers, the council votes. Every council member must either 
vote or explicitly abstain. Members with conflicts of interest on a particular vote 
must abstain. Passing requires a strict majority of non-abstaining council members.

Elections
~~~~~~~~~

Packaging Voting Body: The Packaging Voting Body (PVB) includes the following:

* PyPA committers
* Affiliate project voters

PyPA committers can nominate either non-PyPA projects or individual contributors 
who do not contribute to a PyPA project to be added to the PVB. These voters will 
be called affiliate project voters.

* Any voter nomination has to be seconded by another PyPA committer
* External projects nominated by PyPA committers will get to appoint a representative 
  for each ballot

The initial seed of voters will include all committers for the existing non-archived 
public projects under the `pypa` organisation of GitHub.

The PyPA has to maintain a list of eligible voters and their project affiliation. This 
list should contain the names of PyPA committers and affiliate project voters. 
This canonical list should be maintained by the PyPA with access available to 
all Packaging Voting Body members. This list should not share personal 
information publicly. It is the responsibility of all PyPA projects to 
ensure that the names of any new committers are added to the list of eligible voters 
and any communication spaces for the PyPA.

For the initial election, this voter list will be maintained by the PSF. 
Adding any new voters and subsequent due diligence is the responsibility 
of PyPA committers. Once the new council has been elected, ownership of the 
Packaging Voting Body membership list and the PyPA-voters mailing list will 
be transferred to the PyPA.

PyPA elections will be held in three phases

* Phase 1: Packaging voting body members nominate affiliate project voters. 
  Affiliate project voters are added to the list of eligible voters.
* Phase 2: Candidates advertise their interest in serving. Candidates must be 
  nominated by a PyPA voting body member. Self-nominations are allowed. 
  Candidates need not be a PyPA committer.
* Phase 3: Each PyPA voting body member can vote for zero or more of the candidates. 
  Voting is performed anonymously. Candidates are ranked by the total number of votes 
  they receive. If a tie occurs, it may be resolved by mutual agreement among the candidates, 
  or else the winner will be chosen at random.

Each phase lasts one to two weeks, at the outgoing council’s discretion. For the 
initial election, all three phases will last two weeks. The election for all 
subsequent councils will start in the 12th month since the previous council election.
The election process is managed by a returns officer nominated by the outgoing 
Packaging council. For the initial election, the returns officer will be nominated 
by the PSF Executive Director.

The council should ideally reflect the diversity of Python Packaging contributors 
and users. PVB members are encouraged to vote accordingly.

Term
~~~~

A new council is elected once every year. Each council’s term runs from 
when their election results are finalized until the next council’s term 
starts. There are no term limits. Generally, each council member’s term 
should last 12 months with one exception described below.

Vacancies
~~~~~~~~~

Council members may resign their position at any time. There could also be 
situations that council members have been removed from the council via a 
vote of no confidence. 

Whenever there is a vacancy during the regular council term, the council 
may vote to appoint a replacement to serve out the rest of the term. In 
such a situation, this council seat will be up for election at the next 
council election as long as the replacement has served 12 weeks or longer 
in the council. Therefore, the longest term for any council member will be 
15 months before that seat is up for election.

If a council member drops out of touch and cannot be contacted for a month or 
longer without prior notice, then the rest of the council may vote to replace them.

Conflicts of interest
~~~~~~~~~~~~~~~~~~~~~

In order to avoid any appearance of conflict of interest, at most 
2 members of the council can work for any single employer.
In a council election, if 3 of the top 5 vote-getters work for the same 
employer, then whichever of them ranked lowest is disqualified and the 
6th-ranking candidate moves up into 5th place; this is repeated until 
a valid council is formed.

During a council term, if changing circumstances cause this rule to be 
broken (for instance, due to a council member changing employment), then 
one or more council members must resign to remedy the issue, and the resulting 
vacancies can then be filled as described above.

Ejecting PyPA committer
~~~~~~~~~~~~~~~~~~~~~~~

The Packaging Council or PyPA member may initiate a vote to 
eject a member from the PyPA committer body. A council member 
or PyPA committer can put forward a proposal and call for a vote 
on a public PyPA communication channel. A PyPA committer vote 
is triggered when a PyPA committer (not the proposer) seconds 
the proposal.

The proposal will be put to a vote on the 
`PyPA-Committers <https://mail.python.org/mm3/mailman3/lists/pypa-committers.python.org/>`_ 
mailing list, over a 7-day period. Each PyPA committer and council member 
can vote once, and can choose one of +1 and -1. If at least two 
thirds of recorded votes are +1, then the vote succeeds.

Ejecting PyPA project
~~~~~~~~~~~~~~~~~~~~~

The Packaging Council or PyPA member may initiate a vote to eject 
a project from the PyPA. A council member or PyPA member can put 
forward a proposal and call for a vote on a public PyPA communication 
channel. A PyPA committer vote is triggered when a PyPA committer 
(not the proposer) seconds the proposal.

The proposal will be put to a vote over a 7-day period. Each PyPA 
committer and council member can vote once, and can choose one of +1 and -1. 
If at least two thirds of recorded votes are +1, then the vote succeeds.

A project can also choose to leave the PyPA. If a project is leaving the 
PyPA or has been ejected from the PyPA, it is the responsibility of the council 
to support the transfer of the GitHub repository out of PyPA to a personal repository.

Ejecting an affiliate project voter
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Any Packaging Voting Body member or council member may initiate 
a vote to eject an affiliate voter from the Packaging Voting Body. 
A council member or PyPA committer can put forward a proposal and call 
for a vote on a public PyPA communication channel. A PyPA committer vote 
is triggered when a PyPA committer (not the proposer) seconds the proposal.

The proposal will be put to a vote on the PyPA-voters mailing list, over 
a 7-day period. Each PyPA voting body member can vote once, and can choose 
one of +1 and -1. If at least two thirds of recorded votes are +1, then the 
vote succeeds.

Vote of no confidence
~~~~~~~~~~~~~~~~~~~~~

Any PVB member or Packaging Council member can publicly call 
for one or more Packaging Council members to be removed from the Council 
via a vote of no confidence. 

The vote of no confidence should be called on a project communication 
channel and should be seconded by another PVB member.
The vote lasts for two weeks. PVB members can vote for or against the 
removal. If at least two thirds of voters express a lack of confidence, then 
the vote succeeds.

If the vote of no confidence is for a single member, the council member is 
removed from the council and the vacancy is filled as described above. If 
the vote is for the entire council, the council is dissolved and a new election is held.

PyPA committer
--------------

Role
~~~~

Similar to the Python core team, the PyPA committers is a group 
of volunteers who maintain and support PyPA tools. They have 
authority over the Python Packaging infrastructure, the Python 
Packaging GitHub organization and repositories, the bug tracker, 
the mailing lists, IRC channels, etc.

Prerogatives
~~~~~~~~~~~~

PyPA committers may participate in formal votes, 
typically to nominate new PyPA projects, 
and to elect the Packaging council.

Membership
~~~~~~~~~~

Any Packaging project can request PyPA membership. 

A PyPA member can put forward a proposal to add a project 
to the PyPA and call for a vote on a public PyPA communication channel. 
This proposal must not be opposed by the existing maintainers of the 
project. A PyPA committer vote is triggered when a PyPA committer 
(not the proposer) seconds the proposal.

The proposal will be put to a vote on the PyPA-Committers mailing list, 
over a 7-day period. Each PyPA committer can vote once, and can choose 
one of +1 and -1. If at least two thirds of recorded votes are +1, 
then the vote succeeds.

Once a project has been added to the PyPA organization, the project 
falls under the purview of the PyPA and will be required to meet the 
guidelines as set by the PyPA.

As Packaging contribution requires support and time, it is the 
responsibility of the Packaging Council to ensure there are sufficient 
support mechanisms in the form of (but not limited to) mentorship, internship 
and fellowship to support and guide new PyPA contributors. The Packaging Council 
may work with the PSF to establish such programmes.
 
Acknowledgements
================

This PEP is based on `PEP 13 <https://peps.python.org/pep-0013/>`_ 
which in turn is based on a Django governance 
document authored by Aymeric Augustin.

I am keen on hearing on hearing your thoughts around

  • Whether the council will have the support of the community?
  • Will non-PyPA tools work with such a council?
  • Does the mandate go far enough to make meaningful changes in Python Packaging? Should the powers increase/decrease?
  • Anything else you wish to share

I intend to close discussions the week of Sept 4 and then decide the next steps. Next steps could be re-drafting the proposal or submitting the PEP with changes suggested by the community.

2 Likes
2

I’m not sure if I’m not reading too much into this but I’m certainly looking forward for improvement of the processes that would help third-party stakeholders (Gentoo Linux here) have more streamlined way of influencing the packaging standards.

2 Likes
3

Not that my opinion really counts here, as I won’t be a PyPA member, but the process to eject a member with a public vote by all other members is something I haven’t seen elsewhere, and TBH it sounds pretty scary (as in: could potentially create divisions and uncomfortable public shaming).

In the process for the core Python team (PEP 13), it is the steering council that has the power to vote to eject a member. What is the rationale for making the process different here?

1 Like
4

Also, I thought all mailing lists were being replaced by Discourse, is that not the case?

1 Like
5

Is this text available anywhere else?

It’s difficult to read the large preformatted text block on mobile.

1 Like
6

While the council as described here doesn’t have any specific influence over the broader concept of “Packaging for Python” (i.e., beyond PyPA projects), IMO it would be foolish to assume that people won’t assume that the council sets overall “packaging policy” for Python[1]. As a result, what will happen regarding current PyPA projects? For example, there’s bound to be a presumption that PyPA member projects are in some sense the “official tools”. We’ve had serious credibility problems in the past because tools (specifically pipenv) have claimed to be “the official packaging tool”, and I think it’s important that we avoid this in future.

To be completely open, I’m particularly concerned about hatch here, as the landscape around workflow tools is currently very much undecided, with the various tools competing for “market share”, and I don’t think the PyPA should accidentally provide any tool with an implication of official approval simply as a result of having been a PyPA member prior to the change in governance.

But the point is more general - is it correct to take the existing list of PyPA projects, which was formed on pretty much an “ask and you’re in” basis, and give it the implied stamp of authority that being governed by the official packaging council will confer?

  1. If only because that’s the number one complaint about the existing PyPA, that it seems to have authority, but in practice doesn’t. ↩︎

7 Likes
7

@smm If you’d like to submit it as a draft PR to GitHub - python/peps: Python Enhancement Proposals we’ll get a build preview. Or I can do a build preview on my fork.

@btskinn Or for a local build, git clone https://github.com/python/peps, save the text as pep-9999.rst and then make htmlview

8

I have it on my fork- https://github.com/s-mm/peps/blob/packaging_council/pep-9999.rst

2 Likes
9

This sentence confused me. From skimming the rest of the document, it seems that the structure is as follows:

  • The Python Packaging Authority (PyPA) is the body consisting of the people with commit access to any of the packaging tools under its umbrella (e.g. pip, setuptools, virtualenv, etc.). It is analogous to the “core dev team” for CPython. It does not make formal decisions (other than electing the Council), though individual members who have commit access to particular projects make decisions on those projects.
  • The Python Packaging Council is a 5-member group with defined terms and decision-making authority. It is analogous to the Python Steering Council. It has decision-making authority, e.g. making decisions on packaging PEPs.

The abstract should make this structure clear and say explicitly that these two are different bodies.

4 Likes
11

Some initial editorial suggestions/comments (I’ve only made it to “Powers”, I’ll look through the later sections later!)

A

You will need a sponsor (someone responsible to be a point of contact with the PEP process) who is a PEP editor or core developer. I am happy to volunteer, but you might also want to ask Paul, Hugo, Brett, Jelle, Donald, etc.

I’d put this also in the Packaging topic, note how PEP 609 is listed as both.

Please add a blank line after section headings.

I agree with Jelle – I think you should simplify the abstract and just say that this PEP defines what the “Python Packaging Council” is. I think conflating the Packaging Council with the PyPA will lead to confusion, c.f. Paul’s concern about ‘blessed’ projects.

You can use the :pep: role: :pep`609`.

Perhaps drop ‘tools’ – surely the Packaging Council will also consider standards, and impacts on users, etc.

This paragraph is inconsistent with the one below it – my understanding of the PPC is that it is for packaging in Python, rather than limited to PyPA. If so, I think we should drop reference to PyPA in this section.

Again, I think this section should be titled “Python Packaging Council”, and references to PyPA should be dropped.

Perhaps “monitor” or “oversee” – 5 volunteers shouldn’t be expected to achieve ecosystem wide culture change

12

I haven’t read the current PEP draft but I’ll mention that the idea proposed was that “Python Packaging Authority” will refer to the elected group of 5 individuals, repurposing the name to refer to a body that’d have the decision making powers.

The phrase Python Packaging Community (or something analogous to that) would be referring to what is today “Python Packaging Authority”, since that better represents how the group is organised – as a community of collaborating projects.

13

I understand the idea, but that means the pypa GitHub org must be renamed into pypc. Edit: pypa.io should also become pypc.io. Is that worth the trouble?

14

Doesn’t this mean that there’s going to be a lot of existing assumptions and preconceptions carried across, and those will almost certainly hamper the new council in achieving its actual goals?

4 Likes
15

Please don’t take this the wrong way, but I’d find it a lot more helpful if the discussion was two-way, with you being a lot more active in responding to comments as they are made than you have been in previous discussions.

I’m happy to have a dialogue about this proposal, where you explain what you think about my comments and I then respond to that. But if I’m simply going to make comments only to have them collected for consideration in private, I think that’s just going to result in another frustrating and unproductive discussion like the previous “packaging strategy” ones, and I’m not sure I want to spend energy on more discussions like that.

8 Likes
16

Do you mean the Python user community as a whole? If so, I think the extent to which they will support it depends on the extent to which Python packaging improves following its creation. I do think users would initially support the concept of a body with some power to give direction to the Python packaging world, but that support can quickly erode if it doesn’t translate into material change.

I don’t see why not, although I’m not sure what kind of work is being envisioned.

I’m uncertain about this.

I think that authority to approve packaging PEPs is meaningful, and I think that having a consistent council doing that, with a clear mandate to take a broad view of improving packaging as a whole, could be a positive change.

As I mentioned on some of the other threads, I think documentation is a key issue, and I’m not sure if the documentation powers given to the packaging council are as strong as I personally would like. In my view, the important thing is the documentation on docs.python.org, because, well, that is the Python documentation, and my read on the situation is that people see packaging as part of Python and want Python to improve its packaging (as opposed to seeing “Python packaging” as some separate entity). So I’d either like to see the council get clear jurisdiction over packaging docs on docs.python.org, or, equivalently, have docs.python.org explicitly outsource all packaging docs to packaging.python.org.

Relatedly, I’m not clear on what is envisioning with regard to “governing” the projects that exist under what is currently called PyPA. As others mentioned on this thread, up until now PyPA membership has essentially been free for the asking and imposed no governance or requirements on the member projects. The PEP now says that this new council will “govern the community that supports these tools”, but I don’t really see how that would work, or how any way it might work would actually benefit Python packaging.

For instance the PEP says the council can “can require projects to implement accepted PEP standards or eject maintainers from the project”. Is that saying the council could eject maintainers from individual PyPA projects? That would be a major change if so. But even if it could, nothing restricts users to using only PyPA projects, so maintainers could just leave PyPA and keep doing whatever they want. That wouldn’t be the end of the world, of course, because it’s the same situation we already have! :slight_smile: But for that reason I don’t see that the PEP would change anything in this regard.

I agree with @pf_moore. Reusing the name “Python Packaging Authority” for a new entity seems like it will just lead to confusion. What’s the advantage over just creating a new “Python Packaging Council”?

Agreed, and so I think either the council needs to gain power or its lack of power needs to be more clear.

The way I see it, the problems with Python packaging are of two kinds: one is that tools don’t have the functionality users want[1], and the other is that it’s hard for users to figure out what tool(s) to use because there are so many tools with overlapping functionality.[2] Fixing the first problem requires actually writing better tools, which is hard, and I don’t see that it can really be done without significant funding, because no one writing these tools is going to take marching orders from a packaging council or anyone else. That seems beyond what we can hope to achieve with a packaging council.

Fixing the second problem also requires work, but I think it’s more tractable. In that sense I think designating “official tools” may be one of the best things that can be done. But like @pf_moore I don’t think it’s a good idea to start off with the implication that all the existing PyPA projects are those official tools.

Rather, I think a promising path forward would be for the council to lead a continuing process of understanding user needs, and trying to share understanding about different tools and which of those needs they meet. This can be distilled into official documentation that helps users cut through the thicket of options. The council could also link with relevant tools to hopefully start some fruitful collaboration along the lines of “users seem to want functionality X, your tool does something sort of like that, are you pursuing any development in that direction”. The council could help such projects by giving them a voice in the standards/documentation process, and perhaps by sharing useful user feedback, and the projects in turn could help by implementing features that users want[3].

Even that may be a pipe dream, but it’s about the most optimistic I can be at this point. So with regard to this PEP, I’d say that means distinguishing it from the current loose affiliation model of PyPA and moving more towards an explicit “referee” position. The goal of the council would basically be to facilitate communication among users, tool authors, documentation authors, and standards authors, to make the work of all those groups more mutually beneficial.

Some of that is in the PEP, like approving packaging PEPs and at least some gesture towards handling packaging docs. But for my tastes I’d want some to become “more official” (like docs control) and some to become less (like trying to govern disparate PyPA projects).

I agree with this as well. I found the earlier discussions stimulating and interesting, but I think it’s more satisfying for all concerned when the person who’s going to produce the end product (such as this PEP, in whatever final form it may take) is involved throughout, rather than just at the beginning and the end.

  1. in some cases because different bits of that functionality are divided across multiple tools ↩︎

  2. There is actually a third, which is that users want the tools that do everything they want to be part of Python, but that’s sort of the union of the first two, so I’ll leave it out here. ↩︎

  3. the idea being that these features are close enough to what the project already does that the maintainers would not see it as an imperious demand but as constructive collaboration ↩︎

17

If the whoever “requires” is willing to put the effort necessary into PRs, that is fine. Otherwise “requiring” an implementation to be done by other people, sounds a bit too much.

Note also that “having funds available” for a work to be done, is not a solution either. GitHub - psf/fundable-packaging-improvements: Packaging improvements that could be funded has been curating a list of fundable improvements for a long time, but without someone actively looking to “hire” people, the tasks rarely get done.

This seems to indicate that there needs to be a vote for new committers to be introduced into a project. Is this interpretation correct?

If that is the case, this seems to be getting too much into the terrain of each project’s individual management methodology. For example if a contributor starts to stand out, it should be up to the group of maintainers of that particular project to decide whether to give committer rights to the repository. Adding a vote for that seems to add too much bureaucracy and may be counter-productive.

2 Likes
18

Expelling a member will be done only in extreme cases. And we would not want the council to have complete control over this process either which is why we have included committers to be part of this process.

19

In the PEP 13 model, this is solved by specifying that the SC cannot expel a member while a vote of no-confidence is in place.

20

I would expect the council to make official recommendations as this was one of the most common requests in the survey we conducted last year. These recommendations could include non-PyPA tools if they feel it is the best tool for the task. It is up to the council to communicate these recommendations to mitigate any assumptions that might be made about PyPA tools.

21

Thank you. I will include these changes towards the end of this discussion.