Closing the loop on PyPA Governance / BDFRN

Packaging
1

At PyCon 2018, we had a discussion about forming governance model for the PyPA. In the resulting discussion, I was nominated as a “BDFRN” (benevolent dictator for right now) to attempt to bootstrap a governance model.

On August 7th, 2018, I proposed a “PyPA Lightweight Governance Model”. The goals of this model are twofold:

  • Provide a consensus-driven decision-making process
  • Provide a transparent record of proposed, accepted and rejected decisions

In the resulting discussion, there was some support, and some concerns, namely:

  • How does this overlap/complement/replace our current Packaging BDFL-Delegates?
  • Would this in fact be more or less accessible than the current PEP process?
  • Do we actually need a governance model?

As we’re approaching the 1 year anniversary of this proposal, I’d like to attempt to revive the discussion and bring it to a conclusion by August 7th, 2019.

As I see it, we have a couple options:

  • We accept the currently proposed model, possibly with additional refinements;
  • We accept some other, yet-to-be-proposed model;
  • We determine we don’t need additional governance, I abdicate the current position as BDFRN, and we maintain the current status quo.

In addition, since we don’t have any existing protocol for enabling/enacting such a governance model, I’d like to propose that a given model be considered accepted if it has been:

  • +1’s from all the current Packaging BDFL-Delegates (@pf_moore, @ncoghlan, @dstufft)
  • No “strong objections” from anyone who currently has the commit bit on at least one PyPA project
  • Accepted as a PEP

And… discuss. :slight_smile:

6 Likes
2

You have my vote. :+1:

1 Like
3

I’m OK with this as a general proposal, but see below for some detailed comments.

I just skimmed that discussion and it seems like it covered a whole load of stuff that doesn’t seem to be related to the proposal linked above (such as debates about face to face discussions). I really don’t have the time or inclination to go back over that whole thread (governance discussions bore me senseless) so can someone please summarise that discussion here? Maybe after summarising, we can say that it was all off-topic for the “Lightweight Governance Model” (which as far as I can tell makes no comment about real-time discussions), but I think we need to bring the content of that discussion here, in the spirit of “closing the loop”.

I really don’t care much. I’m not honestly sure what problem is being solved here, beyond “the Python PEP process has changed, so we can’t use it any more” and for that I’m happy with anything that provides:

  1. Somewhere to host our standards (we have that, in the packaging guide).
  2. Some means of reaching consensus (I’m not sure we need more than discussions here plus the existing BDFL-Delegate role).

But if others want additional process, I don’t object (unless it adds extra workload for me that I’m not able/willing to take on).

Some other points on the proposal that I just noticed.

(Hmm, I really don’t know how to easily do a review of a document on Discourse. I’ll limit myself to a few copy&paste bits with comments and hope I dont miss anything).

Creation of a Request for Comments (RFC)

We need something on who can create such an RFC. The current PEP process makes it (deliberately) quite hard for non core devs to raise a PEP. I’m not sure we want to make it too easy for random community members to raise RFCs, or it’ll end up being a route for feature requests. But equally we don’t want to block ideas from the community. I have no good answer here. For an example, see this proposal which is mostly stalled, because of a lack of a sufficiently broad audience to get a critical mass of people to move it through the process. I’d like it if whatever new governance process we had didn’t leave proposals like this

Comments period … as comments on the pull request

I find “comments on PRs” to be a horrible way of discussing changes. I’d much rather that we used some form of proper communication tool (whether Discourse or email, I don’t really care) and limit PR comments to things like grammar corrections and wording details.

If the RFC falls under a specific Interest Area, then the corresponding Interest Area Authority will review the proposal and decide whether it should be accepted or rejected.

We should allow for Interest Area Authorities to delegate decisions if necessary. There will be cases where the IAA doesn’t have the right knowledge to make a final decision, but going to a vote (which means putting it to a group who communally have even less specialised understanding) is the wrong way to go. That should probably simply be “IAAs can appoint a delegate to make the decision if they feel that’s appropriate”.

Conversion of the RFC into an Architectural Decision Record (ADR)

This seems to be replacing the process we’re currently moving to, of having the specs in the packaging guide. Is there a reason for doing this? I don’t think we should repeatedly change the canonical location of standards without good reason (I’m still referring to PEPs rather than the guide :slight_smile:).

Also, there’s no provision for updating agreed specs. Adding one moves us yet further from a “lightweight” model, so I don’t really want to get into endless debate, but we want our standards to move with changes in circumstances, not be locked into stone (“document 0076 as modified by 0089, 0093, 0210 and the addendum in document 0130” :frowning:)

There may be more, but that’s about as much “governance” discussion as I can cope with for today :slight_smile:

4

Anyone with the commit bit on at least one project in the PyPA organization, which should correspond to everyone on the PyPA-Committers mailing list.

I think this is a fine “starting set”, but I think we may want to reserve the right to change this in the future. I know @dstufft and I have had a good conversation once about what makes up the “core” of a project, and focusing on commit bit may tend to discourage contributions that don’t require commit access (such as project management, marketing, getting funding, etc). I don’t think this is the right place to get into the details of what that would look like, but I think maybe throwing an asterisk on that one would be good.

On a related note, I take minor issue with the “Determine who is and isn’t a PyPA member” section. I like the thrust of it, which is that the PyPA itself should be delegating membership discussions, but I think it may be worth allowing for the possibility of adding members not associated with any one project, but still members of the PyPA. Can we change that to “restricting membership in the PyPA”, which is to say that the PyPA itself is not intending to cut out members from individual projects, but reserves the right to add new members above and beyond those.

(Will try to give more feedback soon)

5

I’m in favour of the general idea of this proposal. I do have a few concerns as it stands:

(1) re: voting result metrics

Given the currently proposed voting structure, a single -1 is enough to prevent a vote from getting through. My main concern with that is whether it’d be a good idea to let a single person veto a proposal even if (say) 10 other folks are in favor of the changes? I don’t think so.

I guess we could apply the argument that everyone will be reasonable and know what a -1 is and thus act accordingly. It definitely holds true today but it might not in the future.

A single person with strong opinions on how things should look (or just wanting to block everything, to exaggerate) can veto all proposals and PyPA will basically stall in that situation with no mechanism to take any action for this situation. I don’t think that’s a situation we should find ourselves in.

I think requiring twice as many +1s as -1s, is a simple enough rule to add to protect against this. I think there should be scope for disagreement in this.

(2) re: voting period

It’ll be a good idea to extend it to at least 7 days. I mostly catch up on discussions on weekends and it’d not be nice to know that someone proposed a vote on Monday and I missed it. Maybe I wanted to vote -1. :wink:

Maybe we should also specify the timing of the votes ending? Maybe that should be a process detail (eg. The vote email contains a UTC deadline?)

(3) re: discussions

I don’t think we should have discussions on PRs, keeping them scoped to grammar or tpyos. Any other mechanism (discourse / dedicated GitHub issue / distutils-sig) that is for open ended discussion works for me. I think we shouldn’t specify a communication medium here.

Instead, I think a PEP style discussions-to equivalent, with a list in pypa.io of what discussion locations are okay. We can handle how to update that list as any other RFC. That’ll help separate this discussion from the communication discussion. :slight_smile:

There’s probably more but these are the first things to stick out to me, upon re-reading the proposal.

6

I largely agree with the comments made here and just want to add the additional caveat that it would be good to know who is allowed to, for example, create repositories under the PyPA banner. I think I have the access level, so is it ok for me to add a repository? How does a decision that is fairly unimportant but still not insignificant get adjudicated or whatever? A lot of this type of process related stuff is opaque even to me.

7

I’m still opposed, because I still don’t understand what actual problem is being solved by the proposal.

  • we’re replacing PEPs with RFCs, why? PyPA is far more closely associated with the Python Software Foundation and the Python core development process than it is with IETF, so why would we switch to IETF terminology?
  • we’re making objectively untrue assertions about the PEP process, why? The assertions might have been true once (back in the days when the main PEPs repo was on Mercurial rather than GitHub), but now the last section of the proposal just perpetuating straight up FUD for no good reason.

Since PEP 1 was updated following the CPython governance changes, the Steering Council has added the documentation of the standing delegations to PyPA: https://github.com/python/steering-council/blob/master/process/standing-delegations.md#pypa-delegations (thanks @pradyunsg!)

If it’s just a matter of reorganising PEP 0 to better separate out the PyPA PEPs from the other PEPs, that could be done by enabling filtering on the Discussions-To header (we also want to start displaying the appointed BDFL-Delegate for each PEP: https://github.com/python/peps/issues/1012).

If it’s wanting to remove the requirement to post PEPs to distutils-sig, then that’s already under PyPA’s control at https://www.pypa.io/en/latest/specifications/ (and even for core Python PEPs, the key requirement had been relaxed to notifications and pronouncements being sent to python-dev, with much discussion now taking place on discuss.python.org instead).

Historically, the PyPA had some genuinely valid concerns with the PEP process - there was no mechanism for standing delegations, the process for drafting and submitting a new PEP was arcane, and changes to PEPs were annoyingly difficult to review.

Every one of those concerns has been taken on board and implemented. We have standard delegations now, we have a Provisional PEP status, and the PEP submission and review process is essentially the same as the one PyPA used to run on the ad hoc interoperability-peps repo.

So we’re proposing to go off and reinvent our entire spec approval process… why?

1 Like
8

I’ll also note that the proposed process is actually heavier weight than the PEP process - the PEP process is essentially “the BDFL-Delegate decides whether or not consensus has been reached, and whether or not to accept a proposal based on their assessment of that consensus”. The alternative proposal currently on the table is more in line with PEP 13 - the more formal process that CPython now requires for changes to the governance structure itself (with voting timelines, eligible voter determination procedures, etc).

Now, I don’t think having such a process for PyPA would be a bad thing, which means the idea I’m actually objecting to is only the proposal to conflate the PyPA process for managing governance changes with the process for managing technical design decisions. We deliberately didn’t do that for Python core development (PEP 1 and PEP 13 are different documents that cover different kinds of decisions), and I don’t think we should do it for PyPA either.

So if the proposal were descoped to be only about PyPA governance (e.g. transferring the standing delegations to a new default BDFL-Delegate, or proposing changes to the structure of the PSF Packaging Working Group), and left the handling of technical specifications alone, then I’d be more in favour of it - it would be the PyPA’s equivalent to PEP 13, with technical decisions continuing to be handled under https://www.pypa.io/en/latest/specifications/ and PEP 1.

If it was descoped that way, then it could be formalised as PEP 14 (Python Packaging Authority Governance), rather than needing to define an entirely new standards management process of its own. (We’d also be establishing a potentially useful precedent for other “Interest Area Authorities” within the wider Python ecosystem, like PyCA and PyCQA).

1 Like
9

I agree that we should separate our governance details from the technical discussion detained.

I’m on board for sticking to the PEP process for our technical discussions – the more interesting part (to me anyway) is figuring out the governance around it.

10

We should probably do this anyway, right?

11

I had the feeling that PEPs are for projects/decisions that affect more than one component in the ecosystem. What about decisions/design reviews/call for comments that are for a single component (e.g. pip, pipenv, virtualenv)? I suppose these can happen under the projects repository, but was hopping for having a unified method for all projects under PyPa, a unique portal that we can all follow without needed to follow all issues under a project. An option could be to send mails down distutils mail list with links to the repo issue/discussion board.

12

PEPs are for interoperability specs, basically - anything which multiple tools (either real ones or just potential ones - no-one has written an alternative to pip yet, but we want to allow for the possibility) might need to refer to. Things that only affect a single project internally are handled by the project’s own processes (typically the project’s tracker).

But I do also feel that it would be nice to have a “PyPA” forum, where larger decisions for individual tools can happen1 - not because they involve interoperability issues, but simply to share expertise with other PyPA members. But I don’t think there is such a thing at the moment (unless you count here, but it’s often hard to distinguish between PyPA discussions and “general public” discussions here).

1 I assume you’d be thinking of the “virtualenv next generation” discussion as an example here.

13

Partially that too, but additionally I would expect we have better governance in place to tackle general day to day issues too. For example when pip 19.1 broke editable installs I feel like needing three weeks to release a fix was a bit too long. Much of those three weeks was spent with coming up with a solution we could all agree on, over at least three separate forums (discourse, pip Github issue tracker, distutils, twitter). I would hope we can streamline these as part of PyPi Governance.

14

I’m not 100% sure what you mean here, but I’m deeply concerned about the idea that the PyPA would somehow involve themselves in how individual projects run themselves. Having better means of discussing responses to issues would be great (IMO, part of the problem with the editable install issue was the lack of anywhere we could discuss solutions without endless repeats of the “you broke pip, fix it NOW” complaints) - maybe a PyPA mailing list would help there. But having the PyPA as a group where people can escalate grievances with how individual projects make decisions would, in my opinion, be counter productive.

15

Definitely not grievances, I’m deeply sorry if it came through to like that. The PyPa as it stands down is a very loose thing, with very little governance (for better or worse). People tend to trust projects that live under the PyPa umbrella though. As such we should have a dedicated channel we (periodically) all come together, discuss and come to conclusion in a timely manner of either:

  • major decisions (even at individual tool level),
  • major issues.

Mostly because of the success or the hardships of individual projects that live under the PyPa brand will define how much people trust other projects under the PyPa organization. I hear way too often that python packaging is broken at multiple levels, and would like to change peoples perception on this.

1 Like
16

Strong +1 on this, but this is “communications”, not “governance”.

Some time ago, I suggested (somewhere, I don’t recall where) that getting someone with experience to help us with all aspects of communications might be worthwhile (internal communications, publicising our work, user outreach, etc). I don’t know if that’s something a grant-based project would be useful for?

1 Like
17

I suppose yes it is communication. The place where I’d imagine governance comes in is to not let discussion spiral around endlessly. A moderator sort of (potentially nominated by the governance/PEP delegate) that time boxes things, keeps things on topic, manages the discussion, sums up conclusions.

18

The packaging WG authorised spending PSF money on this a while back. The challenge is that it’s hard to hire someone if you don’t know what to ask for as project deliverables - while you can do time & materials “Help us figure out what we should be asking you to deliver” style contracts, those are also the ones that can be hardest to justify after the fact as money well spent.

I do agree that we have multiple challenges though:

  • clear communications channels between PyPA members both to resolve already existing problems (e.g. our miscommunications around the PEP 517 migration that broke pip 19.1), and to help check design changes for potential problems before they’re published
  • clear communications from PyPA to the rest of the world (it’s OK when there’s a PSF-backed project running with paid comms work as part of it, but we tend to let it fall by the wayside when it’s a purely volunteer activity)
  • a governance model to help make decisions about things like the above two points, as well as other issues mentioned further up this thread

The one area that I think we’ve actually got into a reasonable shape over the past few years is the interoperability specs - PyPA specifications - Python Packaging User Guide is a much nicer overview resource then simply linking to the PEP index, and using the PEP process to suggest changes to an external reference document is a much better fit for the way the PEP process was originally designed than attempting to use the PEPs themselves as reference documents.

19

What we need in such conversations is someone who’d champion the discussion and the work that it entails. I don’t think they need to get the authority to do so from a governance model.

I’m biased here since IMO my first major-ish contribution was managing a long discussion and poking it regularly to keep it from stagnation. It would be (more!) difficult to have that kind of scenario (external contributor who has not done much but spent a lot of time on the sidelines, championing a discussion) if certain amount of authority here comes from whatever governance model we have.

What I’m trying to say is, I might not have made the jump to the dark side if we had such a rule back when I had started contributing. :slight_smile:

@dustin Would you be working on an updated model that separates the governance model from the RFC process change?

20

Pinging @dustin on this again.