Draft PEP: Python Packaging Governance

Packaging
50

I expect that this PEP will undergo a two step process for approval:

  • A PyPA committer vote, since this changes how PyPA projects are governed.
  • An SC review and approval, since this changes the PEP decision making process’ standing delegations.

Supercede. This should be made explicit and clear in the PEP. One of the bits of feedback I had is that the PEP needs to be clear that it’s replacing the existing governance and be a self-contained reference to all the processes.

Yes. This should be made explicit and clear in the PEP.

5 Likes
51

I completely agree that “supercede” needs to be more prominent in the PEP. @smm Please consider adding to the first paragraph of the PEP. It wasn’t clear to me that supercede was expected in the original draft. Thanks!

1 Like
53

Today is the last day that this thread will be alive right?
Did we manage to clarify all the points of the proposal, or should we extend the expire date a bit more?

I suspect that regarding some points that we debated here and are still open, the author’s final proposal is presented in the document as it is, and then (after this thread is closed), it would be up for the PEP delegate to go through the proposal and require changes if they think is necessary. Is that supposition correct?

Which brings to question, who would be the PEP delegate for this proposal, and how does that will interplay with the process highlighted by Pradyun?

I would also like take this opportunity (since it might be my last post in this thread), to highlight again some of the points on the current PEP that I (in principle) disagree with:

  • Committer rights on projects should not be subject to a general vote. I believe that each project should have the capacity to manage themselves and delegate rights/responsibilities in a way that suit their workflow.
  • The wording in The PyPA can require projects to implement accepted PEP standards is problematic. Maybe a better way of putting this would be a variation a long of the lines of The PyPA should liaise with project maintainers and the PSF to secure resources (not only funds, but maybe more importantly, programmers) to implement accepted PEP standards when necessary.

But there is also another whole can of worms to be debated, e.g.:

  • What happens if a project has many more committers than another project? Is it an intentional “design choice”/“feature” of the governance system that this project “has more representativity”? (it might be…)
    • How does this dynamic affect the relationships between the projects? (e.g. what does it mean for “re-packagers” to have more or less representativity than “build-backends”, or what does it mean for “installers” to have more or less representativity than “workflow tools”?) Is that going to bias/affect PyPA in the long term? Is that intentional/desired?
  • How does the new governance actively address the consensus problems as captured by the short hint in Donald’s comment? Is simply adding a group of “elected benevolent co-dictators for mandate” going to actually solve the problem?
    • What is the effect of imposing a decision with the basis of “elected legitimacy” on a group of voluntary workers that disagree with the decision?
2 Likes
54

I hope the thread is extended, as I don’t believe all of the points made here have been addressed. I hope that @smm doesn’t intend to let this thread close, make changes based on what’s been said, and then start another discussion on a revised PEP (or worse, submit the revised PEP without further review). Because that seems to me to be wasteful of people’s time and energy - much better (IMO) to have a more interactive discussion where the PEP is updated as points are made, because that way people can see that their points have been recognised, and feed back immediately (while the details are fresh) if the response doesn’t meet their expectations.

Regarding approval of the PEP, I think @pradyunsg gave the only reasonable approach:

But again, I’d like to see a discussion that ends in at least a reasonable level of consensus before that process occurs. And it seems to me that this thread in particular is nowhere near reaching consensus.

3 Likes
55

Thank you very much @pf_moore , I agree.

@smm , @davidism would it be possible to extend the duration of this thread considering what was discussed above?

56

Yes, this thread can be extended. But I don’t see an option to do it.

1 Like
57

Maybe the moderators can remove the time limit on the thread? As a general point, I think time-limited threads are a very bad idea for discussions like this, as there’s no way of knowing how long it might take to get consensus, and without consensus there’s no real way forward.

5 Likes
58

I agree that many issues are still unresolved.

This is something that had been nagging at me as well. Lucky for us, I don’t think anyone’s going to try to game the system by suddenly giving commit privs to a bunch of people to stuff the ballot box. :slight_smile: But it still seems like some distortions could arise. In particular it could mean that projects that are important to the packaging ecosystem, but are maintained by a small, overworked team, could wind up with less voice than might be warranted.

This gets at what I see as the crux of the issue, which is: who is this packaging council for and what substantive actions will it take? In my mind I have two possible answers to this.

One possibility is that the packaging council is for management of packaging projects. In this case the question of “imposing decisions” becomes very relevant. I’m skeptical that such imposition will be very successful. So in this case it’s unclear to me what meaningful actions the PyPC might take. Maybe it can approve or reject packaging PEPs, but it can’t make anyone follow them.

The other possibility is that the packaging council is for messaging to Python users. In this case the council’s main task is not to manage the projects themselves, but rather to distill the work on those projects into documentation and recommendations that helps users navigate the landscape of tools. My perception is that this also could be fraught, both because there may be reluctance to “pick winners and losers” and because, even if such reluctance could be overcome, it’s unclear there’s consensus on what to recommend.

However, it at least seems in the realm of possibility to me that a packaging council could get to a point where it would release a document that says “Tools X, Y and Z exist to do such-and-such. We recommend X for the following reasons. Here is a guide on how to use it. Watch this space for future updates.” At any rate it seems more plausible than “Tools X, Y and Z exist to do such-and-such. The way X does it seems to be the best. Therefore, we’re telling Y and Z to start doing it that way.”

So the second view (namely, PyPC is for user-directed messaging) seems more doable to me. And under that view the council’s main task would be to create such documentation and make clear recommendations (which might include, for instance, saying “We don’t recommend Tool Z because we standardized on PEP 12345 and Tool Z has not adopted that”).

Because of that, I’m uncertain as to the practical impact of the sections of the PEP about liaising with or managing tools, requiring implementations to do this or that, etc. In the end, anyone maintaining a tool who doesn’t like what the PyPC says can just ignore the PyPC, fork the project if necessary, and do whatever they want. The only real power the council would have is to say “This project is officially recommended and that one isn’t.” So maybe that should be foregrounded more in the PEP.

1 Like
59

This topic was automatically closed after 21 days. New replies are no longer allowed.

60

The key thing I’d like to see (whether here or on the original thread) is some sort of response from @smm on the comments made, explaining what her view is and how she intends to address the points in the PEP - specifically including details and not just “the PEP will be updated to reflect this”.

I think that any other discussion is probably pointless unless @smm engages more actively - I know I’ve mostly given up on posting further as it feels like shouting into the void…

6 Likes
62

“I want to discuss more” is nice to say, but as you’ve observed it doesn’t appear that more discussion is actually having an effect at this point.

I’m not sure how, but you may want to coordinate more with the PSF about the Packaging Project Manager role. Here was the original post Python Software Foundation News: The PSF is hiring a Python Packaging Project Manager! If their job description or duties didn’t match the help packaging maintainers needed, you should probably send that feedback to the PSF so they can amend their job description and tasking/reporting processes in the future.

That said, given how packaging discussions seem to go in general, you may need to accept that no one will be able to fill the roll that everyone else thinks they want the way they all think they want it. At some point, someone will need to make concrete decisions rather than just continue discussion forever.

1 Like
63

FWIW, the funding for that role has run out and @smm is no longer doing this as a job. It can be seen that she’s also no longer listed as PSF staff.

64
65

Partly inspired by the discussion here, I’ve created a separate topic for getting a better sense of what people want from this whole “change packaging governance” conversation:

66

Moving forward to the next step is difficult with a long thread. While preserving the discussion is valuable for history, the long thread makes it hard to determine “where do we have consensus?”

Governance discussions are particularly hard because several topics get melded together:

  • The driving factors for a new governance
  • The scope of the new governance: what is in and out of scope
  • Bylaws or guiding principles of the governance
  • Structure of the governance
  • How to set up the first government

None of this is easy to do. The more clarity that you have on the “why” a new governance, then “how” to structure it, and “who” will steward it the more likely that you will be able to move forward.

A reasonable next step would be to identify and document where there is broad consensus. In other words, breaking the PEP up may be helpful and allow the governance to evolve which is simillar to what we did with core Python.

3 Likes
67

I have tried to answer your questions in the updated draft. One thing I’m not sure about is who is this council accountable to? I have specified the Python Steering Council but I don’t know if that is correct and also if they would want that responsibility. I don’t know who else can oversee the council.

1 Like
68

I have updated the wording.

Not intentional at all. The idea is to give everyone a vote. Same as PEP 609.

I really don’t know what the long-term effects of this would be. Would it make sense to limit the number of votes to one vote per project? But again, the votes would be tilted towards re-packagers/build-backend/installers/workflow tool projects that have high representation.

69

I think the assumption is everyone trusts each other to not let random people have commit privileges and no one project would try to stack the electorate by granting a bunch of people voting rights. I know I personally trust everyone else, and if you did try to do something bad you would get called out for it.

Just as a reminder, we are just a bunch of volunteers trying to figure out an easier way to make decisions. This isn’t an actual government that controls your lives. I saw similar teeth gnashing when we were figuring out the steering council and it all turned out to be for naught as everyone knows each other in the end.

2 Likes
70

I think your (correct) view of our internal trust is orthogonal to the question you’re responding to. My interpretation of what Anderson is talking about is that a project that is as old as pip would have potentially significantly more sway due to the number of volunteers versus a project like flit.

2 Likes
71

My (long) two cents: I see a fundamental problem with the concept of governance that spans multiple projects.

Governance over a single project seems straightforward to me. A project seeks to meet a need in the ecosystems it’s relevant to, and corresponding goals are relatively straightforward to define. (Note that these goals comprise both technical and non-technical aspects; e.g., the latter include diversity and contributor experience.) Stakeholders can disagree with regard to the definition and prioritization of those goals, and on the best methods to achieve them; but, ultimately, there is a singular entity around which goals are defined and that entity is a clear ‘center of gravity’ for the governance.

I don’t see how this works for governance over multiple projects—at least, for governance that tries to direct the projects it’s governing. We have examples of well-established entities that are succeeding in supporting multiple projects: Linux Foundation, Apache Foundation, NumFOCUS, and more. But while they may place some constraints on projects in order for those projects to remain under their umbrellas, they are not governing the projects, in terms of defining the totality of the technical and non-technical goals for those projects to strive toward.

I believe that this is the fundamental and unresolvable problem that’s been styming Python packaging governance discussion: that governance of a collective of projects is not viable.

Why should a pip maintainer (who is not also a twine maintainer) have a direct vote over how twine does things?

Does our pip maintainer have deep knowledge of twine? Of its technical goals? Of its technical needs? Of the health of its community? Etc. It seems to me that the answer to all of these questions is “no,” and thus the answer to the prior question is “they shouldn’t.”

Instead, I think we should redirect our efforts toward a focus on:

  1. Standards
    • See my comment on the ‘What do you want?’ thread for a slice of my thoughts here
  2. User-persona-specific tooling recommendations
    • Like what pyOpenSci is doing for packaging for Python newcomers, but repeated many times over for different user stories
      • This likely needs to be a community effort more than a centralized one
  3. Non-opinionated resources for informing the community w.r.t. what tools are available
    • These are basically awesome lists, but with more focus on ‘list’ than ‘awesome’
    • A big part of the power of open source is innovation
    • But, with rapid innovation comes the information problem of not knowing what tools are out there
      • Note: I know there’s a strong desire for “one-tool-to-rule-them-all”, to reduce this information problem but in my view centralization (as distinct from standardization) is antithetical to innovation
    • So: regularly updated, highly visible information sources about the current state of the tooling ecosystem in a given area are key to mitigate the information problem amid rapid innovation
    • As best I understand, awesome lists are more or less the best tool we currently have to approach this challenge
2 Likes