Draft update to Python Packaging Governance

Partly inspired by the discussion here, I’ve created a separate topic for getting a better sense of what people want from this whole “change packaging governance” conversation:

Moving forward to the next step is difficult with a long thread. While preserving the discussion is valuable for history, the long thread makes it hard to determine “where do we have consensus?”

Governance discussions are particularly hard because several topics get melded together:

  • The driving factors for a new governance
  • The scope of the new governance: what is in and out of scope
  • Bylaws or guiding principles of the governance
  • Structure of the governance
  • How to set up the first government

None of this is easy to do. The more clarity that you have on the “why” a new governance, then “how” to structure it, and “who” will steward it the more likely that you will be able to move forward.

A reasonable next step would be to identify and document where there is broad consensus. In other words, breaking the PEP up may be helpful and allow the governance to evolve which is simillar to what we did with core Python.

3 Likes

I have tried to answer your questions in the updated draft. One thing I’m not sure about is who is this council accountable to? I have specified the Python Steering Council but I don’t know if that is correct and also if they would want that responsibility. I don’t know who else can oversee the council.

1 Like

I have updated the wording.

Not intentional at all. The idea is to give everyone a vote. Same as PEP 609.

I really don’t know what the long-term effects of this would be. Would it make sense to limit the number of votes to one vote per project? But again, the votes would be tilted towards re-packagers/build-backend/installers/workflow tool projects that have high representation.

I think the assumption is everyone trusts each other to not let random people have commit privileges and no one project would try to stack the electorate by granting a bunch of people voting rights. I know I personally trust everyone else, and if you did try to do something bad you would get called out for it.

Just as a reminder, we are just a bunch of volunteers trying to figure out an easier way to make decisions. This isn’t an actual government that controls your lives. I saw similar teeth gnashing when we were figuring out the steering council and it all turned out to be for naught as everyone knows each other in the end.

2 Likes

I think your (correct) view of our internal trust is orthogonal to the question you’re responding to. My interpretation of what Anderson is talking about is that a project that is as old as pip would have potentially significantly more sway due to the number of volunteers versus a project like flit.

2 Likes

My (long) two cents: I see a fundamental problem with the concept of governance that spans multiple projects.

Governance over a single project seems straightforward to me. A project seeks to meet a need in the ecosystems it’s relevant to, and corresponding goals are relatively straightforward to define. (Note that these goals comprise both technical and non-technical aspects; e.g., the latter include diversity and contributor experience.) Stakeholders can disagree with regard to the definition and prioritization of those goals, and on the best methods to achieve them; but, ultimately, there is a singular entity around which goals are defined and that entity is a clear ‘center of gravity’ for the governance.

I don’t see how this works for governance over multiple projects—at least, for governance that tries to direct the projects it’s governing. We have examples of well-established entities that are succeeding in supporting multiple projects: Linux Foundation, Apache Foundation, NumFOCUS, and more. But while they may place some constraints on projects in order for those projects to remain under their umbrellas, they are not governing the projects, in terms of defining the totality of the technical and non-technical goals for those projects to strive toward.

I believe that this is the fundamental and unresolvable problem that’s been styming Python packaging governance discussion: that governance of a collective of projects is not viable.

Why should a pip maintainer (who is not also a twine maintainer) have a direct vote over how twine does things?

Does our pip maintainer have deep knowledge of twine? Of its technical goals? Of its technical needs? Of the health of its community? Etc. It seems to me that the answer to all of these questions is “no,” and thus the answer to the prior question is “they shouldn’t.”

Instead, I think we should redirect our efforts toward a focus on:

  1. Standards
    • See my comment on the ‘What do you want?’ thread for a slice of my thoughts here
  2. User-persona-specific tooling recommendations
    • Like what pyOpenSci is doing for packaging for Python newcomers, but repeated many times over for different user stories
      • This likely needs to be a community effort more than a centralized one
  3. Non-opinionated resources for informing the community w.r.t. what tools are available
    • These are basically awesome lists, but with more focus on ‘list’ than ‘awesome’
    • A big part of the power of open source is innovation
    • But, with rapid innovation comes the information problem of not knowing what tools are out there
      • Note: I know there’s a strong desire for “one-tool-to-rule-them-all”, to reduce this information problem but in my view centralization (as distinct from standardization) is antithetical to innovation
    • So: regularly updated, highly visible information sources about the current state of the tooling ecosystem in a given area are key to mitigate the information problem amid rapid innovation
    • As best I understand, awesome lists are more or less the best tool we currently have to approach this challenge
2 Likes

That seems to argue for a single tool then to own everything to make governance easier. But if that’s going to occur we are back to trying to choose a winner, and people seemingly wanted some governance set up to help either pick a winner or work towards creating a winning tool.

I would argue we already do that and what we are trying to do is make that process easier via the governance proposal.

1 Like

Probably, but I think that’s already true as most of the folks who have been participating the longest and have earned a great deal of trust are from those projects. And so far, those folks have been doing the right thing and not stymieing things in my opinion.

1 Like

I think the ideal future would be where we use standards to add features, and have a default implementation but not choose a winner per se. Aka emphasize that the default is what we recommend people to use if they are just getting started, but is not the only choice. We would need a governance choosing the default though. Similar how to Python itself CPython is the defacto thing people reach to, aka the default; but is not the only choice one can make (PyPy, JPthon, micropython, rustpython, etc. are all good Python one can use for their problems).

2 Likes

Or for simply not trying to “govern” all of the various projects as one. I’m really not sure what problem the proposal to put a council in charge of all PyPA projects is intended to solve. It seems to have come out of the packaging strategy discussions, but to be blunt, I don’t think there was any consensus from those discussions, and in particular there was no consensus that all of the PyPA projects would be governed or owned by a single council. It was suggested, certainly - but so were many other ideas, and I don’t recall any of them getting broad approval.

We don’t need project governance for that. A council that simply has the same powers as the current PEP delegates would do that just as well.

3 Likes

This is the only part of your comment I disagree with. We don’t need “governance” to choose a default, all we need is community consensus. If we can’t get consensus, all the governance in the world won’t help. And if we can, governance adds nothing.

As much as I agree that we need to have a default, I don’t think every (major?) feature needs a standard. I spent some digital ink in Jan this year talking through the situation of how “we need a standard before we implement lockfiles” has played out for pip. I’ll quote a relevant portion from that here:

The reason the default tooling doesn’t improve is that making meaningful improvements to it is blocked on trying to cater to all workflows; in ways that alternative tooling is not.

I don’t think that’s a good state for things to be in and “use standards to add features” is exactly this situation. We’d either need to stop trying to cater to every tool + workflow under the sun when we build “standards”[1] or operate assuming that that every “standard” will be optional unless enforced by some other means (eg: PyPI checking things on upload, pip checks it on install, or something akin to that).

I don’t think that’s a reasonable approach to take here. My opinion is that “community concensus” can’t be achieved here, because we have way-too-many user stories with conflicting/non-overlapping needs in the community.

What an ML developer needs isn’t very similar to what a Linux redistributor needs from the “Python packaging” tools. Expecting these to come to a concensus is… not a good idea. These are two specific user personas I picked – the community has vastly more and is vastly bigger.

Given that, we’re basically going to end up with the ecosystem of packaging tooling being a popularity contest (wait, I guess it is already?) with no checks and balances which… seems counter to the idea of collaborating to making the Python packaging story better.[2]

I’d argue avoiding this is exactly why we need to pick a default somehow. The community can (has, and will) build tools around / on top of the default tools – a shared agreement, amongst the people most “wired-in” on these tools/processes, on a default experience that caters to a (substantial) subset of users[3] could help up focus on removing papercuts that users hit and make that experience excellent.

Edit: Re-reading this, I guess you might have had a smaller group in mind when you said community. However, I still think the broader point of “actually getting to concensus is vanishingly unlikely” still stands, and we’d need to explicitly say “that is not covered, sorry” for certain workflows – having a governance-supported scaffold to help make that call is a good idea.

You’re correct, we could have a named-group that just takes the standing PEP delegations and have the ability to approve PEPs. I don’t think it’s a good idea to do that without additional changes since it’d expose a lot of other underlying issues in how we operate.

To do this as a thought experiment… the smallest change we could make is keep everything the same and have the named-group make the call, with the SC delegating to the named-group. The named-group will have only one “hard” power: ability to decide on standards that tools may or may not follow.

They will have no ability to moderate the projects themselves, have no ability to deal with problematic individuals within the group (even CoC is separate from them), have no authority to liase on funded roles/provide oversight over them, and have no ability to meaningfully make a dent on some of the most explicitly-flagged concerns from the various user studies we’ve done.[4] It also explicitly does nothing to help engage with groups outside of the PyPA (reminder: the named-group will have no authority over the projects themselves and can not represent them, much like how Paul+Donald can’t represent all the PyPA projects).

If we have the named-group voted in by PyPA committers to minimise voting process changes, then this group will be vulnerable to ballot stuffing as well which (while it’s hypothetical) can slowly lead to a more disfunctional group unintentionally – especially if a project like pip will actively having less-and-less ability to advocate for its needs because they’re in a crowded environment with 5 workflow tools that also shipped 4 build-backends as well as additional domain-specific build-backends. If we have the named-group picked by some other mechanism, we have to somehow legitimize the named-group in the view of the projects that ought to listen to them – that’s maybe feasible but still has all the other issues.

If you start going down the route of fixing the council model to account for these things and avoid these pitfalls, you quickly end up in the direction of what the proposal is. Of course, not the exact letter of it – there’s certainly different choices that can be made (and that’s OK?[5]).


  1. At which point I think that’s a misnomer, since it’s much closer to a feature proposal to a tool or, at-best, a well-known set of tools. ↩︎

  2. Is it too political to say that a “free market with no regulation” will lead to some people doing bad things? ↩︎

  3. that they are not locked in! ↩︎

  4. Notably, “I prefer to have a clearly defined official workflow” was the most agreed-to statement in the 2022 packaging survey. (38% strongly agree + 37% agree; with 6% disagree and fewer in strong disagree – somewhere between 1%-5%, the chart doesn’t have a number). ↩︎

  5. I’ll remind folks that it’s a draft by a first-time PEP author tackling a difficult subject. ↩︎

5 Likes

I agree - but I would argue that the scope of the standards that have been considered thus far is much too narrow; and, that there is a large, diverse body of stakeholders in Python distribution and packaging writ large for whom the standards that have been discussed/codified are only slightly relevant at best.

If this is the case, then I’m reading the governance proposal and the surrounding conversation quite differently from how you are. (Which could be on me! Happy to have my mind changed.)

Regardless, I think the PyPA-centricity of the current proposal is profoundly insufficient, and problematic because of that insufficiency. Geospatial, ML, cryptography packaging … the diversity of Python runtimes and their distributions… the space is so much bigger than the PyPA umbrella.

Now—maybe the goal here is to draw a box around the users and use cases that the PyPA projects are currently seeking to serve, and improve the distribution and packaging story within that box. If so, I’m generally okay with that—but, the absolute first step in this process needs to be defining the boundaries of that box. To codify: ‘These are the users and use cases that the PyPA is claiming responsibility for; if these don’t describe you, please look elsewhere.’

I’m not 100% confident it’s possible, but I could believe that there is a PyPA box of this sort that could be drawn…

… and even potentially a “one PyPA tool to rule them all” for values of ‘all’ within that carefully drawn PyPA box. But, I’m quite confident that such a tool would not rule most of the territory outside that PyPA box.

Further—while I understand the allure of the simplicity and comprehensibility of a ‘winning tool,’ one of my biggest concerns about working toward a ‘winning tool’ at all is a sacrifice of innovation in the space. I’m very confident that successfully centralizing around a winner would result in overall less/slower innovation. Perhaps trading away innovation to gain simplicity is a good exchange, but I’m not convinced of that.

Yes please! Let’s narrow the scope of the standards we’re contemplating to whatever degree is necessary to make it possible to reach consensus. Yes, this may require a large number of related, granular standards in some cases – but I’d far prefer that situation to having unending irresolvable discussion and few standards.

2 Likes

Please note that in the case of my previous comment there is no teeth gnashing.
I am not saying that the design being proposed for the governance system is worse or better than a different one in relation to the voting mechanism.

My belief is that, no matter which system we choose, it will be flawed in some form (basically because translating personal choices/preferences/beliefs into “group action/policy/strategy” has been demonstrated to be an unattractable problem for some sets of assumptions and optimisation goals, as studied in the field of social choice theory[1]).

Therefore it is important that we are conscious and aware about the limitations and biases of whichever system we choose. I think this goes well with your other phrase:

… and as such, it will be nice if we can make informed decisions about it (with awareness about the flaws that we are accepting).

While I agree with what you wrote, I believe that the least amount of assumptions[2] needed to make a governance system to work under a set of criteria, the more robust and resilient this system will be.

The point here is not about projects granting a bunch of voting rights to sway results in one direction or the other, but the understanding that people working in different aspects of the ecosystem will have (genuine) different concerns and (genuine) different priorities (individually, each person is only doing what they believe to be the best for the community). If that translates to proportionality in the representation, aspects of the ecosystem that are similarly important maybe end up with disproportional representation.

This is actually a difficult problem to come up with a solution…
While 1-vote-per-project[3] as mentioned by @smm (referenced below), may help with this problem, it may also be prone to gerrymandering…

Unfortunately, I don’t have an answer of which is best (and that is likely to depend on which aspects we want to optimise and which flaws we are willing to accept). But I think that the questions I made in my previous post are some of the interesting questions we can make for us to start to understand which biases and flaws we are accepting under the choice of the governance system[4] (as I said before, I believe that there will always be flaws and biases, regardless of how diligent we are in designing it).

That is also another reason why we don’t need to mimic structures that we find elsewhere in the governance, just because we are used to them. As I mentioned in previous comments, if we have lack of consensus in the community and the best solution that we can come out with is “let’s vote on a group of people that have the legitimacy to vertically impose a particular version”, this approach may ended up corroding the base of volunteers that we have; because fundamentally volunteer work has different “motivation” dynamics…

Ideally, before selecting a governance model, I would like to know:

  1. Which problems we are trying to solve
  2. Why/how the proposal contributes to solving these problems
  3. Why/how the proposal is suitable for the PyPA (and the challenges of being a volunteer organisation that have multiple interacting/potentially-overlapping projects)
  4. What are the flaws and biases of the proposed solution.

That is why I asked the questions in my previous post.


  1. E.g. Arrow’s impossibility theorem, Gibbard’s theorem and so on… ↩︎

  2. about how each member behaves ↩︎

  3. or a different group concept not necessarily tied to a PyPA project, e.g. thematic group. ↩︎

  4. For example, that can be useful if we want to add mechanisms to counter act those biases and flaws. ↩︎

4 Likes

I mostly agree, but the proviso is, of course, that how easily you can get consensus from a group depends on how narrowly you circumscribe that group. :slight_smile: So we do need to leave out certain workflows, but then try to reach some measure of consensus within what’s left.

It’s been mentioned a few times by various people on these threads, but I still have some idea that there is an 80%, or 90%, or who knows, maybe just 75% proportion of use cases that can be improved significantly. The question is how much harder life will get for the other 20% (or 10%, or 25%), but I think even so that can be minimized.[1]

Another point is that, if we consider the overall set of Python users, I think there is a strong likelihood that we can make life better for a lot of them without them necessarily “reaching consensus”, because a fair number of users may be happy with options they hadn’t considered — and in fact that can maybe happen even without full consensus among the smaller number of people involved in these discussions. That’s one reason I think it’s useful to “work backwards” from an end-user perspective. We don’t need everyone who is developing packaging tools to agree on a way to do it. We just need one good way that works for a large number of users, and then we need to communicate that effectively. Maybe “communicate that effectively” is the same as “making it official”, but maybe it just means “have clear, unambiguous documentation and tutorials on docs.python.org for reliable ways to do the things people need to do” (or maybe those are the same thing?).

So yeah, I think governance can help even without full consensus. One way that can potentially happen is through more user surveys and the like, so we can get to a point where we can say “among these various possibilities, users are clearly happiest with option X, so that’s the one we’re making official”.


  1. As an example, “Linux distro redistrbutors” are brought up from time to time, but I tend to think that particular workflow/usecase is so far outside the bounds of the vast majority of Python users that we should feel free to back-burner it. Not so say that we should make it insanely difficult, but just that it would be okay to make things significantly less convenient for Linux packagers if it makes things even slightly easier for regular Python users. ↩︎

2 Likes

As an example, “Linux distro redistrbutors” are brought up from
time to time, but I tend to think that particular workflow/usecase
is so far outside the bounds of the vast majority of Python users
that we should feel free to back-burner it. Not so say that we
should make it insanely difficult, but just that it would be okay
to make things significantly less convenient for Linux packagers
if it makes things even slightly easier for regular Python users.

Are there numbers to back up that impression? Most of the people I
interact with on a regular basis are Linux users and can’t stand
Windows, so my perceptions are biased in such a way that it seems to
me the majority of “regular Python users” are using Linux
distributions and that the very few Windows users I know have no
clue what Python even is. Without a good survey (one that can
account for if not eliminate platform selection bias), it’s very
hard to paint some particular slice of users as “regular Python
users” and others as an inconsequential minority.

Note that by easing packaging and redistribution (by Linux
distributions, Conda, whoever) you’re decreasing their workload and
so increasing the number of packages they can make available to
consumers of those distribution channels. Treating people doing that
packaging work as less important means you’re treating their (vast
swaths of) users as less important too. And it’s entirely possible
that, on the whole, more users of Python are getting packages
through those channels than from PyPI.

1 Like

Pretty sure Brendan is referring to the people who work on RedHat, Ubuntu, etc and make sure that Python is available as a package on their distribution. Not “all linux users”. So I think it’s pretty reasonable to say that there are a lot fewer of them than most other types of users [1].

Although a related point is the ecosystem angle–making things harder for those folks can have downstream effects on a large population of users.


  1. I know there are a lot of distros out there but not that many ↩︎

1 Like

I’m going to say that this Discourse is open to all and leave at that, otherwise this typically ends up going down the rabbit hole of finger-pointing over participation.

Hence the PEP specifies how PyPA projects would be brought under the purview the proposed council to help make sure standards are not done just to spill a ton of digital ink.

Sure, but my concern is striving for perfect/better is going to lead to this never being resolved.

When it came to the SC, we had massive pressure to resolve the situation since Python as a project was effectively frozen until we solved it. But in our case here we can continue to hobble along with the status quo because things are not frozen if we make no changes. So unless we want to do something as drastic as saying, “no packaging PEPs will be approved until we come up with a different governance model” or “no new pip and setuptools releases”, there’s nothing forcing anyone to work towards a solution other than being so upset with the current situation that they personally view it as untenable and refuse to participate until there’s a change and those people being critical enough that there’s an impact from their lack of participation.

And I will say that I find voting to be the perfect allegory of this worry. I honestly think we spent more time arguing over the voting system than the actual governance model for the SC. But because we needed a solution for voting to move forward we all finally came together and compromised on something. But in our current situation, no one has to compromise if the status quo is “good enough” to them, and so we can all argue for what we view as the better solution with no ramifications if we don’t compromise.

2 Likes