This topic was automatically closed after 21 days. New replies are no longer allowed.
The key thing I’d like to see (whether here or on the original thread) is some sort of response from @smm on the comments made, explaining what her view is and how she intends to address the points in the PEP - specifically including details and not just “the PEP will be updated to reflect this”.
I think that any other discussion is probably pointless unless @smm engages more actively - I know I’ve mostly given up on posting further as it feels like shouting into the void…
6 Likes
“I want to discuss more” is nice to say, but as you’ve observed it doesn’t appear that more discussion is actually having an effect at this point.
I’m not sure how, but you may want to coordinate more with the PSF about the Packaging Project Manager role. Here was the original post Python Software Foundation News: The PSF is hiring a Python Packaging Project Manager! If their job description or duties didn’t match the help packaging maintainers needed, you should probably send that feedback to the PSF so they can amend their job description and tasking/reporting processes in the future.
That said, given how packaging discussions seem to go in general, you may need to accept that no one will be able to fill the roll that everyone else thinks they want the way they all think they want it. At some point, someone will need to make concrete decisions rather than just continue discussion forever.
1 Like
FWIW, the funding for that role has run out and @smm is no longer doing this as a job. It can be seen that she’s also no longer listed as PSF staff.
Partly inspired by the discussion here, I’ve created a separate topic for getting a better sense of what people want from this whole “change packaging governance” conversation:
Moving forward to the next step is difficult with a long thread. While preserving the discussion is valuable for history, the long thread makes it hard to determine “where do we have consensus?”
Governance discussions are particularly hard because several topics get melded together:
- The driving factors for a new governance
- The scope of the new governance: what is in and out of scope
- Bylaws or guiding principles of the governance
- Structure of the governance
- How to set up the first government
None of this is easy to do. The more clarity that you have on the “why” a new governance, then “how” to structure it, and “who” will steward it the more likely that you will be able to move forward.
A reasonable next step would be to identify and document where there is broad consensus. In other words, breaking the PEP up may be helpful and allow the governance to evolve which is simillar to what we did with core Python.
3 Likes
I have tried to answer your questions in the updated draft. One thing I’m not sure about is who is this council accountable to? I have specified the Python Steering Council but I don’t know if that is correct and also if they would want that responsibility. I don’t know who else can oversee the council.
1 Like
I have updated the wording.
Not intentional at all. The idea is to give everyone a vote. Same as PEP 609.
I really don’t know what the long-term effects of this would be. Would it make sense to limit the number of votes to one vote per project? But again, the votes would be tilted towards re-packagers/build-backend/installers/workflow tool projects that have high representation.
I think the assumption is everyone trusts each other to not let random people have commit privileges and no one project would try to stack the electorate by granting a bunch of people voting rights. I know I personally trust everyone else, and if you did try to do something bad you would get called out for it.
Just as a reminder, we are just a bunch of volunteers trying to figure out an easier way to make decisions. This isn’t an actual government that controls your lives. I saw similar teeth gnashing when we were figuring out the steering council and it all turned out to be for naught as everyone knows each other in the end.
2 Likes
I think your (correct) view of our internal trust is orthogonal to the question you’re responding to. My interpretation of what Anderson is talking about is that a project that is as old as pip would have potentially significantly more sway due to the number of volunteers versus a project like flit.
2 Likes
My (long) two cents: I see a fundamental problem with the concept of governance that spans multiple projects.
Governance over a single project seems straightforward to me. A project seeks to meet a need in the ecosystems it’s relevant to, and corresponding goals are relatively straightforward to define. (Note that these goals comprise both technical and non-technical aspects; e.g., the latter include diversity and contributor experience.) Stakeholders can disagree with regard to the definition and prioritization of those goals, and on the best methods to achieve them; but, ultimately, there is a singular entity around which goals are defined and that entity is a clear ‘center of gravity’ for the governance.
I don’t see how this works for governance over multiple projects—at least, for governance that tries to direct the projects it’s governing. We have examples of well-established entities that are succeeding in supporting multiple projects: Linux Foundation, Apache Foundation, NumFOCUS, and more. But while they may place some constraints on projects in order for those projects to remain under their umbrellas, they are not governing the projects, in terms of defining the totality of the technical and non-technical goals for those projects to strive toward.
I believe that this is the fundamental and unresolvable problem that’s been styming Python packaging governance discussion: that governance of a collective of projects is not viable.
Why should a pip maintainer (who is not also a twine maintainer) have a direct vote over how twine does things?
Does our pip maintainer have deep knowledge of twine? Of its technical goals? Of its technical needs? Of the health of its community? Etc. It seems to me that the answer to all of these questions is “no,” and thus the answer to the prior question is “they shouldn’t.”
Instead, I think we should redirect our efforts toward a focus on:
- Standards
- See my comment on the ‘What do you want?’ thread for a slice of my thoughts here
- User-persona-specific tooling recommendations
- Like what pyOpenSci is doing for packaging for Python newcomers, but repeated many times over for different user stories
- This likely needs to be a community effort more than a centralized one
- Like what pyOpenSci is doing for packaging for Python newcomers, but repeated many times over for different user stories
- Non-opinionated resources for informing the community w.r.t. what tools are available
- These are basically awesome lists, but with more focus on ‘list’ than ‘awesome’
- A big part of the power of open source is innovation
- But, with rapid innovation comes the information problem of not knowing what tools are out there
- Note: I know there’s a strong desire for “one-tool-to-rule-them-all”, to reduce this information problem but in my view centralization (as distinct from standardization) is antithetical to innovation
- So: regularly updated, highly visible information sources about the current state of the tooling ecosystem in a given area are key to mitigate the information problem amid rapid innovation
- As best I understand, awesome lists are more or less the best tool we currently have to approach this challenge
2 Likes
That seems to argue for a single tool then to own everything to make governance easier. But if that’s going to occur we are back to trying to choose a winner, and people seemingly wanted some governance set up to help either pick a winner or work towards creating a winning tool.
I would argue we already do that and what we are trying to do is make that process easier via the governance proposal.
1 Like
Probably, but I think that’s already true as most of the folks who have been participating the longest and have earned a great deal of trust are from those projects. And so far, those folks have been doing the right thing and not stymieing things in my opinion.
1 Like
I think the ideal future would be where we use standards to add features, and have a default implementation but not choose a winner per se. Aka emphasize that the default is what we recommend people to use if they are just getting started, but is not the only choice. We would need a governance choosing the default though. Similar how to Python itself CPython is the defacto thing people reach to, aka the default; but is not the only choice one can make (PyPy, JPthon, micropython, rustpython, etc. are all good Python one can use for their problems).
2 Likes
Or for simply not trying to “govern” all of the various projects as one. I’m really not sure what problem the proposal to put a council in charge of all PyPA projects is intended to solve. It seems to have come out of the packaging strategy discussions, but to be blunt, I don’t think there was any consensus from those discussions, and in particular there was no consensus that all of the PyPA projects would be governed or owned by a single council. It was suggested, certainly - but so were many other ideas, and I don’t recall any of them getting broad approval.
We don’t need project governance for that. A council that simply has the same powers as the current PEP delegates would do that just as well.
3 Likes
This is the only part of your comment I disagree with. We don’t need “governance” to choose a default, all we need is community consensus. If we can’t get consensus, all the governance in the world won’t help. And if we can, governance adds nothing.
As much as I agree that we need to have a default, I don’t think every (major?) feature needs a standard. I spent some digital ink in Jan this year talking through the situation of how “we need a standard before we implement lockfiles” has played out for pip. I’ll quote a relevant portion from that here:
The reason the default tooling doesn’t improve is that making meaningful improvements to it is blocked on trying to cater to all workflows; in ways that alternative tooling is not.
I don’t think that’s a good state for things to be in and “use standards to add features” is exactly this situation. We’d either need to stop trying to cater to every tool + workflow under the sun when we build “standards”[1] or operate assuming that that every “standard” will be optional unless enforced by some other means (eg: PyPI checking things on upload, pip checks it on install, or something akin to that).
I don’t think that’s a reasonable approach to take here. My opinion is that “community concensus” can’t be achieved here, because we have way-too-many user stories with conflicting/non-overlapping needs in the community.
What an ML developer needs isn’t very similar to what a Linux redistributor needs from the “Python packaging” tools. Expecting these to come to a concensus is… not a good idea. These are two specific user personas I picked – the community has vastly more and is vastly bigger.
Given that, we’re basically going to end up with the ecosystem of packaging tooling being a popularity contest (wait, I guess it is already?) with no checks and balances which… seems counter to the idea of collaborating to making the Python packaging story better.[2]
I’d argue avoiding this is exactly why we need to pick a default somehow. The community can (has, and will) build tools around / on top of the default tools – a shared agreement, amongst the people most “wired-in” on these tools/processes, on a default experience that caters to a (substantial) subset of users[3] could help up focus on removing papercuts that users hit and make that experience excellent.
Edit: Re-reading this, I guess you might have had a smaller group in mind when you said community. However, I still think the broader point of “actually getting to concensus is vanishingly unlikely” still stands, and we’d need to explicitly say “that is not covered, sorry” for certain workflows – having a governance-supported scaffold to help make that call is a good idea.
You’re correct, we could have a named-group that just takes the standing PEP delegations and have the ability to approve PEPs. I don’t think it’s a good idea to do that without additional changes since it’d expose a lot of other underlying issues in how we operate.
To do this as a thought experiment… the smallest change we could make is keep everything the same and have the named-group make the call, with the SC delegating to the named-group. The named-group will have only one “hard” power: ability to decide on standards that tools may or may not follow.
They will have no ability to moderate the projects themselves, have no ability to deal with problematic individuals within the group (even CoC is separate from them), have no authority to liase on funded roles/provide oversight over them, and have no ability to meaningfully make a dent on some of the most explicitly-flagged concerns from the various user studies we’ve done.[4] It also explicitly does nothing to help engage with groups outside of the PyPA (reminder: the named-group will have no authority over the projects themselves and can not represent them, much like how Paul+Donald can’t represent all the PyPA projects).
If we have the named-group voted in by PyPA committers to minimise voting process changes, then this group will be vulnerable to ballot stuffing as well which (while it’s hypothetical) can slowly lead to a more disfunctional group unintentionally – especially if a project like pip will actively having less-and-less ability to advocate for its needs because they’re in a crowded environment with 5 workflow tools that also shipped 4 build-backends as well as additional domain-specific build-backends. If we have the named-group picked by some other mechanism, we have to somehow legitimize the named-group in the view of the projects that ought to listen to them – that’s maybe feasible but still has all the other issues.
If you start going down the route of fixing the council model to account for these things and avoid these pitfalls, you quickly end up in the direction of what the proposal is. Of course, not the exact letter of it – there’s certainly different choices that can be made (and that’s OK?[5]).
At which point I think that’s a misnomer, since it’s much closer to a feature proposal to a tool or, at-best, a well-known set of tools. ↩︎
Is it too political to say that a “free market with no regulation” will lead to some people doing bad things? ↩︎
that they are not locked in! ↩︎
Notably, “I prefer to have a clearly defined official workflow” was the most agreed-to statement in the 2022 packaging survey. (38% strongly agree + 37% agree; with 6% disagree and fewer in strong disagree – somewhere between 1%-5%, the chart doesn’t have a number). ↩︎
I’ll remind folks that it’s a draft by a first-time PEP author tackling a difficult subject. ↩︎
5 Likes
I agree - but I would argue that the scope of the standards that have been considered thus far is much too narrow; and, that there is a large, diverse body of stakeholders in Python distribution and packaging writ large for whom the standards that have been discussed/codified are only slightly relevant at best.
If this is the case, then I’m reading the governance proposal and the surrounding conversation quite differently from how you are. (Which could be on me! Happy to have my mind changed.)
Regardless, I think the PyPA-centricity of the current proposal is profoundly insufficient, and problematic because of that insufficiency. Geospatial, ML, cryptography packaging … the diversity of Python runtimes and their distributions… the space is so much bigger than the PyPA umbrella.
Now—maybe the goal here is to draw a box around the users and use cases that the PyPA projects are currently seeking to serve, and improve the distribution and packaging story within that box. If so, I’m generally okay with that—but, the absolute first step in this process needs to be defining the boundaries of that box. To codify: ‘These are the users and use cases that the PyPA is claiming responsibility for; if these don’t describe you, please look elsewhere.’
I’m not 100% confident it’s possible, but I could believe that there is a PyPA box of this sort that could be drawn…
… and even potentially a “one PyPA tool to rule them all” for values of ‘all’ within that carefully drawn PyPA box. But, I’m quite confident that such a tool would not rule most of the territory outside that PyPA box.
Further—while I understand the allure of the simplicity and comprehensibility of a ‘winning tool,’ one of my biggest concerns about working toward a ‘winning tool’ at all is a sacrifice of innovation in the space. I’m very confident that successfully centralizing around a winner would result in overall less/slower innovation. Perhaps trading away innovation to gain simplicity is a good exchange, but I’m not convinced of that.
Yes please! Let’s narrow the scope of the standards we’re contemplating to whatever degree is necessary to make it possible to reach consensus. Yes, this may require a large number of related, granular standards in some cases – but I’d far prefer that situation to having unending irresolvable discussion and few standards.
2 Likes
Please note that in the case of my previous comment there is no teeth gnashing.
I am not saying that the design being proposed for the governance system is worse or better than a different one in relation to the voting mechanism.
My belief is that, no matter which system we choose, it will be flawed in some form (basically because translating personal choices/preferences/beliefs into “group action/policy/strategy” has been demonstrated to be an unattractable problem for some sets of assumptions and optimisation goals, as studied in the field of social choice theory[1]).
Therefore it is important that we are conscious and aware about the limitations and biases of whichever system we choose. I think this goes well with your other phrase:
… and as such, it will be nice if we can make informed decisions about it (with awareness about the flaws that we are accepting).
While I agree with what you wrote, I believe that the least amount of assumptions[2] needed to make a governance system to work under a set of criteria, the more robust and resilient this system will be.
The point here is not about projects granting a bunch of voting rights to sway results in one direction or the other, but the understanding that people working in different aspects of the ecosystem will have (genuine) different concerns and (genuine) different priorities (individually, each person is only doing what they believe to be the best for the community). If that translates to proportionality in the representation, aspects of the ecosystem that are similarly important maybe end up with disproportional representation.
This is actually a difficult problem to come up with a solution…
While 1-vote-per-project[3] as mentioned by @smm (referenced below), may help with this problem, it may also be prone to gerrymandering…
Unfortunately, I don’t have an answer of which is best (and that is likely to depend on which aspects we want to optimise and which flaws we are willing to accept). But I think that the questions I made in my previous post are some of the interesting questions we can make for us to start to understand which biases and flaws we are accepting under the choice of the governance system[4] (as I said before, I believe that there will always be flaws and biases, regardless of how diligent we are in designing it).
That is also another reason why we don’t need to mimic structures that we find elsewhere in the governance, just because we are used to them. As I mentioned in previous comments, if we have lack of consensus in the community and the best solution that we can come out with is “let’s vote on a group of people that have the legitimacy to vertically impose a particular version”, this approach may ended up corroding the base of volunteers that we have; because fundamentally volunteer work has different “motivation” dynamics…
Ideally, before selecting a governance model, I would like to know:
- Which problems we are trying to solve
- Why/how the proposal contributes to solving these problems
- Why/how the proposal is suitable for the PyPA (and the challenges of being a volunteer organisation that have multiple interacting/potentially-overlapping projects)
- What are the flaws and biases of the proposed solution.
That is why I asked the questions in my previous post.
4 Likes