Proposal to create a Python Packaging Council

Packaging
81

Both of you seem to be seeing this new council as fairly narrowly scoped and fairly passive (fallback decision making / pronouncements). That seems even more narrow than a PyPA-only focus, which would include topics like UX and tool development which do not fall under the PEP process.

For PEP pronouncements: I’m not sure that that’s a real pain point right now, but if so then I think either Paul or the SC can already add more folks to take on this responsibility? It doesn’t seem like we’d need a council with a voting process for that.

I understand excluding Conda from the primary scope, on the basis that it has its own repository, metadata and ecosystem. Poetry and PDM are PyPI-focused tools though, so it worries me that you’d exclude those. At that point, I’d say it’s no longer a Python packaging council focused on PyPI and related standards and tools, but rather a PyPA council. Also potentially useful, but then let’s call it that.

Re voting, I’m comfortable with PyPA members as the initial voting group, and it seems like the most pragmatic choice.

Re scope, I think from narrow to broad these are the choices for the primary scope/focus (each later one also includes the former):

  1. PEPs only
  2. PyPA and its tools/workflows
  3. PyPI-focused Python packaging tools/workflows
  4. Python packaging tools/workflows[1]

My sense is (4) is seen as too broad, and (1) is too narrow - so the choice is between (2) and (3).

A scope can evolve over time, so maybe the even more interesting question is whether this is a mostly-passive council for decision making in the absence of consensus on some topic, or if this is a more active council that gets together regularly and aims to drive things forward in a more proactive manner. I’d personally favor the latter, with council members committing to say >=2 hrs/wk of effort[2], and having meetings (with each other and key projects/stakeholders), writing docs like a roadmap or maintaining a list of key priorities.

Deciding on that now may again be difficult, so maybe it can be resolved by candidates including a short statement with their candidacy that includes what they see as the scope and how they prefer the council to operate? Then voters can take that into account, and the council can take it as a signal and start operating accordingly.

  1. If that seems fuzzy, I’d clarify that it explicitly includes at least the use of build frontends/backends to build binaries and install them directly for other packaging ecosystems. ↩︎

  2. This can be instead of rather than on top of some other activity, it’s voluntary effort after all. To get something impactful done, it seems to me like there’s some lower bound on time investment that will be needed though. ↩︎

1 Like
82

They have excluded themselves, in a fairly important sense. They are welcome to be PyPA projects right now, and that comes with essentially no commitments or constraints. The fact that they haven’t is their choice. And if they want to be included in the sense of your comment above, all they have to do is ask for PyPA membership.

In that sense, the current lack of practical power the PyPA has is beneficial, because it means that projects wanting to be part of the voting group just have to apply for PyPA membership.

Regarding scope, I tend to agree that just PEPs is too narrow. As @brettcannon mentioned me by name, I’ll say that while I don’t find being PEP delegate too onerous, there have been times when having a group of people to discuss a decision with, without the glare of doing so in a public forum, would have been helpful. I see the council’s immediate benefit as being in that area - making statements and pronouncements that need a “single voice” but which shouldn’t be one person’s opinion. The amount of authority that voice will have will grow as the council gains credibility by its actions, but Brett’s “PEP pronouncements and making decisions when consensus can’t be reached” is a good minimum baseline for that.

Beyond that, authority over PyPA tools is natural, as the PyPA committers voted the council in, and so should have, in effect, given the council that authority. I would hope the council wields that authority lightly, though - if things get confrontational between the council and a PyPA project, then IMO the council has failed. This raises one important question, though - will PEP 609 still apply for those situations where it requires a full PyPA vote? I think it should.

Going beyond that to your level (3) or even (4) will, IMO, be largely a consequence of how the natural authority of the council grows. We can’t force anyone outside of the PyPA[1] to accept the council’s pronouncements, but like with packaging PEPs, they will have force to the extent that the council’s voice is respected. That’s why I would rather the council focus on building, growing and strengthening the community, rather than on simply making decisions and pronouncements. For example, better defining what it means to be a non-PyPA packaging project like Poetry or PDM, understanding what those projects gain from not being under the PyPA banner, and how we can best work with them given the fact that they made that choice.

  1. It’s not like we can force PyPA members either, TBH. ↩︎

1 Like
83

I am not sure that that is a fair assessment. Their authors are trying to build good tools, provide value to their users and follow standards as best they can. Here’s the PDM “About” snippet from its repo: “A modern Python package and dependency manager supporting the latest PEP standards”.

To look at another group of tools, build backends, did meson-python and scikit-build, as the two most promising tools for packages with significant compiled code needs, also “exclude themselves” by not putting their repos under the PyPA org? Speaking for meson-python, there was certainly no such intent, we just want to build a good build backend. And we considered both the Meson and PyPA orgs when moving the repo; we went with Meson because we collaborate closely with the Meson maintainers and anticipate upstreaming parts of our code into Meson in the future. The thought of actively not wanting to be part of PyPA did not occur to us[1].

Exactly, you can’t (and shouldn’t) force anyone, that thought of forcing and control is really just out of place here. Each project, whether it lives under the PyPA GitHub org or not, is controlled by its maintainers, who generally all try to participate, implement PEPs, and improve their tools.

I look at the tool landscape roughly like this:

  • build backends for compiled code: 2/3 are not under PyPA, 1/3 is (setuptools)
  • build frontends/installers: pip/build, both under PyPA
  • “higher level workflow tools”: 3 established ones (Poetry, Hatch, PDM) and at least 2 new/experimental ones (Posy/Rye), only 1/5 is under PyPA (hatch)
  • a smattering of other tools and libraries both inside and outside PyPA

So this is about half of the prominent PyPI/Python/sdist/wheel-focused tools not under the PyPA. If a Python packaging council finds that half less important, or out of scope, or as “excluding themselves”, then please call it the PyPA Council rather than the Python Packaging Council.

  1. @FFY00 was a member already of course at the time, and I may still become one if for example pyproject-metadata would be moved under the PyPA org. ↩︎

2 Likes
84

I’m confused. What do you want the council to do with these projects? I’ve already said I think that promoting tools working together is a primary goal, regardless of whether this is PyPA tools or other tools. The council will approve standards based on overall community good, not whether it suits the PyPA. The council might have authority to dictate design decisions on PyPA projects - I’m not sure I think it should, but that may be what people arguing that the council “has ownership of PyPA projects” mean. But what exactly do you expect the council to be able to do with regard to “included” projects that it can’t do with “excluded” ones? I’m not the person who introduced the term “excluded”, so this is a genuine question here - I honestly don’t know what they are being excluded from.

If your only concern is what the council gets called, then I honestly don’t care. What matters to me is results, not names.

85

I’m personally fine with that expansion.

No one has said otherwise, but their level of engagement in the overall process is up to them.

Who ever said that? We are trying to establish who gets to vote in the (initial) council. Having it be PyPA members keeps it simple without arguing over who outside of that org is “important enough” to also get included. Compared to today, it’s no worse and you could argue better since Paul and Donald were appointed for PEP delegation by the SC. At this point we are just trying to get a council to take over for Paul and Donald, and that’s it. Once the council is formed we can see how people choose to listen to it and what “power” people grant it by doing what it does (we aren’t exactly governed by laws here; no one must listen to anyone else).

But if the concern is oversized power, look at the last 27 packaging PEPs and notice 20 of them have PyPA member (co-)authors. I also suspect all the non-Rust projects are using a PyPA project as a dependency. And Nathaniel already participates here so that takes care of Posy. And the setuptools folks also participate. And you, @rgommers , participate for meson-python. And you know you are being listened to (based on the amount of replies here alone :wink:), so I’m not sure where this “less important” feeling is coming from? I’m personally viewing it as a matter of logistics more than anything and to help guarantee something happens here and we don’t derail ourselves on this idea which I’m afraid we are going to do the more we don’t focus on the logistics of voting and getting a PEP written and accepted.

I agree, and I feel like this conversion is starting to get phrased as adversarial instead of simply trying to figure out a voting body to get this whole thing going to take over for PEP delegations.

Ditto.

2 Likes
86

What if we modeled it after the PSF a bit, and said membership in the voting body is open to anyone who spends at least some hours per month (the PSF uses 5) working on improving Python packaging [1], and anyone meeting those guidelines can self certify to be registered as a voting member, with the council reserving the right to deny someone’s self certification if it was determined to be in bad faith.

This gives us a few benefits:

  • It sidesteps the questions about whether a project like PDM or Poetry is eligible to vote without joining the PyPA.
  • It eliminates (or atleast reduces) conflicts of interests in determining what projects should be maintained into the future or not [2].
  • It Allows people who care enough to participate in discussions, but don’t have a project to work on, to have an avenue for their voice to be heard in the make up of the council.
  • It doesn’t have the administrative problems of trying to administer an election open to just anyone, by still having a limited electorate that requires some sort of positive opt-in.
    • Even limiting to PyPA has some administrative concerns, do external contributors count? What about triagers? etc?

With the main downside being that the council (or someone at least) has to occasionally validate self certification.

  1. Working on OSS projects, writing Packaging PEPs, participating in packaging discussions, etc. ↩︎

  2. There’s still the innate conflict of people not wanting the thing they’ve worked on to be deprecated or what not, but at least we don’t add more to it by tying voting membership to it. ↩︎

12 Likes
87

That is a good point and something I’d wondered about as well. Although, as others mentioned, there’s also the question of how much participation there might be from those communities even if they were allowed to vote. An unfortunate cascade effect of the fragmentation of Python packaging is that the communities are also fragmented, so who knows how many people there are in the communities for those other tools who aren’t really aware of the discussion happening here.

I agree. Based on earlier discussion I think that’s more what’s needed. Just having a group to provide fallback decisions doesn’t seem likely to result in a change in direction (although it may reduce the burden on currently overworked people, which would be a benefit).

To me part of the goal of such a group would be to cast a somewhat broader net in terms of vision. That doesn’t mean we need representatives from each tool on the council, but hopefully it would mean that there’s some effort to bring maintainers (and users!) of other tools “into the loop”, for instance to understand more fully what problems are currently being solved outside the official toolchain and how (of whether) they could be solved inside it. I’m not sure what the best way to do that is, but I feel like the packaging council wouldn’t be super impactful if it just maintains the same kind of perspective, purview, etc. that the current informal systems have.

One thing that hasn’t been mentioned here but that I think is important is docs (by which I mean the packaging docs on and accessible from docs.python.org). That is an area where I think some improvements could be made pretty quickly, and where something like a packaging council could be useful in evaluating potential changes.

88

While I agree with this[1], I’m concerned if people think this is blocked on something like having a council - what’s stopping people offering PRs to improve the docs right now? Or if your concern is cases where existing PRs aren’t getting evaluated and/or actioned promptly enough, do you have examples?

I’d be worried if people think that having a council will magically make contributions appear - while the council is a good way forward to address the issue of not having an “official voice” for the packaging community, it’s going to make no difference at all to how much volunteer effort gets contributed to packaging.

  1. Docs changes have been notoriously prone in the past to getting bogged down in debates over details. ↩︎

2 Likes
89

Except I’m not worried about people being mad enough at the PSF to falsely self-certify just to try and get what they want by electing certain people. :wink: It might/probably would be okay, but this is where my mind went when I read this proposal.

90

I’m not really worried about that either TBH, or at least if we’ve pissed off enough people that they feel compelled to do that, then that’s probably a signal that we’ve done something wrong?

If we’re that worried about it though, we could just modify that and let people self propose, and have all existing members vote on acceptance, sort of like the old Fellow model or how core dev gets voted on.

2 Likes
91

Regarding the technical scope of this new council, I have seen responses mostly around packaging and distribution (aka builders and package indexes). There have been some mentions of installers, but I think they were often meaning interpreter installers. Pip has obviously been mentioned, as it is in the PyPA, but I think the council should have remit over package installers in general and their possible interoperability.

I also would very much like to see this new council be given the scope of environment management. Both the definition of and the isolation mechanism. The newer package tools that have come up to challenge the status quo seem to most often come with some level of environment management integrated, showing a strong desire for this by the community. Without this scope, I believe the python ecosystem has no chance of ever standardizing on features like dev dependencies, or definitive test environment declaration and test entrypoints.

It is possible that these responsibilities are already obvious to everyone else and it just wasn’t clear enough for me. It is true that the PyPA organization already has projects that do each of these tasks. Only I did not want these to be left out of a formal definition of powers which would leave the counsel unable to pronounce on them. I should say that I also do not think this should be the whole of the council’s power, just in addition to other points already made.

1 Like
92

FWIW, just chiming in from the point of view of one of the architects of the status quo: I think @pradyunsg’s blog post (referenced early in the thread) summarises the problems with the current state of things well, and I agree with @dstufft that it doesn’t make sense to continue trying to push every Python packaging concern through a governance model that was formed to solve 3 primary problems:

  • PyPI modernisation
  • evolution of interoperability standards (especially for pre-built binary package distribution)
  • community direction of PSF funding for Python packaging related projects (this is the PSF’s packaging working group rather than PyPA, but I’d expect the formation of an elected packaging council to at least significantly alter the operation of the working group, if not supersede it completely)

Telling new Python users how to manage their Python projects definitely wasn’t on the list (aside from the “Don’t run ./setup.py install directly, your future self will thank you” directive that was the impetus to get pip bootstrapping into the standard library).

The challenges and opportunities in the Python packaging ecosystem now aren’t the same ones that we were dealing with 8+ years ago, so it’s reasonable to try to update the related governance structures accordingly (including, as @jezdez pointed out, replacing the current direct-to-individual standing delegations from the Python SC with delegations to the proposed council).

From a voting franchise point of view (although @EWDurbin may dislike the idea of supporting more than one routine election of this magnitude), I think serious consideration should be given to aligning the packaging council voter roll with the one for the PSF Board of Directors (although quorum for a valid election would presumably need to be set much lower). The appropriateness of that will depend on the proposed powers of the council, though (we decided it wasn’t appropriate for the Python Steering Council after all).

Short of that broad franchise, my own preference would be for a PSF-style self nomination (as @dstufft suggested). Rather than being a completely separate form, it could potentially be an opt-in aspect of PSF membership (assuming one of the responsibilities of the council is to take over the working group’s role in directing funds received via the PSF).

1 Like
93

Since my post the other day I’ve been considering the voting franchise question, specifically in the context of how authority derived from “the consent of the governed” applies in the context of Python packaging tools.

There are two key primary Python community examples to draw from now, and I think they’re both informative: PSF board of directors elections, and Python Steering Council elections. (there are potentially other relevant examples as well, such as the formation of NumFocus, but I don’t personally know enough about them to use them as precedent).

Summarising massively, the PSF board originally started with a narrow franchise (the group now known as PSF Fellows), which was sufficient when it started, when it was primarily about the CPython core developers having a legal entity to manage trademarks and other intellectual property. It was reformulated to a broad voting franchise when its role started to change to cover more active community engagement & promotion (there was a bit of give-and-take there: the role started changing first, there were some questions about the legitimacy of those changes given the narrow voting franchise for the PSF Board, then the now more expansive PSF structure was formulated at least partially in response to those concerns).

The origins of the Python Steering Council were different - that was about allowing Guido to take a step back and be an influential voice amongst peers, rather than having to be the final decision maker on essentially every language and standard library design decision (or at least find someone he trusted to make that decision on his behalf). For that purpose, the required voting franchise was the folks that were coming to Guido to get design questions resolved anyway, which meant there was a clear alignment between active involvement in the CPython project and the formation of the new steering council. Even with the narrow voting franchise, the Python Steering Council is also well positioned to advise the PSF on how to best support the language development process with funding.

The proposed formation of a Python Packaging Council contains elements of both situations, but also some aspects that are relatively novel, hence the best way forward being unclear.

Drawing from the PSF situation, one of the proposed goals of the new council is to provide legitimate authority to define the default user experience offered to new Python users that obtain Python directly from python.org rather than via a redistributor. That impacts a lot more than just the volunteers contributing to PyPA projects, hence a genuinely broad voting franchise would seem appropriate. Previously we only attempted to claim enough authority over this aspect of things to break distutils out of the standard library, and eliminate setup.py based interactions with the PyPI service (due to genuine technical problems with both aspects of the tooling UX). The packaging user guide was developed to give folks at least some advice on where to start if they didn’t have anyone else providing more specific guidance, but allowing folks to choose a workflow that suited them rather than dictating anything in particular was an intentional goal in most cases.

The parallel with the Python Steering Council is that any new packaging council will rely entirely on the good graces of the contributors to packaging tools to actually get anything done (while the PSF does have some funding available via flagged donations, it’s only enough to target key enabling and transitional projects, it doesn’t cover ongoing maintenance of essential projects). In that context, a council that contributors see as legimitate will be sufficient to have a genuine impact. The broader Python community will still have indirect influence via the PSF Board, rather than directly influencing the makeup of the packaging council itself.

From a personal perspective, I think following either precedent could work out well, and I doubt either would lead to complete disaster (if a broad franchise elects a council that contributing volunteers ignore that wouldn’t be good, especially if it exacerbated problems with contributor burnout, but the tools themselves would keep functioning, and even the narrower franchises being considered would still be a broader group than the folks that were directly involved in establishing the status quo the better part of a decade ago). I also think there are a few viable “middle of the road” options that could work (such as the “PSF members, but with an extra self-nomination step to participate in packaging council elections” concept).

One other aspect to keep in mind is that it’s likely to be easier to broaden the voting franchise later if a narrow franchise turns out not to be as effective as hoped than it would be to narrow the franchise if a broad voting franchise proves ineffective.

3 Likes
94

I’m not clear what the next step is here. I’m aware that @smm is in the process of developing a proposal, but:

  1. Who will make the decision? Will it be the SC, or the community?
  2. Will there only be a single “take it or leave it” proposal, or will there be a choice? I don’t think we need to go as far as the seven core governance models that led to the SC, but is a single proposal enough, given the relative lack of consensus here on a lot of the details?

My assumption is that, like with PEP 609 (the PyPA governance PEP), it’ll be one PEP that the SC will decide on. Can someone (maybe someone from the SC - @brettcannon?) confirm if that’s the plan?

1 Like
95

The SC can make that decision if that makes the most sense (I would argue it does as it take it to a “neutral” 3rd-party), and I doubt the other SC members would have issues with that.

I think that depends on who is up for putting in the effort to create alternative proposals (if any).

1 Like
96

I think the SC being the decision makers here make sense. Ultimately the decision makers aren’t exactly expected to go with what they personally like, but where it seems like the most consensus is, and I think the SC are greatly positioned to provide that.

2 Likes
97

Nit: 4 if you include maturin as the build backend used for Py03 rust native code

98

FWIW, the PyPA is fiscally sponsored by the PSF and has access to funding on its own through donations to the PSF that are earmarked for use via the fiscal sponsorship program. Because of that, I would argue that a packaging council would not have to rely solely on contributors to packaging tools, but may (if the appropriate PEP process is followed) also seek funding to cover maintenance and R&D.

2 Likes
99

@smm What’s the current status of this proposal?

1 Like
100

I know that @smm has reached out to a few folks and gathered their inputs on a draft proposal (I was one of them) – that draft needs a round of revisions before being put up for public discussion.[1]

FWIW, @smm’s funded role on this ended on July 31. I’m not quite sure what the plan is for the draft at this point TBH (she’s the best person to answer whether she’ll be able to follow up as a volunteer) – if it’ll be revised by her, if it’s OK for someone to take it forward, or something else.

I think it’s reasonable for someone else to pick this up, if nothing moves on this over the next 2-3 weeks.

  1. It’s a PEP written by a first time author, so that’s expected. ↩︎

3 Likes