What do you want in Python packaging's future governance model?

Change: Prioritize the approach “standardise existing behaviour then do follow up improvements” instead of “create a new stardard that is not backwards compatible with existing behaviour (or at least hard to make backwards compatible with)”

Effect: avoid or minimise “growing pains” in the ecosystem.

6 Likes

Change: Improve the process for reaching consensus or (at least reaching consent) for new standards/PEPs in a way that volunteers feel motivated to implement the change and avoid standards/PEPs that still feel controversial/sour.

Effect: Motivate the community to work towards a truly shared vision and minimise the risk of something like the one reported in the following thread to happen again (I am highlighting below 2 important comments that I feel summarise the problem):

8 Likes

Change: Protect the decision making process against ballot stuffing.

Currently, every PyPA committer has one vote (in PyPA committer votes). The PyPA is composed of PyPA projects. However, PyPA-as-a-body has no control over who gets added as a committer to a PyPA project.

In other words, the every-vote-is-equal process is susceptible to a project adding way-too-many committers and resulting in over-representation for that project.

14 Likes

Could you please elaborate a bit more on this one.

Done!

1 Like

Change: Explicitly decouple governance from tool recommendations.

Effect: Make it possible for official documentation to recommend projects that aren’t subject to “packaging governance”, and avoid giving preferential treatment to projects that are.

(Note: This is irrelevant if “Govern everything that interacts with PyPI” is adopted, but if that is rejected, this is a possibly less controversial version).

14 Likes

Change: Require re-application for membership from every project.

Effect: Gives every project an opportunity to re-evaluate whether they are willing to agree to the new governance, and gives the governing body a choice over who to accept. Allows the governing body to set the tone of what it means to be a “packaging project” without historical baggage or accidental bias.

22 Likes

Change: Extend the scope of the term “packaging” to include building/publishing standalone applications, on the one hand, and informal “sharing of scripts” on the other.

Effect: Moves away from the current “everything must be a library” model.

20 Likes

Change: Stop requiring governed projects to live in a single GitHub organisation.

Effect: It’d allow us to resolve CI slowness issues during bursty development (due to a shared Actions runner bucket across all projects) and help with ensuring complete access control to a project (eg: automating releases would allow any admin to make the release of a project, which is part of why pip doesn’t use trusted publishers). It does complicate certain other aspects though notably around money via GH Sponsors and communication around what projects are a member project – but we’ll figure something out. :wink:

7 Likes

Change: Establish a formal agreement with the SC and core devs for “power sharing” with regard to packaging tools in the core distribution. This would cover ensurepip, venv, and the Python launcher/CLI, as well as PEPs like the removal of distutils and PEP 582 (local packages directory).

Effect: Allows the packaging community to take a more holistic approach on “how people use Python”, and avoids the community’s current impotence as soon as a proposal needs core or stdlib changes.

18 Likes

Change: Establish a formal agreement with the PSF, to not only have funds available for packaging improvements but also have a team that actively is looking to disseminate opportunities and recruit programmers to carry out the improvements that the community deem necessary.

Effect: Speed up the implementation process for the backlog of fundable improvements, and avoid having necessary changes that never get implemented because of the lack of available programmers.

Update on the difference of PackagingWG:

3 Likes

Could you update this to clarify how this would be different from https://wiki.python.org/psf/PackagingWG/?

1 Like

I believe that the difference lies on the actively … recruit programmers… part. I understand that the existing group has strong concerns for fundraising and how to direct this funds to the projects that have been previously approved.

But at least at a first glance (and maybe I am wrong), it does not seem to be part of the attributions to actively look for hiring people to “cleanup” the backlog.

The way I interpret (please correct me if I am wrong), is that if someone is interested in carrying out a project, they can apply for the job. I would like to flip this relationship into a more proactive approach and have a group (e.g. a project management group in the PSF) that is aware of the status of each fundable improvement and goes after programmers that can be hired to carry out the implementation.

3 Likes

Change: Re-frame governance to avoid altogether the idea of being a “member” of anything, or to imply any sort of “official” status.

Effect: Avoids the suggestion that projects are forced to accept the governance model to gain some sort of “legitimacy”. Allows projects to compete on merit, and not because of details of how they are governed (which should be irrelevant to end users). Allows user consensus on the best tool to be what drives adoption, rather than labels like “PyPA member”. Allows projects to fail to achieve popularity, without needing to be removed from the packaging governance model simply to avoid confusing users.

11 Likes

Can you clarify what this means? Does it mean that being subject to the governance model directly implies that a tool should be considered the “default”? If so, what does that mean for cases where multiple tools (e.g., build backends or workflow tools) want to work under the governance model?

Change: Include end-users (as opposite of “tool writers”) in the decision making process, but in a way that incentivises them to engage and contribute to the PyPA projects/ecosystem.

One example of a possible way to do that[1] is incentivise end-users to organise themselves community-driven thematic “study groups”[2] which would work kind of a counter part to PyPA projects and provide input to the decision making process (e.g. study group on workflow tools, study group on backwards compatibility, study group on binary extensions, etc…). Between other attributions, these groups could:

  • study the challenges in thematic areas, analyse existing solutions and trade-offs, carry out “user research”/“user studies”/surveys and offer the outcomes as input to help guide the PyPA.
  • liaise with PyPA project maintainers to organise community events such as “hackathons” to help towards tools’ improvement with actual PRs.
  • liaise with PyPA project maintainers to foster contribution (e.g. by helping to write “how to contribute guides”, “project architecture/design overviews”, etc…)

Effect: Right now it is not uncommon to hear that the decisions of the PyPA are biased towards “tool writers” instead of end-user experience. In some sense, this is a bit natural, because “tool writers” dedicate volunteer time to change the packaging landscape directly. We can improve this balance in the decision making process by involving more the users, but it would also be nice if this involvement can be somehow “capitalised” towards direct contribution.

Having pro-active end-user lead study groups would also help to minimise any eventual “gate keeping” (or “perceived gate keeping” sentiment) that might appear.


  1. From the top of my mind, brainstorm style. But there might be better approaches. ↩︎

  2. Here I use “study” in the general sense, not only restricted to “user study” or “user research” (but including them). ↩︎

15 Likes

The motivation is hinted at in Thoughts on the Python packaging ecosystem | Pradyun Gedam. The idea is we’d basically be focusing resources on a specific subset of tools as a group (eg: focus volunteer time for development, community communication, fundraising to hire full time devs on the “default” tools etc).

This is in contrast to trying to somehow manage N different backends/workflow tools that’d all be competing (i.e. wanting to be better than the others at $thing) or implementing the same features in different ways (which is a waste of limited volunteer resources we have as an ecosystem) – this is especially relevant in terms of communicating clearly and for doing fundraising. To be clear, different build-backends that solve different problems are compatible with this but not build-backends that solve the same problem with differently flavoured configuration. :slight_smile:

I have more to say but let’s hold off on an extended discussion around this for now.

2 Likes

Change: When creating standards/tools:

  • avoid[1] breaking people’s existing workflow.
  • avoid[1:1] telling people that they should change their workflow because it is no longer supported (or is no longer “the default”, or is no longer “the recommended”).

The following posts from Paul do a much better job in describing the challenge then my summary above:

Right now users that responded the packaging survey seem to want a unified tool, but, as Paul puts in the comments highlighted above, they implicitly assume that an eventual unified tool will cover their workflow. We should refuse the temptation of cutting things (that currently work) out of the scope in order to achieve “ecosystem simplification” (even if the simplification is purely cognitive).

Effect: Avoid backlashes from the community.


  1. I am using “avoid” intentionally because there might be cases which this cannot be avoided ↩︎ ↩︎

5 Likes

Change: Have a governing body whose purview explicitly includes the adoption of big-picture or long-term goals, and explicitly incorporate consideration of those goals into the approval process for individual changes. That is, individual changes are not to be considered or approved based only on their individual technical merits, but also in terms of their alignment with a coherent, long-term vision.

I know that the big picture does factor into the current process, in the sense that people do think about it when writing PEPs, and people who evaluate PEPs also think about it in evaluating them. But a lot of that is implicit, and I think it would be better if this were explicitly incorporated so that consideration of such issues is unambiguously part of what everyone knows and expects to be part of the whole process. For instance, an addendum to the PEP 1 guidelines saying that packaging PEPs should include a section on “long-term progress” or the like, which addresses “how does this PEP move us towards our long-term goals”.

Effect: Increase the chances that changes to Python packaging will, in aggregate, move us towards where we want to be and not just away from where we are.

6 Likes

Change: Instead of casting this governance discussion as a governance discussion, cast it as a standards definition discussion. (Or, split the discussion into at least two pieces, one of which is a standards definition discussion and one of which might still be a governance discussion.)

By “standards definition discussion” I mean one where a considerably reduced scope of powers is under consideration, but where the scope of applicability and relevance is expanded far beyond only the PyPA.

In particular: instead of working to define a body that sets direction for projects; work to define a body that collaborates to define standards for:

  • Distribution of Python
  • Packaging of Python-containing or Python-interfacing code (as PEP517, PEP518, PEP621, and others already have)
  • Creating of self-contained artifacts containing Python (optionally + other) code/executables
  • Compiled extension modules
  • (various others; specifically avoiding an exhaustive list here)

This standards body should allow for membership by any stakeholder in Python distribution and packaging: Conda, conda-forge, Spack, Conan, Nix, CPython, [insert all the Linuxes], shiv, PyInstaller, build, Hatch, Windows, MacOS, [so many projects and types of project I’m leaving out].

It would be a big, sprawling thing, likely requiring multiple internal working groups, each focused on a specific standards/interop topical area. But I see this as unavoidable, because Python distribution and packaging is a big, sprawling, diverse thing.

(This is something of a remix of posts above by Paul Moore (one, two, three, four) and Anderson (one, two), though I’ve been thinking about it since PyCon US this year.)

Effect: Allows standards discussions to evolve and grow beyond the borders of the concerns of the PyPA project membership, including bringing in currently-walled-out users and stakeholders. These discussions could then proceed independently from any PyPA or PyPA+(C)Python governance discussions we may want to continue with.

4 Likes

Change: explicitly require that packaging PEPs are accompanied by a reference implementation whenever possible.

I mean a reference implementation which is or will be published to pypi or incorporated into an existing pypi package like packaging.

Effect: makes discussion more concrete, and allows tool maintainers to pick up and use that implementation if the PEP is accepted.

Motivated by experiences with build (which taught me what a build frontend vs backend is) and packaging (which is now the only way I want to parse version numbers).

3 Likes