What do you want in Python packaging's future governance model?

[abravalheri (Anderson Bravalheri)]

I believe that the difference lies on the actively … recruit programmers… part. I understand that the existing group has strong concerns for fundraising and how to direct this funds to the projects that have been previously approved.

But at least at a first glance (and maybe I am wrong), it does not seem to be part of the attributions to actively look for hiring people to “cleanup” the backlog.

The way I interpret (please correct me if I am wrong), is that if someone is interested in carrying out a project, they can apply for the job. I would like to flip this relationship into a more proactive approach and have a group (e.g. a project management group in the PSF) that is aware of the status of each fundable improvement and goes after programmers that can be hired to carry out the implementation.

[pf_moore (Paul Moore)]

Change: Re-frame governance to avoid altogether the idea of being a “member” of anything, or to imply any sort of “official” status.

Effect: Avoids the suggestion that projects are forced to accept the governance model to gain some sort of “legitimacy”. Allows projects to compete on merit, and not because of details of how they are governed (which should be irrelevant to end users). Allows user consensus on the best tool to be what drives adoption, rather than labels like “PyPA member”. Allows projects to fail to achieve popularity, without needing to be removed from the packaging governance model simply to avoid confusing users.

[pf_moore (Paul Moore)]

@pradyunsg

Move away from an open tent model, focusing on specifically a “default” experience.

Can you clarify what this means? Does it mean that being subject to the governance model directly implies that a tool should be considered the “default”? If so, what does that mean for cases where multiple tools (e.g., build backends or workflow tools) want to work under the governance model?

[abravalheri (Anderson Bravalheri)]

Change: Include end-users (as opposite of “tool writers”) in the decision making process, but in a way that incentivises them to engage and contribute to the PyPA projects/ecosystem.

One example of a possible way to do that^[1] is incentivise end-users to organise themselves community-driven thematic “study groups”^[2] which would work kind of a counter part to PyPA projects and provide input to the decision making process (e.g. study group on workflow tools, study group on backwards compatibility, study group on binary extensions, etc…). Between other attributions, these groups could:

• study the challenges in thematic areas, analyse existing solutions and trade-offs, carry out “user research”/“user studies”/surveys and offer the outcomes as input to help guide the PyPA.
• liaise with PyPA project maintainers to organise community events such as “hackathons” to help towards tools’ improvement with actual PRs.
• liaise with PyPA project maintainers to foster contribution (e.g. by helping to write “how to contribute guides”, “project architecture/design overviews”, etc…)

Effect: Right now it is not uncommon to hear that the decisions of the PyPA are biased towards “tool writers” instead of end-user experience. In some sense, this is a bit natural, because “tool writers” dedicate volunteer time to change the packaging landscape directly. We can improve this balance in the decision making process by involving more the users, but it would also be nice if this involvement can be somehow “capitalised” towards direct contribution.

Having pro-active end-user lead study groups would also help to minimise any eventual “gate keeping” (or “perceived gate keeping” sentiment) that might appear.

---

1. From the top of my mind, brainstorm style. But there might be better approaches. ↩︎

2. Here I use “study” in the general sense, not only restricted to “user study” or “user research” (but including them). ↩︎

[pradyunsg (Pradyun Gedam)]

@pf_moore

Can you clarify what this means? Does it mean that being subject to the governance model directly implies that a tool should be considered the “default”? If so, what does that mean for cases where multiple tools (e.g., build backends or workflow tools) want to work under the governance model?

The motivation is hinted at in Thoughts on the Python packaging ecosystem | Pradyun Gedam. The idea is we’d basically be focusing resources on a specific subset of tools as a group (eg: focus volunteer time for development, community communication, fundraising to hire full time devs on the “default” tools etc).

This is in contrast to trying to somehow manage N different backends/workflow tools that’d all be competing (i.e. wanting to be better than the others at $thing) or implementing the same features in different ways (which is a waste of limited volunteer resources we have as an ecosystem) – this is especially relevant in terms of communicating clearly and for doing fundraising. To be clear, different build-backends that solve different problems are compatible with this but not build-backends that solve the same problem with differently flavoured configuration.

I have more to say but let’s hold off on an extended discussion around this for now.

[abravalheri (Anderson Bravalheri)]

Change: When creating standards/tools:

• avoid^[1] breaking people’s existing workflow.
• avoid^[1:1] telling people that they should change their workflow because it is no longer supported (or is no longer “the default”, or is no longer “the recommended”).

The following posts from Paul do a much better job in describing the challenge then my summary above:

Python Packaging Strategy Discussion - Part 1

(I’m a maintainer of pip). I agree wholeheartedly with this. And I’d go further and say that for such a tool to be successful, it will be at least as important to decide what workflows we won’t support, as what we will. The lesson we have learned from pip is that if you try to support everything that people come up with, you’ll end up in a mess. And of course, the big problem here will be that many of the respondents wanting a unified tool will have been making an implicit assumption that thei…

Python Packaging Strategy Discussion - Part 1

That’s not what I was saying. I was specifically talking about support of workflows, not of use cases. So, in particular, I am absolutely not in favour of declaring any groups of users or parts of the ecosystem as “out of scope”. I do expect that we may need to ask users to do things in certain ways in order to address their use cases, but I don’t see why that should be unacceptable - there needs to be compromise on both sides if we’re to create a single solution that works for everyone (which …

Right now users that responded the packaging survey seem to want a unified tool, but, as Paul puts in the comments highlighted above, they implicitly assume that an eventual unified tool will cover their workflow. We should refuse the temptation of cutting things (that currently work) out of the scope in order to achieve “ecosystem simplification” (even if the simplification is purely cognitive).

Effect: Avoid backlashes from the community.

---

1. I am using “avoid” intentionally because there might be cases which this cannot be avoided ↩︎ ↩︎

[BrenBarn (Brendan Barnwell)]

Change: Have a governing body whose purview explicitly includes the adoption of big-picture or long-term goals, and explicitly incorporate consideration of those goals into the approval process for individual changes. That is, individual changes are not to be considered or approved based only on their individual technical merits, but also in terms of their alignment with a coherent, long-term vision.

I know that the big picture does factor into the current process, in the sense that people do think about it when writing PEPs, and people who evaluate PEPs also think about it in evaluating them. But a lot of that is implicit, and I think it would be better if this were explicitly incorporated so that consideration of such issues is unambiguously part of what everyone knows and expects to be part of the whole process. For instance, an addendum to the PEP 1 guidelines saying that packaging PEPs should include a section on “long-term progress” or the like, which addresses “how does this PEP move us towards our long-term goals”.

Effect: Increase the chances that changes to Python packaging will, in aggregate, move us towards where we want to be and not just away from where we are.

[btskinn (Brian Skinn)]

Change: Instead of casting this governance discussion as a governance discussion, cast it as a standards definition discussion. (Or, split the discussion into at least two pieces, one of which is a standards definition discussion and one of which might still be a governance discussion.)

By “standards definition discussion” I mean one where a considerably reduced scope of powers is under consideration, but where the scope of applicability and relevance is expanded far beyond only the PyPA.

In particular: instead of working to define a body that sets direction for projects; work to define a body that collaborates to define standards for:

• Distribution of Python
• Packaging of Python-containing or Python-interfacing code (as PEP517, PEP518, PEP621, and others already have)
• Creating of self-contained artifacts containing Python (optionally + other) code/executables
• Compiled extension modules
• (various others; specifically avoiding an exhaustive list here)

This standards body should allow for membership by any stakeholder in Python distribution and packaging: Conda, conda-forge, Spack, Conan, Nix, CPython, [insert all the Linuxes], shiv, PyInstaller, build, Hatch, Windows, MacOS, [so many projects and types of project I’m leaving out].

It would be a big, sprawling thing, likely requiring multiple internal working groups, each focused on a specific standards/interop topical area. But I see this as unavoidable, because Python distribution and packaging is a big, sprawling, diverse thing.

(This is something of a remix of posts above by Paul Moore (one, two, three, four) and Anderson (one, two), though I’ve been thinking about it since PyCon US this year.)

Effect: Allows standards discussions to evolve and grow beyond the borders of the concerns of the PyPA project membership, including bringing in currently-walled-out users and stakeholders. These discussions could then proceed independently from any PyPA or PyPA+(C)Python governance discussions we may want to continue with.

[sirosen (Stephen Rosen)]

Change: explicitly require that packaging PEPs are accompanied by a reference implementation whenever possible.

I mean a reference implementation which is or will be published to pypi or incorporated into an existing pypi package like packaging.

Effect: makes discussion more concrete, and allows tool maintainers to pick up and use that implementation if the PEP is accepted.

Motivated by experiences with build (which taught me what a build frontend vs backend is) and packaging (which is now the only way I want to parse version numbers).