Draft PEP: Python Packaging Governance

[dstufft (Donald Stufft)]

@pf_moore

The issue is simply that there is no one thing to “own”. The Python Steering Council owns CPython, which is a single thing (although even there, the SC doesn’t have direct override authority, they can say “CPython will do X” but the core devs still need to implement that). I can’t think of any other equivalent group to the packaging council. There’s no “Python Web Council”, just project specific groups like the Django board. There’s no “Python ML/AI Council”. There’s no board with overall authority for “scientific Python”.

I don’t think it’s an entirely new thing in the software world, and I don’t even think that it’s a new thing in the Python space.

Yes, the SC primarily officially owns CPython, but that ownership gives them tons of power that allows them to apply requirements to other related projects. For instance, mypy isn’t owned by the SC, but the fact that the SC owns the typing PEPs through their ownership of CPython means that there’s weight behind the SC decisions.

On the rust side, they have the “devtools team”, which is broken up into smaller teams like the “Cargo team”, the “clippy team”, etc. So the individual projects are owned by the overall dev tools team (which is itself owned by the rust project), and they’ve all agreed to use their RFC process for everything, not just standards but pretty much all major decisions.

@pf_moore

Much as I have a personal dislike of conda, I think it’s entirely reasonable for the council to decide that conda is a key component of the packaging toolchain (for certain classes of use). So how would that square with the mandate to “Own… the Python Packaging… Toolchain”?

So there’s a few things about this, and I don’t have a perfect answer, but a few random thoughts:

• I envision this as a mission not a mandate, if the council decides that some particular project should be part of the default tooling and they want to delegate that ownership… that’s probably okay?
• I think that there is benefit to this council owning the experience, but it’s possible the ownership aspect is too sticky, and we should just drop that part. The question then becomes what can they do? Do they own ensurepip/the cpython experience? Do they own packaging.python.org?

@pf_moore

Speaking as a pip maintainer, I doubt we’ll get a choice (pip will be expected to be under the council, to an extent that I don’t think would make it credible for us to say no) but I honestly see no benefit to us from being owned by the council. And yet, it’s not like the council could choose not to make pip the default installer at this point.

I assume PyPI / pip would need to be the “default” for now anyways, it would be up to the council what to do from there. If PyPI/pip wanted to not be owned by the council, I think both projects get that choice, but if either say no, it effectively means that the idea of ownership within the council is DOA, and we just simply can’t include that.

@daniele

I though that what was being discussed was a way to get all the players in the field to agree on how to move the packaging standards forward, not to create a body that will “own” solutions. I don’t see any push back from authors of maintainers of packaging tools in implementing the PyPA standards, thus I don’t see the reason of creating an entity whose constitution seems designed to force these standards on reluctant maintainers.

I think that this is a mischaracterization or misunderstanding of what is being proposed and the reasons for these discussions.

We already have a process for defining standards and it works reasonably well. If the solution to the problems we’re having were just to define another standard, then this discussion wouldn’t exist.

The problem we’re running into is that the only thing we have a process for decision making with is standards, so we’re forced to try and turn everything into a standard, even if it doesn’t make sense for it to be, and anything that can’t be hammered into a standards shape we just sit here and endlessly discuss with no forward movement. Thus the governing body is designed to provide a means to make those much larger or vaguer decisions.

I also think that the idea of code ownership is a relatively small part of the overall idea, and it could be done without it. However, I do think we lose something without it. The key thing is we effectively already have this in the current system, we just have it implicitly. PyPI and pip tend to just implicitly act as the enforcement mechanism, which in large part stems from the fact that Paul and Myself are the default decision makers for those PEPs, and also can ensure that . The code ownership aspect of this idea, is, in my opinion simply formalizing that existing implicit relationship.

I do not think that ownership has anything to do with trying to decide standards through implementation defined standards vs actual standards. Nor do I think that it means that we’re repeating the distutils saga (a and I don’t think the distutils problems had anything to do with ownership). Nor do I think that anything in this means that there will no longer be consensus based decision making.

The ultimate question becomes, when we can’t come to a consensus, what do we do? Right now the answer is that we frustrate everyone involved by long, never ending discussions that will never go anywhere where everyone agrees that something should happen, but nobody agrees on anything else.

[brettcannon]

@dstufft

Of course we would need an initial list of voters to seed the member list with, and I would say we can just use the existing list of PyPA committers for that, maybe with a period of time before the first election for non PyPA committers to get a chance to ask to be members and get voted on.

You can also lean on the SC if we can’t agree on who the initial voters are (I’ve already asked and the other SC members agreed to help out in this process in any way if asked). In general, though, people seem fine with this initial voting group proposal based on all the discussions on this topic.

@dstufft

I think that there is benefit to this council owning the experience, but it’s possible the ownership aspect is too sticky, and we should just drop that part. The question then becomes what can they do? Do they own ensurepip/the cpython experience? Do they own packaging.python.org?

Toss in packaging so there’s a package providing a reference implementation of the standards and I think that’s a good starting point (assuming Pradyun is also okay with this, of course ). If this is about standards and default tooling, you can’t beat accepting/rejecting the PEPs, the website that records those standards and the (hopefully) go-to place for information, and the code that triggers probably the most installs of packaging tools. This keeps things agnostic while not deluding ourselves that recommendations of tools will be made based on which ones are following the standards.

@dstufft

We already have a process for defining standards and it works reasonably well. If the solution to the problems we’re having were just to define another standard, then this discussion wouldn’t exist.

The problem we’re running into is that the only thing we have a process for decision making with is standards, so we’re forced to try and turn everything into a standard, even if it doesn’t make sense for it to be, and anything that can’t be hammered into a standards shape we just sit here and endlessly discuss with no forward movement. Thus the governing body is designed to provide a means to make those much larger or vaguer decisions.

I think the other issue we are having is @pf_moore is one person. Being a dictator is exhausting and it’s hard to make any sort of visionary plan when you’re just trying to tread water and not take flak from all sides for a decision that you’ve been asked to make. Having a group of people to work towards that sort of thing not only helps take the pressure off (a “council” made the decision, not a “person”), but also hopefully lends some weight to any guidance when multiple people who were chosen to make a visionary plan come to an agreement.

[pf_moore (Paul Moore)]

@brettcannon

I think the other issue we are having is @pf_moore is one person. Being a dictator is exhausting and it’s hard to make any sort of visionary plan when you’re just trying to tread water and not take flak from all sides for a decision that you’ve been asked to make.

I don’t think that’s entirely accurate. In order to decide on proposals at all, you have to have a sense of what your long term view is. So I do have a “visionary plan”, at least in the sense of how I want to see packaging develop and grow. I don’t consider it hard to have such a plan - what’s hard is sticking to that plan in spite of others disagreeing (often strongly, and sometimes aggressively) with your decisions. And trying to persuade others to follow your plan (which I’ve very definitely avoided - but not because I haven’t had the time/energy, just because it’s outside of the PEP delegate remit and I don’t want to unilaterally take on something like that).

I’ve generally favoured consensus approval for PEPs for the very simple reason that my vision for packaging is one of multiple, co-operating, and mutually supportive tools. Imposing standards without consensus is in direct contradiction to that vision. But that doesn’t mean I won’t take a stance if I have to. And those decisions are often the ones that cause the most disruption in the community, reinforcing my view that making decisions without community involvement - whether informed by some “visionary plan” or not - is not a workable long term strategy.

I don’t agree with the people who want radical change^[1], and my big concern is that people see the idea of a council as some sort of vehicle for just that. Talking about owning the “official toolset” is an obvious example of this - it’s a significant change in how packaging projects are organised, and it feels like it’s being proposed as “change for change’s sake” rather than because it actually solves a real problem. I don’t think it’s somehow because no-one has ever formulated a vision of “all the packaging tools under one roof”, because that model exists - it’s basically conda - and a significant number of users don’t like it. We can debate about details as much as we like, but organisationally, the “central organisation owns the full integrated toolset” model is what conda offers^[2]. And while it’s great for some, it’s not for everyone - and as far as I can tell, never will be.

Assuming that we do agree to switch to some sort of packaging council, and that people in general have not been unhappy with how I have handled the PEP delegate role, I would strongly request that we start with a council that has minimal powers - basically just the “approve packaging PEPs” responsibility that @dstufft and I currently have - and let the council decide how and when to expand their remit^[3]. One thing that I do agree with @brettcannon over is that gaining any sort of authority in the packaging community is time-consuming and a lot of effort, and no-one who hasn’t actually done the role can honestly have a good view of what they can and can’t achieve. So let’s give the council the chance to work that out for themselves. I can say with some certainty that approving PEPs (at least the way I do it, which means actively participating in essentially every packaging thread relating to standards) is a fair amount of effort^[4], and having that as a starting point would be plenty for a new council to do while they find their own balance and formulate their views. They can then make an informed choice on how they want to expand the role and what goals they can reasonably achieve.

---

1. I favour gradual, incremental improvement over time. ↩︎

2. Please excuse the fact that I don’t actually know how conda is organised, but this is how it feels to an outsider, and it’s how conda is typically presented when discussing tools and trade-offs. ↩︎

3. Before anyone says “but expanding the council’s remit is a whole new governance debate”, I’ll note that if the council can’t get the community to approve a scope increase that they feel is important to the future of Python packaging, they really don’t have much hope of doing anything really hard, like agreeing a core toolset! ↩︎

4. albeit with significant peaks and troughs ↩︎

[BrenBarn (Brendan Barnwell)]

@dstufft

I think that there is benefit to this council owning the experience, but it’s possible the ownership aspect is too sticky, and we should just drop that part. The question then becomes what can they do? Do they own ensurepip/the cpython experience? Do they own packaging.python.org?

I think that if the council had authority over ensurepip-type stuff (i.e., has the ability to include at least a bootstrapper to its toolchain in the stdlib) and the docs, that would be a good concrete start.

@pf_moore

I can’t think of any other equivalent group to the packaging council. There’s no “Python Web Council”, just project specific groups like the Django board. There’s no “Python ML/AI Council”. There’s no board with overall authority for “scientific Python”.

That is true, but I think packaging is far more fundamental than those things. All of those are essentially application domains where Python is used. But they, and virtually all other such domains, rely on packaging. I see packaging as more akin to the something like the sys module than to something like ML/AI or web apps; it’s so core that it’s almost impossible to use Python without it. I think that’s also why packaging issues tend to crop up so widely across the Python user base.

This relates to something @dstufft alluded to in his post, and that’s been mentioned a few times in these various threads, which is that, apparently, the core dev team “isn’t interested in” handling packaging or “doesn’t want to think about” packaging or something along those lines. It seems to me that as long as this remains the case, there’s always going to be a certain tension in the air, because of the large divergence between how central packaging is to the user experience and how peripheral it is in the Python governance/development model. But, as I said, I think if a packaging council had a purview that clearly included the ability to bootstrap packaging tools (with an ensurepip-type thing) and to be the official source for documentation about packaging matters, that would be a step forward.