Should PyPA opt out of the PEP process?

So I was thinking about this last night and I think it might be time for packaging to have its own separate process. For instance, @pradyunsg I’m sure thinks that saying PyPA committers can sign off and sponsor PEPs is simple enough, but that means I have to keep track of who that is on top of core developers and to make sure people who come in proposing PEPs are proposing them in the proper category (which isn’t even officially a thing in PEPs to begin with so that’s yet another deviation from the current PEP process).

I think if the PyPA is going to function as an independent body it might be time to just shift over and manage their own standards process. So I say let’s get distutils out of the stdlib and just hand everything package-related to the PyPA from that perspective and have a cleaner separation. That way the PyPA can have Python Packaging Standards or something and it can be managed in its own way without everyone trying to force things into a single process which doesn’t quite fit for anyone.

I’d not thought of that – and it’s a very valid concern.

I do think we can use automation to take care of this (get the sponsor email/name, if sponsor is in python/<core-developers> or Track == "Packaging" and sponsor is in pypa/<committers-team>).

If we can solve a problem with a GitHub bot, I’d much prefer that we do that, instead of trying to create and manage an entirely new process. Would that help make PEP editors’ job easier (which I definitely don’t want to make more difficult)?

That said, I’ve argued that we don’t have a significant problem in my previous post. :slight_smile:

Not entirely as we now have a GitHub action that must be kept working no matter what just for this use case. Not that there isn’t any other group asking to use a forked process as there is no other group; it is just the core devs and the PyPA.

Sure, but that’s because you’re asking me to manage the current process to meet your needs. :wink:

Honestly, the more I think about this the more I think the PyPA should be run as a separate group with their own process. You can fork off what python-dev does, but the fact that discussions happen entirely separately and the core devs mostly have no idea what’s going on over here shows that things are already run very separately. And to be perfectly frank, I don’t want people thinking that core devs have anything to do with packaging and instead the PyPA is in charge of that because it causes people to not know who to talk to about what.

I have no problem if the power delegation continues to come from the steering council, but I think at that point everything is devolved into being different. Heck, there’s now a separate summit for packaging at PyCon! I think it’s just time to accept the PyPA has “out-grown” being under python-dev in any way and just needs to start operating on its own officially instead of unofficially as it does now.

I will also say @ncoghlan strongly disagrees with me on this. :grin: I believe he would rather have PEP 609 come up with a clear delineation of who the PyPA would accept as sponsors.

Since this was missed in moving threads, I’ll note here that I don’t think the sponsor requirement is a problem worth solving as of today.

The two “groups” of CPython Core Developers and PyPA Committers are closely linked by the work we do IMO and have a decently sized overlap too.

My earlier response: Closing the loop on PyPA Governance / BDFRN

(This is an additional thing that we’re discussing on top of the existing process, which is intentionally out of scope of the PEP – can one of the moderators please move the discussion on PyPA’s use of PEPs + sponsors discussion into a new topic?)

If the choices are many PyPA members needing sponsor for PEPs vs moving to a separate process entirely, I strongly prefer the former.


To be clear, I don’t think status quo is a problem. The fact that I’d need a sponsor for PEPs that I propose, is not a major concern for me personally.

There’s > 5 core developers who are also PyPA committers. They’re all reasonable people. If that count reduces substantially, we can look into ways to address that problem – I’d prefer that we deal with it when it’s a more real issue.

If we’re at a point where no core developer (regardless of whether they’re a PyPA member) is willing to even sponsor a packaging-related PEP, we probably have a different problem that’s manifesting as no sponsor for said PEP. In the worst cases, we’d still have the SC to escalate to if things go south.

As for folks who are PyPA+CPython core developer, it’s fine that they don’t need a sponsor? They’re already wearing two hats, both of them come with responsibility of keeping the process from not loosing effectiveness and I’m willing to trust them on that front.

I’ll echo this sentiment.

I think that a separate process would also be sub-optimal, if we plan to continue calling PyPA an “authority” (see here) and we should to prevent unnecessary churn like that.

3 Likes

According to the recent changes to PEP 1, it looks like you’ll need to find a way to do this anyways:

Ideally, a core developer sponsor is identified, but non-core sponsors may also be selected with the approval of the Steering Council.

So can’t we just put PyPA committers on this same list?

The main reasons I advocate strongly for PyPA to continue using the PEP process are so that folks outside Python core development and the PyPA can continue to ignore the fact that these are distinct-but-overlapping groups, and so that we don’t get into “Which process should we use?” debates for proposals which affect, and need to be reviewed and approved by, both groups (e.g. the past PEPs to add the ensurepip module, or the likely future PEP to make setuptools the reference implementation of the distutils API and switch it to an ensurepip style bundling model).

I think of it as being similar to the different working groups within IETF: the internal processes may vary a bit between groups, but the general approach and the communication to the outside world via the RFC process is broadly consistent.

There are certainly improvements we could make on that front (e.g. I like the idea of more explicitly separating language standards from packaging standards in the metadata, especially if we did it in a way that let us better distinguish CPython implementation proposals from Python language proposals), but I view those as process maturation issues, where PyPA and the typing SIG are marking a trail that should make it easier for other related-but-separate groups to consider using the PEP process for publishing their interoperability specifications (e.g. PyCA, PyCQA, perhaps a future ratification of the ASGI spec).

Edit: note that when PyPA first started using the PEP process, the packaging PEPs could be identified based on a Discussions-To entry that referred to distutils-sig. Similarly, typing PEPs usually referred to typing-sig. That system doesn’t really work any more, as most PEPs these days are pointing to specific discuss.python.org forum threads, regardless of which interest group is managing them.

1 Like

Up to this point I know all the people who would qualify as a non-core sponsor. Now if people trust me to know who the PyPA committers are then it’s not more work. Otherwise people need to either trust people aren’t lying that the person they listed is a PyPA committer or I need to know who to ask for verification.

I don’t think we’d need make it an ad hoc thing. Instead, I’d suggest adding a list of specific people that are willing and able to serve as packaging PEP sponsors to the standing delegation document in the steering council repo (beyond the folks that are also core developers).

3 Likes

Way back when we originally started discussing formalizing the PyPA governance discussion, I advocated for moving the PyPA standards process out of the PEP process, for basically all the reasons listed here. I believe I even explicitly called out the fact that since the PEP process was targeting the Python language, and in many cases, more specifically the CPython implementation of said language, that there was always going to be a bit of a square peg, round hole scenario. I called out a risk that the Python PEP process would evolve in a direction that made it more difficult for the PyPA projects to continue to use it.

It appears that as the Python governance has shifted, that the PEP process is in fact drifting further away from a process that is wholly workable for the PyPA. While we can continue to try and force it, I think it’s a much saner idea to establish our own processes.

Seems like we’re hitting exactly this roadblock in Draft PEP: Recording the origin of distributions installed from direct URL references.

edit: Note that, in this situation – we need the mentoring that’s part of the PEP Sponsor role. Even in a dedicated PyPA process, we’d still need that part but all PyPA folks would be allowed to be sponsors (vs a small overlapping subset). If folks think that’s the issue here (we have someone who’s PyPA but not CPython core dev who can mentor), that’d be an argument in favor of adopting a dedicated, separated process.

My main concern with a separate process is that as things stand right now, even core developers are quick to jump on the packaging ecosystem, sometimes publicly. This is because python has historically focused on other aspects of the language design and considered packaging as either a secondary aspect or as one that isn’t really integral to the language. The result is that there is a strong separation between packaging and the language itself, and it’s underscored by the request to move the packaging process out of the PEP process.

Having a completely independent process will IMO be a symbolic gesture that says basically: packaging isn’t considered integral to the language and isn’t a consideration in language design.

On the other hand there is a wide array of functionality in the standard library and subsections of the core development team have expertise in different areas of the standard library. So the alternative is to treat packaging as any other subsection of the language with a bunch of experts on the particular topic even if not everyone is interested. Certainly it has a massive impact and I tend to think it would be a mistake to discount it from discussions that impact language design.

6 Likes

I disagree with this. I think if anything it’s a nice “break with the past” that indicates that Python packaging is becoming such a priority that we can no longer piggy-back on the PEP process and we as an organization take specifications and communication seriously.

I also think it’s useful for there to be some distance between core dev and PyPA because core dev is focused on CPython, whereas PyPA has to serve the needs of everyone, including projects like PyPy and Beeware, which have their own packaging challenges.

I’d prefer to see more distance between core dev and CPython, frankly.

The PEP process is predominantly about Python language and ecosystem. As the reference implementation, CPython is the baseline for “can it be implemented” as well as a required target, but the language design applies to all implementations. As such, I like having the packaging PEPs be in there, even if the SC establishes an alternative review and approval process for packaging PEPs (which they can totally do, as that’s basically the only power our governance model gives them :wink:)

2 Likes

Yep, I agree with Steve - if the PEP process isn’t working for PyPA, it’s the PEP process that needs to change, not PyPA.

The requests I’m seeing in this thread are:

  • a way to clearly subdivide “Standards Track” PEPs based on their target scope (e.g. “Python language”, “CPython implementation”, “Python packaging”)
  • having an easy to maintain way for PyPA to specify who is willing and able to sponsor packaging PEPs

(I know the SC having to ratify the standing delegations when they change irks some folks, but that’s been purely a formality to date, and it seems unlikely to become a problem in the future)

2 Likes

I agree with Nick regarding the requests in this thread, for changes to the PEP process.


Looking at how this discussion has evolved till now, it probably makes sense to escalate this issue to the SC, trusting them to make the judgement call on how “broad” the PEP process should be.

I’ll file an issue on the SC issue tracker, requesting the SC for a decision/inputs, in early Jan 2020 (i.e. after the year-end vacations), assuming everyone here is OK with me doing this (holler if you aren’t) and there’s no active discussion here about this, at that time.

Please be VERY clear about what you’re asking about/for. We have no idea who will be on the council come January and so there may be no people on there with any context as to this entire discussion. So please have agreement here on what is going to be asked before making the request, else it’s just going to be a bunch of back-and-forth and questions as to why anything is being asked to begin with if the ask wasn’t agreed upon here already.

Yep, I’ll definitely run the exact request (and likely text) in the issue across this topic, before filing that issue.

My calendar notification about moving this forward, popped up a few days ago.

I’m still going through all the discussions related to this, to try to summarize the situation (and what all we’ve discussed). I’m hoping that it’ll better surface the concerns and trade-offs that we’re dealing with here. I’ll be posting that here once that’s done, for further discussion.


As per my current understanding, there are “smaller” questions here, but the main question we want to ask the SC is:

Should related-but-separate groups within the Python community (like the PyPA) use the PEP process for publishing their interoperability specifications?

(We should definitely provide context about why we’re asking this, which I’m hoping the aforementioned summary+discussion would help provide.)

  • Is this not the main question we want to ask the SC?
  • Do we want to ask for something else too?
  • Does someone else have a better phrasing for this question?