Should PyPA opt out of the PEP process?

(This is an additional thing that we’re discussing on top of the existing process, which is intentionally out of scope of the PEP – can one of the moderators please move the discussion on PyPA’s use of PEPs + sponsors discussion into a new topic?)

If the choices are many PyPA members needing sponsor for PEPs vs moving to a separate process entirely, I strongly prefer the former.


To be clear, I don’t think status quo is a problem. The fact that I’d need a sponsor for PEPs that I propose, is not a major concern for me personally.

There’s > 5 core developers who are also PyPA committers. They’re all reasonable people. If that count reduces substantially, we can look into ways to address that problem – I’d prefer that we deal with it when it’s a more real issue.

If we’re at a point where no core developer (regardless of whether they’re a PyPA member) is willing to even sponsor a packaging-related PEP, we probably have a different problem that’s manifesting as no sponsor for said PEP. In the worst cases, we’d still have the SC to escalate to if things go south.

As for folks who are PyPA+CPython core developer, it’s fine that they don’t need a sponsor? They’re already wearing two hats, both of them come with responsibility of keeping the process from not loosing effectiveness and I’m willing to trust them on that front.

I’ll echo this sentiment.

I think that a separate process would also be sub-optimal, if we plan to continue calling PyPA an “authority” (see here) and we should to prevent unnecessary churn like that.

3 Likes

According to the recent changes to PEP 1, it looks like you’ll need to find a way to do this anyways:

Ideally, a core developer sponsor is identified, but non-core sponsors may also be selected with the approval of the Steering Council.

So can’t we just put PyPA committers on this same list?

The main reasons I advocate strongly for PyPA to continue using the PEP process are so that folks outside Python core development and the PyPA can continue to ignore the fact that these are distinct-but-overlapping groups, and so that we don’t get into “Which process should we use?” debates for proposals which affect, and need to be reviewed and approved by, both groups (e.g. the past PEPs to add the ensurepip module, or the likely future PEP to make setuptools the reference implementation of the distutils API and switch it to an ensurepip style bundling model).

I think of it as being similar to the different working groups within IETF: the internal processes may vary a bit between groups, but the general approach and the communication to the outside world via the RFC process is broadly consistent.

There are certainly improvements we could make on that front (e.g. I like the idea of more explicitly separating language standards from packaging standards in the metadata, especially if we did it in a way that let us better distinguish CPython implementation proposals from Python language proposals), but I view those as process maturation issues, where PyPA and the typing SIG are marking a trail that should make it easier for other related-but-separate groups to consider using the PEP process for publishing their interoperability specifications (e.g. PyCA, PyCQA, perhaps a future ratification of the ASGI spec).

Edit: note that when PyPA first started using the PEP process, the packaging PEPs could be identified based on a Discussions-To entry that referred to distutils-sig. Similarly, typing PEPs usually referred to typing-sig. That system doesn’t really work any more, as most PEPs these days are pointing to specific discuss.python.org forum threads, regardless of which interest group is managing them.

1 Like

Up to this point I know all the people who would qualify as a non-core sponsor. Now if people trust me to know who the PyPA committers are then it’s not more work. Otherwise people need to either trust people aren’t lying that the person they listed is a PyPA committer or I need to know who to ask for verification.

I don’t think we’d need make it an ad hoc thing. Instead, I’d suggest adding a list of specific people that are willing and able to serve as packaging PEP sponsors to the standing delegation document in the steering council repo (beyond the folks that are also core developers).

3 Likes

Way back when we originally started discussing formalizing the PyPA governance discussion, I advocated for moving the PyPA standards process out of the PEP process, for basically all the reasons listed here. I believe I even explicitly called out the fact that since the PEP process was targeting the Python language, and in many cases, more specifically the CPython implementation of said language, that there was always going to be a bit of a square peg, round hole scenario. I called out a risk that the Python PEP process would evolve in a direction that made it more difficult for the PyPA projects to continue to use it.

It appears that as the Python governance has shifted, that the PEP process is in fact drifting further away from a process that is wholly workable for the PyPA. While we can continue to try and force it, I think it’s a much saner idea to establish our own processes.

Seems like we’re hitting exactly this roadblock in Draft PEP: Recording the origin of distributions installed from direct URL references.

edit: Note that, in this situation – we need the mentoring that’s part of the PEP Sponsor role. Even in a dedicated PyPA process, we’d still need that part but all PyPA folks would be allowed to be sponsors (vs a small overlapping subset). If folks think that’s the issue here (we have someone who’s PyPA but not CPython core dev who can mentor), that’d be an argument in favor of adopting a dedicated, separated process.

My main concern with a separate process is that as things stand right now, even core developers are quick to jump on the packaging ecosystem, sometimes publicly. This is because python has historically focused on other aspects of the language design and considered packaging as either a secondary aspect or as one that isn’t really integral to the language. The result is that there is a strong separation between packaging and the language itself, and it’s underscored by the request to move the packaging process out of the PEP process.

Having a completely independent process will IMO be a symbolic gesture that says basically: packaging isn’t considered integral to the language and isn’t a consideration in language design.

On the other hand there is a wide array of functionality in the standard library and subsections of the core development team have expertise in different areas of the standard library. So the alternative is to treat packaging as any other subsection of the language with a bunch of experts on the particular topic even if not everyone is interested. Certainly it has a massive impact and I tend to think it would be a mistake to discount it from discussions that impact language design.

6 Likes

I disagree with this. I think if anything it’s a nice “break with the past” that indicates that Python packaging is becoming such a priority that we can no longer piggy-back on the PEP process and we as an organization take specifications and communication seriously.

I also think it’s useful for there to be some distance between core dev and PyPA because core dev is focused on CPython, whereas PyPA has to serve the needs of everyone, including projects like PyPy and Beeware, which have their own packaging challenges.

I’d prefer to see more distance between core dev and CPython, frankly.

The PEP process is predominantly about Python language and ecosystem. As the reference implementation, CPython is the baseline for “can it be implemented” as well as a required target, but the language design applies to all implementations. As such, I like having the packaging PEPs be in there, even if the SC establishes an alternative review and approval process for packaging PEPs (which they can totally do, as that’s basically the only power our governance model gives them :wink:)

2 Likes

Yep, I agree with Steve - if the PEP process isn’t working for PyPA, it’s the PEP process that needs to change, not PyPA.

The requests I’m seeing in this thread are:

  • a way to clearly subdivide “Standards Track” PEPs based on their target scope (e.g. “Python language”, “CPython implementation”, “Python packaging”)
  • having an easy to maintain way for PyPA to specify who is willing and able to sponsor packaging PEPs

(I know the SC having to ratify the standing delegations when they change irks some folks, but that’s been purely a formality to date, and it seems unlikely to become a problem in the future)

2 Likes

I agree with Nick regarding the requests in this thread, for changes to the PEP process.


Looking at how this discussion has evolved till now, it probably makes sense to escalate this issue to the SC, trusting them to make the judgement call on how “broad” the PEP process should be.

I’ll file an issue on the SC issue tracker, requesting the SC for a decision/inputs, in early Jan 2020 (i.e. after the year-end vacations), assuming everyone here is OK with me doing this (holler if you aren’t) and there’s no active discussion here about this, at that time.

Please be VERY clear about what you’re asking about/for. We have no idea who will be on the council come January and so there may be no people on there with any context as to this entire discussion. So please have agreement here on what is going to be asked before making the request, else it’s just going to be a bunch of back-and-forth and questions as to why anything is being asked to begin with if the ask wasn’t agreed upon here already.

Yep, I’ll definitely run the exact request (and likely text) in the issue across this topic, before filing that issue.

My calendar notification about moving this forward, popped up a few days ago.

I’m still going through all the discussions related to this, to try to summarize the situation (and what all we’ve discussed). I’m hoping that it’ll better surface the concerns and trade-offs that we’re dealing with here. I’ll be posting that here once that’s done, for further discussion.


As per my current understanding, there are “smaller” questions here, but the main question we want to ask the SC is:

Should related-but-separate groups within the Python community (like the PyPA) use the PEP process for publishing their interoperability specifications?

(We should definitely provide context about why we’re asking this, which I’m hoping the aforementioned summary+discussion would help provide.)

  • Is this not the main question we want to ask the SC?
  • Do we want to ask for something else too?
  • Does someone else have a better phrasing for this question?

I’m not sure we want to ask the SC whether we “should” use the PEP process. How would they know if it suited our needs? IMO, the only question along those lines that is appropriate would be whether the SC has any objection to PyPA using the PEP process.

If they do, we need to switch to a process of our own (which we’d need to define). If they don’t, then there’s no need to change, but if we decide that the PEP process doesn’t suit our requirements, we can still do so.

A possible supplementary question would be whether the SC want us to use the PEP process - or to put it another way, whether they want to have some level of control over what we decide. My feeling is that they are happy to leave us to make our own decisions, but that may be because the “standing BDFL delegates” (@dstufft and myself, currently) along with the use of the PEP process, are sufficient for them.

1 Like

Given packaging is such a critical part of most people’s introduction to Python these days, I’d expect this to be the important question and that the SC will likely want the PEP process to be used for anything purporting to be a “community standard” (not simply a feature or design document for a particular tool). That’s certainly what I’d prefer, if only so outside observers have an easier time figuring out what’s officially endorsed and what’s not.

4 Likes

If you think the following is what we want to ask the SC, use the like button below to indicate that. Otherwise, please write a comment describing why you think this is not. :slight_smile:

Does the SC have any objections toward the use of the PEP process by related-but-separate groups within the Python community (like the PyPA) for publishing their interoperability specifications? If there are no objections, does the SC feel that it is preferable that these groups use the PEP process instead of a separate process?

1 Like

It’s missing context, specifically, a clear definition of the PEP process you’re referring to. Also whether the plan is to clone the process and publish everything on a separate site, or if you intend to submit packaging PEPs to the Python SC for approval (and allow them to decide what criteria to use to approve them).

You probably also need to clearly define the scope of (what would currently be) PEPs you are going to take elsewhere. Changes that affect pip may impact the CPython distribution, and so probably need sign off from a CPython “representative” if you don’t want pushback when trying to update the version included with CPython (because someone thinks it’s a breaking change that isn’t appropriate for the 3.8.x or whatever release), while changes that only affect PyPI likely don’t - who decides for each not-a-PEP whether it’s in or out?

Providing that criteria or decision making process (or an intent to develop such a process) along with the question will give the SC something to actually respond to.

1 Like

@steve.dower: I agree, about the need for additional context.

My thought process was that this question would be asked when presenting the SC with PEP 609. The context on would take the form of a summary of the entire “PyPA process formalization” discussion and would be provided to the SC along with PEP 609 (either as part of the PEP, or along with the issue on python/steering-council for that PEP). That summary would provide the context on why we’re asking this question, what points/concerns have been raised about the PyPA using/not using the PEP process and what “position” PEP 609 represents.


I’d volunteered to do the task of summarizing this discussion but I’ve not been able to make time to do so properly over the past few months. Further, this is not a priority item for me right now and I won’t have the time in the coming days to pick this back up. I welcome others who are interested in moving this discussion forward to write down something to provide the relevant context on this topic for the SC.


I think what’s needed here is a summary of this overall “PyPA process formalization” discussion, which is something that’s been discussed in two discuss.python.org topics (this topic and https://discuss.python.org/t/1776) as well as and two pypa-committers mailing list threads (1, and 2). I’ll try to find my notes from back when I was working on this, but at this point, I’m not even sure how/where I’d made/kept them. If I do find them, I’ll share them here.