The existing PyPA governance rules state that any change to a packaging spec (other than textual clarifications with no interoperability implications) must go through the PEP process, to agree the change that will be made to the spec.
Recently, there have been a number of relatively small changes, or proposals for changes, to the specs - this discussion and this one about the metadata format (also referenced here), one about compatibility tags, and a few others.
I’m ambivalent about whether changes like these should need the PEP process. On the one hand, they are typically relatively small and typically uncontroversial changes, but on the other hand, the PEP process is our only formal way of making sure there’s a public notification of a change.
I’m tending to stick to the letter of the law at the moment, and ask for a PEP when there’s any doubt, but I feel that if we do use the PEP process for smaller changes, we could become a nuisance, using more than our fair share of PEPs and making the process over-bureaucratic for people wanting to help out.
As per the PyPA governance rules PEP 609 I’d like to have a discussion on whether we’d be OK with smaller packaging spec changes not needing the PEP process, and what the process would be for deciding on exemptions (I’d be fine with “PEP delegate decides”, but I’d also be OK with another process if people prefer). We should also agree on how we ensure that interested parties get notified, if we move away from the PEP process.
From a perspective of someone who had to learn most of this stuff relatively recently: is there some concrete single source of truth & communication for all of this?
It is already quite difficult to keep track of all of the different PEPs, their updates, their new versions, all of the recommendations, etc.
E.g. there’s no mention of PEP-639 in PEP-566 - one has to actively look for a new version of metadata to realize there is one.
The sources for information seem quite fragmented and its hard to wrap one’s head around what to look for and where. The list of PEPs one needs to read to e.g. create a build system is a lengthy one, it would be really nice if there was one list with all of the sources & specs.
I’ve seen this already, but to me it looked like it isn’t complete. E.g. only metadata 2.1 is linked up in Core Metadata Specifications, with no mention of 2.2.
If it’s the place to look for it - maybe it could use a bit of reorganizing? Some sort of a reference list of related PEPs would be really helpful.
The goal is to have every specification under PyPA specifications - Python Packaging User Guide, so that location is the equivalent of the Python language spec, with PEPs simply being proposal documents describing changes to those documents. Under that model, there’s no requirement to link back to the PEPs from the documentation, although having them as a form of changelog might be useful.
However, we haven’t reached that goal yet, so yes there is some confusion. The current rules allow for “readability improvements that don’t affect software interoperability” to be made as simple pull requests, so clarifying PRs are welcome.
This discussion is intended to ask whether changes that do affect interoperability can also, in some cases, be handled without a PEP. And to ask the supplemental question, if we do make changes without a PEP, how do we make sure people know about them?
Probably both. The former is helpful in keeping up with all the changes, the latter is helpful with implementing things up to spec.
Honestly I hadn’t looked at it that way, maybe that’s where my perspective stems from. My primary source for all-things-packaging was PEPs, solely because I consider them to be the most complete source on each respective topic.
In the same way, if I was going to create my own implementation of Python, and was at the stage of implementing coroutines, I wouldn’t go to “Coroutines and Tasks” in the Python docs, I’d go straight to PEP-342 and PEP-492.
I realize this is somewhat off-topic here, I just wanted to voice my doubts given my very recent experiences with learning the subject. My worry is that if you reduce the bureaucracy around the specs, it might get harder to keep up, since it would essentially mean that some things would only be known to people who read all of the surrounding conversations, and those tend to get lengthy.
I’m fine with “PEP delegate decides” for the process here. In my head, the process is:
If it’s a “major” change or a controversial change, get a PEP written down for it.
If it’s a “minor” uncontroversial change, just do a PR and a few posts around our communication channels for visibility (which should be linked to in the PR, and maybe the text).
Someone needs to decide the “major” vs “minor” and “controversial” vs “uncontroversial” for these changes, and I’m happy for that person to be a Steering Council’s PEP-delegate for Python Packaging.
I thought that’s what we had already.
Edit: I feel like the concerns that Paul has is around “major and uncontroversial” changes, but like, I don’t think we get a lot of those here. And given our main documentation for this stuff is a little meh right now, we should just err on the side of writing a PEP, since this documents it and allows for clearer avenues to comment on this major change.
If we feel the need to have granularity here, to aid with communicating with people involved… IMO that means we should get our own RFC-style process independent of PEPs, to allow for the increased “traffic”.
(such a change needs a pypa-committers vote)
Regarding reorganisation of stuff, I think if someone posted a bunch of PRs moving all the current specifications one-by-one, those PRs would certainly be accepted. At the very least, you’ll have me advocating to accept them.
Regarding changelogs linking back to PEPs, I certainly think that’d be worthwhile! Someone needs to do the work though. Last I checked, there isn’t a lot of energy available for improving our documentation situation.
To give some examples of recent cases which seem to me to be borderline:
Changes to the wheel spec
a. To cover storing timestamps and file modes. Relatively uncontroversial.
b. To include symlinks. More controversial, not least because the standard zip format doesn’t support symlinks.
Tightening of the calling convention for PEP 517 hooks to require them to always be called with keyword args. I pretty much said “no” to this flat out, but that’s not my prerogative¹, so the proposer may still want to try to get this accepted.
All of these would need a PEP under our current rules.
Thinking some more about this, I think my concern is that we currently have too strong a link between “publicising the proposal, having a discussion and getting consensus” and the PEP process. I’m fine with not having a PEP for smaller changes, but not if that means we don’t ensure consensus on the change. And we lack a way of making sure that interested parties even see the smaller discussions.
At the moment, someone interested in the standards can monitor new PEPs and know they won’t miss anything important. If we allow changes that affect interoperability without a PEP, how do those people know there’s a change upcoming, and have their say? Expecting everyone with an interest to monitor all of the threads on the Packaging discourse is (IMO) too much. And judging when we have consensus is a problem in a thread where maybe only a few people have commented and there’s no way of knowing if others don’t have anything to say, or have missed the discussion. The PEP process is a way of enforcing due dilligence on the consensus building process.
If we had some other tracking process (a low-traffic list solely for announcements of proposed standards changes, for example) then that would cover my concerns. Otherwise I think the lack of significant feedback here² suggests to me that there’s not much interest in changing our processes, and I’ll stick to my existing policy of following the documented process, and anything other than clarification-style changes need a PEP. What I will do, though, is try to push people to not propose smaller changes, but bundle them with larger ones - so for example, I’ll take the view that the wheel changes mentioned above cannot go into Wheel 1.0, but could be incorporated in the Wheel 2.0 activity, when that gets moving again.
¹ It is, sort of, but IMO if the proposer can get consensus they have a right to request me to reconsider.
² Which is itself a good example of how it’s hard to know if we have consensus if too few people are participating in the discussion
I thought it had been discontinued. And I’m pretty sure a lot of people unsubscribed from it because of spam (I know I did). But yeah, something like that would do fine. Or a packaging-announcements list.
I don’t have a problem with this, if people want to do it this way. What would need to happen (IMO):
Someone works out how exactly we’ll do this. As PEP-delegate, I’d need to know what to tell people proposing a change that they have to do, if it’s not “write up a PEP and follow the PEP process as documented in the Python devguide”.
Consensus from PyPA members that this is a reasonable change. PEP 609 says “The PyPA will use PEPs for defining, and making changes to, the interoperability specifications maintained by the PyPA”, so I think we probably need a PyPA vote to change it.
Update the PyPA rules to document the new process.
I’m not entirely clear on how the PyPA governance stuff works, and I don’t personally mind requiring stuff to go through the PEP process. I also don’t really want to handle the bureaucracy of doing all of this myself, so I guess I’ll leave it to someone else to take this forward if they want to.
(My personal interest ends with being able to say “I flagged up that the PEP process is a bit heavyweight for some changes, but no-one was willing to define a simpler process, so yes, you do need to write a PEP” when asked )
Closing the loop here, the relevant paragraph in the specification rules is:
If a change being considered this way has the potential to affect software interoperability, then it must be escalated to the distutils-sig mailing list for discussion, where it will be either approved as a text-only change, or else directed to the PEP process for specification updates.
Assume “distutils-sig mailing list” is replaced by “Packaging area on Discourse”.
I’m going to assume that “approval as a text-only change” here means:
Consensus on the thread that the change is uncontroversial, and a PEP isn’t needed.
Consensus on the thread that it’s been sufficiently well publicised (where relevant).
The PEP-delegate agrees (i.e. the PEP delegate can insist on a PEP for anything they feel needs it).
It’s important here to note that the maintainers of the spec documents will have to be responsible for ensuring these rules are followed. I have trouble following the github permissions maze, but that includes the “packaging user guide editors” group, most of whom aren’t active here, so I don’t know if they are OK with this responsibility. One thing I will say, though, is that as PEP delegate, I won’t be monitoring every PR for packaging.python.org to confirm if it’s a specification change that affects interoperability.
I do have a concern that we could gradually make significant changes to the specs via a series of “small, uncontroversial” changes. So I’ll probably apply a few personal rules here:
Backward incompatible changes probably need a PEP (compatible with the de facto behaviour of existing tools is probably good enough here).
Specs that are “volatile” (in the sense if getting a lot of proposals for changes) will need PEPs more frequently.
