Splitting out the packaging PEPs, on the PEP site

It seems to me like what @pradyunsg is proposing is similar to @pganssle’s option #1 in the OP of that thread:

and what I suggested here:

1 Like

We could do awful things filtering on Discussions-To — I’m wary on adding a new header, as we’ll need to update and categorise many (all?) PEPs to do a ‘proper’ implementation.

A

If we use the Track (naming tbc) approach, it may be easier as we assume it is opt-in for packaging, typing, WSGI(?), etc — perhaps a good compromise.

A

1 Like

At least from his OP here, it had sounded to me like @pradyunsg was only suggesting a technical change in how the PEPs were displayed on PEP 1, rather than anything that would substantively change the process, so I didn’t have in mind something much more substantive, as the linked threads appear to be discussing, and would seem to be the implied intent here.

The most immediately feasible technical approach to specifically separate packaging PEPs that would be more or less compatible with how things are currently done, and that would not require adding a new header, using hacky heuristics or changing other PEPs, would be adding a new Packaging Type equal to Standards Track, Informational and Process, as I believe @dustin is suggesting. The Type is already one of the two key discriminators, along with Status, used to determine the different categories shown on PEP 0, and more substantively, used in PEP 1 (and 12) to describe specific elements of the PEP process that do/don’t apply to such PEPs, so that would be fairly consistent with existing use.

The changes on PEP 0 would be simple to implement technically, though we would need SC approval to introduce a new PEP type, and we would need to update PEP 1 (and 12/the template) to reflect its existence—though this would also be a perfect opportunity to more explicitly describe the distinctions in process for packaging PEPs that aren’t really formally documented right now. I could take the lead on the technical and writing implementation side, since I’ve already been working on these areas a lot lately.

As opposed to a dedicated Category header, it doesn’t scale so well to more categories, if such is contemplated. But I suppose we can cross that bridge if and when we come to it.

I do think without a sortable table, however, it will become rather unweidly and more difficult to navigate; since per the existing structure, we are stuck with duplicating many of the existing separate tables for packaging PEPs; that would allow us to put them all into one table, sorted by type. But that would take someone more effort from a technical perspective and may be something we’d want to put broader thought into—especially prior to PEP 676’s full implementation, in which we’d have to do everything twice in two different codebases (whereas just adding a few new tables would be way less work).

I thought that all that was being suggested was to have packaging PEPs listed in a separate section (probably entitled “Packaging PEPs”) in PEP 0, nothing more than that?

If I’m wrong, and this is intended to open up the governance question again, then I stand corrected (but in that case, I don’t think the original post was sufficiently clear…)

2 Likes

Summary of possible backend implementation strategies to achieve this:

  • Add Packaging option to existing Type header (Standards Track, Informational, etc)
    :+1: Arguably most natural fit within the existing frontend and backend
    :+1: Allows more easily codifying current and future process differences
    :-1: Requires more PEP 1/PEP 12 changes, both substantive and mechanical
    :-1: Carries potential governance implications

  • New Category header
    :+1: Useful and extensible to other PEPs
    :+1: Relatively lightweight from a policy perspective
    :-1: Implies categorizing other PEPs and requires a few more backend changes
    :-1: A bit odd to special-case packaging with it without more frontend changes

  • Use heuristics on Discussions-To header or hardcode list of packaging PEPs
    :+1: Requires fewer changes than some other options
    :-1: Awful, hacky and inherently unstable and unreliable

This was also my initial impression as well, based on the limited information in the OP. To note though, implementation wise, perhaps the most practical way that was suggested to achieve this within the existing system without hacks, excessive hardcoding/special-casing or affecting other PEPs is adding a new Packaging value for Type, which inherently carries at least some implied degree of significance for process/governance, for better (documenting existing practice and making future changes smoother) or worse (revisiting a bigger debate).

Also, to note, there’s been a concerted and growing push recently to make the PEP process as a whole friendlier, less arcane, more streamlined and better documented, coupled with a tighter focus on PEPs as change proposals rather than living specifications or documentation while working to find better homes for the latter (modeled on the mostly-successful effort to move PyPA specs to the PyPUG).

Note that my proposal here is NOT about opting out of the PEP process.

That’s indeed my intent here. Apologies for not stating it more verbosely earlier.

Looking at the discussion that @dustin posted, yea, I think Track: Packaging is indeed what I’m thinking of as a good solution to this.

If anyone has reservations to doing something like that (that thread doesn’t seem to have any, although I might’ve missed them), please holler. Otherwise, this can go into a bucket of “PyPA standards story improvement” that one of us eventually tackles. :slight_smile:

1 Like

It would make sense to coordinate with typing-sig – Typing is the other big active Track.

1 Like

Just noting that when I proposed filtering on the Discussions-To header, we weren’t using Discourse yet, so there were only a few potential entries, and every packaging PEP used “distutils-sig@python.org”.

Parsing the Discourse category out of the topic URL isn’t an option, so new metadata in the PEP headers is the only really viable option now.

Fortunately, the new rendering pipeline will make that much easier to implement than would have previously been the case (no python.org code to get confused by the addition).

Somewhat ironically, the process for changing PEP 1 isn’t clearly defined, but the following is at least “not wrong”:

  • discuss here the change that we’d like to see (specifying subcategories of “Standards Track” would get my vote)
  • create a draft PR with the details
  • post a Steering Council ticket in the SC Github repo
  • take any other steps the SC request

For actually listing the subcategories in the PEP headers, they could potentially go in parentheses after the existing types, and we could allow one PEP to span multiple subcategories.

For example, given these subcategories:

  • Language
  • Builtins
  • Standard Library
  • Gradual Typing
  • Packaging
  • CPython
  • C API

A PEP like PEP 558 might say “Type: Standards Track (Builtins, CPython, C API)” since it:

  • changes the locals() builtin
  • changes the way CPython works
  • changes the public C API
2 Likes

Thinking about it some more, I initially assumed you meant Type: Packaging (the header that Standards Track, Informational and Process are currently sorted under), but an actual new header, Track, with values e.g. Core, Packaging and (perhaps) Typing combines most of the advantages of both main approaches summarized above.

It is a cleaner and more extensible solution than co-opting the Type header while carrying similar weight, and actually requires fewer individual changes to PEP 1/12/template [1]. Meanwhile, it side-steps the problems with the broader Category header by being more narrowly scoped to just special tracks (Packaging, Typing, and any future ones) and only requiring those PEPs be tweaks (the rest can have an assumed default of Core or whatever we want to call it).

So from both a PEP editor/infra/meta-PEP maintainer perspective, and a packaging community member and PEP writer perspective, (and a “person who’s probably going to implement this” perspective) as a solution to this I’d strongly support a new Track: Packaging header (the proposal as-written). It doesn’t actually directly change anything governance-related about packaging PEPs, it just documents the present status quo and makes it easier to implement and document any future changes the SC and PyPA may agree on.

Implementation-wise, I could knock the above out fairly quickly, since I’m familiar with exactly what would need to be added/changed all the different places this would touch (PEP 1, PEP 12, template, header processing, PEP 0 processing, linting/validation, and existing PEPs).

The remaining blockers are:

  • Ensure consensus and SC approval for the above proposal
  • PEP 676 system (now officially Accepted) goes live (to avoid duplicate effort)
  • PEP editors and packaging community agree on how we want to organize PEP 0

The latter is necessarily more complex than just sticking the packaging PEPs into a separate table, because right now there’s the Index by Category, which is organized by the combination of two different factors (Type and Status), plus the Numerical Index, so there’s a lot of different ways of adding a third, none of them ideal.

On the latter, the the most straightforward-to-implement viable option would seem to be adding a new Packaging subheading to the Index by Category, with four tables (open, accepted, active/final and withdrawn/superceded/rejected/deferred) like we have for the other PEPs, moving the Packaging PEPs there and out of the other “by Category” tables. Additionally, to be more consistent and scale better, we could put the “Core” track (or whatever we call them) PEPs under a subheading too. Ideally, we’d have one sortable table, but that would be a much bigger change.


  1. as the Type header value comes into play a lot of different places ↩︎

For for any PEP, including packaging PEPs, I google "pep " (for example, “pep version format”). This is my primary way of getting to PEPs, and for the most part also for discovery. Is PEP 0 (index of PEPs) a common way for others to go to, discover and keep track of packaging standards changes?

I think I don’t know exactly why the packaging PEPs need to be split out in the index. I understand the benefit of tagging packaging PEPs for future programmatic query, outside of a user-displayed list.

Good point; if I’m trying to find a specific PEP that I know by approximate or exact name, I generally just google it (by number, I just Ctrl-Tab to one of the PEP tabs I have open and type it into the URL, but I’m probably not the norm on that). Aside from giving Google something to index, the index page is rather useful though to discover new PEPs and keep track of existing ones in a specific area of interest, which separating out the packaging PEPs on the displayed page could, I imagine, assist with. But I’m not sure how broad the utility for that is, though, beyond just the relatively niche pool of packaging PEP writers, reviewers and implementors rather than most users, though, who can and should reference the canonical specifications in the PyPUG rather than the original PEPs.

To note, its the latter, the backend issue, that is pretty straightforward to implement (and allows us to do all manner of things in the future). The former, the frontend concern, is technically quite doable, but raises a number of UI, UX and conceptual questions.

One thing that does make Packaging PEPs genuinely different is that they’re covered by standing delegations from the Steering Council (formerly from Guido back when he was still BDFL). The other potential topics kicked around earlier in the thread don’t have that level of process separation.

PEP 1 doesn’t spell out where those delegations are to be recorded, just that they will be recorded and published. The current location of that record is the SC GitHub repo: steering-council/standing-delegations.md at main · python/steering-council · GitHub

I think that’s also where the front end changes come in:

  • the SC (and core developers in general) have an interest in being able to easily tell which posted PEPs are covered by a standing delegation, since they’re automatically somebody else’s problem
  • PyPA members (especially the standing delegates) have an interest in being able to easily tell which posted PEPs are covered by the PyPA standing delegations, since those PEPs are by definition PyPA’s problem
1 Like

As far as display goes, something I’ve been contemplating is that rather than trying to change PEP 0 itself, we could instead emit separate track specific pages for the standards track PEPs. So if you wanted to look at only packaging PEPs, you would go to https://peps.python.org/track/packaging/, while the default track might use “Python” as its implied name and be found at https://peps.python.org/track/python/

That would then scale reasonably well if further tracks were defined (e.g. a “CPython” track for PEPs that change CPython without also changing the language, or a dedicated “Typing” track).

2 Likes

Indeed, and a Track header would reflect that, and allow making that more clear in PEP 1. Of course, this does mean we would want SC assent to the overall idea, beyond just being a display/backend change.

We could explicitly link that in PEP 1, since I don’t imagine it is expected to change too often.

UI/UX wise, that would be a lot cleaner and less disruptive solution than trying to add a bunch more headings on the PEP 0 page, and not require bikeshedding over any necessary frontend decisions, since it could just use the format of the existing PEP 0 (and not need to modify the same, aside from adding links to the track subpages).

From a technical perspective, it would only require a relatively low number and complexity of changes to the existing generation code [1] and would fit well with the existing architecture without being hacky or disruptive. All told, it would be a few dozen lines of code/text and, combined with the tweaks to add, parse and validate the track header, is something I could probably knock out in a few hours once consensus was reached.

It also shouldn’t have any meaningful performance impact, since the PEPs would only be read and parsed once (by far the most intensive part of the process), and only the actual writing out of the indices would be needed.

In fact, it should scale seamlessly to multiple tracks, as adding a new track name would be a only require changing a couple lines in the code (adding the name to a list of valid tracks in constants.py and the pre-commit config, and optionally a custom intro paragraph if it was desired to describe the particulars of the track more specifically), while not affecting existing pages.


  1. add the header to those parsed in parser.PEP.__init__; filter the PEPs by track and feed them to the PEP generation and writing code in a loop in pep_index_generator.create_pep_zero, plus modifying the filepath generation logic accordingly; and tweaking writer.PEPZeroWriter.write_pep0 to accept a track arg that controls using an appropriately modified intro paragraph ↩︎

1 Like

I actually don’t want to touch the workflows for core developers or any other SIG. I’d rather that someone else step forward, say it’s useful to them, and then take point on the relevant concensus building discussions with the relevant group. :slight_smile:

I’m happy to have a design that accomodates for extensions in the future, but I don’t want to block on trying to establish that this is useful for other groups. As it stands, this can be a self-contained change that benefits a well-defined subset of users, of the PEP process, without affecting others’ workflows. I’d like to keep it that way and get this resolved in a reasonable timeframe.


Relatedly, I don’t think there’s any need to try to establish concensus at python-dev or whatever for something that PyPA would benefit from in the PEP process.

If the PEP editors are OK with such a change, I think this can get landed with small changes to PEP 1 and the Packaging PEPs. This does not need to bubble up all the way to the SC, if PEP editors and PyPA members are in agreement. If we don’t reach an agreement, this can bubble up to the SC.

1 Like

I think you’re more likely to upset people and get push back by trying to make the change that way than you are by posting up front to say “We’re proposing to make a change to the PEP rendering process that emits a sub index with just the packaging PEPs, these are the mechanics of how we’re planning to make it work, here’s how it could be expanded to other PEP categories in the future if folks want to do that”.

I do agree that actually generalising the process and formalising the idea of standards track categories in PEP 1 can wait, but PEP rendering is a shared resource and it’s courteous to keep other users of that resource informed when making significant changes to it.

That’s part of why I favour keeping the category information in parentheses inside the existing Type field: it’s far more defensible as a rendering pipeline enhancement that doesn’t require PEP 1 and PEP template changes the way the addition of a new field in the PEP header would.

1 Like

I definitely agree here. I don’t think there will be much resistance or lengthy debate to a narrowly-scoped, technically-focused proposal, but I think its much less likely to elicit a negative reaction to just be up front with it and get a formal okay than go ahead and make the change unilaterally. In the mean time, I can get a PR ready; I expect technical and content review of that and any further discussions here will likely take longer than getting a quick SC signoff anyway.

Actually, its in fact less clean conceptually, more complex to implement and more potentially disruptive to the existing uses to change the syntax of the existing Type header than to add a new header tailored to this purpose, as well as less flexible, extensible and forward-compatible with potential future changes/applications. This is why after careful analysis of the various facets this would touch, I mapped out the planned implementation above with a separate Track header.

To summarize, the current Type header and its valid values is used/referred to a bunch of different places in the checkers, processing code, PEP index, JSON API and PEP 1/12/template, all of which would have to be reviewed and many/most of which would have to be carefully modified to accommodate and handle the new, more complicated Type [(Track)] syntax (which is before even considering the actual core changes needed to generate different index pages).

By contrast, adding a new field only requires a few very specific and localized changes to PEP 1/12/the template, a simple new linter check and adding it to the desired packaging PEPs. It should require essentially no code changes, aside from that needed anyway to generate sub-index pages—which is a fair bit simpler not having to extract a field within a field. It also makes it trivially easy to not actually show the field in the header table (to further minimize initial impact), doesn’t require further parsing in the structured API, and is generally easier to extend to accommodate future additions/changes.

2 Likes

Adding parentheses to Type would also require changes to PEP 1. OTOH, PEP 1 changes aren’t that big a big deal.

I agree with CAM, Track: Packaging would be better than parentheses. Easier to implement, and easier to consume.

3 Likes