Splitting out the packaging PEPs, on the PEP site

Packaging
1

I think it would be useful to do that, for making these PEPs more discoverable on that index.

Is there any reason to not do something like this, or is this just a matter of someone figuring out how to do this?

3 Likes
2

Previously discussed in Should PyPA opt out of the PEP process?

3

It is certainly possible, the code that does this is fairly simple and I was just mucking with it a few hours ago. However, it is perhaps a bit bigger change than just special-casing packaging; it would be pretty hacky that way, not fit the existing layout and beg the question as to why not for Typing or other things too. There is no unambitious indicator of what counts as a packaging PEP; we’d have to add a new Category field to the PEP header for this unless we hardcode a list somewhere, and they don’t really intersect well with the current way sections are structured, i.e. by PEP status.

I think having a Category header field with a few basic categories like Core, Typing, Stdlib, Packaging, Release, etc. would be very helpful and allow users to find PEPs they are interested in much more easily. To display them, we could make the index page even longer with tables of PEPs by category, such that it would have to be a separate set of sections (which have been rejected in the past for other things).

However, a much better approach would be having the main table of PEPs sortable by each of the headers (number, date, category, status, type, etc). This would add requested and useful functionality without making the page longer (theoretically, we could even omit the currently duplicative tables and just rely on that).

Of course, as to that aspect I believe @AA-Turner would be the best judge. But this all isn’t really a packaging discussion, and might be a better fit for, e.g., the PEPs repo.

Could you point us to the specific comment there where this was discussed? At least skimming through, I couldn’t find it.

1 Like
4

Could you point us to the specific comment there where this was discussed? At least skimming through, I couldn’t find it.

It came up here (in a different thread): Closing the loop on PyPA Governance / BDFRN - #8 by ncoghlan

2 Likes
5

It seems to me like what @pradyunsg is proposing is similar to @pganssle’s option #1 in the OP of that thread:

and what I suggested here:

1 Like
6

We could do awful things filtering on Discussions-To — I’m wary on adding a new header, as we’ll need to update and categorise many (all?) PEPs to do a ‘proper’ implementation.

A

7

If we use the Track (naming tbc) approach, it may be easier as we assume it is opt-in for packaging, typing, WSGI(?), etc — perhaps a good compromise.

A

1 Like
8

At least from his OP here, it had sounded to me like @pradyunsg was only suggesting a technical change in how the PEPs were displayed on PEP 1, rather than anything that would substantively change the process, so I didn’t have in mind something much more substantive, as the linked threads appear to be discussing, and would seem to be the implied intent here.

The most immediately feasible technical approach to specifically separate packaging PEPs that would be more or less compatible with how things are currently done, and that would not require adding a new header, using hacky heuristics or changing other PEPs, would be adding a new Packaging Type equal to Standards Track, Informational and Process, as I believe @dustin is suggesting. The Type is already one of the two key discriminators, along with Status, used to determine the different categories shown on PEP 0, and more substantively, used in PEP 1 (and 12) to describe specific elements of the PEP process that do/don’t apply to such PEPs, so that would be fairly consistent with existing use.

The changes on PEP 0 would be simple to implement technically, though we would need SC approval to introduce a new PEP type, and we would need to update PEP 1 (and 12/the template) to reflect its existence—though this would also be a perfect opportunity to more explicitly describe the distinctions in process for packaging PEPs that aren’t really formally documented right now. I could take the lead on the technical and writing implementation side, since I’ve already been working on these areas a lot lately.

As opposed to a dedicated Category header, it doesn’t scale so well to more categories, if such is contemplated. But I suppose we can cross that bridge if and when we come to it.

I do think without a sortable table, however, it will become rather unweidly and more difficult to navigate; since per the existing structure, we are stuck with duplicating many of the existing separate tables for packaging PEPs; that would allow us to put them all into one table, sorted by type. But that would take someone more effort from a technical perspective and may be something we’d want to put broader thought into—especially prior to PEP 676’s full implementation, in which we’d have to do everything twice in two different codebases (whereas just adding a few new tables would be way less work).

9

I thought that all that was being suggested was to have packaging PEPs listed in a separate section (probably entitled “Packaging PEPs”) in PEP 0, nothing more than that?

If I’m wrong, and this is intended to open up the governance question again, then I stand corrected (but in that case, I don’t think the original post was sufficiently clear…)

2 Likes
10

Summary of possible backend implementation strategies to achieve this:

  • Add Packaging option to existing Type header (Standards Track, Informational, etc)
    :+1: Arguably most natural fit within the existing frontend and backend
    :+1: Allows more easily codifying current and future process differences
    :-1: Requires more PEP 1/PEP 12 changes, both substantive and mechanical
    :-1: Carries potential governance implications

  • New Category header
    :+1: Useful and extensible to other PEPs
    :+1: Relatively lightweight from a policy perspective
    :-1: Implies categorizing other PEPs and requires a few more backend changes
    :-1: A bit odd to special-case packaging with it without more frontend changes

  • Use heuristics on Discussions-To header or hardcode list of packaging PEPs
    :+1: Requires fewer changes than some other options
    :-1: Awful, hacky and inherently unstable and unreliable

This was also my initial impression as well, based on the limited information in the OP. To note though, implementation wise, perhaps the most practical way that was suggested to achieve this within the existing system without hacks, excessive hardcoding/special-casing or affecting other PEPs is adding a new Packaging value for Type, which inherently carries at least some implied degree of significance for process/governance, for better (documenting existing practice and making future changes smoother) or worse (revisiting a bigger debate).

Also, to note, there’s been a concerted and growing push recently to make the PEP process as a whole friendlier, less arcane, more streamlined and better documented, coupled with a tighter focus on PEPs as change proposals rather than living specifications or documentation while working to find better homes for the latter (modeled on the mostly-successful effort to move PyPA specs to the PyPUG).

11

Note that my proposal here is NOT about opting out of the PEP process.

That’s indeed my intent here. Apologies for not stating it more verbosely earlier.

12

Looking at the discussion that @dustin posted, yea, I think Track: Packaging is indeed what I’m thinking of as a good solution to this.

If anyone has reservations to doing something like that (that thread doesn’t seem to have any, although I might’ve missed them), please holler. Otherwise, this can go into a bucket of “PyPA standards story improvement” that one of us eventually tackles. :slight_smile:

1 Like
13

It would make sense to coordinate with typing-sig – Typing is the other big active Track.

1 Like
14

Just noting that when I proposed filtering on the Discussions-To header, we weren’t using Discourse yet, so there were only a few potential entries, and every packaging PEP used “distutils-sig@python.org”.

Parsing the Discourse category out of the topic URL isn’t an option, so new metadata in the PEP headers is the only really viable option now.

Fortunately, the new rendering pipeline will make that much easier to implement than would have previously been the case (no python.org code to get confused by the addition).

Somewhat ironically, the process for changing PEP 1 isn’t clearly defined, but the following is at least “not wrong”:

  • discuss here the change that we’d like to see (specifying subcategories of “Standards Track” would get my vote)
  • create a draft PR with the details
  • post a Steering Council ticket in the SC Github repo
  • take any other steps the SC request

For actually listing the subcategories in the PEP headers, they could potentially go in parentheses after the existing types, and we could allow one PEP to span multiple subcategories.

For example, given these subcategories:

  • Language
  • Builtins
  • Standard Library
  • Gradual Typing
  • Packaging
  • CPython
  • C API

A PEP like PEP 558 might say “Type: Standards Track (Builtins, CPython, C API)” since it:

  • changes the locals() builtin
  • changes the way CPython works
  • changes the public C API
2 Likes
15

Thinking about it some more, I initially assumed you meant Type: Packaging (the header that Standards Track, Informational and Process are currently sorted under), but an actual new header, Track, with values e.g. Core, Packaging and (perhaps) Typing combines most of the advantages of both main approaches summarized above.

It is a cleaner and more extensible solution than co-opting the Type header while carrying similar weight, and actually requires fewer individual changes to PEP 1/12/template [1]. Meanwhile, it side-steps the problems with the broader Category header by being more narrowly scoped to just special tracks (Packaging, Typing, and any future ones) and only requiring those PEPs be tweaks (the rest can have an assumed default of Core or whatever we want to call it).

So from both a PEP editor/infra/meta-PEP maintainer perspective, and a packaging community member and PEP writer perspective, (and a “person who’s probably going to implement this” perspective) as a solution to this I’d strongly support a new Track: Packaging header (the proposal as-written). It doesn’t actually directly change anything governance-related about packaging PEPs, it just documents the present status quo and makes it easier to implement and document any future changes the SC and PyPA may agree on.

Implementation-wise, I could knock the above out fairly quickly, since I’m familiar with exactly what would need to be added/changed all the different places this would touch (PEP 1, PEP 12, template, header processing, PEP 0 processing, linting/validation, and existing PEPs).

The remaining blockers are:

  • Ensure consensus and SC approval for the above proposal
  • PEP 676 system (now officially Accepted) goes live (to avoid duplicate effort)
  • PEP editors and packaging community agree on how we want to organize PEP 0

The latter is necessarily more complex than just sticking the packaging PEPs into a separate table, because right now there’s the Index by Category, which is organized by the combination of two different factors (Type and Status), plus the Numerical Index, so there’s a lot of different ways of adding a third, none of them ideal.

On the latter, the the most straightforward-to-implement viable option would seem to be adding a new Packaging subheading to the Index by Category, with four tables (open, accepted, active/final and withdrawn/superceded/rejected/deferred) like we have for the other PEPs, moving the Packaging PEPs there and out of the other “by Category” tables. Additionally, to be more consistent and scale better, we could put the “Core” track (or whatever we call them) PEPs under a subheading too. Ideally, we’d have one sortable table, but that would be a much bigger change.

  1. as the Type header value comes into play a lot of different places ↩︎

16

For for any PEP, including packaging PEPs, I google "pep " (for example, “pep version format”). This is my primary way of getting to PEPs, and for the most part also for discovery. Is PEP 0 (index of PEPs) a common way for others to go to, discover and keep track of packaging standards changes?

I think I don’t know exactly why the packaging PEPs need to be split out in the index. I understand the benefit of tagging packaging PEPs for future programmatic query, outside of a user-displayed list.

17

Good point; if I’m trying to find a specific PEP that I know by approximate or exact name, I generally just google it (by number, I just Ctrl-Tab to one of the PEP tabs I have open and type it into the URL, but I’m probably not the norm on that). Aside from giving Google something to index, the index page is rather useful though to discover new PEPs and keep track of existing ones in a specific area of interest, which separating out the packaging PEPs on the displayed page could, I imagine, assist with. But I’m not sure how broad the utility for that is, though, beyond just the relatively niche pool of packaging PEP writers, reviewers and implementors rather than most users, though, who can and should reference the canonical specifications in the PyPUG rather than the original PEPs.

To note, its the latter, the backend issue, that is pretty straightforward to implement (and allows us to do all manner of things in the future). The former, the frontend concern, is technically quite doable, but raises a number of UI, UX and conceptual questions.

18

One thing that does make Packaging PEPs genuinely different is that they’re covered by standing delegations from the Steering Council (formerly from Guido back when he was still BDFL). The other potential topics kicked around earlier in the thread don’t have that level of process separation.

PEP 1 doesn’t spell out where those delegations are to be recorded, just that they will be recorded and published. The current location of that record is the SC GitHub repo: steering-council/standing-delegations.md at main · python/steering-council · GitHub

I think that’s also where the front end changes come in:

  • the SC (and core developers in general) have an interest in being able to easily tell which posted PEPs are covered by a standing delegation, since they’re automatically somebody else’s problem
  • PyPA members (especially the standing delegates) have an interest in being able to easily tell which posted PEPs are covered by the PyPA standing delegations, since those PEPs are by definition PyPA’s problem
1 Like
19

As far as display goes, something I’ve been contemplating is that rather than trying to change PEP 0 itself, we could instead emit separate track specific pages for the standards track PEPs. So if you wanted to look at only packaging PEPs, you would go to https://peps.python.org/track/packaging/, while the default track might use “Python” as its implied name and be found at https://peps.python.org/track/python/

That would then scale reasonably well if further tracks were defined (e.g. a “CPython” track for PEPs that change CPython without also changing the language, or a dedicated “Typing” track).

2 Likes
20

Indeed, and a Track header would reflect that, and allow making that more clear in PEP 1. Of course, this does mean we would want SC assent to the overall idea, beyond just being a display/backend change.

We could explicitly link that in PEP 1, since I don’t imagine it is expected to change too often.

UI/UX wise, that would be a lot cleaner and less disruptive solution than trying to add a bunch more headings on the PEP 0 page, and not require bikeshedding over any necessary frontend decisions, since it could just use the format of the existing PEP 0 (and not need to modify the same, aside from adding links to the track subpages).

From a technical perspective, it would only require a relatively low number and complexity of changes to the existing generation code [1] and would fit well with the existing architecture without being hacky or disruptive. All told, it would be a few dozen lines of code/text and, combined with the tweaks to add, parse and validate the track header, is something I could probably knock out in a few hours once consensus was reached.

It also shouldn’t have any meaningful performance impact, since the PEPs would only be read and parsed once (by far the most intensive part of the process), and only the actual writing out of the indices would be needed.

In fact, it should scale seamlessly to multiple tracks, as adding a new track name would be a only require changing a couple lines in the code (adding the name to a list of valid tracks in constants.py and the pre-commit config, and optionally a custom intro paragraph if it was desired to describe the particulars of the track more specifically), while not affecting existing pages.

  1. add the header to those parsed in parser.PEP.__init__; filter the PEPs by track and feed them to the PEP generation and writing code in a loop in pep_index_generator.create_pep_zero, plus modifying the filepath generation logic accordingly; and tweaking writer.PEPZeroWriter.write_pep0 to accept a track arg that controls using an appropriately modified intro paragraph ↩︎

1 Like