# PEP 676: PEP Infrastructure Process **URL:** \ **Category:** PEPs\ **Created:** [September 23, 2021, 12:14am UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774 "2021-09-23T00:14:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![AA-Turner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aa-turner/32/4979_2.png) [@AA-Turner](https://discuss.python.org/u/AA-Turner)\ **Post date:** [September 23, 2021, 12:14am UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/1 "2021-09-23T00:14:07Z")
# One-line summary This discusses rationale for creating `peps.python.org` as a stand alone site to host PEPs, replacing the current systems. # Summary: Python Enhancement Proposals (PEPs) are currently rendered by running `docutils` on each PEP document and are refreshed on [python.org/dev/peps](https://www.python.org/dev/peps/) every 15 minutes. This proposal would seek to make rendering of PEPs self-contained. This would: - reduce the amount of distributed configuration for supporting PEPs - enable quality-of-life improvements for those who read, write, and review PEPs - solve a number of outstanding issues, and lay the path for potential improvements - save volunteer time in maintaining the current system The proposed end state is that PEPs are accessed through [peps.python.org](http://peps.python.org) at the top-level namespace (for example [peps.python.org/pep-9999](http://peps.python.org/pep-9999)), and all tooling to support rendering PEPs is hosted in the [`python/peps`](https://github.com/python/peps) repository. # Current status: The proposed implementation has been merged into the `python/peps` repository, and a rendered preview of all PEPs is available ([python.github.io/peps](https://python.github.io/peps/)). I [was asked](https://github.com/python/peps/pull/2023#issuecomment-924434537) to write up this briefing by the PEP editors for consideration. # Reducing distributed configuration and simplifying tooling: Currently, three Python repositories are involved in orchestrating the rendering process, using three separate technologies (`docutils`, Django, Saltstack). The proposed implementation renders the PEPs using Sphinx and a custom Sphinx plugin. As it is proposed to stand alone, the integration with `pythondotorg` and associated update machinery can be removed, reducing cognitive load when reviewing issues with PEP rendering. Use of CI will also surface all build logs in one place, again useful when investigating problems. By removing configuration in these other projects, volunteer time does not need to be spent maintaining more fragile linkages and complex distributed systems. It may also reduce the barrier to entry to adding new features, as the scope of the PEP rendering tooling is well defined. # Quality-of-life improvements and resolving issues: There are a number of requests for additional features in viewing PEPs, including syntax highlighting, ` .. code-block::` directives, support for SVG images, typographic quotation marks, additional footer information, and inter-sphinx functionality. These are “easy wins” from this proposal, and would serve to improve the quality-of-life for consumers of PEPs (including reviewers and writers). Equally, there are a small number of broken items, for example list styles not being respected or support for updating images being challenging with the current `pythondotorg` system. These would be solved by default in the proposal. Of note, I found a lot more such items that had been fixed in various ways by volunteers – which shows both their dedication and that such time investigating a fix might be better used elsewhere. # Potential alternatives It would likely be possible to amend the current rendering process to include a lot of the quality-of-life improvements and issue mitigations mentioned above. However, I do not believe that this would solve the distributed tooling issue. It would be possible to use the output from the proposed rendering system and import it into `pythondotorg`. I would argue however that this would be the worst of both worlds, as a great deal of complexity is added, and none is removed. # Specific proposed implementation As the proposed system uses Sphinx, any static file hosting solution could be used for [peps.python.org](http://peps.python.org) – this could also be behind a CDN for example as there is no dynamic content, and the only optional JavaScript is for a single cosmetic change. Commercial vendors such as Read the Docs, Inc can provide additional services such as preview rendering, which may align with the quality-of-life piece. The rendered PEPs would be made available at [peps.python.org](http://peps.python.org), with the current redirect configuration updated to point to the new canonical URLs. A Thanks, Adam # References: ## General ### Initial 2016 issues [peps#2](https://github.com/python/peps/issues/2) [peps#3](https://github.com/python/peps/issues/3) [peps#17](https://github.com/python/peps/pull/17) [peps#25](https://github.com/python/peps/pull/25) ### This proposal’s implementation [peps#1930](https://github.com/python/peps/pull/1930) [peps#1931](https://github.com/python/peps/pull/1931) [peps#1932](https://github.com/python/peps/pull/1932) [peps#1933](https://github.com/python/peps/pull/1933) [peps#1934](https://github.com/python/peps/pull/1934) ## Quality of life improvements ### Syntax highlighting and ` .. code-block::` [pythondotorg#1063](https://github.com/python/pythondotorg/pull/1063) [pythondotorg#1206](https://github.com/python/pythondotorg/issues/1206) [pythondotorg#1638](https://github.com/python/pythondotorg/pull/1638) [peps#159](https://github.com/python/peps/issues/159) [comment in peps#1571](https://github.com/python/peps/pull/1571#discussion_r478701944) [peps#1577](https://github.com/python/peps/pull/1577) ### SVG [peps#701](https://github.com/python/peps/issues/701) ### Typographic quotation marks [peps#165](https://github.com/python/peps/issues/165) ### Footer information [pythondotorg#1564](https://github.com/python/pythondotorg/issues/1564) ### Inter-Sphinx [comment in peps#2](https://github.com/python/peps/issues/2#issuecomment-339195595) ## Current issues ### List styling [peps#1387](https://github.com/python/peps/issues/1387) ### Image updating [pythondotorg#824](https://github.com/python/pythondotorg/issues/824) [pythondotorg#1556](https://github.com/python/pythondotorg/pull/1556) ---
**Author:** ![dustin](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/dustin/32/149_2.png) [@dustin](https://discuss.python.org/u/dustin)\ **Post date:** [September 23, 2021, 2:37pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/2 "2021-09-23T14:37:45Z")
I’m generally in favor – there’s been a lot of requests for improvements here, and this gives us some additional flexibility. Thanks for working on this! > [@AA-Turner](#): > > It would be possible to use the output from the proposed rendering system and import it into `pythondotorg` . I would argue however that this would be the worst of both worlds, as a great deal of complexity is added, and none is removed. I wonder if there’s some middle ground here. I think there are advantages to having PEPs be part of [python.org](http://python.org) (e.g. they get the same “official” styling, links to the rest of [python.org](http://python.org), etc). Maybe @EWDurbin has some ideas? ---
**Author:** ![EWDurbin](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ewdurbin/32/3103_2.png) [@EWDurbin](https://discuss.python.org/u/EWDurbin)\ **Post date:** [September 23, 2021, 4:10pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/3 "2021-09-23T16:10:12Z")
I heartily agree with the need to address the pipeline for PEP publication and support this effort. As far retaining placement under [PEP 0 -- Index of Python Enhancement Proposals (PEPs) | Python.org](https://www.python.org/dev/peps/), I’m flexible as to how we try to accomplish this. However, similar to documentation and the developers guide it is hard to make a case that retaining styling consistency and navigation are as important as maintainability. We can easily front [peps.python.org](http://peps.python.org) or similar with our CDN or work with redirects to ensure that existing links aren’t permanently broken. My concern with trying to import the resulting documents into [python.org](http://python.org)’s CMS, which is not a huge departure from what we do now, is that it then creates two places for this information… which is canonical? ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [September 23, 2021, 6:59pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/4 "2021-09-23T18:59:53Z")
> [@EWDurbin](#): > > We can easily front [peps.python.org](http://peps.python.org) or similar with our CDN or work with redirects to ensure that existing links aren’t permanently broken. My concern with trying to import the resulting documents into [python.org](http://python.org)’s CMS, which is not a huge departure from what we do now, is that it then creates two places for this information… which is canonical? I would advocate for hosting on Read the Docs and not doing any CMS importing (if I’m understanding what Ee is asking). ---
**Author:** ![AA-Turner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aa-turner/32/4979_2.png) [@AA-Turner](https://discuss.python.org/u/AA-Turner)\ **Post date:** [September 23, 2021, 7:03pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/5 "2021-09-23T19:03:11Z")
> [@dustin](#): > > I wonder if there’s some middle ground here. I think there are advantages to having PEPs be part of `python.org` (e.g. they get the same “official” styling, links to the rest of `python.org`, etc). I suppose it gets to the point of what `python.org` is intended for – if the website is primarily an advert for the project as a whole and the charity, then it may make sense (and I would argue that it does) to put PEPs on a subdomain also, similar to `docs.python.org`. It seems at the monement that there seems to be a good precendent for hosting things on subdomains as opposed to the main site, but I guess PEPs might be “sufficiently different”, and warrant staying on the main website. _list of subdomain things:_ - issue tracker (`bugs.python.org`) - automatic builds (`buildbot.python.org`) - developer’s guide (`devguide.python.org`) - documentation (`docs.python.org`) - email list archives (`mail.python.org`) - packaging guide (`packaging.python.org`) - performance statistics (`speed.python.org`) - status (`status.python.org`) - wiki (`wiki.python.org`) > [@EWDurbin](#): > > My concern with trying to import the resulting documents into `python.org's` CMS, which is not a huge departure from what we do now, is that it then creates two places for this information… which is canonical? I’d agree with this – there’s no point doing both, there should be one canonical source – for the reasons in my original paper I would favour `peps.python.org`, but we shouldn’t create that if we continue integrating into the main `python.org` website. A ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [September 23, 2021, 7:08pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/6 "2021-09-23T19:08:25Z")
> [@AA-Turner](#): > > I suppose it gets to the point of what `python.org` is intended for – if the website is primarily an advert for the project as a whole and the charity, then it may make sense (and I would argue that it does) to put PEPs on a subdomain also, similar to `docs.python.org` . My only concern is that the existing URL structure for PEPs ([https://www.python.org/dev/peps/pep-0XXX/](https://www.python.org/dev/peps/pep-0XXX/)) continues to work, as breaking that would wreck _so_ much stuff. ---
**Author:** ![AA-Turner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aa-turner/32/4979_2.png) [@AA-Turner](https://discuss.python.org/u/AA-Turner)\ **Post date:** [September 23, 2021, 7:18pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/7 "2021-09-23T19:18:19Z")
> [@pf\_moore](#): > > My only concern is that the existing URL structure for PEPs ([https://www.python.org/dev/peps/pep-0XXX/](https://www.python.org/dev/peps/pep-0XXX/)) continues to work, as breaking that would wreck _so_ much stuff. Yes, this should be a zero-disruption change. I’d propose redirects here (similar to how `https://python.org/peps/pep-0100.html` goes to the right place) – would you prefer that the links don’t change at all? A reverse proxy could be used to the `/dev/peps` namespace, but that would likely require a lot more work (& not sure if it’s even currently possible!) A ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [September 23, 2021, 9:10pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/8 "2021-09-23T21:10:20Z")
> [@AA-Turner](#): > > I’d propose redirects here (similar to how `https://python.org/peps/pep-0100.html` goes to the right place) – would you prefer that the links don’t change at all? I don’t think I care as long as the links work. ---
**Author:** ![steven.daprano](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steven.daprano/32/1083_2.png) [@steven.daprano](https://discuss.python.org/u/steven.daprano)\ **Post date:** [September 24, 2021, 3:26am UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/9 "2021-09-24T03:26:22Z")
Hi Adam, I have difficulty telling whether I should care about this proposal or not. You say that it will impact anyone who reads, writes or reviews PEPs. I often read PEPs, and have sometimes written PEPs. Can you expain how this proposal will affect people like me? I’ll be honest, on the basis of a fast read through, it sounds more like a back-end infrastructure change that should have a minimal affect on PEP readers or writers. Is this wrong? Some user-stories might help. Suppose I want to read a specific PEP, and I know either some keywords or the PEP number. Right now, I google for “Python PEP …” or (more rarely) go straight to [PEP 0 -- Index of Python Enhancement Proposals (PEPs) | Python.org](https://www.python.org/dev/peps/) and look at the PEP. How will your proposal change this? I want to read PEPs about a keyword, say, anything to do with dicts. My workflow here is more or less the same. How will your proposal affect me? As a reader of PEPs, is the only change that the URL will change? The last time I actively wrote a PEP was back in the hg days. The change to github derailed me like a car on a railway line and due to many work/ life/technology factors I have not yet caught up with the brave new world of github. (I expect that I will probably do so three months before we shift to a new and improved system _wink_) But, if I were to write another PEP, how would this change impact me? ---
**Author:** ![steven.daprano](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steven.daprano/32/1083_2.png) [@steven.daprano](https://discuss.python.org/u/steven.daprano)\ **Post date:** [September 24, 2021, 3:30am UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/10 "2021-09-24T03:30:23Z")
Hi Paul, You say: "My only concern is that the existing URL structure for PEPs ([https://www.python.org/dev/peps/pep-0XXX/](https://www.python.org/dev/peps/pep-0XXX/)) continues to work, as breaking that would wreck _so_ much stuff." Are you referring to link rot? Or did you have some other form of breakage in mind? ---
**Author:** ![pf\_moore](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/pf_moore/32/35_2.png) [@pf\_moore](https://discuss.python.org/u/pf_moore)\ **Post date:** [September 24, 2021, 3:52am UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/11 "2021-09-24T03:52:50Z")
I’m not sure what you mean by “link rot”, but I’m referring to all the places that link to PEPs, as well as things like “smart bookmarks” that construct a URL from a PEP number based on the existing URL names. @AA-Turner says that the existing URLs will continue to work though, so that doesn’t seem to be an issue. As a PEP reader and occasional author, I agree with your other comment - I can’t see any way in which this change would affect me. But I choose to interpret that as a good thing, the change can be made without harming people in my position. ---
**Author:** ![steven.daprano](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/steven.daprano/32/1083_2.png) [@steven.daprano](https://discuss.python.org/u/steven.daprano)\ **Post date:** [September 24, 2021, 4:07am UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/12 "2021-09-24T04:07:43Z")
Paul: “”" I’m not sure what you mean by “link rot” “”" > **[Link rot](https://en.wikipedia.org/wiki/Link_rot)** > > Link rot (also called link death, link breaking, or reference rot) is the phenomenon of hyperlinks tending over time to cease to point to their originally targeted file, web page, or server due to that resource being relocated to a new address or becoming permanently unavailable. A link that no longer points to its target, often called a broken or dead link (or sometimes orphan link), is a specific form of dangling pointer. > The rate of link rot is a subject of study and research due to its signi... We’re talking about the same thing 🙂 ---
**Author:** ![hugovk](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hugovk/32/14505_2.png) [@hugovk](https://discuss.python.org/u/hugovk)\ **Post date:** [September 24, 2021, 7:23am UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/13 "2021-09-24T07:23:08Z")
Link rot is a valid and extremely important concern, and shouldn’t happen: - Either PEPs will remain at `https://www.python.org/dev/peps/pep-0008/` - Or they will be moved to something like `https://peps.python.org/pep-0008/` * * * As a reader, nothing should really change too much. The most noticeable will be things like styling, formatting (I’m looking forward to coloured syntax highlighting!), menus. For example, compare the demo page: - Demo: [PEP 8 – Style Guide for Python Code | peps.python.org](https://peps-aaturner.readthedocs.io/pep-0008/) - Current: [PEP 8 -- Style Guide for Python Code | Python.org](https://www.python.org/dev/peps/pep-0008/) Because it’ll be using Sphinx/RTD in the background, which are well known and supported in the Python community, this should make things easier to maintain and customise. Is it fair to say it should be more accessible? * * * For the googling use case, this is my usual way of finding PEPs too. Google will start to pick up the new links and update their results, and because we’ll have redirects from the old links to the new location, things will still work in the meantime. * * * As a PEP author, your workflow would be the same. You will create a PR at [GitHub - python/peps: Python Enhancement Proposals](https://github.com/python/peps), and it’ll be reviewed as usual. When merged it’ll be auto-deployed as before (although in a different manner, but as a PEP author you don’t need to worry about how). * * * One new and really useful benefit, RTD can create preview builds of docs for PRs. That way the author and reviewers can check the changes look good and render properly (little things that can be easy to get wrong, like code formatting, tables, links). For example, this [PR from another project](https://github.com/python-pillow/Pillow/pull/5713) has a CI check from RTD and a build preview - " **docs/readthedocs.org:pillow** — Read the Docs build succeeded!" - [Installation — Pillow (PIL Fork) 8.4.0.dev0 documentation](https://pillow--5713.org.readthedocs.build/en/5713/installation.html#continuous-integration-targets) As it happens for this PR, the reviewer saw from the preview that some table cells could be merged, and suggested such a change. After applying, we checked it worked as desired, and merged. ---
**Author:** ![AA-Turner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aa-turner/32/4979_2.png) [@AA-Turner](https://discuss.python.org/u/AA-Turner)\ **Post date:** [September 24, 2021, 1:03pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/14 "2021-09-24T13:03:19Z")
Thanks for your comments Steven – Hugo covered most of I would have said far more eloquently! > [@steven.daprano](#): > > ou say that it will impact anyone who reads, writes or reviews PEPs. I often read PEPs, and have sometimes written PEPs. Can you expain how this proposal will affect people like me? To re-emphasise though - changes for readers are that there would be a different canonical URL (with auto-redirects from current URLs) and different styling. - changes for reviewers are the above (as reviewers read PEPs) and potential quality-of-life improvements like auto-built previews, to enable reviewing what the rendered document would look like (for potential rendering issues, like the table markup case Hugo mentioned) - changes for writers are the above (as writers read and review their own PEPs) and also that PEPs would be deployed on merge, instead of waiting for a synchronisation process. As with Paul/Hugo these aren’t massive changes, but I wanted to perhaps be overly conservative in what constitutes a “change” in my original paper. A ---
**Author:** ![ncoghlan](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ncoghlan/32/14266_2.png) [@ncoghlan](https://discuss.python.org/u/ncoghlan)\ **Post date:** [October 4, 2021, 1:51am UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/15 "2021-10-04T01:51:12Z")
I’m in favour of the change, using the “new canonical hosting URL with redirects” approach. With a distinct subdomain we don’t have to keep the styling identical to either the regular docs or the main [python.org](http://python.org) site - we can use whichever hybrid of the two we feel makes the most sense for the content. For hosting, I would suggest keeping the URLs behind the main [python.org](http://python.org) CDN, but switching the app layer backend to RTD should be fine (I don’t know the actual PEP reading traffic volume, but even with only text documents I assume it’s high enough that offloading it all to RTD’s CDN would be a bad idea, whereas for the PSF it’s a drop in the ocean compared to PyPI’s traffic. That said, given some of the project docs that RTD hosts, the PEP traffic would probably be at worst a drop in a large sea for them, so the exact details likely don’t matter much) For record keeping purposes, the proposal posted here should probably also be recorded as a PEP in its own right. ---
**Author:** ![AA-Turner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aa-turner/32/4979_2.png) [@AA-Turner](https://discuss.python.org/u/AA-Turner)\ **Post date:** [November 1, 2021, 5:24pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/16 "2021-11-01T17:24:44Z")
Apologies for the delay, I had intended to leave this for a week or so to see if any others had thoughts and then events conspired against me. People seem largely in favour of this (or not against it) – so perhaps the next step is to formally write this up as a meta-PEP as Nick suggested. I don’t know what the best process is here – if I should just have a stab at putting the content from the original post (perhaps with stronger language about how redirects etc will be preserved) into the PEP template and submit it as a PR? [PEP 1 - Submitting a PEP](https://www.python.org/dev/peps/pep-0001/#submitting-a-pep) suggests that I would need a sponsor for the PEP – I know a number of the participants in the various discussions around this have been valid PEP sponsors – advice on this would be appreciated. A ---
**Author:** ![brettcannon](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/brettcannon/32/34895_2.png) [@brettcannon](https://discuss.python.org/u/brettcannon)\ **Post date:** [November 1, 2021, 8:04pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/17 "2021-11-01T20:04:01Z")
> [@AA-Turner](#): > > I don’t know what the best process is here – if I should just have a stab at putting the content from the original post (perhaps with stronger language about how redirects etc will be preserved) into the PEP template and submit it as a PR? I would draft it and post the draft somewhere to discuss it (here would be fine since this isn’t really a python-ideas sort of thing). You shouldn’t post a PEP via a PR without a sponsor. > [@AA-Turner](#): > > [PEP 1 - Submitting a PEP](https://www.python.org/dev/peps/pep-0001/#submitting-a-pep) suggests that I would need a sponsor for the PEP – I know a number of the participants in the various discussions around this have been valid PEP sponsors – advice on this would be appreciated. You will need a sponsor. You can either see if one comes up here or write the PEP and explicitly mention in the posting of the draft PEP that you’re looking for a sponsor. ---
**Author:** ![Mariatta](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/mariatta/32/30581_2.png) [@Mariatta](https://discuss.python.org/u/Mariatta)\ **Post date:** [November 1, 2021, 8:32pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/18 "2021-11-01T20:32:57Z")
I’ll be happy to sponsor this PEP @AA-Turner. ---
**Author:** ![hugovk](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/hugovk/32/14505_2.png) [@hugovk](https://discuss.python.org/u/hugovk)\ **Post date:** [November 6, 2021, 5:26pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/19 "2021-11-06T17:26:22Z")
@AA-Turner Let me know if I can help draft. Feel free to [DM me here](https://discourse.jupyter.org/t/how-to-send-messages-in-discourse/646) or Discord (`hugovk#6128`), or open an issue on your or my `peps` forks. ---
**Author:** ![AA-Turner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aa-turner/32/4979_2.png) [@AA-Turner](https://discuss.python.org/u/AA-Turner)\ **Post date:** [November 30, 2021, 5:59pm UTC](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774/20 "2021-11-30T17:59:56Z")
With thanks to Hugo for a some initial reviews, please find the current draft text below, and a [rendered preview](https://aa-turner.github.io/peps/pep-9999/) online. (There is currently an ugly duplication of the _footnotes_ and _references_ sections, but [PR 2155](https://github.com/python/peps/pull/2155) will fix this.) A ```auto PEP: 9999 Title: PEP Infrastructure Process Author: Adam Turner Sponsor: Mariatta Discussions-To: https://discuss.python.org/t/10774 Status: Draft Type: Process Content-Type: text/x-rst Created: 01-Nov-2021 Post-History: 23-Sep-2021 Abstract ======== This PEP addresses the infrastructure around rendering PEP files from reStructuredText_ files to HTML webpages. We aim to specify a self-contained and maintainable solution for PEP readers, authors, and editors. Motivation ========== As of November 2021, Python Enhancement Proposals (PEPs) are rendered in a multi-system, multi-stage process. A continuous integration (CI) task runs a docutils_ script to render all PEP files individually. The CI task then uploads a tar archive to a server, where it is retrieved and rendered into the `python.org`_ website periodically. This places a constraint on the `python.org`_ website to handle raw HTML uploads and handle PEP rendering, and makes the appropriate place to raise issues unclear in some cases [1]_. This PEP provides a specification for self-contained rendering of PEPs. This would: * reduce the amount of distributed configuration for supporting PEPs * enable quality-of-life improvements for those who read, write, and review PEPs * solve a number of outstanding issues, and lay the path for improvements * save volunteer maintainers' time We propose that PEPs are accessed through ``peps.python.org`` at the top-level namespace (for example ``peps.python.org/pep-0008/``), and that all custom tooling to support rendering PEPs is hosted in the `python/peps`_ repository. Rationale ========= Simplifying and Centralising Infrastructure ------------------------------------------- As of November 2021, to locally render a PEP file, a PEP author or editor needs to create a full local instance of the `python.org`_ website and run a number of disparate scripts, following documentation_ that lives outside of the `python/peps`_ repository. The proposed implementation provides a single Makefile_ and a Python script to render all PEP files, with options to target a web-server or local filesystem environment. Using a single repository to host all tooling will clarify where to raise issues, reducing volunteer time spent in triage. Simplified and centralised tooling may also reduce the barrier to entry to further improvements, as the scope of the PEP rendering infrastructure is well defined. Quality-of-Life Improvements and Resolving Issues ------------------------------------------------- There are several requests for additional features in reading PEPs, such as: * syntax highlighting [2]_ * use of ``.. code-block::`` directives [2]_ * support for SVG images [3]_ * typographic quotation marks [4]_ * additional footer information [5]_ * intersphinx functionality [6]_ These are "easy wins" from this proposal, and would serve to improve the quality-of-life for consumers of PEPs (including reviewers and writers). For example, the reference implementation no longer requires a scheduled render process to run, instead initiating rendering on every commit to the `python/peps`_ repository. Equally, there are a small number of broken items, for example list styles not being respected or support for updating images being challenging with the system [7]_. These would be solved by default in the proposal. Commercial providers such as `Read the Docs`_ can additionally enhance this experience, for example by providing rendered previews of changes in pull requests. Specification ============= The proposed specification for rendering the PEP files to HTML is as per the `reference implementation`_. That the HTML files should be made available under ``peps.python.org``. The rendered files may be hosted just as static files, and behind a content delivery network (CDN). The following redirect rules must be created from the `python.org`_ domain: * /peps/ -> https://peps.python.org/ * /dev/peps/ -> https://peps.python.org/ * /peps/(.*)\.html -> https://peps.python.org/$1 * /dev/peps/(.*) -> https://peps.python.org/$1 .. code-block:: nginx location ~ ^/dev/peps/?(.*)$ { return 308 https://peps.python.org/$1/; } location ~ ^/peps/?(.*)\.html$ { return 308 https://peps.python.org/$1/; } location ^/(dev/)?peps(/.*)?$ { return 308 https://peps.python.org/; } Redirects must be implemented to preserve `URL fragments`_ for backward compatability purposes. Backwards Compatibility ======================= Due to server-side redirects to new canonical URLs, there are no backwards compatability concerns. Links in previously published materials referring to any old URL scheme will be guaranteed to work. Security Implications ===================== No security implications, and the main `python.org`_ website will no longer take raw HTML uploads, closing a potential threat vector. How to Teach This ================= The new canonical URLs will be publicised in the documentation. However, this is mainly a backend infrastructure change, and there should be minimal end-user impact. Reference Implementation ======================== The proposed implementation has been merged into the `python/peps`_ repository in a series of pull requests [8]_. This automatically renders all PEPs on every commit. Rejected Ideas ============== It would likely be possible to amend the current (as of November 2021) rendering process to include a lot of the quality-of-life improvements and issue mitigations mentioned above. However, we do not believe that this would solve the distributed tooling issue. It would be possible to use the output from the proposed rendering system and import it into `python.org`_. We would argue however that this would be the worst of both worlds, as a great deal of complexity is added, and none is removed. Open Issues =========== None. Acknowledgements ================ Thanks to Hugo van Kemenade, Pablo Galindo Salgado, and Éric Araujo for support since April 2020. Footnotes ========= .. _documentation: https://pythondotorg.readthedocs.io/pep_generation.html .. _docutils: https://docutils.sourceforge.io .. _Makefile: https://www.gnu.org/software/make/manual/make.html#Introduction .. _python.org: https://www.python.org .. _python/peps: https://github.com/python/peps .. _Read the Docs: https://readthedocs.org .. _reStructuredText: https://docutils.sourceforge.io/rst.html .. _URL fragments: https://url.spec.whatwg.org/#concept-url-fragment .. [1] For example, `pythondotorg#1024 `__, `pythondotorg#1038 `__, `pythondotorg#1387 `__, `pythondotorg#1388 `__, `pythondotorg#1393 `__, `pythondotorg#1564 `__, `pythondotorg#1913 `__, .. [2] Requested: `pythondotorg#1063 `__, `pythondotorg#1206 `__, `pythondotorg#1638 `__, `peps#159 `__, `comment in peps#1571 `__, `peps#1577 `__, .. [3] Requested: `peps#701 `__.. [4] Requested: `peps#165 `__.. [5] Requested: `pythondotorg#1564 `__.. [6] Requested: `comment in peps#2 `__.. [7] As of November 2021, see `peps#1387 `__, `pythondotorg#824 `__, `pythondotorg#1556 `__, .. [8] Implementation PRs: `peps#1930 `__, `peps#1931 `__, `peps#1932 `__, `peps#1933 `__, `peps#1934 `__ Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: ``` [Next page](https://discuss.python.org/t/pep-676-pep-infrastructure-process/10774.md?page=2)