Request for comment: Making PEP rendering self-contained

One-line summary

This discusses rationale for creating peps.python.org as a stand alone site to host PEPs, replacing the current systems.

Summary:

Python Enhancement Proposals (PEPs) are currently rendered by running docutils on each PEP document and are refreshed on python.org/dev/peps every 15 minutes.

This proposal would seek to make rendering of PEPs self-contained. This would:

  • reduce the amount of distributed configuration for supporting PEPs
  • enable quality-of-life improvements for those who read, write, and review PEPs
  • solve a number of outstanding issues, and lay the path for potential improvements
  • save volunteer time in maintaining the current system

The proposed end state is that PEPs are accessed through peps.python.org at the top-level namespace (for example peps.python.org/pep-9999), and all tooling to support rendering PEPs is hosted in the python/peps repository.

Current status:

The proposed implementation has been merged into the python/peps repository, and a rendered preview of all PEPs is available (python.github.io/peps). I was asked to write up this briefing by the PEP editors for consideration.

Reducing distributed configuration and simplifying tooling:

Currently, three Python repositories are involved in orchestrating the rendering process, using three separate technologies (docutils, Django, Saltstack). The proposed implementation renders the PEPs using Sphinx and a custom Sphinx plugin.

As it is proposed to stand alone, the integration with pythondotorg and associated update machinery can be removed, reducing cognitive load when reviewing issues with PEP rendering. Use of CI will also surface all build logs in one place, again useful when investigating problems.

By removing configuration in these other projects, volunteer time does not need to be spent maintaining more fragile linkages and complex distributed systems.

It may also reduce the barrier to entry to adding new features, as the scope of the PEP rendering tooling is well defined.

Quality-of-life improvements and resolving issues:

There are a number of requests for additional features in viewing PEPs, including syntax highlighting, .. code-block:: directives, support for SVG images, typographic quotation marks, additional footer information, and inter-sphinx functionality. These are “easy wins” from this proposal, and would serve to improve the quality-of-life for consumers of PEPs (including reviewers and writers).

Equally, there are a small number of broken items, for example list styles not being respected or support for updating images being challenging with the current pythondotorg system. These would be solved by default in the proposal.

Of note, I found a lot more such items that had been fixed in various ways by volunteers – which shows both their dedication and that such time investigating a fix might be better used elsewhere.

Potential alternatives

It would likely be possible to amend the current rendering process to include a lot of the quality-of-life improvements and issue mitigations mentioned above. However, I do not believe that this would solve the distributed tooling issue.

It would be possible to use the output from the proposed rendering system and import it into pythondotorg. I would argue however that this would be the worst of both worlds, as a great deal of complexity is added, and none is removed.

Specific proposed implementation

As the proposed system uses Sphinx, any static file hosting solution could be used for peps.python.org – this could also be behind a CDN for example as there is no dynamic content, and the only optional JavaScript is for a single cosmetic change.

Commercial vendors such as Read the Docs, Inc can provide additional services such as preview rendering, which may align with the quality-of-life piece.

The rendered PEPs would be made available at peps.python.org, with the current redirect configuration updated to point to the new canonical URLs.

A

Thanks,
Adam

References:

General

Initial 2016 issues

peps#2
peps#3
peps#17
peps#25

This proposal’s implementation

peps#1930
peps#1931
peps#1932
peps#1933
peps#1934

Quality of life improvements

Syntax highlighting and .. code-block::

pythondotorg#1063
pythondotorg#1206
pythondotorg#1638
peps#159
comment in peps#1571
peps#1577

SVG

peps#701

Typographic quotation marks

peps#165

Footer information

pythondotorg#1564

Inter-Sphinx

comment in peps#2

Current issues

List styling

peps#1387

Image updating

pythondotorg#824
pythondotorg#1556

5 Likes

I’m generally in favor – there’s been a lot of requests for improvements here, and this gives us some additional flexibility. Thanks for working on this!

I wonder if there’s some middle ground here. I think there are advantages to having PEPs be part of python.org (e.g. they get the same “official” styling, links to the rest of python.org, etc). Maybe @EWDurbin has some ideas?

1 Like

I heartily agree with the need to address the pipeline for PEP publication and support this effort.

As far retaining placement under PEP 0 -- Index of Python Enhancement Proposals (PEPs) | Python.org, I’m flexible as to how we try to accomplish this. However, similar to documentation and the developers guide it is hard to make a case that retaining styling consistency and navigation are as important as maintainability.

We can easily front peps.python.org or similar with our CDN or work with redirects to ensure that existing links aren’t permanently broken. My concern with trying to import the resulting documents into python.org’s CMS, which is not a huge departure from what we do now, is that it then creates two places for this information… which is canonical?

1 Like

I would advocate for hosting on Read the Docs and not doing any CMS importing (if I’m understanding what Ee is asking).

1 Like

I suppose it gets to the point of what python.org is intended for – if the website is primarily an advert for the project as a whole and the charity, then it may make sense (and I would argue that it does) to put PEPs on a subdomain also, similar to docs.python.org.

It seems at the monement that there seems to be a good precendent for hosting things on subdomains as opposed to the main site, but I guess PEPs might be “sufficiently different”, and warrant staying on the main website.

list of subdomain things:

  • issue tracker (bugs.python.org)
  • automatic builds (buildbot.python.org)
  • developer’s guide (devguide.python.org)
  • documentation (docs.python.org)
  • email list archives (mail.python.org)
  • packaging guide (packaging.python.org)
  • performance statistics (speed.python.org)
  • status (status.python.org)
  • wiki (wiki.python.org)

I’d agree with this – there’s no point doing both, there should be one canonical source – for the reasons in my original paper I would favour peps.python.org, but we shouldn’t create that if we continue integrating into the main python.org website.

A

My only concern is that the existing URL structure for PEPs (https://www.python.org/dev/peps/pep-0XXX/) continues to work, as breaking that would wreck so much stuff.

2 Likes

Yes, this should be a zero-disruption change.

I’d propose redirects here (similar to how https://python.org/peps/pep-0100.html goes to the right place) – would you prefer that the links don’t change at all?

A reverse proxy could be used to the /dev/peps namespace, but that would likely require a lot more work (& not sure if it’s even currently possible!)

A

1 Like

I don’t think I care as long as the links work.

1 Like

Hi Adam,

I have difficulty telling whether I should care about this proposal or

not. You say that it will impact anyone who reads, writes or reviews

PEPs. I often read PEPs, and have sometimes written PEPs. Can you expain

how this proposal will affect people like me?

I’ll be honest, on the basis of a fast read through, it sounds more like

a back-end infrastructure change that should have a minimal affect on

PEP readers or writers. Is this wrong?

Some user-stories might help.

Suppose I want to read a specific PEP, and I know either some keywords

or the PEP number. Right now, I google for “Python PEP …” or (more

rarely) go straight to PEP 0 -- Index of Python Enhancement Proposals (PEPs) | Python.org and look at the

PEP. How will your proposal change this?

I want to read PEPs about a keyword, say, anything to do with dicts. My

workflow here is more or less the same. How will your proposal affect

me?

As a reader of PEPs, is the only change that the URL will change?

The last time I actively wrote a PEP was back in the hg days. The change

to github derailed me like a car on a railway line and due to many work/

life/technology factors I have not yet caught up with the brave new

world of github. (I expect that I will probably do so three months

before we shift to a new and improved system wink)

But, if I were to write another PEP, how would this change impact me?

Hi Paul,

You say:

"My only concern is that the existing URL structure for PEPs

(https://www.python.org/dev/peps/pep-0XXX/) continues to work, as

breaking that would wreck so much stuff."

Are you referring to link rot? Or did you have some other form of

breakage in mind?

I’m not sure what you mean by “link rot”, but I’m referring to all the places that link to PEPs, as well as things like “smart bookmarks” that construct a URL from a PEP number based on the existing URL names. @AA-Turner says that the existing URLs will continue to work though, so that doesn’t seem to be an issue. As a PEP reader and occasional author, I agree with your other comment - I can’t see any way in which this change would affect me. But I choose to interpret that as a good thing, the change can be made without harming people in my position.

1 Like

Paul:

“”"
I’m not sure what you mean by “link rot”
“”"

We’re talking about the same thing :slight_smile:

Link rot is a valid and extremely important concern, and shouldn’t happen:

  • Either PEPs will remain at https://www.python.org/dev/peps/pep-0008/

  • Or they will be moved to something like https://peps.python.org/pep-0008/

    • AND redirects will be put in place from the old https://www.python.org/dev/peps/pep-0008/ to https://peps.python.org/pep-0008/, so anyone visiting the old one will be automatically redirected.

As a reader, nothing should really change too much. The most noticeable will be things like styling, formatting (I’m looking forward to coloured syntax highlighting!), menus. For example, compare the demo page:

Because it’ll be using Sphinx/RTD in the background, which are well known and supported in the Python community, this should make things easier to maintain and customise. Is it fair to say it should be more accessible?


For the googling use case, this is my usual way of finding PEPs too. Google will start to pick up the new links and update their results, and because we’ll have redirects from the old links to the new location, things will still work in the meantime.


As a PEP author, your workflow would be the same. You will create a PR at GitHub - python/peps: Python Enhancement Proposals, and it’ll be reviewed as usual. When merged it’ll be auto-deployed as before (although in a different manner, but as a PEP author you don’t need to worry about how).


One new and really useful benefit, RTD can create preview builds of docs for PRs. That way the author and reviewers can check the changes look good and render properly (little things that can be easy to get wrong, like code formatting, tables, links).

For example, this PR from another project has a CI check from RTD and a build preview

As it happens for this PR, the reviewer saw from the preview that some table cells could be merged, and suggested such a change. After applying, we checked it worked as desired, and merged.

2 Likes

Thanks for your comments Steven – Hugo covered most of I would have said far more eloquently!

To re-emphasise though

  • changes for readers are that there would be a different canonical URL (with auto-redirects from current URLs) and different styling.
  • changes for reviewers are the above (as reviewers read PEPs) and potential quality-of-life improvements like auto-built previews, to enable reviewing what the rendered document would look like (for potential rendering issues, like the table markup case Hugo mentioned)
  • changes for writers are the above (as writers read and review their own PEPs) and also that PEPs would be deployed on merge, instead of waiting for a synchronisation process.

As with Paul/Hugo these aren’t massive changes, but I wanted to perhaps be overly conservative in what constitutes a “change” in my original paper.

A

3 Likes

I’m in favour of the change, using the “new canonical hosting URL with redirects” approach. With a distinct subdomain we don’t have to keep the styling identical to either the regular docs or the main python.org site - we can use whichever hybrid of the two we feel makes the most sense for the content.

For hosting, I would suggest keeping the URLs behind the main python.org CDN, but switching the app layer backend to RTD should be fine (I don’t know the actual PEP reading traffic volume, but even with only text documents I assume it’s high enough that offloading it all to RTD’s CDN would be a bad idea, whereas for the PSF it’s a drop in the ocean compared to PyPI’s traffic. That said, given some of the project docs that RTD hosts, the PEP traffic would probably be at worst a drop in a large sea for them, so the exact details likely don’t matter much)

For record keeping purposes, the proposal posted here should probably also be recorded as a PEP in its own right.

2 Likes