PEP 676: PEP Infrastructure Process

PEPs
#1

One-line summary

This discusses rationale for creating peps.python.org as a stand alone site to host PEPs, replacing the current systems.

Summary:

Python Enhancement Proposals (PEPs) are currently rendered by running docutils on each PEP document and are refreshed on python.org/dev/peps every 15 minutes.

This proposal would seek to make rendering of PEPs self-contained. This would:

  • reduce the amount of distributed configuration for supporting PEPs
  • enable quality-of-life improvements for those who read, write, and review PEPs
  • solve a number of outstanding issues, and lay the path for potential improvements
  • save volunteer time in maintaining the current system

The proposed end state is that PEPs are accessed through peps.python.org at the top-level namespace (for example peps.python.org/pep-9999), and all tooling to support rendering PEPs is hosted in the python/peps repository.

Current status:

The proposed implementation has been merged into the python/peps repository, and a rendered preview of all PEPs is available (python.github.io/peps). I was asked to write up this briefing by the PEP editors for consideration.

Reducing distributed configuration and simplifying tooling:

Currently, three Python repositories are involved in orchestrating the rendering process, using three separate technologies (docutils, Django, Saltstack). The proposed implementation renders the PEPs using Sphinx and a custom Sphinx plugin.

As it is proposed to stand alone, the integration with pythondotorg and associated update machinery can be removed, reducing cognitive load when reviewing issues with PEP rendering. Use of CI will also surface all build logs in one place, again useful when investigating problems.

By removing configuration in these other projects, volunteer time does not need to be spent maintaining more fragile linkages and complex distributed systems.

It may also reduce the barrier to entry to adding new features, as the scope of the PEP rendering tooling is well defined.

Quality-of-life improvements and resolving issues:

There are a number of requests for additional features in viewing PEPs, including syntax highlighting, .. code-block:: directives, support for SVG images, typographic quotation marks, additional footer information, and inter-sphinx functionality. These are “easy wins” from this proposal, and would serve to improve the quality-of-life for consumers of PEPs (including reviewers and writers).

Equally, there are a small number of broken items, for example list styles not being respected or support for updating images being challenging with the current pythondotorg system. These would be solved by default in the proposal.

Of note, I found a lot more such items that had been fixed in various ways by volunteers – which shows both their dedication and that such time investigating a fix might be better used elsewhere.

Potential alternatives

It would likely be possible to amend the current rendering process to include a lot of the quality-of-life improvements and issue mitigations mentioned above. However, I do not believe that this would solve the distributed tooling issue.

It would be possible to use the output from the proposed rendering system and import it into pythondotorg. I would argue however that this would be the worst of both worlds, as a great deal of complexity is added, and none is removed.

Specific proposed implementation

As the proposed system uses Sphinx, any static file hosting solution could be used for peps.python.org – this could also be behind a CDN for example as there is no dynamic content, and the only optional JavaScript is for a single cosmetic change.

Commercial vendors such as Read the Docs, Inc can provide additional services such as preview rendering, which may align with the quality-of-life piece.

The rendered PEPs would be made available at peps.python.org, with the current redirect configuration updated to point to the new canonical URLs.

A

Thanks,
Adam

References:

General

Initial 2016 issues

peps#2
peps#3
peps#17
peps#25

This proposal’s implementation

peps#1930
peps#1931
peps#1932
peps#1933
peps#1934

Quality of life improvements

Syntax highlighting and .. code-block::

pythondotorg#1063
pythondotorg#1206
pythondotorg#1638
peps#159
comment in peps#1571
peps#1577

SVG

peps#701

Typographic quotation marks

peps#165

Footer information

pythondotorg#1564

Inter-Sphinx

comment in peps#2

Current issues

List styling

peps#1387

Image updating

pythondotorg#824
pythondotorg#1556

7 Likes
#2

I’m generally in favor – there’s been a lot of requests for improvements here, and this gives us some additional flexibility. Thanks for working on this!

I wonder if there’s some middle ground here. I think there are advantages to having PEPs be part of python.org (e.g. they get the same “official” styling, links to the rest of python.org, etc). Maybe @EWDurbin has some ideas?

1 Like
#3

I heartily agree with the need to address the pipeline for PEP publication and support this effort.

As far retaining placement under PEP 0 -- Index of Python Enhancement Proposals (PEPs) | Python.org, I’m flexible as to how we try to accomplish this. However, similar to documentation and the developers guide it is hard to make a case that retaining styling consistency and navigation are as important as maintainability.

We can easily front peps.python.org or similar with our CDN or work with redirects to ensure that existing links aren’t permanently broken. My concern with trying to import the resulting documents into python.org’s CMS, which is not a huge departure from what we do now, is that it then creates two places for this information… which is canonical?

1 Like
#4

I would advocate for hosting on Read the Docs and not doing any CMS importing (if I’m understanding what Ee is asking).

1 Like
#5

I suppose it gets to the point of what python.org is intended for – if the website is primarily an advert for the project as a whole and the charity, then it may make sense (and I would argue that it does) to put PEPs on a subdomain also, similar to docs.python.org.

It seems at the monement that there seems to be a good precendent for hosting things on subdomains as opposed to the main site, but I guess PEPs might be “sufficiently different”, and warrant staying on the main website.

list of subdomain things:

  • issue tracker (bugs.python.org)
  • automatic builds (buildbot.python.org)
  • developer’s guide (devguide.python.org)
  • documentation (docs.python.org)
  • email list archives (mail.python.org)
  • packaging guide (packaging.python.org)
  • performance statistics (speed.python.org)
  • status (status.python.org)
  • wiki (wiki.python.org)

I’d agree with this – there’s no point doing both, there should be one canonical source – for the reasons in my original paper I would favour peps.python.org, but we shouldn’t create that if we continue integrating into the main python.org website.

A

1 Like
Styling of PEPs
#6

My only concern is that the existing URL structure for PEPs (https://www.python.org/dev/peps/pep-0XXX/) continues to work, as breaking that would wreck so much stuff.

2 Likes
#7

Yes, this should be a zero-disruption change.

I’d propose redirects here (similar to how https://python.org/peps/pep-0100.html goes to the right place) – would you prefer that the links don’t change at all?

A reverse proxy could be used to the /dev/peps namespace, but that would likely require a lot more work (& not sure if it’s even currently possible!)

A

1 Like
#8

I don’t think I care as long as the links work.

1 Like
#9

Hi Adam,

I have difficulty telling whether I should care about this proposal or

not. You say that it will impact anyone who reads, writes or reviews

PEPs. I often read PEPs, and have sometimes written PEPs. Can you expain

how this proposal will affect people like me?

I’ll be honest, on the basis of a fast read through, it sounds more like

a back-end infrastructure change that should have a minimal affect on

PEP readers or writers. Is this wrong?

Some user-stories might help.

Suppose I want to read a specific PEP, and I know either some keywords

or the PEP number. Right now, I google for “Python PEP …” or (more

rarely) go straight to PEP 0 -- Index of Python Enhancement Proposals (PEPs) | Python.org and look at the

PEP. How will your proposal change this?

I want to read PEPs about a keyword, say, anything to do with dicts. My

workflow here is more or less the same. How will your proposal affect

me?

As a reader of PEPs, is the only change that the URL will change?

The last time I actively wrote a PEP was back in the hg days. The change

to github derailed me like a car on a railway line and due to many work/

life/technology factors I have not yet caught up with the brave new

world of github. (I expect that I will probably do so three months

before we shift to a new and improved system wink)

But, if I were to write another PEP, how would this change impact me?

#10

Hi Paul,

You say:

"My only concern is that the existing URL structure for PEPs

(https://www.python.org/dev/peps/pep-0XXX/) continues to work, as

breaking that would wreck so much stuff."

Are you referring to link rot? Or did you have some other form of

breakage in mind?

#11

I’m not sure what you mean by “link rot”, but I’m referring to all the places that link to PEPs, as well as things like “smart bookmarks” that construct a URL from a PEP number based on the existing URL names. @AA-Turner says that the existing URLs will continue to work though, so that doesn’t seem to be an issue. As a PEP reader and occasional author, I agree with your other comment - I can’t see any way in which this change would affect me. But I choose to interpret that as a good thing, the change can be made without harming people in my position.

1 Like
#12

Paul:

“”"
I’m not sure what you mean by “link rot”
“”"

We’re talking about the same thing :slight_smile:

#13

Link rot is a valid and extremely important concern, and shouldn’t happen:

  • Either PEPs will remain at https://www.python.org/dev/peps/pep-0008/

  • Or they will be moved to something like https://peps.python.org/pep-0008/

    • AND redirects will be put in place from the old https://www.python.org/dev/peps/pep-0008/ to https://peps.python.org/pep-0008/, so anyone visiting the old one will be automatically redirected.

As a reader, nothing should really change too much. The most noticeable will be things like styling, formatting (I’m looking forward to coloured syntax highlighting!), menus. For example, compare the demo page:

Because it’ll be using Sphinx/RTD in the background, which are well known and supported in the Python community, this should make things easier to maintain and customise. Is it fair to say it should be more accessible?

For the googling use case, this is my usual way of finding PEPs too. Google will start to pick up the new links and update their results, and because we’ll have redirects from the old links to the new location, things will still work in the meantime.

As a PEP author, your workflow would be the same. You will create a PR at GitHub - python/peps: Python Enhancement Proposals, and it’ll be reviewed as usual. When merged it’ll be auto-deployed as before (although in a different manner, but as a PEP author you don’t need to worry about how).

One new and really useful benefit, RTD can create preview builds of docs for PRs. That way the author and reviewers can check the changes look good and render properly (little things that can be easy to get wrong, like code formatting, tables, links).

For example, this PR from another project has a CI check from RTD and a build preview

As it happens for this PR, the reviewer saw from the preview that some table cells could be merged, and suggested such a change. After applying, we checked it worked as desired, and merged.

2 Likes
#14

Thanks for your comments Steven – Hugo covered most of I would have said far more eloquently!

To re-emphasise though

  • changes for readers are that there would be a different canonical URL (with auto-redirects from current URLs) and different styling.
  • changes for reviewers are the above (as reviewers read PEPs) and potential quality-of-life improvements like auto-built previews, to enable reviewing what the rendered document would look like (for potential rendering issues, like the table markup case Hugo mentioned)
  • changes for writers are the above (as writers read and review their own PEPs) and also that PEPs would be deployed on merge, instead of waiting for a synchronisation process.

As with Paul/Hugo these aren’t massive changes, but I wanted to perhaps be overly conservative in what constitutes a “change” in my original paper.

A

3 Likes
#15

I’m in favour of the change, using the “new canonical hosting URL with redirects” approach. With a distinct subdomain we don’t have to keep the styling identical to either the regular docs or the main python.org site - we can use whichever hybrid of the two we feel makes the most sense for the content.

For hosting, I would suggest keeping the URLs behind the main python.org CDN, but switching the app layer backend to RTD should be fine (I don’t know the actual PEP reading traffic volume, but even with only text documents I assume it’s high enough that offloading it all to RTD’s CDN would be a bad idea, whereas for the PSF it’s a drop in the ocean compared to PyPI’s traffic. That said, given some of the project docs that RTD hosts, the PEP traffic would probably be at worst a drop in a large sea for them, so the exact details likely don’t matter much)

For record keeping purposes, the proposal posted here should probably also be recorded as a PEP in its own right.

3 Likes
#16

Apologies for the delay, I had intended to leave this for a week or so to see if any others had thoughts and then events conspired against me.

People seem largely in favour of this (or not against it) – so perhaps the next step is to formally write this up as a meta-PEP as Nick suggested.

I don’t know what the best process is here – if I should just have a stab at putting the content from the original post (perhaps with stronger language about how redirects etc will be preserved) into the PEP template and submit it as a PR?

PEP 1 - Submitting a PEP suggests that I would need a sponsor for the PEP – I know a number of the participants in the various discussions around this have been valid PEP sponsors – advice on this would be appreciated.

A

1 Like
#17

I would draft it and post the draft somewhere to discuss it (here would be fine since this isn’t really a python-ideas sort of thing).

You shouldn’t post a PEP via a PR without a sponsor.

You will need a sponsor. You can either see if one comes up here or write the PEP and explicitly mention in the posting of the draft PEP that you’re looking for a sponsor.

1 Like
#18

I’ll be happy to sponsor this PEP @AA-Turner.

2 Likes
#19

@AA-Turner Let me know if I can help draft. Feel free to DM me here or Discord (hugovk#6128), or open an issue on your or my peps forks.

1 Like
#20

With thanks to Hugo for a some initial reviews, please find the current draft text below, and a rendered preview online.

(There is currently an ugly duplication of the footnotes and references sections, but PR 2155 will fix this.)

A

PEP: 9999
Title: PEP Infrastructure Process
Author: Adam Turner <python@quite.org.uk>
Sponsor: Mariatta <mariatta@python.org>
Discussions-To: https://discuss.python.org/t/10774
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 01-Nov-2021
Post-History: 23-Sep-2021

Abstract
========

This PEP addresses the infrastructure around rendering PEP files from
reStructuredText_ files to HTML webpages. We aim to specify a self-contained
and maintainable solution for PEP readers, authors, and editors.

Motivation
==========

As of November 2021, Python Enhancement Proposals (PEPs) are rendered in a
multi-system, multi-stage process. A continuous integration (CI) task runs a
docutils_ script to render all PEP files individually. The CI task then uploads
a tar archive to a server, where it is retrieved and rendered into the
`python.org`_ website periodically.

This places a constraint on the `python.org`_ website to handle raw HTML
uploads and handle PEP rendering, and makes the appropriate place to raise
issues unclear in some cases [1]_.

This PEP provides a specification for self-contained rendering of PEPs. This
would:

* reduce the amount of distributed configuration for supporting PEPs
* enable quality-of-life improvements for those who read, write, and review
  PEPs
* solve a number of outstanding issues, and lay the path for improvements
* save volunteer maintainers' time

We propose that PEPs are accessed through ``peps.python.org`` at the top-level
namespace (for example ``peps.python.org/pep-0008/``), and that all custom
tooling to support rendering PEPs is hosted in the `python/peps`_ repository.

Rationale
=========

Simplifying and Centralising Infrastructure
-------------------------------------------

As of November 2021, to locally render a PEP file, a PEP author or editor needs
to create a full local instance of the `python.org`_ website and run a number
of disparate scripts, following documentation_ that lives outside of the
`python/peps`_ repository.

The proposed implementation provides a single Makefile_ and a Python script to
render all PEP files, with options to target a web-server or local filesystem
environment.

Using a single repository to host all tooling will clarify where to raise
issues, reducing volunteer time spent in triage.

Simplified and centralised tooling may also reduce the barrier to entry to
further improvements, as the scope of the PEP rendering infrastructure is well
defined.

Quality-of-Life Improvements and Resolving Issues
-------------------------------------------------

There are several requests for additional features in reading PEPs, such as:

* syntax highlighting [2]_
* use of ``.. code-block::`` directives [2]_
* support for SVG images [3]_
* typographic quotation marks [4]_
* additional footer information [5]_
* intersphinx functionality [6]_

These are "easy wins" from this proposal, and would serve to improve the
quality-of-life for consumers of PEPs (including reviewers and writers). For
example, the reference implementation no longer requires a scheduled render
process to run, instead initiating rendering on every commit to the
`python/peps`_ repository.

Equally, there are a small number of broken items, for example list styles not
being respected or support for updating images being challenging with the
system [7]_. These would be solved by default in the proposal.

Commercial providers such as `Read the Docs`_ can additionally enhance this
experience, for example by providing rendered previews of changes in pull
requests.

Specification
=============

The proposed specification for rendering the PEP files to HTML is as per the
`reference implementation`_.

That the HTML files should be made available under ``peps.python.org``. The
rendered files may be hosted just as static files, and behind a content
delivery network (CDN).

The following redirect rules must be created from the `python.org`_ domain:

* /peps/            -> https://peps.python.org/
* /dev/peps/        -> https://peps.python.org/
* /peps/(.*)\.html  -> https://peps.python.org/$1
* /dev/peps/(.*)    -> https://peps.python.org/$1

.. code-block:: nginx

    location ~ ^/dev/peps/?(.*)$ {
        return 308 https://peps.python.org/$1/;
    }

    location ~ ^/peps/?(.*)\.html$ {
        return 308 https://peps.python.org/$1/;
    }

    location ^/(dev/)?peps(/.*)?$ {
        return 308 https://peps.python.org/;
    }

Redirects must be implemented to preserve `URL fragments`_ for backward
compatability purposes.

Backwards Compatibility
=======================

Due to server-side redirects to new canonical URLs, there are no backwards
compatability concerns. Links in previously published materials referring to
any old URL scheme will be guaranteed to work.

Security Implications
=====================

No security implications, and the main `python.org`_ website will no longer
take raw HTML uploads, closing a potential threat vector.

How to Teach This
=================

The new canonical URLs will be publicised in the documentation. However, this
is mainly a backend infrastructure change, and there should be minimal
end-user impact.

Reference Implementation
========================

The proposed implementation has been merged into the `python/peps`_ repository
in a series of pull requests [8]_. This automatically renders all PEPs on every
commit.

Rejected Ideas
==============

It would likely be possible to amend the current (as of November 2021)
rendering process to include a lot of the quality-of-life improvements and
issue mitigations mentioned above. However, we do not believe that this would
solve the distributed tooling issue.

It would be possible to use the output from the proposed rendering system and
import it into `python.org`_. We would argue however that this would be the
worst of both worlds, as a great deal of complexity is added, and none is
removed.

Open Issues
===========

None.

Acknowledgements
================

Thanks to Hugo van Kemenade, Pablo Galindo Salgado, and Éric Araujo for support
since April 2020.

Footnotes
=========

.. _documentation: https://pythondotorg.readthedocs.io/pep_generation.html
.. _docutils: https://docutils.sourceforge.io
.. _Makefile: https://www.gnu.org/software/make/manual/make.html#Introduction
.. _python.org: https://www.python.org
.. _python/peps: https://github.com/python/peps
.. _Read the Docs: https://readthedocs.org
.. _reStructuredText: https://docutils.sourceforge.io/rst.html
.. _URL fragments: https://url.spec.whatwg.org/#concept-url-fragment

.. [1] For example,
       `pythondotorg#1024 <https://github.com/python/pythondotorg/issues/1204>`__,
       `pythondotorg#1038 <https://github.com/python/pythondotorg/issues/1038>`__,
       `pythondotorg#1387 <https://github.com/python/pythondotorg/issues/1387>`__,
       `pythondotorg#1388 <https://github.com/python/pythondotorg/issues/1388>`__,
       `pythondotorg#1393 <https://github.com/python/pythondotorg/issues/1393>`__,
       `pythondotorg#1564 <https://github.com/python/pythondotorg/issues/1564>`__,
       `pythondotorg#1913 <https://github.com/python/pythondotorg/issues/1913>`__,
.. [2] Requested: `pythondotorg#1063 <https://github.com/python/pythondotorg/pull/1063>`__,
       `pythondotorg#1206 <https://github.com/python/pythondotorg/issues/1206>`__,
       `pythondotorg#1638 <https://github.com/python/pythondotorg/pull/1638>`__,
       `peps#159 <https://github.com/python/peps/issues/159>`__,
       `comment in peps#1571 <https://github.com/python/peps/pull/1571#discussion_r478701944>`__,
       `peps#1577 <https://github.com/python/peps/pull/1577>`__,
.. [3] Requested: `peps#701 <https://github.com/python/peps/issues/701>`__
.. [4] Requested: `peps#165 <https://github.com/python/peps/issues/165>`__
.. [5] Requested: `pythondotorg#1564 <https://github.com/python/pythondotorg/issues/1564>`__
.. [6] Requested: `comment in peps#2 <https://github.com/python/peps/issues/2#issuecomment-339195595>`__
.. [7] As of November 2021, see
       `peps#1387 <https://github.com/python/peps/issues/1387>`__,
       `pythondotorg#824 <https://github.com/python/pythondotorg/issues/824>`__,
       `pythondotorg#1556 <https://github.com/python/pythondotorg/pull/1556>`__,
.. [8] Implementation PRs:
       `peps#1930 <https://github.com/python/peps/pull/1930>`__,
       `peps#1931 <https://github.com/python/peps/pull/1931>`__,
       `peps#1932 <https://github.com/python/peps/pull/1932>`__,
       `peps#1933 <https://github.com/python/peps/pull/1933>`__,
       `peps#1934 <https://github.com/python/peps/pull/1934>`__

Copyright
=========

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


..
 Local Variables:
 mode: indented-text
 indent-tabs-mode: nil
 sentence-end-double-space: t
 fill-column: 70
 coding: utf-8
 End:
3 Likes