PEP 676: PEP Infrastructure Process

Excellent write-up! I think that could be submitted to the SC as soon as it has been assigned an official PEP number.

3 Likes

PR created! (PEP 9999: PEP Infrastructure Process by AA-Turner · Pull Request #2173 · python/peps · GitHub)

1 Like

This is now PEP 676: PEP 676 -- PEP Infrastructure Process | Python.org

And rendered according to PEP 676: PEP 676 – PEP Infrastructure Process | peps.python.org

3 Likes

First off, on a technical note, the auto-generated references section looks particularly odd, confusing and repetitive with the very intentional footnotes section; merging my PR python/peps#2155 that you and Guido have already reviewed will fix that.

As a longtime frequent PEP reader, and more recently a (re-)writer, reviewing and minor contributor to the CIs and build system, I strongly support this change.

As a user who reads PEPs frequently, the better they can be presented, with more useful information, clean styling and better aesthetics, and the easier it is for authors to do this, the better.

As a new PEP (re-)writer, like many Python developers, I am very familiar and comfortable with Sphinx (being the creator and maintainer of the Spyder IDE documentation and others, as well as our own detailed docs style guide and theme), but not with legacy docutils. The workflow is different, the syntax is much more limited, and often I’ll either unknowingly use Sphinx-only syntax, or hesitate to use syntax that will actually work due to uncertainty, and overall its a significant mental burden. Furthermore, it doubles testing time, as I have to build and examine PEPs using both build systems rather than just one, and in some edge cases things render differently, creating inconsistency.

As someone who’s occasionally helped a little with the backend side, build system changes (like my python/peps#2155) have to be duplicated twice, once in modern Sphinx and once in the legacy custom docutils build. This makes them twice as much work to implement, review and test, and leads to duplicated code and a higher chance of bugs in one or the other.

Furthermore, the existing rst2html.py, while still functional, is a relic of Python as it was perhaps nearly two decades ago, and is much more of a chore to work with than the modern, modular, well organized, designed and styled Sphinx build system that @AA-Turner has helped implement, and full of hacks to work around quirks and limitations of docutils at the time. The sooner we can move forward on the former without the latter holding us back, the better, in my opinion.

As for site hosting, is there a reason to use an external managed service like RTD, though? It should work with any static site generator that supports custom domains; even GH pages would do (would be easiest from a CD perspective), or can be deployed to somewhere on Python/PSF controlled infra if that is preferred. And a service like Netlify can be easily used to show the rendered previews, without the need to go the hosted RTD route, as we do for e.g. Spyder-Docs and our other docs and website repos. But that’s ultimately someone up to the people who manage that. Previews would be a huge plus, though.

3 Likes

Agreed, build previews are definitely a huge plus, they make a big difference for me on other repos which have then turned on: as reviewer (and indeed author) you can check things rendered properly, and you can review the intended output rather the RST code.

RTD is already used for the devguide, so using a familiar service may be a helpful.

I’ve used Netlify for previews of other non-RTD sites, and it was easy to set up and use, and deploying to GH Pages is straighforward via CI/CD.

2 Likes

Netlify + GH Pages via GH Actions is what we do on Spyder-Docs as well, and its super streightforward (most of the complexity for us is with the multiversion support, which isn’t needed here). We use short bash scripts for the build and install commands to avoid duplication where practical between GH Actions and Netlify, but its pretty simple and straightforward. IMO the biggest benefit of RTD is the multiversion support (plus the theme), which we had to take care of ourselves, but neither is an issue for the PEPs, since mutliple versions would be counterproductive. We also have a standalone Python script that can generate redirects for every page, though since that would mainly be important on the main Python domain, I imagine it would be easier to use .htaccess or something else built-in to redirect the whole tree.

1 Like

I see that intersphinx is mentioned as a quality-of-life improvement, but there is no explicit mention of Sphinx in the PEP. Is that something that should be included?

On the other hand, I’m wondering if the hosting decision (RTD vs GHA + Netlify and/or GH Pages) is something that folks intend to include in the PEP as well?

To add a few more points in favor of Read the Docs (disclaimer: they’re my employer), apart from the pull request previews that have been mentioned, as @ericholscher mentioned in python/peps#2 RTD gives more sophisticated search and content embedding (see for example this blog post, a newer one is in the works). I agree that versioning would be undesirable in this case, and luckily it can be disabled.

1 Like

I see that intersphinx is mentioned as a quality-of-life improvement, but there is no explicit mention of Sphinx in the PEP. Is that something that should be included?

Great point; that does seem to be a pretty substantial omission, considering the main backend change is moving from docutils + big bespoke script to Sphinx, and many of the mentioned features under Quality of Life Improvements and elsewhere are specifically due to Sphinx. I could see an argument for not wanting to rigidly specify a particular backend in the PEP, but given many of the specific benefits cited therein depend on it, I’d certainly think it would make sense.

On the other hand, I’m wondering if the hosting decision (RTD vs GHA + Netlify and/or GH Pages) is something that folks intend to include in the PEP as well?

I was thinking something along the same lines, though I’m not sure whether or not that qualifies as a PEP-level question as opposed to one of implementation.

To add a few more points in favor of Read the Docs

Seems like RTD as an integrated solution might make a lot of sense here, considering it is already in use elsewhere for hosting. The content previews in particular are super cool. Just to clarify (others here might already be aware), are any of the specific RTD benefits mentioned in this thread tied to using the RTD theme or domain, and if RTD is being used as the host on a custom domain, does it impose any constraints as to the URL path structure of the resulting output? Thanks!

1 Like

Very kind!

This was my reticence towards specifying any particular vendor – Hugo convinced me to add some language around e.g. PR previews in the PEP (which mentions that RtD is a vendor that can provide them), but I wanted this to be on a more principles based thing of simplifications and making things easier, which happens to involve using Sphinx as the back-end.

Agree with both points – there may be value in running e.g. RtD for one and Netifly for another to reduce potential lock-in, although there may be points I’m missing.

This was deliberate – I picked up intersphinx as it was explicitly noted in the tracker, but I wanted to be as tooling agnostic as possible and focus on the principles of self-contained tooling & simplifications.

I’d say that this is a low level point of hosting – ideally the vendor shouldn’t be visible to most all users, except perhaps those who use PR previews. If it is sufficiently contentious though, maybe?

A

1 Like

Nope and nope :slight_smile: Both custom domains and custom themes are supported and the features should be the same. sphinx_rtd_theme doesn’t do anything that other themes can’t do.

No more constraints than the ones set by Sphinx itself. Single-versioning the docs removes the /<language>/<default_version>/ part. For special cases, custom redirections can be put in place too, although the UX is admittedly not great.

2 Likes

Fair enough - I’m not familiar with the PEP process, so I’ll let others decide on these two points.

1 Like

See post above (just missed yours Chris!). My intention was/is to abstract this away, though – perhaps I should add a note in reference implementation that it uses Sphinx + custom extensions & theme, implemented through PRs XXX - YYY.

I would say no. But I have limited experience!

Also to note – the current PEP 676 implementation entirely disables search, entirely deliberately – on the notion that a google[1] search for site:peps.python.org blah will always be better than a custom hosted search engine (through JS or RtD’s backend (sphinx-search?).

Neither am I to be clear! Would also welcome thoughts from others.

A


  1. other search engines are available! ↩︎

1 Like

Just as another note, the new build system allows including supplementary material to the PEP (dependencies, examples, user guides, background material, etc) written as reST, included in the PEP’s subdirectory, linked from the PEP and rendered separately. This helps keep PEPs shorter, more focused and easier to read, while still allowing authors to include additional resources and content not part of the PEP itself without tacking on long appendices or having to host it elsewhere, on a link that may change, break, get taken down, etc. over time.

This will be extremely valuable for my work on PEP 639, as it has accumulated way too much supplementary material (over 2/3rds its length are in appendices, examples, user guidance, rejected ideas, etc) that would be much better off in separate document(s). This would shorten the PEP by over two thirds (in conjunction with the references section elimination and other changes), making it much easier to read and navigate while while still retaining the value in being easily accessible alongside the PEP to those interested.

I can do a copyediting and formatting pass, and let me know if I can be of other assistance (Sphinx/build system stuff, writing, mundane tasks, etc) in helping it move along!

The PEP has now been announced to python-dev at Mailman 3 PEP 676 -- PEP Infrastructure Process - Python-Dev - python.org (It is showing my email not my name, even though I have an account with mailman, but not a large issue).

@Mariatta - assuming no substantial issues are raised, how long would you suggest to leave before submitting to the Council?

A

1 Like

Replying to @barry’s comments on python-dev to avoid splitting the discussion. Sorry for being awkward! (sent a similar note of apology on python-dev, but I believe there’s moderation, so it may take a while to send.)

This looks great Adam, +1

Thanks :slight_smile:

dark mode theme

I’ll have a look, and will ping you on the PR for feedback. It may take a few days, though – if only to decide on the colour scheme!

fully documented

Yes, a good point. Do you have any suggestions on where said documentation should end up? I don’t expect it to be very large, but it doesn’t really fit on docs.python.org, I wouldn’t think.

hosting on RTD … rejected

It hasn’t been rejected, more that no-one has made a passionate argument in favour of RTD over e.g. Netifly, or nothing at all. I suppose my personal view is that PR previews are the only thing that PEPs would miss from such a service, and there are several options to get PR previews (Netlifly, RTD, a more complex GH actions setup, etc).

Equally, I view it as an implementation detail – it shouldn’t actually matter where PEPs are hosted, and there may be benefits to the infrastructure team to pick one solution over another. I wouldn’t want to constrain that choice set by overly specifying a detail that (in my view) doesn’t matter that much.

Perhaps the PEP could be updated to say that PR preview rendering SHOULD be provided with whatever provider(s) are used?

A

Personally I’d be in favor of RTD since we have been using RTD for other docs like the DevGuide.

I was thinking we should have an organization RTD account owned by The PSF to manage our various RTD projects, but I think that’s out of scope of this PEP.

Perhaps the PEP could be updated to say that PR preview rendering SHOULD be provided with whatever provider(s) are used?

Yes I think this is a good way to go about it.

2 Likes

:+1: to that…so long as we don’t get too caught up on what color to paint the bikeshed website :grinning_face_with_smiling_eyes:

I’d defer to @Mariatta or someone else more experienced around here for more concrete advice, of course, but just for some background, the equivalent docs for the main Python.org site are hosted on a RTD subdomain. This could do the same, but In the spirit of making everything self-contained (per the title), it might make more sense to host the source in the PEPs repo (perhaps in a docs subdirectory), build it together with the PEPs and expose them in a subdir, perhaps something like peps.python.org/docs (which would naturally follow from the source location).

FWIW, seems like @astrojuanlu did (though as he mentioned they are his employer) and it seems there was general agreement with RTD as the static host, though not particularly strong vs. other static hosts. I brought up Netlify and GHA for building and previews, though that had more to do with having a lot of experience deploying docs/sites like this with them and relatively little with RTD, rather than any strong preference for them, and the reasons for the latter tend to make sense.

That would definitely be a huge win with this proposal, in my perspective—now that I’ve deployed rendered previews on all my various static docs and websites, its hard to live without, especially so for those with many contributors and a heavily customized build system. To note, the rendered previews don’t have to be built by the same provider as the actual site build, CI checks and deploy (for example, for historical reasons, most docs/static sites I maintain use Netlify for previews and GHA for everything else). Having both rendered previews and production builds/deploys with one provider does simplify maintenance (which seems to be an advantage for RTD), but isn’t a hard constraint on the production deploy provider.

1 Like

The devguide has version-independent info on documenting Python, and a page about changing the language that sums up the PEP process. I think that if documenting the PEP docs system does not fit in a singe README file in the PEPs repo (automatically rendered and hosted!), then the devguide would be a good host.

1 Like

I thought about suggesting the devguide above, but nearly all of it focuses on the development of the Python language and its documentation, rather than the backing infrastructure surrounding it, and given the very similarly focused documentation for the main website infra is hosted separately from it, I’m not sure it would would make sense for it to go there…but again, I don’t really have any authority on this.

Definitely a big plus one to deploy previews.

True, but it’s usually a good idea to have test and production sites with as close setups as possible because you never know what little difference will cause different output. And simplifying maintenance is a worthy goal.

1 Like