Even if you lost sight, that does not mean that you can not contribute. The only difference is that, it will take you more time to do it!
I strongly support a simplified commonmark variant. I find markdown the easiest to read even when uninterpreted (i.e. **bold** instead of bold). I do not think we need the full commonmark experience for the stdlib.
If the docstring format in this file is Markdown, pydoc will try to render it as bold. The same way as it already uses bold for section headers and defined names. In a browser it uses HTML elements.
man and pydoc use less for displaying output. It supports bold and underscored text. On my terminal bold looks like blue on black. If I set TERM=dumb, less will wrant that terminal is not fully functional, but still shows bold the same way. I do not know what happens if run it on terminal that does not support bold.
I do not know were it displays docstrings, except tooltips, in which case it only extract signatures from docstring, stripping prose. I do not know what it uses display signatures, Label or Text. If latter, it can be made to use bold in signatures, but this is not related to docstring formatting.
It can, and often does. But pydoc does not know what formatting is used and cannot render it, so it displays it all as plain text.
An UI chooses what and how to render. It can simply show plain text, like it do currently.
There is no input which cannot be parsed as Markdown. If we choose different format, for example HTML, it can reject some input, but current HTML is also very permissived and can eat any garbage you feed it.
Yes, by default format is plain text. But many modules already use some uncommon convention, for example Tkinter uses UPPERCASE for parameter. There is desire to rewrite it. If we can support some rich text format, it will be good opportunity.
Thanks for finding that!
However, just as we have changed other things (hosting, bug tracker, version control systems), we can change our preferred markup language for docstrings.
Two comments about PEP 287:
- Markdown/Commonmark was not invented yet;
- I think reStructuredText fails the first goal of the PEP:
- It must be readable in source form by the casual observer.
What features described in PEP 287 do you find not readable, and how would those constructs look in Markdown?
Jut tried writing one. And found out that common mark is pretty much the opposite of “There should be one-- and preferably only one --obvious way to do it.” and decided not to.
I think something more lightweight and less ambiguous would be much more appropriate.
https://djot.net (which is written by the same guy as CommonMark) looks quite good, but it is not really markdown anymore…
Personally, if you decided to actually implement CommonMark (with GFM and plugins) and maintain it in stdlib, I would be very pleased and grateful. ![]()
This is a hard problem with many conflicting priorities:
- Established standard in Python: PEP-287 defines that docstrings should be reStructuredText. Doing anything else is a massive breaking change. Libraries and tooling in the ecosystem is built on that assumption.
- Established standards in other programming languages: Basically all major languages use a from of markdown (only exception is Nim, which also uses reStructuredText).
- Disadvantages of reStructuredText: It’s a little more verbose and a IMHO a bit quirky at times. It’s less common so harder to read/write for contributors.
- Disadvantages of Markdown: There are multiple standards with different feature sets. Specification is quite lax, making correct parsing difficult. (djot tries to address this).
- Consistency / requirements: Docstrings vs. prose documentation: Ideally, we do not have different formatting conventions between docstrings and prose documentation, but requirements are different. Readability of raw source is very important in docstrings, we do not need as many features. In contrast, prose documentation needs many more features (sections, footnotes crosslinks to sections, crosslinks to API, etc.).
- Parser complexity in the standard lib vs. features of the format: The stdlib wants a lightweight implementation, but any full markup format is quite complex due to its features.
I see two possible routes forward:
- stick with reStructuredText (with all its limitations) and define a subset format “reST light” that can be used for docstrings and implemented in the stdlib.
- move to markdown for docstrings. - This must be a major concerted effort. While we could do this for CPython only, it’s not a good sign to the community. So that would mean there’d need to be a strategy for the whole ecosystem, including good specification and a usage/migation strategy, in particular including a format specifier for docstrings to inform tooling (e.g. a trailing comment
``` # format: markdown) and tooling to convert the markdown docstring to a reST docstring so that existing tools can easily handle the new format by converting it back to reST, maybe reverse tooling, so that libraries can easily swich their reST docstrings.
The other way around: if just a subset will do, what markdown features can be safely omitted? And will it noticeably reduce the complexity of the module to justify missing features?
Reading the CommonMark specs, I would say the “ATX” headers are preferred over “setext” headers (whatever those names mean) making the latter not required. If we are concerned mainly about terminal output, than perhaps all HTML specific stuff could remain unsupported. Links (names for URLs make little sense without an <a> tag), raw HTML blocks, embedded images. But that’s probably all. Inline marking, bulleted and numbered lists, tables, horizonal rules, code blocks, etc. all that could have a purpose in docs.
I wonder if we could use a special tstring to denote md docstrings for rendering vs the old style. Maybe not even worth it.
We could if t-string is extended in some way to describe the kind of its content, as noted in Mechanism to Describe the “Kind” of Template of PEP-750 as a “hope” for future development.
Maybe a magic-bytes style approach inside the docstring is a less intrusive way of doing this, like a __$LANG__ on the opening line of the """:
class Gadget:
"""__md__
Does **good** stuff.
"""
Still a bit ugly, especially repeating it in every docstring.
Perhaps instead it could be a field docstring-language=markdown placed in pyproject.toml to indicate the format of docstrings across an entire module or package?
The Python interpreter currently does not read pyproject.toml.
Or a module-level constant like __markdown_docstrings__ = True.
Module-level granularity is possibly good enough, both from the perspective that it forces using the same style in all docstrings of the file, and that the setting must be repeated in every file.
It also has the advantage that the setting is inside the file, so it’s fully self-contained.
There’s “PEP 258 – Docutils Design Specification”, that defines the module level variable __docformat__ for specifying the format of docstrings in a module, so for example:
__docformat__ = "reStructuredText"
Out of curiosity, I tried to answer the question : what if we start by allowing only the common subset between restructuredtext and markdown ? It could be an improvement over raw text event if it’s not a full fledged solution.
Here is what I found (with Claude’s help) :
title
=====
subtitle
--------
*italic*
**bold**
+ item 1
+ item 2
* item 2.1 (with blank line due to rst)
* item 2.2
* item 2.2.1
* item 2.2.2
- item 2.2.2.1
- item 2.2.2.2
1. numbered item 1
2. numbered item 2
``inline code``
\*escape\*
In practice I think that it’s too tempting to use additional syntax since it’s usually used “everywhere” (this everywhere depending on if you come from the RST or MD world).
I agree. The numpy docstring style explicitely states:
Our docstring standard uses re-structured text (reST) syntax
The other common style is Google style but it only defines blocks (like Args:, Returns:, Raises:), not any other formatting.
Here’s something that I did in reply to this thread:
https://pypi.org/project/simplemark/
It is, basically, a subset of CommonMark’s constructs, with one narrowed inline rule: emphasis delimiters must be flanked by whitespace or punctuation, so emphasis never happens inside a word.
Doc at:
https://simplemark.lab.abilian.com/
Source code:
https://git.sr.ht/~sfermigier/simplemark
S.
I also had a quick thought about this. The major issue is that Google Style and numpydoc both use indentation. For numpydoc style this is formally a reST definition list - each parameter is a separate definition. I suppose Google style is a proprietary format and the indentation is their own invention, not following any other markup spec.
But in Markdown, indentation defines code-block. This means, interpreting numpydoc or Google Style docstrings as plain markdown turns parameters into plain text, and it can have funny code-block effects if parameter descriptions have multiple paragraphs.
numpydoc
"""
Divide two values.
Parameters
----------
a : float
The dividend of the division.
Multiline paragraph.
Second paragraph.
b : float
Divisor.
"""
would render as
Divide two values.
Parameters
a : float
The dividend of the division.
Multiline paragraph.
Second paragraph.
b : float
Divisor.
google style
"""
Divide two values.
Args:
a (float): The dividend of the division.
Multiline paragraph.
Second paragraph.
b (float): Divisor.
"""
would render as:
Divide two values.
Args:
a (float): The dividend of the division.
Multiline paragraph.
Second paragraph.
b (float): Divisor.
This means, this does not work as is for a large part of the ecosystem.
The turn side is that markdown does not have a definition list syntax at all (well only as some extra extension), so that structured parameter lists are fundamentally not possible.
It should be mentioned that cpython docstrings are different from most ecosystem docstrings, because the ecosystem widely uses structured docstrings in numpdoc or google style, but cpython only uses paragraphs with light markup. We’d need to clarify what the scope is: cpython vs. potentially the whole ecosystem. While it can be a choice to make this a cpython specific topic, it would be nice to create something that would suit the whole ecosystem, and in particular the idea to make it work for reST and markdown has little value if it’s limited to cpython, because in this limited context, we could choose just one.