A standard for non-displaying executable lines of code in documentation

Ideas
1

Hello world,

Would Python ever consider adopting a standard for non-renering executable lines of code in documentation?

Context

Doctests are a great way of ensuring that code documentation is accurate to a library, and that those examples work. However, there are times when it is valuable to show a small snippit of code that require larger chunks of setup to validate, or where testing is desired but the Python console syntax is not.

Idea

The minimum requirement would be Python documentation that states, "lines in documentation beginning with the control sequence ##! are intended to be hidden by any tool that renders the code, but run (without the control sequence) by any tool intended to execute it.

My proposed control sequence was ##!. It needs to start with a comment so that nothing breaks when trying to run it. ! tends to hint executable-related in POSIX world ($! for PID, !cmd or !! to re-execute a command, and shebang) but #! is used for shebang, so I landed on ##!.

Allowing this syntax has the benefit of breaking nothing; tools that can execute it would be strictly opt-in. Anything that does not support the syntax will just ignore the ##!'d lines of code.

Example

Assume the following code is in a docstring or documentation source file:

##! from typing import Optional, TYPE_CHECKING
##! if TYPE_CHECKING:
##!     from mymodule.something import CoolClass
def fn(x: Optional[int]) -> CoolClass:
    return CoolClass(x)
##! assert fn(5) == CoolClass(5)

Anything that renders the code to readers (IDE hints, documentation tool, etc) simply removes lines starting with the control sequences. So, the above example displays as the following terse snippet:

def fn(x: Optional[int]) -> CoolClass:
    return CoolClass(x)

Anything that wants to use the full executable code just removes the control sequence itself. So a linter, type checker, or test runner would wind up with:

from typing import Optional, TYPE_CHECKING
if TYPE_CHECKING:
    from mymodule.something import CoolClass
def fn(x: Optional[int]) -> CoolClass:
    return CoolClass(x)
assert fn(5) == CoolClass(5)

The main use case for this is likely plain Python code blocks that are outside the scope of doctest. However, there is no reason that doctest or a doctest runner wouldn’t also be able to understand the syntax; >>> ##! foo() or ... ##! foo = "bar" would just get run as if the ##! were not there.

Inspiration

This is inspired from the way that Rust does documentation tests and allows hiding lines, as shown here: Documentation tests - The rustdoc book. In my opinion, there are quite a few excellent things that they do with documentation, this being one of the nice little things, and I think seeing some of those brought to Python would encourage users to write better docs.

The best existing alternative for hiding code lines is sphinx.ext.doctest. However, that requires boilerplate (.. testsetup/.. testcleanup/.. testcode), is RST-only, doesn’t allow for hiding code lines not at the beginning or end (e.g. if you wanted to show from __future__ import annotations, hide imports, and show the rest of the code) and requires the pycon syntax that I believe to be somewhat limiting (e.g., copy/paste from working code is not straightforward, IDEs generally do not predict >>>/... continuation).

I considered a few possible ways that a thin wrapper could make use of this:

  • black could format the example as-is without worrying about maintaining >>>/...
  • After stripping the control sequences, Mypy could verify the snippet is typed correctly
  • flake8 could verify stripped code
  • A pytest plugin could execute the stripped code
  • Any of these would easily work across RST, Markdown, generic docstrings, and others.

Other considerations

I started a discussion for this at the Sphinx project and included some rough implementation suggestions (with examples) in this comment. But, after consideration, I figured that discussion may be worth having here.

I understand that this may seem like something best implemented by plugins or tools. My motivation for asking here is that I feel something like this has the potential to be quite useful, but also subject to fragmentation, meaning that some tools may begin to adapt control sequences that do not work well with one another. So, my order of acceptance preference from least to most potential for fragmentation is as follows:

  1. Python style suggestion (here)
  2. Sphinx (linked)
  3. Plugin / other tools

Other tags like rustdoc has could also be useful (ignore, should_panic, no_run, edition2018), but these may fit better into the relevant tools that would consume them. As opposed to this nondisplay tag, which would be usable by numerous tools.

I appreciate any thoughts, thank you for reading.

Edit: moved from “documentation” to a more fitting “ideas” with “documentation” tag.
Edit 2: grammar

5 Likes
2

To comment from Sphinx’s perspective - we are supportive of Trevor’s idea, but concerned about making sure we don’t reinvent the wheel if some other similar standard currently exists that we’re unaware of in different programming or markup languages. (re: xkcd 927)

If two different magic comment strings get used by different tools, and you want to use both tools, you’re out of luck.

A

3

Is there a Sphinx syntax that can be interpreted as a code block by doctest, but ignored by Sphinx? Does the following work?

..
   .. code-block:: python
      from typing import Optional, TYPE_CHECKING
      if TYPE_CHECKING:
          from mymodule.something import CoolClass

.. code-block:: python
   def fn(x: Optional[int]) -> CoolClass:
       return CoolClass(x)

..
   .. code-block:: python
      assert fn(5) == CoolClass(5)
4

I am not sure if I completely understand your idea. Are you proposing that all the tools executing the code embedded in a documentation must be changed to interpret the special sequence ##!?

For example you propose that doctest (and similar tools) will execute the code commented out using the special sequence?

import doctest

def example_function() -> None:
    """Just provide an example of doctest.

    >>> result = print('test1')
    >>> ##! assert result is None
    test1
    """
   
doctest.run_docstring_examples(example_function, globals(), verbose=True)

I understand it differently. As I see it the change breaks compatibility with all tools which need to run the code because the code will become comments for them. If you just need to change the rendering of the code, the change should be required just in the documentation generators (like Sphinx).

a) using a Sphinx markup like in the example of Serhiy

b) if not possible, use special directive in comments after the code like usual directives, rough example:

assert result is None # nodoc
5

# nodoc is a reasonable idea, although perhaps not the most discoverable.

A

6

No, all the text is just inside a Docutils comment node. We could make it work, but it would be very verbose, e.g.:

.. .. code-block:: python

   from typing import Optional, TYPE_CHECKING
   if TYPE_CHECKING:
       from mymodule.something import CoolClass

.. code-block:: python
   def fn(x: Optional[int]) -> CoolClass:
       return CoolClass(x)

..  .. code-block:: python

   assert fn(5) == CoolClass(5)

It is a reasonable idea though, and I like the use of reST syntax for it.

A

7

If it is not work, it may be simpler to add an option similar to :emphasize-lines: to hide specified lines or show only specified lines (and add support of ranges).

8

Hey Sehiy and Václav,

I’m on board with something like # nodoc at the end of the line, and it’s certainly not without precedent. I did consider this, so let me explain context -

I’m planning that the main application that would be hiding the lines would be specifically documentation generators, but also know that there are plenty of places with code viewers that provide syntax highlighting but wouldn’t hide them - which is very OK, even preferred (plain rst viewers, markdown online, rst/md editors used by doc writers, etc). Comment-first has the benefit that in these cases, the code displays closer to the intent. Looking at the first and third block of code from my original post, I think the first block (commented) displays the intent of the code better than the third block (with # nodoc at the end) would. It just keeps the emphasis on the interesting lines rather than the setup.

Additionally, I’m looking mostly at use cases outside the scope of doctest, things that need a runner anyway. I mentioned that doctest could accept the syntax, but don’t see it as the main use case by any means; doctest already does the asserts for you so that’s one use case gone. It seems in general, doctest/pycon style is used mostly for quick-setup examples that run contiguously - times when you likely don’t care about things like strict typing, especially for avoiding confusion with beginners. My goal is moreso support for the less introductory examples where e.g.,:

  • You want to show calling examples foo(a) and validate it runs but don’t care about how it returns (or can’t, if it’s something that returns e.g. a repr with memory locations)
  • The above, but you’d like to assert a few things as sanity checks
  • You want to display typing but don’t want to make things confusing by displaying the code in console form
  • You have >2 indented blocks, which are rough to write in pycon

Quick examples:

  • source to this python function page at the Equivalent to:: example for all(). Just easily add a ##! assert(not all([True, False])) and T,T to validate the written example
  • source to this mypy page at this example:
     class A:
     y: ClassVar = 0  # Type implicitly Any!
    Add ##! from typing import ClassVar and run type checking. There’s no pycon/doctest examples at all on that page

I also don’t really think it’s breaking since it’s strictly opt-in - any test/formatting/linting runner would have to support it and could easily be wrapped if needed.

^ again to reiterate, I’m not against # nodoc and would gladly prefer it to nothing

Regarding the examples for sphinx - I am positive that something like this would be workable right away with some small tweaks or a simple plugin. However, couple things -

  1. That’s limited to RST, there’s at least some traction with MyST markdown and other non-RST doc generators
  2. You don’t know whether the code blocks are expected to be connected or disjoint without some extra work
  3. It’s less easy to read and write the source when it’s split up into different blocks, and that’s one main thing I am hoping to avoid. Easier to write a single line of code with a simple tag than needing to add a separate block with the proper tag(s) and indentation (and if you alternate hidden and nonhidden lines, it kind of blows up in size)

People are more likely to use features to write good code in documentation if they’re painless - one of the reasons I think a lot of projects don’t use doctest, it’s not easy to write unless you copy from console (and at that, it’s tricky to format). Ease of use in validating/testing examples is definitely one of the reasons that most Rust projects don’t suffer from nonworking examples in the docs, whereas with Python I’ve come across at least a handful of those (Python is 10x easier to write in the first place, but that’s besides the point).

Thanks for the feedback!

9

How does doctest find code to test? Is it explicitly looking for .. code-block:: directives?

10

I’ll let Adam confirm but I think within sphinx, it just combines testsetup followed by doctest or testcode → testoutput blocks (in any order), followed by testcleanup - all of those within groups (based on the source here.

It probably hands those groups more or less directly off to python’s doctest, which just grabs the code to run and output to validate via regex.

11

stdlib doctest uses a regex as Trevor noted. reST doctests also search for >>> although as part of reST parsing in the state machine. Sphinx’s sphinx.ext.doctest uses explicit .. doctest:: directives with optional setup (example below stolen from docs):

.. testsetup:: *

   import parrot

The parrot module is a module about parrots.

.. doctest::

   >>> parrot.voom(3000)
   This parrot wouldn't voom if you put 3000 volts through it!

I don’t believe there’s anything which looks for literal blocks or the .. code-block:: directive, though I may have misunderstood the question.

A

2 Likes
12

I assume you mean apart from those in the doctest extension, which does handle the specific use case you presented (though not the more general one described by @tgross35 of hiding arbitrary lines).

13

I’m sorry, I’m going to question the entire premise of this discussion.

Doctests are first and foremost documentation. If you are hiding critical parts of the documentation, how does that help the reader? I maintain that makes it worse documentation. At best it is incomplete and at worst it is actively misleading.

Trevor’s example, with setup code hidden:

def fn(x: Optional[int]) -> CoolClass:
    return CoolClass(x)

leaves the reader unable to tell where CoolClass comes from. If they are a beginner, they may not be able to guess that Optional came from typing. They certainly can’t copy that snippet into their own interactive interpreter and run it, at least not in any meaningful way.

I don’t think that hiding parts of the doctest helps the reader, who should be our first priority. Anything which makes the documented code that they see different from the code which is actually run is a problem.

There is already a UI problem with Sphinx hiding doctest directives from the reader. The directive is an important part of the documentation and suppressing it makes the examples harder to understand, not easier but at least it is only a directive, it is not part of the actual code being run.

So I am not convinced that this helps the reader of the documention or makes better docs.

1 Like
14

I disagree. I think including boilerplate in code examples feels like I’m insulting the reader’s intelligence, and that it adds fluff (which must still be read, then ignored) which doesn’t advance the concepts which the docs are trying to convey. I don’t think such boilerplate is critical to the documentation. I don’t have empirical evidence for my belief however.

2 Likes
15

Laurie covered the main points, just that every line really isn’t necessary for every single example.

A frequent documentation pattern is to start with a bigger blob of code with all the context (imports) and maybe a simple example, then alternate between text descriptions and code blocks showing more in-depth usage, within the context of the big block. I don’t think there’s any clarity gained by displaying the few lines of setup in each of the small code blocks. However, these lines might be necessary to the author, to verify their documentation contains valid code. This pattern is pretty frequent even in official python docs - see below, three separate code blocks and only one shows where Fraction comes from.

Source from Fractions docs 
   .. method:: limit_denominator(max_denominator=1000000)

      Finds and returns the closest :class:`Fraction` to ``self`` that has
      denominator at most max_denominator.  This method is useful for finding
      rational approximations to a given floating-point number:

         >>> from fractions import Fraction
         >>> Fraction('3.1415926535897932').limit_denominator(1000)
         Fraction(355, 113)

      or for recovering a rational number that's represented as a float:

         >>> from math import pi, cos
         >>> Fraction(cos(pi/3))
         Fraction(4503599627370497, 9007199254740992)
         >>> Fraction(cos(pi/3)).limit_denominator()
         Fraction(1, 2)
         >>> Fraction(1.1).limit_denominator()
         Fraction(11, 10)


   .. method:: __floor__()

      Returns the greatest :class:`int` ``<= self``.  This method can
      also be accessed through the :func:`math.floor` function:

        >>> from math import floor
        >>> floor(Fraction(355, 113))
        3

Another thought - hidden lines of code could actually be leveraged to make these small blocks of code easier for beginners. If the needed context for small blocks is usually hidden (using the control sequence), it could be optionally displayed via something like a toggle or popup. So setup is kept out of sight when it would be considered extraneous, but available if the user needs assistance with that exact example.

16

I’m somewhere between the two schools of thought. I like examples to be complete (and being able to hide “irrelevant” lines is an attractive nuisance for authors in that sense, as it makes it too easy to decide something is “irrelevant” when it could actually be of benefit to the reader) but I also see that a bunch of imports is of limited value.

Maybe have two directives - doctest and doctest-preamble? The former is displayed, the latter isn’t. This would encourage authors to group their “hidden preamble” stuff in one place without cluttering the explanation with it (maybe the preamble could even be rendered as hidden by default but expandable if the user really wants to see it).

In the original example:

##! from typing import Optional, TYPE_CHECKING
##! if TYPE_CHECKING:
##!     from mymodule.something import CoolClass
def fn(x: Optional[int]) -> CoolClass:
    return CoolClass(x)
##! assert fn(5) == CoolClass(5)

the imports at the top could go in a preamble block. The assert, IMO, should not be hidden, as it demonstrates what the code does. If you don’t like the look of the assert, then either improve the way you present your documentation, or use a normal unit test rather than a doctest - doctests should be documentation first IMO.

3 Likes
17

What should or should not be in the documentation for a project is always subject to debate.
If the doctest contains assert foo._privateattr is None, I can certainly see an argument for keeping it out of the docs.

Such a feature would potentially be misused by some. If you hide critical imports and never show them anywhere in your docs, you wrote bad docs. But the feature doesn’t mandate such misuse. I would tend to look at the benefits of this feature used well, rather than the harm of it being misused.

I think it’s a good idea, but I’m not sure the language itself needs to get involved. If sphinx and a few other tools can establish consensus, wouldn’t that be sufficient? What if sphinx were to support both the prefix syntax and # nodoc for compatibility?

1 Like
18

Great! I think it is the solution, and we should use it in the stdlib docs if it is needed to hide some doctest code.

19

I’m a little confused—how is this different from the existing testetup/doctest/testcleanup trio of directives in Sphinx?

1 Like
20

It probably isn’t, I didn’t even know they existed. I’m really just arguing against the idea of hiding code that’s part of the doctest, unless it’s some sort of clearly defined “setup” that the reader can be assumed to know about. But I’m old-school, and think of doctest very much as described in its own documentation - it “executes those sessions to verify that they work exactly as shown” (my emphasis).

Mostly I don’t care how sphinx users want to mark up their documentation, though.