How to set doctest directives globally?

Python Help
1

I am looking for a way to globally set doctest directives embedded in the document, instead of having to repeat them. Is there such a thing?

I love doctest, especially for literate programming style documentation. But one thing that gets me down is the need to repeat directives over and over again. For example, I have a document that repeats #doctest: +NORMALIZE_WHITESPACE forty times in not quite 100 test lines :frowning_face:

The documentation says of directives:

“An example’s doctest directives modify doctest’s behavior for that single example.” (emphasis added)

I have searched for some way to set the directive once, at the beginning of the document, and have it take effect for all examples. I cannot find anything that allows that. Have I missed something?

I know that the person running the doctests using python -m doctest filename.txt can use command line switches such as -o NORMALIZE_WHITESPACE to enable them for the entire document. But that requires the caller to know what directives to apply.

1 Like
2

I have not found a way that will work for all invocations of doctest. The closest I have found is by configuring pytest and running doctests that way.

Edit: Guess I should also mention that sphinx.ext.doctest has doctest_default_flags as well.

3

I confess, I always run doctest from a shell script:

mostly so that i can go doctest module.name or doctest path/to/foo.py without thinking much.
It would be easy to shoehorn default options into such a script, even
options gleaned from a quick scan of the target file.

And interactively I’ve got an alias named doctest which, if I supply
no arguments, runs doctest against the set of “modified” files
according to my VCS.

Cheers,
Cameron Simpson cs@cskk.id.au

4

You can import the doctest module and pass the flags to doctest.testmod. Like this:

import doctest
doctest.testmod(module, optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS)
5

Thanks, but that solution is not something that can be imbedded in the doc string itself, so it still requires the caller to perform it. And it is much less convenient than passing the flags with -o.

6

Why should it be in the docstring if you want to apply it globally? For a module, I like to add a main section like this:

if __name__ == "__main__":
    import doctest
    failures, tests = doctest.testmod(
        optionflags=doctest.NORMALIZE_WHITESPACE | doctest.ELLIPSIS)
    print(f"{__file__}: Tests: {tests}; Failures: {failures}")

python mymodule.py runs the doctests.
And for an application I add a --doctest command line option that runs the tests on all (or the specified) modules.

7

Executive summary:

  • Existing solutions to the problem of having to repeat doctest directives over and over again are not terrible, but they have limitations and problems of their own.

  • Having the ability for doctest documents to specify their own default options would be a nice improvement.

Details follow.

When I say globally, I mean global to a single document. I don’t mean to apply to every document on my computer.

And by document, I mean either an actual .txt or .rst file, but sometimes also an individual docstring in a .py file. But usually the first. (See further discussion on this later.)

The directive belongs to the document because it is part of the design of the examples in that document.

I trust that we agree that if you want to set a flag for a single example, the right place for that directive is next to that example? It would be a horrible design if we had to put that directive in a different file, or give it as a command-line option:

python -m doctest --line 58 +NORMALIZE_WHITESPACE --line 125 +NORMALIZE_WHITESPACE some_document

Aside from the impracticality, this is awful because metadata about the example (the directive needed) is moved far away from the example, in a different file (this would be bad) or even worse, in some README file that the users have to keep referring back to in order to look up the directives they need to prevent the doctests from failing.

So let us agree: when a document containing doctest examples requires certain directives, those directives should be embedded in the document. Agreed?

Often the right place for the directive is right next to the example, and that’s great. But sometimes all the examples, or at many of them, require the same directive, and it becomes annoying and impractical to annotate every example with the same directive.

There’s got to be a better way than to repeat yourself over and over and over and over and over again.

I’m suggesting that there ought to be a way for a document to specify the default settings once, and have those settings apply globally (in that document) unless overridden with a directive.

Cons:

  • additional complexity in the doctest framework, of course;
  • it may be a little bit less obvious why a test example passes instead of failing;
  • if you copy the test example into another document, the global settings may be different and it may fail.

Pros:

  • To the reader of the tests: the directives are usually just noise. Removing them from the example makes the example prettier and easier to read.
  • For the writer of the tests: Don’t Repeat Yourself. Avoids needing to add the same directive over and over and over and over and over again.
  • Helps avoid long lines or ugly work-arounds like moving the directive to the next line.

Sure, I’ve done that too. It’s not a bad solution, for a module at least the settings are in the same file, but it has some problems and doesn’t cover all the cases.

For a module, your option flags are global to all the individual docstrings. If one docstring needs whitespace normalised and another doesn’t, there’s no one global setting you can set the optionflags to which will work correctly for both docstrings.

When my docstrings are short, I don’t mind adding a few directives here and there, and the issue usually doesn’t come up.

I’ve been using the term “document” because the case I most care about is when I have a document (usually ReST) containing embedded doctests. In other words, “literate testing”. In this case, the document is a .txt or .rst file, its not executable.

(We can generalise this case to individual docstrings in a .py file, but that’s not the motivating example for me.)

So that’s the first problem: I have to move the directives out of the document where they belong, into a second file which gets executed as a script to run doctest over the document.

I hope that we agreed earlier that this was bad.

The document might not even be testing a single module. It might be a tutorial, or how to guide, or something else. So in this case I would have to create a separate script as an executable facade that calls doctest with the right option flags.

Now this isn’t an awful solution, but it is an annoyance.

And users who famously never read the documentation :wink: have to remember to only run the facade ./rundoctests document.rst instead of the more idiomatic python -m doctest document.rst.

8

So, I understand that your document is like one big docstring. And I think that even in a module it can be convenient to be able to set options globally for a whole docstring. So I guess this could be solved by some doctest function to dynamically change (some of?) the options. Like we can do now for numpy using numpy.set_printoptions. So we could have something like

"""This docstring changes the doctest options

>>> doctest.set_options(doctest.NORMALIZE_WHITESPACE)
>>> print("a     b")
a b
"""

It would be good to have the function available in the environment where the doctest is executed, so that no extra ‘import doctest’ line is needed.