Testing doctests breaks src/ layout?

So I’m sure we’ve all heard that the src/ layout is the greatest thing since sliced lists. One of its selling points is that testing a src/-layout project necessarily involves installing in non-development mode, with the result that, when you write import packagename in a tests/test_*.py file, you know you’re importing the installed code rather than what’s in src/, so you know you’re getting what your users are getting.

However, if your code contains any doctests, you’d probably run your tests with pytest --doctest-modules src tests, and it turns out that this eliminates the abovementioned benefit of the src/ layout: pytest, in running the doctests in src, imports the src version of the code, and then when pytest gets around to running the tests in tests, it ends up reusing those imports. You can see this for yourself by adding a function to your code that returns __file__ and then adding a test that checks the function’s return value for '*/site-packages/*'.

Fixing this problem is tricky. The obvious solution of switching the command arguments (to produce pytest --doctest-modules tests src) produces an ImportMismatchError from pytest. The only solution I’ve found (and it only works if you’re running your tests in tox) is to change the src in the command to {envsitepackagesdir}/MODULE, where MODULE is the name of the module file (e.g., packagename.py) or module package that contains your code. The downside to this is that it’s ugly and it causes the report from pytest-cov to list your source files as “.tox/py*/lib/python*/site-packages/packagename/...” rather than as just “src/packagename/...” or “packagename/...”.

Is there a better way of dealing with this?

Edit: I came up with a solution to the “.tox/py*/lib/python*/site-packages/packagename/...” thing. I have to set [coverage:run]parallel = True, NOT set [coverage:run]source, and set [coverage:paths]source to src and .tox/*/site-packages. I don’t really understand coverage’s options.

Is it possible to run doc-tests separately from unit tests? ie, run

pytest --doctest-modules src
pytest tests

Yes, that’s the only other solution I’ve come up with.

For those who are interested, the two basic setups that I’ve found that work while also measuring code coverage are given as minimal tox.ini files below:

  1. As described in the OP: Use {envsitepackagesdir}/MODULE instead of src:

     [tox]
     isolated_build = True
    
     [testenv]
     deps =
         pytest~=5.0
         pytest-cov~=2.0
     commands = pytest {envsitepackagesdir}/foobar.py test
    
     [pytest]
     addopts = --cov=foobar --no-cov-on-fail --doctest-modules
    
     [coverage:run]
     parallel = True
     #source # DO NOT SET
    
     [coverage:paths]
     source =
         src
         .tox/*/site-packages
    
  2. Alternatively, test src and test in separate pytest processes. This requires using coverage instead of pytest-cov as the latter is unable to accumulate coverage data across runs.

     [tox]
     isolated_build = True
    
     [testenv]
     deps =
         coverage~=5.0
         pytest~=5.0
     commands =
         coverage run -m pytest --doctest-modules src
         coverage run -m pytest test
         coverage combine
         coverage report
    
     [coverage:run]
     parallel = True
     source = foobar
    
     [coverage:paths]
     source =
         src
         .tox/*/site-packages
1 Like