A test design pattern to reuse integration tests on newly-built distribution packages?

Packaging
1

Are there design patterns for a project’s unit tests and integration tests, which let some of those tests be re-used easily to test distribution packages as well?

I am asking for myself. I write some simple little utility command-line programs in Python. I write lots of unittest test fixtures, even for the little programs. Some are truly “unit tests”. They exercise a single function or class within the program. But some are better called “integration tests” or “end-to-end tests”. They run the entire command-line program via subprocess.run(), or they call the a high-level entry point to which a [project.scripts] declaration refers.

After a long time jury-rigging ways to run these programs, e.g. by linking to their software development directory, I am learning how to package these applications. I am reading the Python Packaging Guide. I am writing pyproject.toml. I am invoking setuptools to make wheels. I am generating application files with shiv.

Did I get the packaging right? It seems to me that the integration tests ought to be able to test the source distribution or wheel installed to a temporary location, or the application file created by a tool like shiv or pex. It might be just a matter of abstracting the invocation in the correct way, or of putting the right decorator on the test classes and test methods.

I’m sure I could figure out, by myself, a way to do this which works for me. But I would much rather learn from someone who has already figured it out. Are others reusing their integration tests this way? What design pattern do you apply to your testing code to make it easier to do?

I did a little reading, and found these threads, which are almost but not quite what I want:

The thread Should sdists include docs and tests?, here on Discuss, talks a bit about people who distribute packages maybe wanting to do integration testing on the distribution packages in their new environment. They might want to re-use the integration tests and unit tests in a source distribution in the new environment.

The Conda-Forge note on testing Python packages seems to talk about a way to declare test invocations in a conda recipe file. Some of the ideas seem applicable. It does not seem to be a reusable design pattern.

Anything else you would recommend that I read?

2

If your goal is to test against an installed copy of your project instead of against the code in your repository, the solution is to use either tox or nox to install your project in a virtual environment and run your tests in that environment. Be sure you’re also using a src/ layout, or else you run the risk of your tests accidentally importing the copy of your code in your repository instead of the installed copy.

2 Likes
3

Basically, this breaks down into two parts. First, to ensure your tests can be used by downstreams to test against the distributed sdist, just make sure to include your tests in the sdist. This will be true by default if you use a build backend or plugin (e.g. Setuptools-SCM or most non-Setuptools backends) that include all the files checked into VCS in the sdist, and they aren’t otherwise excluded; otherwise, this will depend on where you’ve located your tests, whether you have any non-Python data files and what backend you are using; check your backend’s docs for details. You can check by simply seeing if they are present in the .tar.gz sdist archive.

The other component, which is arguably a lot more important since it applies to both your own testing and that of downstream redistributors, is ensuring you’re testing against your installed package rather than your local source tree in your project directory. For this, in order of highest benefit with lowest cost, you should (some of these overlap, but there’s no harm in doing as many as possible):

  • Use a src directory layout in your package, as recommended by the PyPUG. This ensures that, at the minimum, your local copy is not getting picked up automatically instead of whatever is installed
  • Always invoke your test runner with python -I, e.g. python -I -m pytest, to run in “Isolated mode”, which doesn’t add the current working directory to sys.path (even without src layout) and avoids a few other environment/site dependent error sources (or, better yet, python -I -b -X dev -m <pytest,unittest, etc> to also report warnings and perform extra useful checks, or python -I -bb -X dev -W error -m <pytest,unittest, etc> to treat them as errors and fail the relevant test(s))
  • If using pytest, use --import-mode importlib (ideally in your pytest.ini) to only add your test modules to your sys.path rather than whole directories, or (if you need helper functions outside of fixtures to be importable), at least --import-mode append, will ensure any installed packages/modules get imported ahead of any local ones.
  • On your CIs, since you’re installing a fresh env anyway, instead of running against the source tree or editable install, always build your package with python -m build, install from the built wheel with pip install dist/*.whl, and use the aforementioned method(s) to ensure you’re testing against the wheel. This ensures your package always builds, installs and runs correctly from a clean environment without having to do anything extra locally. As a bonus, run twine check --strict dist/* and pip check following install to perform a few extra packaging checks.
  • Use a tool like Tox or Nox to handle set up and manage a consistent testing environments and workflow, locally and remotely. While not actually absolutely essential if you perform the previous steps and are running CIs before merging to your main branch, this can be very helpful especially on larger projects or with complex workflows.

A src directory is best, of course, but doesn’t Tox automatically, or at least with an option, execute in a temporary working directory and/or strip the CWD from sys.path to avoid this issue? I’m not terribly familiar with this, as I haven’t worked much with any projects that both use a non-src layout and also use Tox/Nox, and my primary line of defense is building and installing from a wheel in CI with either src or python -I pytest + importlib import mode to catch packaging issues before they are ever merged to main).

4 Likes
4

pytest would be the tool that prevents CWD to end up in PYTHONPATH, but only if you call pytest, not if you call python -m pytest. Tox/nox have flags to change directory before running, but you have to activate them.

If you don’t use the src layout you have to constantly be aware of that and actively take measures throughout your entire test suite… For example if you run a subprocess in your tests, you have to remember that CWD might leak (if you are calling sys.executable in your subprocess, then you also have to pass -I to it).

Moreover -I is “very dramatic” and might not be easy to use for simple tests (-P is more appropriate, but it is not available in all versions of Python).

In general, tools nowadays allow you to counter ballance the disadvantages of a “flat” layout, but you still have to be “vigilant”. src-layout is just less error prone for testing.

2 Likes
5

@JDLH , tox allow you to pass a --installpkg to run the tests in a virtual environment with a pre-built wheel installed (nox probably has a similar CLI option).

You can see an example on how to use this flag in GItHub actions here, or in other CIs.

These are some posts that may be helpful:

Both tox and nox have great documentation that may also help a lot.

2 Likes
6

Isn’t that a side-effect of using any standalone entry point, not anything specific to pytest? And can’t the same effect as that be achieved by simply adding -I or -P to the invocation? Also, don’t you mean sys.path (specifically sys.path[0]), rather than PYTHONPATH?

I thought so, thanks; I just wasn’t super familiar with that particular functionality and whether it was default or optional.

Indeed, and in fact no one measure is enough to reliably prevent leakage in all common scenarios, which is why I suggest multiple lines of defense above (with the src layout as the first one):

  • Using a src layout ensures your local package source tree doesn’t leak into sys.path with default Python invocations and (at least in many cases) with pytest --import-mode prepend.
  • An explicit -P does the same (but only when explicitly passed), but also ensures any other Python files in the CWD or any non-source subdirs with __init__,py (e.g. test dirs under certain Pytest import modes) don’t leak either
  • On top of that, using -I instead additionally ensures any custom site configuration and PYTHON* env vars that may be set (including PYTHONPATH) don’t leak into the test env either, which is rarely if ever desirable
  • I’m not 100% sure of the exact interaction, but even with both -I and src, if the test files are inline with the package, pytest’s default --import-mode prepend may still add code under test to the path; --import-mode append avoids shadowing and --import-mode importlib avoids this completely (though the latter at the cost of not being able to import other non-installed, non-fixture test support code).

All it does outside of -P is not running any custom user- or system-specific sitecustomize.py and doesn’t use any PYTHON* environment variables that may have been set somewhere; outside of perhaps very specific edge cases, I’m not sure how I see how that would be harmful (as opposed to beneficial) when testing whether a package installs and runs correctly in a clean env. But maybe there’s something I’m missing here?

Couldn’t agree more there—though even so, there are still some cases that a bit of extra vigilance will still help prevent, even with src layout.

1 Like
7

I am not sure about the internals, maybe it is as you say. However, tox/nox alone will not guarantee CWD will not end up in sys.path. Regarding PYTHONPATH and sys.path, sorry I tend to use these 2 interchangeably, but they are not, you are correct :sweat_smile:.

It is as you said, there are some edge cases when people actually want PYTHON* to be passed. This might be true for PYTHONWARNINGS and other env vars that can be set in a CI environment. Another situation happens when a project ends up being distributed by a OS (e.g. fedora). People involved in “repackaging” may attempt to run the test suite against a OS/temp-dir specific installation (instead of a virtual environment installation) by setting PYTHONPATH. If someone writes all tests that require subprocesses with subprocess.run([sys.executable, "-I", ...]), then these tests will have to be ignored by the OS repackagers.

8

Thank you, @abravalheri . This was very helpful reading, especially Hynek Schlawack’s “Testing & Packaging” post. I appreciate the links.

1 Like