# A test design pattern to reuse integration tests on newly-built distribution packages? **URL:** \ **Category:** Packaging\ **Created:** [July 3, 2022, 4:46am UTC](https://discuss.python.org/t/a-test-design-pattern-to-reuse-integration-tests-on-newly-built-distribution-packages/17006 "2022-07-03T04:46:21Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![JDLH](https://avatars.discourse-cdn.com/v4/letter/j/77aa72/32.png) [@JDLH](https://discuss.python.org/u/JDLH)\ **Post date:** [July 3, 2022, 4:46am UTC](https://discuss.python.org/t/a-test-design-pattern-to-reuse-integration-tests-on-newly-built-distribution-packages/17006/1 "2022-07-03T04:46:21Z")
Are there design patterns for a project’s unit tests and integration tests, which let some of those tests be re-used easily to test distribution packages as well? I am asking for myself. I write some simple little utility command-line programs in Python. I write lots of `unittest` test fixtures, even for the little programs. Some are truly “unit tests”. They exercise a single function or class within the program. But some are better called “integration tests” or “end-to-end tests”. They run the entire command-line program via `subprocess.run()`, or they call the a high-level entry point to which a `[project.scripts]` declaration refers. After a long time jury-rigging ways to run these programs, e.g. by linking to their software development directory, I am learning how to package these applications. I am reading the Python Packaging Guide. I am writing `pyproject.toml`. I am invoking **setuptools** to make wheels. I am generating application files with `shiv`. Did I get the packaging right? It seems to me that the integration tests ought to be able to test the source distribution or wheel installed to a temporary location, or the application file created by a tool like `shiv` or `pex`. It might be just a matter of abstracting the invocation in the correct way, or of putting the right decorator on the test classes and test methods. I’m sure I could figure out, by myself, a way to do this which works for me. But I would much rather learn from someone who has already figured it out. Are others reusing their integration tests this way? What design pattern do you apply to your testing code to make it easier to do? I did a little reading, and found these threads, which are almost but not quite what I want: The thread [**Should sdists include docs and tests?**](https://discuss.python.org/t/should-sdists-include-docs-and-tests/14578), here on Discuss, talks a bit about people who distribute packages maybe wanting to do integration testing on the distribution packages in their new environment. They might want to re-use the integration tests _and_ unit tests in a source distribution in the new environment. The Conda-Forge note on [**testing Python packages**](https://conda-forge.org/docs/maintainer/adding_pkgs.html#testing-python-packages) seems to talk about a way to declare test invocations in a conda recipe file. Some of the ideas seem applicable. It does not seem to be a reusable design pattern. Anything else you would recommend that I read? ---
**Author:** ![jwodder](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/jwodder/32/236_2.png) [@jwodder](https://discuss.python.org/u/jwodder)\ **Post date:** [July 3, 2022, 5:11am UTC](https://discuss.python.org/t/a-test-design-pattern-to-reuse-integration-tests-on-newly-built-distribution-packages/17006/2 "2022-07-03T05:11:26Z")
If your goal is to test against an installed copy of your project instead of against the code in your repository, the solution is to use either [tox](https://tox.readthedocs.io) or [nox](https://nox.thea.codes) to install your project in a virtual environment and run your tests in that environment. Be sure you’re also using a `src/` layout, or else you run the risk of your tests accidentally importing the copy of your code in your repository instead of the installed copy. ---
**Author:** ![CAM-Gerlach](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cam-gerlach/32/3688_2.png) [@CAM-Gerlach](https://discuss.python.org/u/CAM-Gerlach)\ **Post date:** [July 3, 2022, 6:50am UTC](https://discuss.python.org/t/a-test-design-pattern-to-reuse-integration-tests-on-newly-built-distribution-packages/17006/3 "2022-07-03T06:50:11Z")
Basically, this breaks down into two parts. First, to ensure your tests can be used by downstreams to test against the distributed sdist, just make sure to include your tests in the sdist. This will be true by default if you use a build backend or plugin (e.g. Setuptools-SCM or most non-Setuptools backends) that include all the files checked into VCS in the sdist, and they aren’t otherwise excluded; otherwise, this will depend on where you’ve located your tests, whether you have any non-Python data files and what backend you are using; check your backend’s docs for details. You can check by simply seeing if they are present in the `.tar.gz` sdist archive. The other component, which is arguably a lot more important since it applies to both your own testing and that of downstream redistributors, is ensuring you’re testing against your _installed package_ rather than your _local source tree_ in your project directory. For this, in order of highest benefit with lowest cost, you should (some of these overlap, but there’s no harm in doing as many as possible): - Use a `src` directory layout in your package, as recommended by the PyPUG. This ensures that, at the minimum, your local copy is not getting picked up automatically instead of whatever is installed - Always invoke your test runner with `python -I`, e.g. `python -I -m pytest`, to run in “Isolated mode”, which doesn’t add the current working directory to `sys.path` (even without `src` layout) and avoids a few other environment/site dependent error sources (or, better yet, `python -I -b -X dev -m ` to also report warnings and perform extra useful checks, or `python -I -bb -X dev -W error -m ` to treat them as errors and fail the relevant test(s)) - If using `pytest`, use `--import-mode importlib` (ideally in your `pytest.ini`) to only add your test modules to your `sys.path` rather than whole directories, or (if you need helper functions outside of fixtures to be importable), at least `--import-mode append`, will ensure any installed packages/modules get imported ahead of any local ones. - On your CIs, since you’re installing a fresh env anyway, instead of running against the source tree or editable install, always build your package with `python -m build`, install from the built wheel with `pip install dist/*.whl`, and use the aforementioned method(s) to ensure you’re testing against the wheel. This ensures your package always builds, installs and runs correctly from a clean environment without having to do anything extra locally. As a bonus, run `twine check --strict dist/*` and `pip check` following install to perform a few extra packaging checks. - Use a tool like Tox or Nox to handle set up and manage a consistent testing environments and workflow, locally and remotely. While not actually absolutely essential if you perform the previous steps and are running CIs before merging to your main branch, this can be very helpful especially on larger projects or with complex workflows. > [@jwodder](#): > > Be sure you’re also using a `src/` layout, or else you run the risk of your tests accidentally importing the copy of your code in your repository instead of the installed copy. A `src` directory is best, of course, but doesn’t Tox automatically, or at least with an option, execute in a temporary working directory and/or strip the CWD from `sys.path` to avoid this issue? I’m not terribly familiar with this, as I haven’t worked much with any projects that both use a non-src layout and also use Tox/Nox, and my primary line of defense is building and installing from a wheel in CI with either `src` or `python -I pytest` + `importlib` import mode to catch packaging issues before they are ever merged to `main`). ---
**Author:** ![abravalheri](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/abravalheri/32/4990_2.png) [@abravalheri](https://discuss.python.org/u/abravalheri)\ **Post date:** [July 3, 2022, 7:21am UTC](https://discuss.python.org/t/a-test-design-pattern-to-reuse-integration-tests-on-newly-built-distribution-packages/17006/4 "2022-07-03T07:21:37Z")
> [@CAM-Gerlach](#): > > A `src` directory is best, of course, but doesn’t Tox automatically, or at least with an option, execute in a temporary working directory and/or strip the CWD from `sys.path` to avoid this issue? `pytest` would be the tool that prevents CWD to end up in PYTHONPATH, but only if you call `pytest`, not if you call `python -m pytest`. Tox/nox have flags to change directory before running, but you have to activate them. If you don’t use the `src` layout you have to constantly be aware of that and actively take measures throughout your entire test suite… For example if you run a subprocess in your tests, you have to remember that CWD might leak (if you are calling `sys.executable` in your subprocess, then you also have to pass `-I` to it). Moreover `-I` is “very dramatic” and might not be easy to use for simple tests (`-P` is more appropriate, but it is not available in all versions of Python). In general, tools nowadays allow you to counter ballance the disadvantages of a “flat” layout, but you still have to be “vigilant”. `src`-layout is just less error prone for testing. ---
**Author:** ![abravalheri](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/abravalheri/32/4990_2.png) [@abravalheri](https://discuss.python.org/u/abravalheri)\ **Post date:** [July 3, 2022, 7:36am UTC](https://discuss.python.org/t/a-test-design-pattern-to-reuse-integration-tests-on-newly-built-distribution-packages/17006/5 "2022-07-03T07:36:27Z")
@JDLH , `tox` allow you to pass a `--installpkg` to run the tests in a virtual environment with a pre-built wheel installed (`nox` probably has a similar CLI option). You can see an example on how to use this flag in GItHub actions [here](https://github.com/pyscaffold/pyscaffold/blob/master/src/pyscaffold/templates/github_ci_workflow.template), or in [other CIs](https://github.com/pyscaffold/pyscaffold/blob/master/.cirrus.yml). These are some posts that may be helpful: - [Managing a Project’s Virtualenvs with tox | seanh.cc](https://www.seanh.cc/2018/09/01/tox-tutorial/) - [Testing & Packaging](https://hynek.me/articles/testing-packaging/) - [Testing your python package as installed](https://blog.ganssle.io/articles/2019/08/test-as-installed.html) Both tox and nox have great documentation that may also help a lot. ---
**Author:** ![CAM-Gerlach](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/cam-gerlach/32/3688_2.png) [@CAM-Gerlach](https://discuss.python.org/u/CAM-Gerlach)\ **Post date:** [July 3, 2022, 7:54am UTC](https://discuss.python.org/t/a-test-design-pattern-to-reuse-integration-tests-on-newly-built-distribution-packages/17006/6 "2022-07-03T07:54:53Z")
> [@abravalheri](#): > > `pytest` would be the tool that prevents CWD to end up in PYTHONPATH, but only if you call `pytest`, not if you call `python -m pytest`. Isn’t that a side-effect of using any standalone entry point, not anything specific to `pytest`? And can’t the same effect as that be achieved by simply adding `-I` or `-P` to the invocation? Also, don’t you mean `sys.path` (specifically `sys.path[0]`), rather than `PYTHONPATH`? > [@abravalheri](#): > > Tox/nox can have flags to change directory before running, but you have to activate them. I thought so, thanks; I just wasn’t super familiar with that particular functionality and whether it was default or optional. > [@abravalheri](#): > > If you don’t use the `src` layout you have to constantly be aware of that and actively take measures throughoutyour entire test suite… Indeed, and in fact no one measure is enough to reliably prevent leakage in all common scenarios, which is why I suggest multiple lines of defense above (with the `src` layout as the first one): - Using a `src` layout ensures your local package source tree doesn’t leak into `sys.path` with default Python invocations and (at least in many cases) with pytest `--import-mode prepend`. - An explicit `-P` does the same (but only when explicitly passed), but also ensures any other Python files in the CWD or any non-source subdirs with ` __init__ ,py` (e.g. test dirs under certain Pytest import modes) don’t leak either - On top of that, using `-I` instead additionally ensures any custom `site` configuration and `PYTHON*` env vars that may be set (including `PYTHONPATH`) don’t leak into the test env either, which is rarely if ever desirable - I’m not 100% sure of the exact interaction, but even with both `-I` and `src`, if the test files are inline with the package, pytest’s default `--import-mode prepend` _may_ still add code under test to the path; `--import-mode append` avoids shadowing and `--import-mode importlib` avoids this completely (though the latter at the cost of not being able to import other non-installed, non-fixture test support code). > [@abravalheri](#): > > Moreover `-I` is “very dramatic” and might not be useful for common tests (`-P` is more appropriate, but it is not available in all versions of Python). All it does outside of `-P` is not running any custom user- or system-specific `sitecustomize.py` and doesn’t use any `PYTHON*` environment variables that may have been set somewhere; outside of perhaps very specific edge cases, I’m not sure how I see how that would be harmful (as opposed to beneficial) when testing whether a package installs and runs correctly in a clean env. But maybe there’s something I’m missing here? > [@abravalheri](#): > > In general, tools nowadays allow you to counter ballance the disadvantages of a “flat” layout, but you still have to be “vigilant”. `src`-layout is just less error prone for testing. Couldn’t agree more there—though even so, there are still some cases that a bit of extra vigilance will still help prevent, even with `src` layout. ---
**Author:** ![abravalheri](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/abravalheri/32/4990_2.png) [@abravalheri](https://discuss.python.org/u/abravalheri)\ **Post date:** [July 3, 2022, 8:28am UTC](https://discuss.python.org/t/a-test-design-pattern-to-reuse-integration-tests-on-newly-built-distribution-packages/17006/7 "2022-07-03T08:28:46Z")
> [@CAM-Gerlach](#): > > Isn’t that a side-effect of using any standalone entry point, not anything specific to `pytest`? And can’t the same effect as that be achieved by simply adding `-I` or `-P` to the invocation? Also, don’t you mean `sys.path` (specifically `sys.path[0]`), rather than `PYTHONPATH`? I am not sure about the internals, maybe it is as you say. However, `tox`/`nox` alone will not guarantee CWD will not end up in `sys.path`. Regarding `PYTHONPATH` and `sys.path`, sorry I tend to use these 2 interchangeably, but they are not, you are correct 😅. > [@CAM-Gerlach](#): > > All it does outside of `-P` is not running any custom user- or system-specific `sitecustomize.py` and doesn’t use any `PYTHON*` environment variables that may have been set somewhere; outside of perhaps very specific edge cases, I’m not sure how I see how that would be harmful (as opposed to beneficial) when testing whether a package installs and runs correctly in a clean env. But maybe there’s something I’m missing here? It is as you said, there are some edge cases when people actually want `PYTHON*` to be passed. This might be true for `PYTHONWARNINGS` and other env vars that can be set in a CI environment. Another situation happens when a project ends up being distributed by a OS (e.g. fedora). People involved in “repackaging” may attempt to run the test suite against a OS/temp-dir specific installation (instead of a virtual environment installation) by setting `PYTHONPATH`. If someone writes all tests that require subprocesses with `subprocess.run([sys.executable, "-I", ...])`, then these tests will have to be ignored by the OS repackagers. ---
**Author:** ![JDLH](https://avatars.discourse-cdn.com/v4/letter/j/77aa72/32.png) [@JDLH](https://discuss.python.org/u/JDLH)\ **Post date:** [July 3, 2022, 7:43pm UTC](https://discuss.python.org/t/a-test-design-pattern-to-reuse-integration-tests-on-newly-built-distribution-packages/17006/8 "2022-07-03T19:43:23Z")
> [@abravalheri](#): > > …These are some posts that may be helpful … Thank you, @abravalheri . This was very helpful reading, especially [Hynek Schlawack’s “Testing & Packaging” post](https://hynek.me/articles/testing-packaging/). I appreciate the links.