Pre-PEP: Standardizing test dependency and command specification

Packaging Standards
51

With some of the conflicting requirements being raised here, this is starting to feel a bit like a frontend/backend problem akin to the PEP 517 build system one, only for task management systems like tox/nox/hatch/etc, rather than for building release artifacts.

Suppose the pyproject.toml addition looked like this:

[test-system]
requires = ["tox"]
test-backend = "tox.backend:tests"

Command execution hooks:

  • run_tests: run the test suite in an externally prepared environment
  • run_static_checks: run static checks (linting, typechecking, etc) in an externally prepared environment

Optional environment preparation hooks:

  • get_requires_for_run_tests: report the requirements that need to be available in the run_tests prepared environment
  • prepare_files_for_run_tests: export the test data files from the source tree that are needed when executing the tests
  • prepare_env_for_run_tests: set up the environment for run_tests the way to backend normally would
  • get_requires_for_run_static_checks: report the requirements that need to be available in the run_static_checks prepared environment
  • prepare_env_for_run_static_checks: set up the environment for run_tests the way to backend normally would

(Hook naming scheme intentionally based on the one used in PEP 517)

tox and nox should probably stay as pure backends for this approach, but could optionally decide to support acting as frontends as well. Project management tools like hatch, pdm, etc might decide to offer test subcommands, with --static-checks-only and --no-static-checks options to decide on the scope of the tests to execute. Distro build systems would be frontends (mapping requirements to system packages rather than upstream ones), as would IDEs.

The advantage I see to the narrower proposal focused on just testing is that it means any backend API definition can be more semantically meaningful (just like PEP 517), rather than being so absurdly generic as to be meaningless (which is a problem affecting my corresponding task-runner suggestion in the task management thread).

If we went down this path, future expansion out to also define a generic docs building interface would be via a docs-backend table, again for the semantic benefits that brings to the backend API design process (such as allowing the environment setup to be handled by the frontend, as distros would prefer, even if the normal build process uses a separate automatically created virtual environment).

2 Likes