I would like to discuss a proposal for a test backend in
pyproject.toml. What I wrote here is how I see it from a Nixpkgs distribution point of view.
A proposal for defining the entry point and the test dependencies in
In the past it was common that projects would use the same invocation,
python setup.py test, to run tests. This is however no longer the case, with many projects using different test runners or custom entry points.
For many end-users this is not relevant, because they would only use packages. Distributions, on the other hand, typically also run (parts of) the test suites of packages to test whether their build and packaging was correct.
A well-specified method for describing how to run the test suite would ease the process for distributions, and allow for further automation.
As extension to
build-system in the
pyproject.toml file I propose we also add a
test-system section for running tests using the current interpreter by calling what is in the field
test-backend, in an environment that has the package for which the
pyproject.toml exists, along with additional test dependencies (the field
[test-system] requires = [ pytest mock ] # PEP 508 specifications. test-backend = "pytest"
The test-backend should only run tests in the environment (interpreter version, operating system) that is provided by the front-end.
No specification for a test-system
Not standardizing would make automation difficult and lead to further wild growth of entry points.
Some may argue that Tox handles this, so why bother? Tox takes charge of the environment, and that is something distributions do not like because they provide the environment for testing and eventually for the end-user. Using Tox for a single environment is not a problem, however.
Note this was brought up on the
setuptools issue tracker: