Proposal for tests entry point in pyproject.toml

Hi,

I would like to discuss a proposal for a test backend in pyproject.toml. What I wrote here is how I see it from a Nixpkgs distribution point of view.

Abstract

A proposal for defining the entry point and the test dependencies in pyproject.toml.

Rationale

In the past it was common that projects would use the same invocation, python setup.py test, to run tests. This is however no longer the case, with many projects using different test runners or custom entry points.

For many end-users this is not relevant, because they would only use packages. Distributions, on the other hand, typically also run (parts of) the test suites of packages to test whether their build and packaging was correct.

A well-specified method for describing how to run the test suite would ease the process for distributions, and allow for further automation.

Specification

As extension to build-system in the pyproject.toml file I propose we also add a test-system section for running tests using the current interpreter by calling what is in the field test-backend, in an environment that has the package for which the pyproject.toml exists, along with additional test dependencies (the field requires).

An example:

[test-system]
requires = [ pytest mock ]  # PEP 508 specifications.
test-backend = "pytest"

The test-backend should only run tests in the environment (interpreter version, operating system) that is provided by the front-end.

Alternatives

No specification for a test-system

Not standardizing would make automation difficult and lead to further wild growth of entry points.

Use Tox

Some may argue that Tox handles this, so why bother? Tox takes charge of the environment, and that is something distributions do not like because they provide the environment for testing and eventually for the end-user. Using Tox for a single environment is not a problem, however.

Note this was brought up on the setuptools issue tracker:

Maintainer of the tox here (note should be written always with lowercase t - just as pytest). tox by default takes full charge of the environment management via virtualenv, however note this doesn’t have to be the case. the tool is flexible enough to allow any environment management to be plugged in. For example this could be a simple shim instructing the OS to prepare an environment with following packages and then run this as a test. I’m currently in process of reorganizing the internals to make this even easier. Once done, we try out in practice, and if works good we can standardize it. Your current proposal misses tunes of nuances though to make it workable (like environment variables, working directory, test dependencies, test commands, parameterization of the commands, etc).

Yes, I should clarify that in the post. I intend to update the post along the way.

Indeed. After I wrote it I realized e.g. that the current working directory should be usable in case of a runtests.py script and runtests test-backend. Thank you for your feedback.

Hi! Fedora packager here.
We had chat about exactly this issue at EuroPython, and since then we’ve been working on a plugin that makes tox use the environment it is running under: https://github.com/fedora-python/tox-current-env#tox-current-env
Please read the README there as my reply :‍)

I’m looking forward to standardizing something like [test-system], but it seems that doing it right now would be premature. PEP 517 [build-system] is still provisional, after all.

Could you move it to a GitHub Gist or something equivalent? That way it’ll be easier to have context as we go and we can also clearly see what the changes made are (vs the Discourse UI).

That’s not how TOML works. :upside_down_face:


I think it’ll be useful to figure out “what’s needed” for a test runner and then trying to design an interface with that understanding – instead of the other way around.