Providing a way to specify how to run tests (and docs?)

[bernatgabor (Bernát Gábor)]

@CAM-Gerlach

This proposed syntax would seem to require the project to use a Python task runner

It’s already the case with package build backend and frontends, so if anything we would just remain consistent and not deviate.

@CAM-Gerlach

There would need to be a hook to invoke a task without (at least Python) environment isolation,

task runners can/should provide a mode to run in host mode, e.g. for tox see GitHub - fedora-python/tox-current-env: tox plugin to run tests in current Python environment that makes this a non-problem.

@CAM-Gerlach

There would need to be sufficient standardization of at least the core tasks that downstreams need ( tests , docs , etc) so they can be consistently programmatically invoked.

Would be fairly trivial to standardize those via defining your test and docs targets for the task runner:

[project.tasks]
requires = ["tox>=4"]
targets.test = ["py-unit", "py-integration"]
targets.docs = ["docs"]

@CAM-Gerlach

• Both downstream tooling and task runners would need to implement support for these interfaces, and project authors adopt compatible versions (or switch task runner).

This would be fairly simple tough on both ends. For task runners we only really need tox, nox and pyinvoke. There’s precedence here, adopting PEP-517 was fairly easy for flit/setuptools. They just had to expose what they already did under a new common API.

@CAM-Gerlach

However, it runs into the issue discussed above—there’s conflation between two levels of the stack, where either a testing tool or a task runner may be invoked here.

Hence why I don’t like it that much. If the user sets up a testing (or documentation) tool here it easily can fail downstream or on another machine because you never addressed all the other factors at play here:

@bernatgabor

@brettcannon

• Install the dependencies (either via an extra or hard-coded in nox or tox)

Other stuff that you most likely also support for a minimal viable product before running the test:

• Discover targets defined (and select the ones to run)
• Altering the current working directory
• Setting environment variables
• Passing through environment variables (which you should always pass and which you should remove)

And then the not must have, but probably nice to have concepts for a more robust/powerful usage:

• per target temporary folder
• setup/teardown commands
• environment reuse between runs

I think it’s similarly important that whatever we come up can live together with task runners and not cannibalize it. It would be a bad place where some of your test setup/teardown logic is in the tasks section and the rest in nox/tox configuration files (ini, toml or python file).