Providing a way to specify how to run tests (and docs?)

[bernatgabor (Bernát Gábor)]

@brettcannon

To be even more concrete, VS Code needs access to the command used to run pytest. Why? Because we need to have pytest tell us what tests there are to populate the test explorer.

This sounds to me like you want to standardize the test runners’ interface, not define generic task runners. How do you get from how to run the test suite to what tests would be run? E.g.:

    pytest \
      --cov "{envsitepackagesdir}/tox" \
      --cov-config "{toxinidir}/tox.ini" \
      --junitxml {toxworkdir}/junit.{envname}.xml \
      {posargs:.}

For pytest you can hardcode some flag in there, but what if the user uses another test runner? Or more importantly how do you know which flags are needed for running the test suite (e.g. coverage flags) vs which ones influence the test discovery (which you really care about I feel like).

@CAM-Gerlach

And indeed, it seems to be this environment management functionality that, without a standardized way for consumers to determine a test command will trigger it, and signal that it should be bypassed, appears to be the main potential pitfall for the primary consumers of the metadata proposed here—distros, packaging ecosystems and other downstreams who want to test the package in their respective real-world environments, not the isolated one produced by the upstream’s particular tooling.

I think for a task runners it’s critical to define not just how to run the task but what environment it should be run into. tox e.g. might use this information to create a runtime environment to run the task in it, while downstream build systems (like Fedora/Debian) might use it to make sure their current environment setup satisfies that and fail hard otherwise. See for example this feature addition by the Fedora team to achieve this Ability to disable provisioning · Issue #1921 · tox-dev/tox · GitHub
I don’t think downstream people don’t care about the environment the task needs to be run, just that they’d use it as a check rather than a setup.

@CAM-Gerlach

Could you share a source on that? As I cited above, per the official 2020 Python Developer survey , only 7% of Python developers used Tox in any of their projects, whereas nearly 70% used a testing framework of some kind.

You’re comparing apples to oranges here. The survey wasn’t made in between maintainers of projects that get repackaged downstream. I’m pretty sure that projects that end up being repackaged by various distributions the number is more like 70% use tox, nox, hatch or escons; and 95% have test suites. E.g. the data science community (included in that survey) tends to not write test due to the explorative nature of their task,

@CAM-Gerlach

My point is that we shouldn’t require upstream projects adopt a whole environment management and task running tool just to be able to expose their test dependencies and invocation in a standardized,

We can have a default task runner that basically implements your built-in assumption on how the environment is set up (pip install project and pip install requirements.txt if present, run task with cwd set to project root and inherit all env-vars from the host), but I truly believe that defining how to call targets without specifying in what environment to run it will have very limited benefit.