Providing a way to specify how to run tests (and docs?)

[brettcannon]

@bernatgabor

This sounds to me like you want to standardize the test runners’ interface, not define generic task runners.

I asked that already once and was swiftly shot down, so I’m explicitly not asking that.

@bernatgabor

How do you get from how to run the test suite to what tests would be run?

Right now we run pytest for you, so we can inject the appropriate flags for our discovery (which will be a pytest plug-in when we rewrite that bit of code). See Testing Python in Visual Studio Code for the configuration, but we have you specify what to add on after pytest, not the full command to run.

@bernatgabor

For pytest you can hardcode some flag in there, but what if the user uses another test runner?

Then VS Code won’t support it. Not much I can do about that w/o an API and as I said, got shot down for that idea already. Plus, let’s be honest, I can support the vast majority of people bothering with testing by setting up pytest, and a bit more for unittest. But basically it’s my problem to worry about that detail.

@bernatgabor

Or more importantly how do you know which flags are needed for running the test suite (e.g. coverage flags) vs which ones influence the test discovery (which you really care about I feel like).

Because most people don’t set up their tests to also run coverage with us. My bet is most people make coverage a separate task for faster test execution in the REPL developer loop they are typically in and not something they run quickly and frequently.

@pf_moore

What you actually need, surely, is a way to ask the project “what tests are there?”

As mentioned above, pytest didn’t like that idea.

@pf_moore

After all, the project might not even be using pytest.

To be frank, there are not enough folks who fall outside of pytest and unittest for me to worry about that case by trying harder to get a common test API. Only so much time in the day.

@pf_moore

It sounds like your real requirement is “have a way to get a machine readable list of tests from the project”

I would say that’s ideal, but not a requirement because …

@pf_moore

rather than “know how to run the tests (without any requirements on what, if any, output is produced)”.

… I can at least work with this scenario. Pragmatically, we could check the command for pytest and then do the right thing in our case (same goes for unittest).