My experience with CI is next to nil, but what I was thinking was more along the lines of:
- Build Python
- Run the test suite (verifies everything that should pass still does)
- For each open PR
a. extract a specially-named file in that PR (e.g.
testthis.py) (skip if not found) (should be no applying of patch as we are extracting a file)
b. run test and take action based on results
If that’s not feasible, then going with the approach you outlined:
- Add new support decorator
open_bug(xxx) which does the label updates (maybe closing PR in some cases), but whose results are not reported as a failure to CI.
- Perhaps only have these types of tests run with a command-line switch, which is scheduled for once a day or week.
- Merge these tests into the main repo.
So, two questions:
Of course, the final question is whether this is all worth it, but having an idea of the possible solutions would help with that answer as well.