Run test suite with -Werror on CI

#1

From time to time there are reports of DeprecationWarning and other warnings in tests. This is mostly due to CI not running with warnings to cause error. Unless it’s executed by someone with -Wall it’s not caught. I think it’s not much expensive and catches issues with tests in this mode. If the warnings are problematic or false positives we can at least restrict the warnings to DeprecationWarning only. I can’t find old threads about this so I thought to open one. Some sample issues

3 Likes
#2

Yes, it sounds like a good idea. I always wanted to do that. We need a buildbot worker to run such job, and modify the buildbot config to write the job config. Ask in the python-buildbot config.

#3

Why not in Travis or Azure? Doing it in one of the primary CI catches things earlier before merging. An example of this few days back : https://github.com/python/cpython/pull/16148 . I think doctest in Travis already runs in similar mode.

#4

I’m not sure if we block to block the workflow (don’t allow to merge any PR anymore) if Python starts to emit a warning. Warnings should not block the workflow, they can be fixed later.

#5

IMO DeprecationWarning represents the fact that the test/code already contains deprecated code where the warning might be added in the PR. So fixing them there makes sense instead of primary CI passing and the later buildbot fails to create another PR eventually for the fix. We don’t need to turn on all warnings as failure in that case perhaps restrict it to only certain ones like DeprecationWarning that is frequent.

#6

Is it possible to emit a lower priority warning that provides log information without causing a CI failure and blocking the PR? DeprecationWarnings seem like the perfect scenario to utilize something like that.

#7

If it’s just in the CI logs then it’s really easy to get skipped over. I often just make a PR and wait for the green tick, only looking at the logs if a failure occurred.

What might be useful is a bot (or Github action) that adds a comment in the PR if a warning occurred.

1 Like
#8

About the Python test suite: it must pass with: python3 -Werror -m test (…).