# Run test suite with -Werror on CI **URL:** \ **Category:** Core Workflow\ **Created:** [September 13, 2019, 4:49pm UTC](https://discuss.python.org/t/run-test-suite-with-werror-on-ci/2333 "2019-09-13T16:49:24Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![xtreak](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/xtreak/32/267_2.png) [@xtreak](https://discuss.python.org/u/xtreak)\ **Post date:** [September 13, 2019, 4:49pm UTC](https://discuss.python.org/t/run-test-suite-with-werror-on-ci/2333/1 "2019-09-13T16:49:24Z")
From time to time there are reports of DeprecationWarning and other warnings in tests. This is mostly due to CI not running with warnings to cause error. Unless it’s executed by someone with -Wall it’s not caught. I think it’s not much expensive and catches issues with tests in this mode. If the warnings are problematic or false positives we can at least restrict the warnings to DeprecationWarning only. I can’t find old threads about this so I thought to open one. Some sample issues - [https://bugs.python.org/issue38120](https://bugs.python.org/issue38120) (Deprecation warning since August 2019) - [https://bugs.python.org/issue37409](https://bugs.python.org/issue37409) (ImportWarning in tests) - While merging [https://bugs.python.org/issue36373](https://bugs.python.org/issue36373) - [https://bugs.python.org/issue37393](https://bugs.python.org/issue37393) ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 17, 2019, 7:31pm UTC](https://discuss.python.org/t/run-test-suite-with-werror-on-ci/2333/2 "2019-09-17T19:31:41Z")
Yes, it sounds like a good idea. I always wanted to do that. We need a buildbot worker to run such job, and modify the buildbot config to write the job config. Ask in the python-buildbot config. ---
**Author:** ![xtreak](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/xtreak/32/267_2.png) [@xtreak](https://discuss.python.org/u/xtreak)\ **Post date:** [September 17, 2019, 9:23pm UTC](https://discuss.python.org/t/run-test-suite-with-werror-on-ci/2333/3 "2019-09-17T21:23:03Z")
Why not in Travis or Azure? Doing it in one of the primary CI catches things earlier before merging. An example of this few days back : [https://github.com/python/cpython/pull/16148](https://github.com/python/cpython/pull/16148) . I think doctest in Travis already runs in similar mode. ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 17, 2019, 9:46pm UTC](https://discuss.python.org/t/run-test-suite-with-werror-on-ci/2333/4 "2019-09-17T21:46:30Z")
I’m not sure if we block to block the workflow (don’t allow to merge any PR anymore) if Python starts to emit a warning. Warnings should not block the workflow, they can be fixed later. ---
**Author:** ![xtreak](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/xtreak/32/267_2.png) [@xtreak](https://discuss.python.org/u/xtreak)\ **Post date:** [September 17, 2019, 9:59pm UTC](https://discuss.python.org/t/run-test-suite-with-werror-on-ci/2333/5 "2019-09-17T21:59:41Z")
IMO DeprecationWarning represents the fact that the test/code already contains deprecated code where the warning might be added in the PR. So fixing them there makes sense instead of primary CI passing and the later buildbot fails to create another PR eventually for the fix. We don’t need to turn on all warnings as failure in that case perhaps restrict it to only certain ones like DeprecationWarning that is frequent. ---
**Author:** ![aeros](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/aeros/32/4140_2.png) [@aeros](https://discuss.python.org/u/aeros)\ **Post date:** [September 21, 2019, 12:45am UTC](https://discuss.python.org/t/run-test-suite-with-werror-on-ci/2333/6 "2019-09-21T00:45:36Z")
> [@vstinner](#): > > I’m not sure if we block to block the workflow (don’t allow to merge any PR anymore) if Python starts to emit a warning. Warnings should not block the workflow, they can be fixed later. Is it possible to emit a lower priority warning that provides log information without causing a CI failure and blocking the PR? `DeprecationWarning`s seem like the perfect scenario to utilize something like that. ---
**Author:** ![ammaraskar](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/ammaraskar/32/32_2.png) [@ammaraskar](https://discuss.python.org/u/ammaraskar)\ **Post date:** [September 21, 2019, 1:43am UTC](https://discuss.python.org/t/run-test-suite-with-werror-on-ci/2333/7 "2019-09-21T01:43:14Z")
> [@aeros](#): > > without causing a CI failure and blocking the PR If it’s just in the CI logs then it’s really easy to get skipped over. I often just make a PR and wait for the green tick, only looking at the logs if a failure occurred. What might be useful is a bot (or Github action) that adds a comment in the PR if a warning occurred. ---
**Author:** ![vstinner](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/vstinner/32/15130_2.png) [@vstinner](https://discuss.python.org/u/vstinner)\ **Post date:** [September 24, 2019, 10:46am UTC](https://discuss.python.org/t/run-test-suite-with-werror-on-ci/2333/8 "2019-09-24T10:46:53Z")
About the Python test suite: it must pass with: python3 -Werror -m test (…).