Add `assertNotWarns` method to `unittest.TestCase`

sobolevn · June 2, 2023, 11:30am

Feature or enhancement

Pitch

Right now most of the methods in unittest.TestCase have assertNot* alternative:

And:

But, for some reason assertNotWarns is not there.
I was not able to find any existing discussions about it.

I think that adding this feature will make quite a lot of sense. Here’s an example:

import warnings

def function(arg):
    if arg is None:
       warnings.warn('Passing None is deprecated', DeprecationWarning)
    return str(arg)  # whatever other logic

So, the test case for None is quite obvious:

with self.assertWarns(DeprecationWarning):
    function(None)

But, there’s no way to ensure that DeprecatedWarning is not raised for other arguments.
I propose to add a new helper that can ensure that there are no warnings:

with self.assertNotWarns(DeprecatedWarning):  # or just `self.assertNotWarns()`
    function(1)

So, warnings might go unseen for quite sometime and mess up the output for users.
I think that the argument can be optional and by default check all Warning subtypes, but if passed - only check specified Warning subtypes.

What alternatives do we have to assertNotWarns? We use warnings.catch_warnings in CPython itself. Like this:

with warnings.catch_warnings(record=True) as warnings_log:
    function(1)

self.assertEqual(warnings_log, [])

So, basically we do the same thing but with some extra verbosity.

I also want to show a contrast with self.assertRaises, which does not have not alternative, because raise stops the program execution and does not require some extra helpers: it either raises or not.

And one last note about assertWarnsRegex, I don’t think that it needs a not alternative, because it is only helpful when asserting the warning message and can easily be exchanged for just assertNotWarns in the negative case.

If others agree that this is a good idea, I would like to send a PR with this feature, since I am interested in unittest module

storchaka · June 2, 2023, 12:02pm

You can write it less verbosely:

with warnings.catch_warnings():
    warnings.simplefilter("error", DeprecationWarning)
    ...

Or just run tests with -We.

But how common is this case? How many times it is tested that no warning is raised in CPython tests, in comparison with uses of assert[Regex]Warns?

iritkatriel · June 2, 2023, 12:12pm

We don’t need assertNotWarns for the same reason that we don’t need assertNotRaises (it fails the test, at least if you use -We).

sobolevn · June 2, 2023, 12:31pm

My use-case was not in CPython’s core tests, but in my custom project, where I wanted to be sure that I do not show warnings in some cases (since one of my users reported a bug in this part).

I also found that we have a special test-only helper called check_no_warnings: cpython/warnings_helper.py at 41de54378d54f7ffc38f07db4219e80f48c4249e · python/cpython · GitHub

» ag check_no_warnings     
Lib/test/test_grammar.py
320:    from test.support.warnings_helper import check_no_warnings
1288:        with self.check_no_warnings(category=SyntaxWarning):

Lib/test/support/warnings_helper.py
110:def check_no_warnings(testcase, message='', category=Warning, force_gc=False):
148:    with check_no_warnings(testcase, category=ResourceWarning, force_gc=True):

Lib/test/test_file.py
209:            with warnings_helper.check_no_warnings(self,

Lib/test/test_http_cookiejar.py
658:        with warnings_helper.check_no_warnings(self):

Right now there are multiple simplefilter("error", ...) usages:

» ag 'simplefilter\(.error'
Lib/test/test_coroutines.py
1682:            warnings.simplefilter("error")
1759:                warnings.simplefilter("error")

Lib/test/test_descr.py
1849:            warnings.simplefilter('error', DeprecationWarning)
1857:            warnings.simplefilter('error', DeprecationWarning)

Lib/test/test_xml_etree.py
3394:            warnings.simplefilter('error', DeprecationWarning)
3414:            warnings.simplefilter('error', DeprecationWarning)
3415:            warnings.simplefilter('error', RuntimeWarning)
3432:            warnings.simplefilter('error', DeprecationWarning)
3433:            warnings.simplefilter('error', RuntimeWarning)

Lib/test/test_itertools.py
33:            warnings.simplefilter("error", category=DeprecationWarning)

Lib/test/test_importlib/test_util.py
372:            warnings.simplefilter('error')

Lib/test/test_importlib/import_/test_fromlist.py
153:                warnings.simplefilter('error', BytesWarning)
163:                warnings.simplefilter('error', BytesWarning)

Lib/test/test_grammar.py
242:                warnings.simplefilter('error', SyntaxWarning)
1297:            warnings.simplefilter('error', SyntaxWarning)
1485:            warnings.simplefilter('error', SyntaxWarning)
1569:            warnings.simplefilter('error', SyntaxWarning)

Lib/test/test_fnmatch.py
210:            warnings.simplefilter('error', Warning)

Lib/test/test_runpy.py
494:            warnings.simplefilter("error", RuntimeWarning)

Lib/test/test_warnings/__init__.py
347:            self.module.simplefilter("error", category=UserWarning)
350:            self.module.simplefilter("error", category=UserWarning)
365:            self.module.simplefilter("error", append=True)
378:            self.module.simplefilter("error")

Lib/test/test_unittest/test_case.py
1501:            warnings.simplefilter("error", RuntimeWarning)
1548:            warnings.simplefilter("error", RuntimeWarning)
1595:            warnings.simplefilter("error", RuntimeWarning)
1639:            warnings.simplefilter("error", RuntimeWarning)

Lib/test/support/warnings_helper.py
39:        warnings.simplefilter('error', SyntaxWarning)

Lib/test/test_re.py
1581:            warnings.simplefilter('error', BytesWarning)
2131:            warnings.simplefilter('error', BytesWarning)

Lib/test/test_codeop.py
288:            warnings.simplefilter('error', SyntaxWarning)
293:            warnings.simplefilter('error', SyntaxWarning)

Lib/test/test_string_literals.py
124:            warnings.simplefilter('error', category=SyntaxWarning)
149:            warnings.simplefilter('error', category=SyntaxWarning)
201:            warnings.simplefilter('error', category=SyntaxWarning)
225:            warnings.simplefilter('error', category=SyntaxWarning)

Lib/test/test_hmac.py
355:            warnings.simplefilter('error', RuntimeWarning)

Lib/test/test_random.py
189:            warnings.simplefilter("error", DeprecationWarning)

Doc/conf.py
44:warnings.simplefilter('error')

And catch_warnings(record=True) hack is also present:

So, I can say that this is a quite popular thing to solve in CPython tests as well.
And the existence of check_no_warnings proves that this is a desired feature.

@iritkatriel I agree that for CPython this is a working alternative, but unittest has many users outside of CPython and it is not easy to just fail on all warnings for many projects. Because of how many warnings there are in average projects with multiple runtime / tests deps.

For example, even my current project I am working on has these warnings in tests:

warning : /Users/sobolev/Desktop/coverage-conditional-plugin/.venv/lib/python3.11/site-packages/pytest_cov/plugin.py:256: PytestDeprecationWarning: The hookimpl CovPlugin.pytest_configure_node uses old-style configuration options (marks or attributes).
Please use the pytest.hookimpl(optionalhook=True) decorator instead
 to configure the hooks.
 See https://docs.pytest.org/en/latest/deprecations.html#configuring-hook-specs-impls-using-markers
  def pytest_configure_node(self, node):

warning : /Users/sobolev/Desktop/coverage-conditional-plugin/.venv/lib/python3.11/site-packages/pytest_cov/plugin.py:265: PytestDeprecationWarning: The hookimpl CovPlugin.pytest_testnodedown uses old-style configuration options (marks or attributes).
Please use the pytest.hookimpl(optionalhook=True) decorator instead
 to configure the hooks.
 See https://docs.pytest.org/en/latest/deprecations.html#configuring-hook-specs-impls-using-markers
  def pytest_testnodedown(self, node, error):

So, I would need to set up filterwarnings in pytest correctly to do a single assert. This is quite complex. We can do better.

But, I don’t push too hard for this feature, because there are some alternatives, but I still think that there are use-cases for it and the implementation is very simple (again, we can take a look at cpython/warnings_helper.py at 41de54378d54f7ffc38f07db4219e80f48c4249e · python/cpython · GitHub)

mgorny · June 2, 2023, 1:05pm

As a distro maintainer, I’d like to point out that “does not warn” kind of tests are one of the worst kinds of PITA for us. It is a nice feature if you’re the project maintainer and want to make sure you catch all the warnings. It is not nice when you deal with a release from 12 months ago that supposedly still works fine but the test suite suddenly starts failing all over the place because some remote dependency deprecated some minor feature.

Again, this is useful information for project maintainers. But it’s not useful for users who just want to make sure that the particular version still works.

Using -Werror has the advantage that we can at least easily filter them out.

sobolevn · June 2, 2023, 2:01pm

Yes, I agree, it is not suited for distro maintainers. My use-case is for direct project maintainers that test their own code.

oscarbenjamin · June 2, 2023, 2:42pm

The point is that the distro maintainer does not write a separate test suite. If your project is included in a distro then both the distro maintainer and the project maintainer will run the same test suite. If assertNotWarns is not suited for distro maintainers then it is also not suited for any project that is included in a distro. Using -Werror means that how to handle warnings is controllable at the time of running the tests: the project maintainer and the distro maintainer can make different choices.

sobolevn · June 2, 2023, 3:17pm

Sorry, then I don’t quite understand (since I am not a distro maintainer).
Can you please specify how my example:

def function(arg):
    if arg is None:
       warnings.warn('Passing None is deprecated', DeprecationWarning)
    return str(arg)  # whatever other logic

and

with self.assertNotWarns(DeprecatedWarning):
    function(1)

affects distro maintainers when this test is included in a test suite?

oscarbenjamin · June 3, 2023, 12:06am

See the point above:

Your code calls something from another library. A new release of that library deprecates the called function and emits a warning. Your test suite turns that warning from the other library into a test failure (because you used assertNotWarns). The distro maintainer cannot run your test suite without this unless they change your test suite (which they do not want to do).

mgorny · June 3, 2023, 4:49am

Just imagine that # whatever other logic calls some function from Python stdlib that is deprecated in a future version of Python.

Catching such deprecation warning is useful to you because it tells you that you need to update the code. However, you are not catching them in general but accidentally catching them when you were looking for something else.

A distro maintainer will not run the test suite on new Python version and have it fail because of this new deprecation. It will be entirely meaningless failure to them since the package clearly works with the new Python version (albeit it started emitting warnings). They aren’t the ones who should see the test failure, yet they now have to either deal with it via patching the test suite or waiting for a fixed release (that may not happen in a long time, or be blocked because some other package pins to an old version).

gpshead · June 3, 2023, 3:59pm

In a broader sense, this is the general problem with warnings: We as code authors write them with good intent, but anytime they show up to the wrong person at the wrong time it is bad (and occasionally too late… as that means they’ve shipped).

From an assertNotWarns API perspective… you really want to encourage people to match their and only their warning so perhaps the only API of that nature should do something to encourage authors to always check the warning message contents rather than only matching the class. ie: only offer assertNotWarnsRegex?

That’d prevent the false positive of a transitive dep of your code causing a DeprecationWarning or whatnot bubbling up and triggering the wrong negative assertion.

Zac-HD · June 9, 2023, 5:28am

I ran into similar issues last year, and made a PR to support with warnings.catch_warnings(action="error" [, category=Warning]): (in 3.11+). Perhaps we should also go through and clean up all the needlessly verbose old-style context managers in the CPython repo, based on your searches!

In any case, I’d strongly prefer to direct users to this functionality, rather than effectively duplicating it in another module and waiting even longer for it to be reliably available.