The support for testing exceptions with e.g. self.assertRaises in unittest is good for regular exceptions. However, for exception groups, there appears very little machinery to do this cleanly. Compare this with the level of support PyTest has: How to write and report assertions in tests - pytest documentation
It seems like unittest could be improved in this regard.
What exactly do you need?
It is more of an observation than a specific need. I’ve recently been working more with exception groups, and see some machinery for writing tests aimed at these exists in PyTest. However, I prefer to write tests using unittest (standard libraries preferred over third party packages), and when I went looking for these I saw the functionaity and feature set for this was not present.
I imagine constructs similar to those running around in PyTest’s RaisesGroup, or perhaps just some documentation and examples of using the existing raises() with exception groups, and specifications of defined/expected behaviour and otherwise. In the documentation there are examples of capturing exceptions using the context manager and then performing additional checks on them afterwards, but this feels a little “clunky”.
New methods are added for common checks. For example, recently added assertHasAttr() and assertIsSubclass() are now used hundred times in the Python tests. They also generate better error reports than general `assertTrue(…)`.
assertRaises(ExceptionGroup) only occurrs 23 times (mostly in test_taskgroups). And it is not clear that a helper similar to pytest.RaisesGroup() would help in all these cases. Does it take into account the order of exceptions? How does it handle several exceptions with same type? You need additional checks if it does not match your use case exactly.
Maybe when ExceptionGroup is used more widely and we have more uses in tests, we will recognize the most common pattern and add a method for it.
Exception groups are used widely outside of the standard library, but tend to be in application stacks that already just use pytest because that is the far more powerful norm across so much of the Python ecosystem.
There is room for better exception groups support in unittest itself even if just documenting best practices. PRs welcome. I filed a tracking issue - Create `unittest` assertRaises conventions for exception groups · Issue #137311 · python/cpython · GitHub
There is an internal utility that could possibly be a starting point for this:
Currently we have 4 different APIs to work with Exceptions and ExceptionGroups:
with self.assertRaises(E)self.assertRaises(E, callable)with self.assertRaisesRegex(E, msg)self.assertRaisesRegex(E, msg, callable)This code works as expected:
class ExceptionGroupTests(unittest.TestCase):
def test_exception_group(self):
with self.assertRaises(ExceptionGroup) as exc:
raise ExceptionGroup('A', [TypeError('message1')])
self.assertEqual(len(exc.exception.exceptions), 1)
def raises():
raise ExceptionGroup('A', [TypeError('message2')])
self.assertRaises(ExceptionGroup, raises)
It works as expected, I don’t think that we need to change / add anything here.
One can also check all internals of exc.exception to use native ExceptionGroup APIs like .split, etc.
But, we don’t have a way to make a clean assert that TypeError('message1') (for example) is raised as a part of the group. Now we would have to write something like:
self.assertEqual(
len(exc.exception.subgroup(
lambda exc: (
isinstance(exc, TypeError)
and exc.args[0] == 'message1'
)
).exceptions),
1,
)
which is not really great in terms of usability.
pytest already has an API that we are looking for How to write and report assertions in tests - pytest documentation
There are several use-cases that we want to cover:
raise CustomExceptionGroup('my message', [sublist]) from None was raised with give entries in sublistsublist contains of specific types with specific messages / patternsExceptionGroup object, it would be nice to be able to check whether or not some sub-exception matches. Like pytest does with RaisesGroup(ExceptionGroup('a', [TypeError('b')])).matches(TypeError('b')) will return TrueMy proposal is to add:
.assertRaisesGroup and .assertRaisesGroupRegex methods that will behave very similarly to pytest’s RaisesGegexExamples:
# Asserting custom exception group type:
with self.assertRaisesGroup(TypeError, ValueError, msg=r'group \w+', group=CustomExceptionGroup):
...
# Asserting flattened exception group:
with self.assertRaisesGroup(TypeError, ValueError, flatten=True):
...
# Post-processing exception group after successful raise:
with self.assertRaisesGroup(TypeError, ValueError) as cm:
...
self.assertEqual(len(cm.exception.exceptions), 2)
RaisedExc helper to be able to call self.assertRaisesGroup with specific sub-exception with specific messages:with self.assertRaisesGroup(unittest.RaisedExc(TypeError, msg=r'Got \d+ values')) as cm:
...
Basically, all pytest’s features, except:
check= param, because we can post-process exceptions and assert any things we wantallow_unwrapped= param, because why would we want to mix things up?Does self.assertRaisesGroup(TypeError, ValueError) require that the exception group contans TypeError or ValueError or that it contans TypeError and ValueError?
I do not think that the group would be so useful. Custom exception groups are rare, and you always can post-process exceptions.
Some our tests check more detailed structure of the exception group – they would not benefit much from general assertRaisesGroup() helper. But I do not know how common this in third-party tests. In any case, I suggest trying to convert all existing tests to use the new API and then see how much it helps. Considering that our tests are not representative.
I think the most immediate improvement in the short term would be to improve the documentation/specification over best-practices and expected behaviour of exception groups in unittest, and possibly also some cook-book style recipes of “how would I best test XYZ?”. If nothing else, this might give a nice side by side comparison of a before and after any new functionality is added, and how much or little it might clear things up.
One can also check all internals of
exc.exceptionto use nativeExceptionGroupAPIs
^ This I don’t think is obvious, and would benefit from better documentation.
For items like does this group contain X and Y versus X or Y, and possible ordering concerns, etc., I think that just requires some attempts to use this in third party applications and see which fits best, but ideally to have sensible functionality exposes for most (all?) sensible sounding use cases.
I do not know how common this in third-party tests
I think there is a small bit of a chicken and egg situation going on here. All the while that testability of exception groups is not a first class citizen, that third parties will drift away from raising and catching exception groups, as oppose to regular exceptions.
It means that: it contains just two error instances: TypeError and ValueError in no particular order.
The logic is the same as with assertRaises. This will mean or: self.assertRaisesGroup((TypeError, IndexError), ValueError). So, here we expect two instances: one of them is either TypeError or IndexError and the second one is ValueError in no particular order.
I do not think that the
groupwould be so useful.
Good point! I agree.
What if it contains 3 TypeError and 2 ValueError?
In this case it will require to specify TypeError 3 times and ValueError 2 times. Otherwise, it would fail the test. Otherwise it would be very easy to have more exceptions than expected.
I am not sure this is what would be expected in most case. except *ValueError catches all ValueError, not only a one of them. You need to know in advance the number of exceptions and then write something monstruous like self.assertRaisesGroup(*([ValueError]*n)).
Also, checking if the exception group maptches self.assertRaisesGroup((TypeError, IndexError), (IndexError, ValueError), (ValueError, TypeError)) may be not easy.