Asserting exact exception instances

Ideas
1

There are two helpers in unittest.TestCase for asserting that exceptions are raised and properties of them:

  • assertRaises checks that an exception is raised and asserts its type
  • assertRaisesRegex checks that an exception is raised and matches its type and str

I generally try to assert something about the message just to make sure the test doesn’t accidentally pass by raising the same exception in a different way, so I almost always use assertRaisesRegex.

But it’s a bit awkward to convert the string I want to match to a regex (and even more clumsy to use re.escape() purely for this.

How about allowing assertRaises to take an exception instance for exact match?

with self.assertRaises(ValueError("object is already real")):
    reify(x)

This is currently rejected with TypeError.

It’s comparatively awkward to use the context object. This doesn’t work

with self.assertRaises(ValueError) as cm:
    reify(x)
self.assertEqual(cm.exception, ValueError("object is already real"))

because

>>> ValueError('foo') == ValueError('foo')
False

This seems like an oversight? Exception.__eq__ could return type(a) == type(b) and a.args == b.args.

The other path that might be helpful is something like assertRaisesStartsWith() or assertRaises(exc, *, startswith: str = "") because a lot of exception messages are like “something bad happened: exact details” so this would allow

with self.assertRaises(Exception, startswith="something bad happened:"):
    unbundle_from(stream)
5 Likes
2

It’s probably a better idea to define custom exception types that don’t need parseable error messages to carry the same information.

class AlreadyRealError(ValueError):
    pass

...

with self.assertRaises(AlreadyRealError):
    reify(x)
1 Like
3

This is just wrong. You probably meant something like

with self.assertRaises(ValueError, errormsg="object is already real"):
    reify(x)

Right? Your code tries to create a new exception instance and use it as an argument of the assertRaises method. That’s certainly not what you wanted.

And even if we accept that, then the question is why not to do:

with self.assertRaisesRegex(ValueError, "object is already real"):
    reify(x)

Remember, the old adage of vi users, “every string is also a regular expression”? What’s wrong with that?

And even if you need to do something fancy with the caught exception (and its message) see the second example in the documentation for assertRaises method. You have the caught exception available, so you can test on that.

Also, my experience with assertRaisesRegex for checking methods OVER WHICH YOU HAVE NO CONTROL is rather mixed. Their maintainers tend to change error messages, and then you get an incredibly brittle test suite (e.g., see for the similar situation Pygments). Quite often, I have to switch it off, Fix etree XMLPullParser tests for Expat >=2.6.0 with reparse deferral · openSUSE-Python/cpython@d7133c7 · GitHub.

4 Likes
4

Sometimes, but in a test we want to validate that exception messages are constructed correctly as well as that the correct type is raised.

Regular expressions have a lot of special characters that need escaping, and those are exactly the characters that appear in the repr of common Python objects. I don’t know about you but I always have to spend ages looking at a failing assertRaisesRegex test to work out what I’ve failed to escape. I know exactly the message that should be raised, the problem is regex. It’s less convenient about 80% of the time imho.

1 Like
5

Testing against an exact error message is usually not a good ide, since error messages are intended to be read by humans. Readability is more important than stability. This is especially important if you have no control on generated error messages. For example, Tkinter raises TclError for all errors, but error messages can vary between versions of the underlying Tcl/Tk library. They can be changed even in a bugfix release.

It is more common to use regular expressions that contains important keywords and ignore the rest of the error message. You can also test other exception attributes (errno, filename, line, offset) to be less depending on error message.

So, this feature has not be implemented because it is less useful than you think.

There are also other problems with this idea. Exceptions usually do not implement __eq__, so different exceptions ara always not equal. How are you going to test the exception? If you only test the type and the string representation, take into account that the string representation is not always equal to the first string argument. It can be constructed from all aruments, and it can depends on the hidden attributes. It may be not stable – depending on the order of keyword arguments. It can include timestamp of the exception creation or IDs of internal objects, unique for every exception. The constructor of the exception can be private. So this solution is not universal.

2 Likes
6

I’m exclusively talking about testing the code in which I construct the error messages, and I disagree that it’s not a good idea. I want to test that the error condition is reported correctly. And we have assertRaisesRegex to do this; my complaint is that writing regexes is often inconvenient.

But yes, I would not expect that users of my library make assertions about the text of errors that they don’t raise.

Exactly, let’s make them implement __eq__. I don’t see any reason why comparing exceptions could not compare their value part (args) and ignore contextual stuff, __traceback__, __cause__ etc.

The same applies for assertRaisesRegex() but we have that, so…? assertRaisesRegex() does something like assert re.match(pat, str(exc)) so assertRaisesStartsWith() would do assert str(exc).startswith(prefix).

So? If it’s my exception type and I can’t test it with assertRaises(MyException(blah)) then and it’s on me to find a way to test it, probably assertRaises(MyException._private_keep_out(blah)). Same if I don’t implement _eq_().

7

with pytest I can use the following code:

    with pytest.raises(
        TypeError,
        match="missing 1 required positional argument: 'url_websocket'",
    ):

which is useful.

If that’s currently hard to achieve with unittest, adding it as a feature sounds like a good idea.

1 Like
8

Unfortunately pytest.raises(match=...) also takes a regex pattern, and asks you to use re.escape() if you don’t want that.

But I think you’re right, comparing exceptions directly might be more challenging and there’s an easy path to improve ergonomics by adding an assert method like

def assertRaisesMessage(
    self,
    expected_exception: Type[BaseException],
    callable: Callable | None = None,
    args: tuple = (),
    kwargs: dict = {},
    msg: str | None = None,
    *,
    equals: str | None  = None,
    startswith: str | None  = None,
    endswith: str | None  = None,
    contains: str | None = None,
) -> _AssertRaisesContext:

That could be used as:

with assertRaisesMessage(ValueError, startswith="object is already real:"):
    reify(x)

or

with assertRaisesMessage(ValueError, contains=repr(x)):
    reify(x)

equals, startswith, endswith, contains seem like a basic set of tests on strings but could be extended with fnmatch or even regex/match to unify and align with assertRaisesRegex/pytest.raises.

1 Like
9

This seems like a relatively easy helper to write (based on assertRaisesRegex). Is it a helper you write in your own code[1], or do you have evidence that people write similar things in other projects? That would be useful information, when deciding whether it’s worth adding it to the stdlib (if no-one else thinks it’s worth the effort of writing, why should we?)

  1. as opposed to just writing the test inline ↩︎

10

I don’t know what’s so complicated about something like this:

with self.assertRaises(SomeException) as cm:
    do_something()

self.assertEqual(str(cm.exception), "something")
1 Like
11

Notably, pytest.raises is nearly identical to the unittest behavior. I think that’s a compliment to unittest.

I sometimes find myself mildly annoyed because I forgot to escape some symbols in the regex match. And then I escape them and happily move on with my day. I’m not sure that there’s a demonstrated need in this thread, and if there is it would be interesting to discuss solutions with pytest before trying to change unittest.

1 Like
12

There are a lot of occurrences in the stdlib tests, seemingly hundreds. Many of these hits seem correct:

Beyond CPython there are more than 10k hits for assertRaisesRegex with re.escape:

This is just cases that are literally using re.escape(). There may be cases that have a manually escaped pattern literal, at the expense of readability.

I’m happy to write it; I think it’s worth the effort. The question is whether it would be accepted, whether it’s worth the cost of maintaining it in the stdlib.

2 Likes
13

I think you’re missing my point. Why is it worth writing to put in the stdlib, but not worth writing for all those cases you identified? It seems like most people (including you) are getting along just fine using re.escape…

14

That can be explained purely by rational application of

For most individual cases it’s not worth writing the helper, it saves 5 seconds a day maybe. Put it in the stdlib and give it to a million Python programmers and it saves years.

5 Likes
15

Here’s a different answer - somebody did write it in the CPython repo (jeffrey.yasskin in r59974 in 2008):

It’s also in Django:

4 Likes
16

I want to express sympathy for this proposal! I very often would have preferred to test against startswith or contains rather than having to deal with regular expressions. I also resort to manually escaping characters or just re.escape(). Having to deal with regular expressions is a conceptual step up that I am sure brings discomfort to the many non-software-developers people that end up writing Python code.

4 Likes
17

Take a look at my plugin pytest-raisin