PEP 785 – New methods for easier handling of ExceptionGroups

PEPs
1

As PEP 654 ExceptionGroup has come into widespread use across the Python community, some common but awkward patterns have emerged. We therefore propose adding two new methods to exception objects:

  • BaseExceptionGroup.flat_exceptions(), returning the ‘leaf’ exceptions as a list, with each traceback composited from any intermediate groups.
  • BaseException.preserve_context(), a context manager which saves and restores the self.__context__ attribute of self, so that re-raising the exception within another handler does not overwrite the existing context.

We expect this to enable more concise expression of error handling logic in many medium-complexity cases. Without them, exception-group handlers will continue to discard intermediate tracebacks and mis-handle __context__ exceptions, to the detriment of anyone debugging async code.

Comments and questions welcome!

7 Likes
2

I will say the name flat_exceptions() confused me initially as a “flat exception” isn’t a thing.

3

Ah, right. The origin of that name was helpers named flatten_*, then seeking a closer match to the .exceptions attribute, and clarifying that it returns list[BaseException] rather than BaseExceptionGroup.

On reflection, I’ll update this to be the .leaf_exceptions() method, matching the terminology used in PEP-654.

6 Likes
4

Curious if @iritkatriel or @yselivanov have thoughts on PEP 785 ExceptionGroups methods?

5

Thanks for the ping. I agree that some utilities to help manipulate ExceptionGroups would be handy, so thanks for opening this up for discussion. I will focus for now on the first proposal, leaf_exceptions.

The appeal of leaf_exceptions() is in its apparent simplicity, but I think the PEP is glossing over some of the complexities in it. For example, ExceptionGroups should be trees of exceptions, but then again the context chain should be a chain and the interpreter still handles cycles in it. What will these utilities do when the tree is not a tree? The PEP should specify that.

The destructive nature of leaf_exceptions seems problematic. You wouldn’t necessarily want this:

>>> from leaf import leaf_exceptions
... 
... def raise_group():
...     excs = []
...     for t in [ValueError, TypeError]:
...         try:
...             raise t() 
...         except Exception as e:
...             excs.append(e)
...     raise ExceptionGroup("eg", excs)
... 
... try:
...     raise_group()
... except* Exception as e:
...     leaf_exceptions(e)
...     raise
... 
[ValueError(), TypeError()]
  + Exception Group Traceback (most recent call last):
  |   File "<python-input-0>", line 13, in <module>
  |     raise_group()
  |     ~~~~~~~~~~~^^
  |   File "<python-input-0>", line 10, in raise_group
  |     raise ExceptionGroup("eg", excs)
  | ExceptionGroup: eg (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "<python-input-0>", line 13, in <module>
    |     raise_group()
    |     ~~~~~~~~~~~^^
    |   File "<python-input-0>", line 10, in raise_group
    |     raise ExceptionGroup("eg", excs)
    |   File "<python-input-0>", line 7, in raise_group
    |     raise t()
    | ValueError
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "<python-input-0>", line 13, in <module>
    |     raise_group()
    |     ~~~~~~~~~~~^^
    |   File "<python-input-0>", line 10, in raise_group
    |     raise ExceptionGroup("eg", excs)
    |   File "<python-input-0>", line 7, in raise_group
    |     raise t()
    | TypeError
    +------------------------------------
>>>

In order to predict how leaf_exceptions will impact the different tracebacks in an exception group, you need to know how it is implemented. How will this be documented?

I think an iterator utility should be non-destructive, returning immutable views or complete copies. It’s not hard to copy the traceback, but it may not be possible to copy leaf exceptions (see all the problems with pickling exceptions). We presented a non-destructive iterator version of leaf_exceptions in PEP 654. It emits exc, tb pairs, without modifying the traceback on any exceptions. It has its problems too.

Our view when we wrote PEP 654 is that iteration is not, typically, the right tool for handling ExceptionGroups. This is why we decided not to make them iterable containers. Our thinking was that split and subgroup will be used to sift through the groups. That this would be less error prone, and usually more efficient.

In the motivating example, you wanted to split out the “first” exception from the subgroup of HTTPExceptions. Split/subgroup can do things like this, because they can take a predicate function. Can we design some utilities that will make split and subgroup easier to use for such tasks? Here’s one for your case:

>>> class OneItem:
...     def __init__(self):
...         self.count = 0
...     def __call__(self, e):
...         if not isinstance(e, BaseExceptionGroup):
...             self.count += 1
...             return self.count == 1
...         return 0
...
>>> g
ExceptionGroup('eg', [HTTPException(1), HTTPException(2)])
>>> g.split(OneItem())
(ExceptionGroup('eg', [HTTPException(1)]), ExceptionGroup('eg', [HTTPException(2)]))

Rather than (destructively) denormalising the whole tree just to determine whether there are multiple HTTPException instances, we could provide a Count() utility to pass to subgroup or split.

I can also see us adding a utility to (destructively) transform a single-leaf exception group into a singleton exception with a flat traceback. It should look like a destructive transformation rather than an iteration utility.

6

Regarding the .preserve_context() method, it seems that the intended use of that is always

with e.preserve_context():
    raise e

It would be a nicer API if that could be a single line. Can we make something like raise e.with_preserved_context() work?

7

Another point that needs to be considered is that exception groups can have cause, context and notes. What happens to them when the group is flattened?

8

Thanks Irit, I really appreciate the writeup.

It’s easy to create a context cycle - e = ValueError() ; e.__context__ = e ; raise e, but I’m not sure how I’d go about creating a cyclic ExceptionGroup given that .exceptions is a tuple and the attribute can’t be reassigned. On the other hand a ‘diamond’ group is also easy to create:

v = ValueError()
raise ExceptionGroup("", [ExceptionGroup("", [v]), ExceptionGroup("", [v])])

and it’s clear that there’s no correct way to update the tracebacks in such cases.

I agree this is problematic, though I’d accepted it after looking at the destructive nature of the existing .with_traceback() method.

After reflecting on your proposal though, and working through some examples with dag-not-tree ExceptionGroups, I’m leaning towards .leaf_exceptions() returning an iterable of (exception, traceback) pairs - much like the example code in PEP-654, but with the tracebacks assembled into a single new traceback rather than a list of parts, so that users can:

for exc, tb in group.leaf_exceptions():
    with exc.preserve_context():
        raise exc.with_traceback(tb)

I’m not wholly convinced though, because I expect that many users will omit the .with_traceback(tb), and so we’d be giving up a fair chunk of the motivating benefit.

I don’t think we can do this with the raise statement. A .raise_preserving_context() -> NoReturn method would be feasible, but I continue to prefer the more visibly unusual context manager form.

The intermediate groups are not leaves, so none of this information is returned from .leaf_exceptions(). Of course, this is an additional reason to encourage users towards the .split()/.subgroup() methods, or except*, whenever those can be used to solve the problem at hand.

9

This is why PEP 654 did not make exception groups iterable. In most (all?) use cases for iteration we could think of there is a better way to do it, and support for iteration would nudge users towards the wrong way.