Accepting PEP 654 (Exception Groups and except*)

My congratulations!

3 Likes

I’m hoping that we can start a discussion on how exactly these will be implemented. We have just one opportunity to get it right.

1 Like

I’m particularly interested in backporting exception groups to earlier Python versions so we can get an early start with trio and anyio. Since the new syntax won’t be available until 3.11, we would at least need some helper functions. I have a 99% complete PEP 654 implementation here.

I’d also love a backports.exceptiongroup package for Pytest and Hypothesis, though contra this advice I think we should just provide the builtin version if it’s available - having to catch either-of-maybe-two implementations sounds awful. Let me know if I can help out somehow?

Code review and experimenting with the code would be great help.

The code will be committed in three stages:

  1. ExceptionGroup / BaseExceptionGroup classes
  2. Updates to traceback machinery to display them nicely
  3. except*

We are now reviewing the PR for the first step: bpo-45292: [PEP 654] add the ExceptionGroup and BaseExceptionGroup classes by iritkatriel · Pull Request #28569 · python/cpython · GitHub

How would you use ExceptionGroups in 3.10 without except*?

The part that I really want is an ecosystem-wide “collection of exceptions” type, which enables nice features like good reporting, debugger support, and so on.

Most of that for me is Hypothesis’ report_multiple_bugs setting: we collect exceptions from each failing input, and want to show the user each distinct error. Currently that means we just print each (trimmed) traceback - it works, but misses out on things like Pytests’s configurable colored and expanded tracebacks, debugger suport, and so on.

Review incoming - I found it very surprising that BaseExceptionGroup("", [Exception()]) returns a non-Base exceptiongroup object, but the rest looks good.

From a few hours of experimenting, the only thing I’m missing is a way to annotate each nested exception. In my case that’s something like Falsifying example: test(x=False), but I think the same consideration applies to use motivated by callbacks (which does the exception come from?) or async tasks - the traceback alone is not always sufficient identifying information.

raise FalsifyingExample("test(x=False)") from err for each of the nested exceptions provides a workaround, but that does feel like an ugly hack.

3 Likes

The plan was to update GitHub - python-trio/exceptiongroup with the PEP 654 code and documentation, and to add any necessary utility functions to support their use without the except* syntax. But since the code would be overwritten anyway, any repo would do I suppose.

@njs thoughts?

Perhaps a new backports.exceptiongroup package?

We now have pytest and better-exceptions integration for MultipleFailures, which demonstrates that this will be easy to support upstream for ExceptionGroup - if a little fiddlier for nested than flat groups.

BPO-45607 suggests adding an assignable __note__ attribute, which would give us a much cleaner alternative :smiling_face_with_three_hearts:

I have a backport here now. I’m soliciting comments on it before I publish it to PyPI.

One open question is how to name the project. The package itself is backports.exceptiongroups but there is an existing PyPI name used as a placeholder, exceptiongroup which I have been given ownership of.
The options would thus be the following:

  1. Create a new project on PyPI: backports.exceptiongroups and possibly delete the placeholder exceptiongroup project
  2. Use the existing project name, and possibly restructure the package to match

Then there is the question of the catch() function: should I try to make it possible for the catcher to support multiple handlers, rather than having users use multiple context managers?

Since there isn’t going to be an exceptiongroup module in the stdlib then might as well use the name.

1 Like

I’ve published v1.0.0rc1 now. I’ve made two changes as of late:

  1. The catch() function takes a dict, with the exception type(s) as key and the handler callback as value. This allows for near perfect emulation of the except* syntax.
  2. There is now an environment variable that you can use to opt out of the exception hook / monkey patching of TracebackException: EXCEPTIONGROUP_NO_PATCH (set to 1 to enable).

If anybody has a problem with the API, this is the last call for changes so speak up.

I can take a look at integrating this into Quattro.

My first impression is that the catch context manager has very poor usability since it takes a callback function. This is awkward since in that callback function you cannot easily change locals in the calling function, or return from the calling function.

Why not have a filtering helper, and make users do:

try:
    await something
except ExceptionGroup as eg:
    if exc := exceptiongroup.contains(eg, ValueError, KeyError):
        print('Caught exception:', type(exc))
    else:
        raise

I don’t really see how requiring a callback translates to poor usability. Your example requires catching the ExceptionGroup, and it also only catches the first ValueError or KeyError that it finds, rather than handling them all. And what if the handling code raises an exception? You would have to catch that and then wrap all other raised exceptions in a new ExceptionGroup. Doing all this by hand is tedious and error prone, compared to the catch() context manager which manages all this for you, approximating except* as best as can be done on Python < 3.11.

This is awkward since in that callback function you cannot easily change locals in the calling function, or return from the calling function.

The reason why I didn’t want to add any extra helpers is because I want this library to be just a stopgap measure until Python 3.11 is mainstream. I therefore don’t want to add any functionality that would remain in use in the long term.

As for not being able to change locals in the calling function, why not use nonlocal?

Further, returning from except* is prohibited anyway. From PEP 654:

continue, break, and return are disallowed in except* clauses, causing a SyntaxError.

Yeah you’re probably right that trying to re-implement this in a clever way is a fool’s errand. Consider my objection withdrawn.

I was very surprised to learn it’s not possible to return from an except* clause, and other users will be too. I was left wondering why?

I’m in the process of adding extended validation to the cattrs library, and this use case really needs an ExceptionGroup-like thing, but it looks to me like this exception group implementation is inadequate.

In particular, an array of exceptions is insufficient. I need at least a dict of strings to exceptions. Which is a shame since I could have used the fancy traceback support.

@Tinche, Would PEP 678 notes solve this problem?

CC @Zac-HD

Hm, probably yeah actually. Why do the notes have to be strings though? The root exception group will probably need to be parsed by tools, and having richer data in the notes would be better. @Zac-HD

Why not just have the note be anything, and when printing it out just repr whatever is in there.

Mostly because __note__ is intended for display to users, and requiring that notes are strings makes it easy to concatenate them (etc).

If you want programmatic handling of structured data, that’s not really what __note__ is for. Can you say more about the problem you’re trying to solve / API you want to offer?