Accepting PEP 654 (Exception Groups and except*)

PEPs
1

Irit, Guido, Yury, Nathaniel,

(In the interest of visibility, this was also emailed to the authors and posted to python-dev.)

After careful consideration and rather lengthy discussion, the Steering Council has decided to accept PEP 654 (Exception Groups and except*). We do have two requests, one of which we mentioned before. We’re very pleased with the amount of care and discussion that went into this PEP, and we’re satisfied that it solves a real-world problem in the least disruptive way. We do remain open for further tweaking of the design, incorporating some of Nathaniel’s ideas, or new ones, but we want to move forward with PEP 654’s proposal and let people get hands-on experience with it. To that end, our request is that we get at least one non-toy use of ExceptionGroups somewhere in the standard library (targeted to 3.11, like PEP 654), and preferably get some third-party experimentation with it as well. The other request is that PEP 654 be updated with a reference to Nathaniel’s ideas (a link to the Discourse discussion), outlining why PEP 654 rejects those ideas (it could just be a copy-paste of some of the discussion, just so long as it’s recorded in the PEP).

Regarding Nathaniel’s alternative proposal, I want to be clear on the main reason why we’re accepting PEP 654 rather than waiting for a more complete alternative PEP with concrete examples. The SC agrees that exception groups solve a real problem, albeit a relatively rare and low-level one. We don’t think the problem it solves is big enough that it warrants fundamental changes to the language; changes that would affect the way everyone would write code, or that would make learning Python or reasoning about Python code harder. PEP 654 just about squeaks by those criteria: the new syntax is something new to learn about Python, but it’s not something that’s required unless consuming an API that uses ExceptionGroups. ExceptionGroups behave like regular exceptions in code that does not expect them, and do not change the behaviour of existing try/except. In code that potentially produces ExceptionGroups (or its documentation, examples, tutorials, etc) there is enough opportunity to document that, show examples of how to use it, explain the implications and refer to documentation about the concept of ExceptionGroups.

By contrast, Nathaniel’s proposal that ExceptionGroups affect how existing try/except statements behave, fails those criteria. By sheer nature of doing something meaningful (if confusing) in try/except, it would encourage API designs that let ExceptionGroups escape to unsuspecting code. It would introduce subtle bugs in code that – entirely legitimately – expects only one except block to be executed, or only executed once. These problems would be very hard to detect since they would only show up if called code could in practice raise ExceptionGroups. We don’t consider this a viable design choice.

The SC does remain open to proposals to extend PEP 654 (like adding an API to iterate over ExceptionGroups), or change some of the semantics (like flat versus nested ExceptionGroups), and we do not want to discourage discussion on any of this. We do want to move forward with PEP 654 as-is in the meantime, in particular to get practical experience in the standard library.

With thanks for all your hard work,
For the whole SC,
Thomas.

7 Likes
2

Thank you Thomas and the SC. I’ll start working on incorporating TaskGroups into asyncio in the next few weeks.

5 Likes
3

Thank you Thomas and the SC. Making this decision now leaves us with plenty of time to implement the PEP carefully for 3.11 and to work with third-party users - a couple who have already expressed an interest in exception groups, as well as any others who will.

3 Likes
4

My congratulations!

3 Likes
5

I’m hoping that we can start a discussion on how exactly these will be implemented. We have just one opportunity to get it right.

6

I’m particularly interested in backporting exception groups to earlier Python versions so we can get an early start with trio and anyio. Since the new syntax won’t be available until 3.11, we would at least need some helper functions. I have a 99% complete PEP 654 implementation here.

7

I’d also love a backports.exceptiongroup package for Pytest and Hypothesis, though contra this advice I think we should just provide the builtin version if it’s available - having to catch either-of-maybe-two implementations sounds awful. Let me know if I can help out somehow?

8

Code review and experimenting with the code would be great help.

The code will be committed in three stages:

  1. ExceptionGroup / BaseExceptionGroup classes
  2. Updates to traceback machinery to display them nicely
  3. except*

We are now reviewing the PR for the first step: https://github.com/python/cpython/pull/28569

How would you use ExceptionGroups in 3.10 without except*?

9

The part that I really want is an ecosystem-wide “collection of exceptions” type, which enables nice features like good reporting, debugger support, and so on.

Most of that for me is Hypothesis’ report_multiple_bugs setting: we collect exceptions from each failing input, and want to show the user each distinct error. Currently that means we just print each (trimmed) traceback - it works, but misses out on things like Pytests’s configurable colored and expanded tracebacks, debugger suport, and so on.

Review incoming - I found it very surprising that BaseExceptionGroup("", [Exception()]) returns a non-Base exceptiongroup object, but the rest looks good.

From a few hours of experimenting, the only thing I’m missing is a way to annotate each nested exception. In my case that’s something like Falsifying example: test(x=False), but I think the same consideration applies to use motivated by callbacks (which does the exception come from?) or async tasks - the traceback alone is not always sufficient identifying information.

raise FalsifyingExample("test(x=False)") from err for each of the nested exceptions provides a workaround, but that does feel like an ugly hack.

3 Likes
10

The plan was to update GitHub - python-trio/exceptiongroup with the PEP 654 code and documentation, and to add any necessary utility functions to support their use without the except* syntax. But since the code would be overwritten anyway, any repo would do I suppose.

@njs thoughts?

11

Perhaps a new backports.exceptiongroup package?

We now have pytest and better-exceptions integration for MultipleFailures, which demonstrates that this will be easy to support upstream for ExceptionGroup - if a little fiddlier for nested than flat groups.

BPO-45607 suggests adding an assignable __note__ attribute, which would give us a much cleaner alternative :smiling_face_with_three_hearts:

12

I have a backport here now. I’m soliciting comments on it before I publish it to PyPI.

One open question is how to name the project. The package itself is backports.exceptiongroups but there is an existing PyPI name used as a placeholder, exceptiongroup which I have been given ownership of.
The options would thus be the following:

  1. Create a new project on PyPI: backports.exceptiongroups and possibly delete the placeholder exceptiongroup project
  2. Use the existing project name, and possibly restructure the package to match

Then there is the question of the catch() function: should I try to make it possible for the catcher to support multiple handlers, rather than having users use multiple context managers?

13

Since there isn’t going to be an exceptiongroup module in the stdlib then might as well use the name.

1 Like
14

I’ve published v1.0.0rc1 now. I’ve made two changes as of late:

  1. The catch() function takes a dict, with the exception type(s) as key and the handler callback as value. This allows for near perfect emulation of the except* syntax.
  2. There is now an environment variable that you can use to opt out of the exception hook / monkey patching of TracebackException: EXCEPTIONGROUP_NO_PATCH (set to 1 to enable).

If anybody has a problem with the API, this is the last call for changes so speak up.

15

I can take a look at integrating this into Quattro.

My first impression is that the catch context manager has very poor usability since it takes a callback function. This is awkward since in that callback function you cannot easily change locals in the calling function, or return from the calling function.

Why not have a filtering helper, and make users do:

try:
    await something
except ExceptionGroup as eg:
    if exc := exceptiongroup.contains(eg, ValueError, KeyError):
        print('Caught exception:', type(exc))
    else:
        raise
16

I don’t really see how requiring a callback translates to poor usability. Your example requires catching the ExceptionGroup, and it also only catches the first ValueError or KeyError that it finds, rather than handling them all. And what if the handling code raises an exception? You would have to catch that and then wrap all other raised exceptions in a new ExceptionGroup. Doing all this by hand is tedious and error prone, compared to the catch() context manager which manages all this for you, approximating except* as best as can be done on Python < 3.11.

17

This is awkward since in that callback function you cannot easily change locals in the calling function, or return from the calling function.

The reason why I didn’t want to add any extra helpers is because I want this library to be just a stopgap measure until Python 3.11 is mainstream. I therefore don’t want to add any functionality that would remain in use in the long term.

As for not being able to change locals in the calling function, why not use nonlocal?

18

Further, returning from except* is prohibited anyway. From PEP 654:

continue, break, and return are disallowed in except* clauses, causing a SyntaxError.

19

Yeah you’re probably right that trying to re-implement this in a clever way is a fool’s errand. Consider my objection withdrawn.

I was very surprised to learn it’s not possible to return from an except* clause, and other users will be too. I was left wondering why?

20

I’m in the process of adding extended validation to the cattrs library, and this use case really needs an ExceptionGroup-like thing, but it looks to me like this exception group implementation is inadequate.

In particular, an array of exceptions is insufficient. I need at least a dict of strings to exceptions. Which is a shame since I could have used the fancy traceback support.