Accepting PEP 654 (Exception Groups and except*)

Irit, Guido, Yury, Nathaniel,

(In the interest of visibility, this was also emailed to the authors and posted to python-dev.)

After careful consideration and rather lengthy discussion, the Steering Council has decided to accept PEP 654 (Exception Groups and except*). We do have two requests, one of which we mentioned before. We’re very pleased with the amount of care and discussion that went into this PEP, and we’re satisfied that it solves a real-world problem in the least disruptive way. We do remain open for further tweaking of the design, incorporating some of Nathaniel’s ideas, or new ones, but we want to move forward with PEP 654’s proposal and let people get hands-on experience with it. To that end, our request is that we get at least one non-toy use of ExceptionGroups somewhere in the standard library (targeted to 3.11, like PEP 654), and preferably get some third-party experimentation with it as well. The other request is that PEP 654 be updated with a reference to Nathaniel’s ideas (a link to the Discourse discussion), outlining why PEP 654 rejects those ideas (it could just be a copy-paste of some of the discussion, just so long as it’s recorded in the PEP).

Regarding Nathaniel’s alternative proposal, I want to be clear on the main reason why we’re accepting PEP 654 rather than waiting for a more complete alternative PEP with concrete examples. The SC agrees that exception groups solve a real problem, albeit a relatively rare and low-level one. We don’t think the problem it solves is big enough that it warrants fundamental changes to the language; changes that would affect the way everyone would write code, or that would make learning Python or reasoning about Python code harder. PEP 654 just about squeaks by those criteria: the new syntax is something new to learn about Python, but it’s not something that’s required unless consuming an API that uses ExceptionGroups. ExceptionGroups behave like regular exceptions in code that does not expect them, and do not change the behaviour of existing try/except. In code that potentially produces ExceptionGroups (or its documentation, examples, tutorials, etc) there is enough opportunity to document that, show examples of how to use it, explain the implications and refer to documentation about the concept of ExceptionGroups.

By contrast, Nathaniel’s proposal that ExceptionGroups affect how existing try/except statements behave, fails those criteria. By sheer nature of doing something meaningful (if confusing) in try/except, it would encourage API designs that let ExceptionGroups escape to unsuspecting code. It would introduce subtle bugs in code that – entirely legitimately – expects only one except block to be executed, or only executed once. These problems would be very hard to detect since they would only show up if called code could in practice raise ExceptionGroups. We don’t consider this a viable design choice.

The SC does remain open to proposals to extend PEP 654 (like adding an API to iterate over ExceptionGroups), or change some of the semantics (like flat versus nested ExceptionGroups), and we do not want to discourage discussion on any of this. We do want to move forward with PEP 654 as-is in the meantime, in particular to get practical experience in the standard library.

With thanks for all your hard work,
For the whole SC,
Thomas.

8 Likes

Thank you Thomas and the SC. I’ll start working on incorporating TaskGroups into asyncio in the next few weeks.

5 Likes

Thank you Thomas and the SC. Making this decision now leaves us with plenty of time to implement the PEP carefully for 3.11 and to work with third-party users - a couple who have already expressed an interest in exception groups, as well as any others who will.

3 Likes

My congratulations!

3 Likes

I’m hoping that we can start a discussion on how exactly these will be implemented. We have just one opportunity to get it right.

1 Like

I’m particularly interested in backporting exception groups to earlier Python versions so we can get an early start with trio and anyio. Since the new syntax won’t be available until 3.11, we would at least need some helper functions. I have a 99% complete PEP 654 implementation here.

I’d also love a backports.exceptiongroup package for Pytest and Hypothesis, though contra this advice I think we should just provide the builtin version if it’s available - having to catch either-of-maybe-two implementations sounds awful. Let me know if I can help out somehow?

Code review and experimenting with the code would be great help.

The code will be committed in three stages:

  1. ExceptionGroup / BaseExceptionGroup classes
  2. Updates to traceback machinery to display them nicely
  3. except*

We are now reviewing the PR for the first step: bpo-45292: [PEP 654] add the ExceptionGroup and BaseExceptionGroup classes by iritkatriel · Pull Request #28569 · python/cpython · GitHub

How would you use ExceptionGroups in 3.10 without except*?

The part that I really want is an ecosystem-wide “collection of exceptions” type, which enables nice features like good reporting, debugger support, and so on.

Most of that for me is Hypothesis’ report_multiple_bugs setting: we collect exceptions from each failing input, and want to show the user each distinct error. Currently that means we just print each (trimmed) traceback - it works, but misses out on things like Pytests’s configurable colored and expanded tracebacks, debugger suport, and so on.

Review incoming - I found it very surprising that BaseExceptionGroup("", [Exception()]) returns a non-Base exceptiongroup object, but the rest looks good.

From a few hours of experimenting, the only thing I’m missing is a way to annotate each nested exception. In my case that’s something like Falsifying example: test(x=False), but I think the same consideration applies to use motivated by callbacks (which does the exception come from?) or async tasks - the traceback alone is not always sufficient identifying information.

raise FalsifyingExample("test(x=False)") from err for each of the nested exceptions provides a workaround, but that does feel like an ugly hack.

2 Likes

The plan was to update GitHub - python-trio/exceptiongroup with the PEP 654 code and documentation, and to add any necessary utility functions to support their use without the except* syntax. But since the code would be overwritten anyway, any repo would do I suppose.

@njs thoughts?