Feature Request: Expose Handling Frame Information in Python's Traceback Module

Ideas
1

Feature or enhancement

Proposal:

Problem: When dealing with chained exceptions in Python, there is no straightforward way to identify the “outermost handling frame”: the frame in the except/finally block that starts the part of the stacktrace that’s unique to the “during the handling” exception.

This information is not visible in the textual stacktrace, nor is it accessible programmatically via Python’s traceback module (or similar tools).

Since this “handling frame” represents half of the link between 2 exceptions when shown in “chained mode”, the fact that this information is missing makes it very hard to see how 2 chained exceptions actually relate to each other.

Proposal: Extend Python’s traceback-related modules (e.g., traceback, sys.exc_info) to expose handling frame information. This would allow developers and tools to programmatically determine where exceptions are caught and re-raised in the call stack.

(It would seem to me that just changing the underlying datastructures leads to the least disruption in peoples’ workflow, while still allowing for some users to extract the information if they need it)

Current Behavior:

Using the following example:

class OriginalException(Exception):
    pass

class AnotherException(Exception):
    pass

def raise_another_exception():
    raise AnotherException()

def show_something():
    try:
        raise OriginalException()
    except OriginalException:
        raise_another_exception()

show_something()

Running this code produces:

Traceback (most recent call last):
  File "example.py", line 15, in show_something
    raise OriginalException()
__main__.OriginalException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "example.py", line 20, in <module>
    show_something()
  File "example.py", line 17, in show_something
    raise_another_exception()
  File "example.py", line 10, in raise_another_exception
    raise AnotherException()
__main__.AnotherException

In this traceback, line 17 (raise_another_exception()) is the handling frame, but there is no direct way to programmatically extract this information.

Proposed Improvement:

Enhance the traceback-related API(s) to expose handling frame information.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

I wrote extensively about this problem (as well as more generally about how confusing chained stacktraces can be) on my blog.

This is obviously not actually a discussion since it doesn’t involve other people, but I do think it’s useful context (and it’s better to keep the longer story out of the feature-request)

==

Previously here (the above is a verbatim copy of that issue to save you a click)

2

It’s not clear to me what you’re missing? You can already get frame information for both the exceptions:

class OriginalException(Exception):
    pass

class AnotherException(Exception):
    pass

def raise_another_exception():
    raise AnotherException()

def show_something():
    try:
        raise OriginalException()
    except OriginalException:
        raise_another_exception()

try:
    show_something()
except Exception as e:
    print(e.__traceback__.tb_frame)  # The location of the AnotherException()
    print(repr(e.__context__))  # The OriginalException() 
    print(e.__context__.__traceback__.tb_frame)  # Where OriginalException() is raised
3

The relationship between the 2 exceptions cannot be established from the stacktrace, because all frames from the OriginalException that are also part of the AnotherException are pruned, but the point-of-pruning (“handling frame”) is not indicated in the unpruned stacktrace (the one from AnotherException). This has multiple semantically equivalent results:

  1. You can’t reconstruct the full stacktrace of OriginalException by copying (in your head or otherwise) part of the AnotherException’s stacktrace over (because where should you stop copying?). This makes OriginalException much harder to read, because you are dropped in the middle of the story (“how did you end up there?”)
  2. You can’t point at the line in AnotherException which was reached through virtue of it being in the handling (except/finally) block of the OriginalException. Makes it much harder to see the relationship between the 2 exceptions.
1 Like
4

I don’t think we currently have this information. ISTM that it would require the interpreter attaching the frame to the exception when it is caught (and clear it when the same exception is raised again).

This would probably make it possible to fix things like: Problems with recursive automatic exception chaining · Issue #63061 · python/cpython · GitHub

5

I think so too

Regarding the linked issue: I’ve read it with interest, but I haven’t figured out yet how the two issues are related.

6

I think we do. When an exception is handled, its traceback would have its frame pointing at the handler rather than the line where the exception occurs. The current traceback code uses tb_lasti for positions, however, which points to where the exception occurs, so it loses the position of the handler.

I’ve created a PR that produces a traceback output with the handler frame added to each chained exception, so the following snippet:

class OriginalException(Exception):
    pass

class AnotherException(Exception):
    pass

def raise_exception():
    raise OriginalException()

def raise_another_exception():
    raise AnotherException()

def show_something():
    try:
        raise_exception()
    except OriginalException:
        raise_another_exception()

show_something()

would now with this PR produce something like:

Traceback (most recent call last):
  File "test.py", line 15, in show_something
    raise_exception()
  File "test.py", line 8, in raise_exception
    raise OriginalException()
  File "test.py", line 17, in show_something
    raise_another_exception()
OriginalException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 19, in <module>
    show_something()
  File "test.py", line 17, in show_something
    raise_another_exception()
  File "test.py", line 11, in raise_another_exception
    raise AnotherException()
AnotherException

Changing the traceback output breaks a bunch of existing tests though. I’ll modify those tests if you agree it’s the way to go.

1 Like
7

That’s interesting. I need to think about it some more, but my first reaction/concerns are:

  1. What can we actually assume about the relationship between an exception and its __context__, given that the __context__ field is mutable and anyone can set it to whatever they want?

  2. What about __cause__?

1 Like
8

I thought about it some more, and then figured the information is available, but in an indirect way. (Am I saying the same thing here as Ben but in terms of Python rather than C?)

The thing is: for each exception that is ultimately handled the first (i.e. no tb_next called) frame will be the same one as the handling frame of the next exception in the chain.

The thing I didn’t realize is: at the level of Python objects, we’re literally talking about the same frame object (funnily enough the tbline_no of that object points to the handling point even for the frame that’s part of the raising stack; presumably the traceback formatting has access to another attr for the other lineno)

To get the information I need, all I need to do is push the information from one exception in the chain to the next and compare them.

Here’s a PoC on the sentry-SDK

https://github.com/vanschelven/sentry-python/commit/f90f067348f5b37cac3e354308e395e176241876

Maybe not end-of-discussion quite yet? Open questions being the textual representation, and whether my hacked solution is durable for edge cases? In particular: does this hold up in the case of context processors?

9

Good points. I’ve rewritten the PR so that it now follows __cause__ and __context__, and simply formats the handler frame with an additional note of (from context) instead like this:

Traceback (most recent call last):
  File "test.py", line 15, in show_something
    raise_exception()
  File "test.py", line 8, in raise_exception
    raise OriginalException()
OriginalException

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 19, in <module>
    show_something()
  File "test.py", line 17, in show_something (from context)
    raise_another_exception()
  File "test.py", line 11, in raise_another_exception
    raise AnotherException()
AnotherException

And also with a snippet that raises from a cause:

def show_something():
    try:
        raise ValueError()
    except ValueError as e:
        exc = e
    raise RuntimeError from exc

show_something()

it would output the referenced frame with a note of (from cause):

Traceback (most recent call last):
  File "test.py", line 3, in show_something
    raise ValueError()
ValueError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 8, in <module>
    show_something()
  File "test.py", line 6, in show_something (from cause)
    raise RuntimeError from exc
RuntimeError

Does that look better?

10

This still assumes that the __context__ is what the interpreter put there, and was not overwritten. Right?

11

It assumes that __context__.__traceback__.tb_frame is among the stack of frames of the traceback of the current exception being handled, so as long as that consistency is there when __context__ is modified the frame of the exception handler can be identified. If not, the worst that can happen is that there would not be a note of (from context) for the handler frame, which is what the developer should expect when modifying __context__ arbitrarily, which I don’t think should happen in the real world anyway.

12

The worst that can happen is that you get the wrong result.

__context__ is mutable, and the only restriction currently is that what’s assigned there needs to be an exception. We need to think this through without making assumptions about what people do with that.

13

Well we can document how the handler frame is identified so people can set their expectations when modifying __context__ and reading the traceback output.

But yeah I agree that if we do want to be completely accurate even when __context__ is modified with an arbitrary exception then we need to add two attributes to the frame object to bind its cause and/or its context. I wonder if __context__ is modified often enough to warrant the cost of the two additional attributes though.

14

Or add one attribute to BaseException which is the frame in which it was most recently caught.

This is a change to a builtin though, so it would require a PEP.

15

An exception can be both the cause and the context of another though, so we still need two attributes.

try:
    try:
        raise ValueError('foo')
    except ValueError as e:
        raise RuntimeError from e
except Exception as e:
    print(e.__cause__, e.__context__) # foo foo
16

I mean each exception has a field indicating where it was last caught by an except. If we have that, I think we don’t need to use the __context__ and __cause__ fields anymore. We just add this frame to the traceback output.

17

Yeah I totally see how this can be useful as a replacement to __context__, but I still don’t see how it can help identify the frame that raises from a cause because it isn’t necessarily the same as where the exception is caught.

18

Actually I think your approach can work, as long as you don’t pretend that it’s more than it is. You just document it accurately, say “this is the top frame in __context__”, and give an interpretation of what that means for unmodified tracebacks and then it’s not your problem what happens when they modify (it does the same thing, but it doesn’t necessarily mean the same thing).

Question is - do we just change the traceback output, or do we add a kwarg to the traceback module functions to include this information? The latter is safer.

1 Like
19

Yup. Agreed 100%.

In my current implementation I’ve added two new kwargs of cause and context to FrameSummary.__init__. I think that should be enough for the purpose.

20

I’m not sure, because this is invoked internally. I’m talking about a new kwarg to traceback.print_exception, etc.

In the last few years we protected even less visible changes under new kwargs (see compact). We can always decide in the future to make the new traceback the default, but initially I would make sure you don’t need to change any existing tests, just add new tests for the new functionality.

1 Like