Show builtin functions and classes in tracebacks and profiles

Currently builtin (C) functions (both in the stdlib and third-party libraries) form an important part of many Python applications, but they are invisible in stack traces.
The same applies to classes, whether Python or builtin.

Some sampling profilers show builtin functions, but they need to use platform specific unwinding information, making them non-portable. CProfile also shows calls to builtin functions, but cProfile is so slow as to be useless.

We can add builtin functions to the stack trace at very little cost, so let’s do it.

Example:

 class C:
    def __init__(self):
        math.sqrt(-1)

Calling C() gives this traceback:

  File "example.py", line 7, in <module>
    C()
  File "example.py", line 5, in __init__
    math.sqrt(-1)
ValueError: math domain error

What would like is something like:

  File "example.py", line 7, in <module>
    C()
  File "example.py", line 4, in <module>
    class C:
  File "example.py", line 5, in __init__
    math.sqrt(-1)
  File "mathmodule.c", in math.sqrt
ValueError: math domain error

Compatibility issues

Currently all traceback objects have an attached frame, and that frame has a code object.
Traceback objects for builtin functions, would not have frames, or if they did, those frames would not have code objects.

This would break existing code.
One possibility would be to change the traceback from a single linked lists, to a pair of interleaved linked lists, the first accessible by ex.__traceback__ containing the current traceback, and another accessible by ex.__traceback_full__ containing the full traceback with classes and builtin functions.

Estimated costs

Always on – Builtin functions are visible in all stack traces

About 0.5% of current performance. This fraction may increase as we add new optimizations, but not by much and might even decrease.

Only for exception tracebacks

This has essentially zero cost. It makes raising an exception slightly more expensive, but has no cost otherwise

Limitations

Only builtin functions called from Python, or through the Python C-API would show up in stack traces.
Builtin functions and other C functions called directly from C code would need to explicitly add themselves to the stack trace if they wanted to be visible.

9 Likes

As a C programmer and extension writer, I’d find this useful, but don’t you think that it would look confusing to a Python programmer ?

People might start to wonder where this mathmodule.c can be found when debugging the code.

Having an optional switch to turn on such functionality would be nice (with the default being off). Extension writers could then use this to more easily pin point errors when debugging extensions.

1 Like

Having an optional switch to turn on such functionality would be nice

The visibility of the frames could be turned on or off at runtime.
It is faster to always have the frames, than to have them conditionally.

Given the backwards compatibility issues, I think the new frames would have to be invisible by default.

However, the frames would be there and would be visible to statistical profilers and such, or if the user wanted to see them.

With that, I’d be +1 on such an enhancement. Debuggers and tracers could then make use of those extra frames, even when they are not shown.