Pdb does not stop at break points after maximum recursion depth has been exceeded

Python Help
1

My program evaluates cases by applying a series of rules. Some of these can produce deeply nested function calls. Therefore I evaluate each rule within a try . . except block. If the “Maximum recursion depth exceeded” error occurs, it is caught, the program logs the fact and proceeds to the next rule. But I have found that once that error has occurred, pdb does not stop at any further break points. Is this a “feature”, or a bug? Is there any way round it?

2

In sys.settrace() it says

If there is any error occurred in the trace function, it will be unset, just like settrace(None) is called.

The following might not be the right way to go around that, as the warning below says, but it works in this case.

import sys


def f():
    f()

try:
    t = sys.gettrace()  # Getting the trace function
    f()  # The deep recursion causes the trace function to be unset
except RuntimeError:
    sys.settrace(ot)  # Re-setting the trace function back
    print('breakpoint here')  # A breakpoint here 


PYDEV DEBUGGER WARNING:
sys.settrace() should not be used when the debugger is being used.
This may cause the debugger to stop working correctly.
If this is needed, please check: 
http://pydev.blogspot.com/2007/06/why-cant-pydev-debugger-work-with.html
to see how to restore the debug tracing back correctly.
Call Location:
  File "/path/to/script.py", line 11, in <module>
    sys.settrace(t)

breakpoint here
3

I gather that pdb is built on bdb, and bdb uses sys.settrace(). The documentation for the last states that “if any error occurred in the trace function, it will be unset”, but what is included in “any error”? In the EAFP style of programming, handled exceptions occur frequently. They do not cause the debugger to stop tracing. Is a handled RuntimeError of a different order from, say, a handled KeyError or AttributeError? Of course, if the exception is not handled, execution will stop.

4

They mean inside the trace function. It is not that RuntimeError is different, it is that the location of the error is special.

I don’t know which error happened exactly, but it is understandable that a RecusionError is a problem managing the execution stack and the trace function (I think) gets called on each stack frame, and it itself consumes stack space.

A RuntimeError in another location, or that it did not affect the debuggers’s trace function should not have the same effect. For example, the following doesn’t cause the breakpoint to disappear.

try:
    raise RecursionError
except RuntimeError as _:
    print('breakpoint here')
5

Cut down to essentials, my code is as follows:

try:
   result = _evaluate_rule(rule)
except RuntimeError as e:
   if e.args[0].startswith('maximum recursion depth exceeded'):
      msg = 'Maximum recursion depth exceeded: call stack %s' %           \
            '; '.join(['%s, %s, %s' % (item[0].name, item[1], item[2])
                       for item in thr.rule_call_stack])
      session_log(msg, 'W')
   else:
      raise

I was trying to debug unexplained behaviour that occurred later on, but the debugger did not stop at my break points. To test the hypothesis that that was the result of the RuntimeError I put a break point on the session_log line. The debugger did not stop there, but the message was written to the log, so I know that execution had passed that point - indeed, it continued to the end of the program.

6

Did you try adding ot = sys.gettrace() inside the try, before the result = _evaluate_rule(rule), and sys.settrace(ot) inside the except?

7

No. Since raising this query I have discovered that on this occasion the recursion limit was exceeded because of a simple bug, which I have now fixed. But I shall remember your tip for next time. Thank you for the information.

8

I have now added the code that you suggested, and tested it. It works. Thank you again. This can now be marked as solved.