How to debug at recursion limit

Python Help
1

I have an issue which I believe is caused by hitting the recursion limit inside of a getattr call. However, I have no idea how to debug this. Take this simple contrived example:

def f(i):
    try:
        f(i+1)
    except RecursionError:
        print(f"Reached max recursion depth before breakpoint {i}")
        breakpoint()
        print(f"Reached max recursion depth after breakpoint {i}")
  • The first print happens at i=996 (or a partial print at i=997, somewhere around here)
  • Then there are many more prints as the breakpoint function itself fails to work correctly and raises an RecursionError
  • at i=949 we actually get a debugger finally, but this is ofcourse far away from where I can actually play around at the recursion limit

Even just raw print debugging might not work and is probably at least one layer removed from where the problem happens.

Note on the original issue: getattr(self, generated_name) returns None despite the object having the name for sure and this having succeeded many times before. Also note: Increasing the recursion limit is not a solution. There are potentially infinite recursions which I am trying to catch with an RecursionError.

2

Hmm. My first thought is sys.setrecursionlimit() but that is itself a call. It might help though?

3

Well, even if it works, then I am no longer at recursion limit and can no longer observe the behavior I want to observe…

4

Oh, but I also randomly figured out the original issue… This is quite the edge case xd

  • name isn’t a str instances, but a subclass of str (called Token) with a custom __eq__, but one that does defer to str.__eq__ if the other isn’t an instance of Token
  • getattr(self, name) potentially falls through to a __getattr__, which provides a default value.

So I guess what happens is:

  • getattr(self, name) is called, which tries to look up name in self.__dict__. This calls __eq__, which causes an RecursionError because of the call to str.__eq__ (or earlier, doesn’t really matter)
  • For some reason, getattr doesn’t propagate this error, treats it as an AttributeError and calls self.__getattr__, which succeeds, returning the wrong value

This should definitely be fixed, I will try to make a proper reproducer and report it to CPython. (In my case a workaround of not using the subclass is ok)

But the OP question is IMO still interesting, if someone has a nice idea, I am interested.

5

Sorry, lemme clarify. I wasn’t clear in what I was saying. I meant that using setrecursionlimit inside the exception handler might solve it. Something like this:

def f(i):
    try:
        f(i+1)
    except RecursionError:
        sys.setrecursionlimit(sys.getrecursionlimit() + 10)
        print(f"Reached max recursion depth before breakpoint {i}")
        breakpoint()

rather than as a global change.

6

The exception handler then gets called 5 times till we are far enough away from the recursion limit so that breakpoint() works now. But that still means I can’t easily observe the behavior of the recursion limit itself. Ideally, I would like to be able to type print("XYZ") and instead of it printing XYZ it would raise an recursion error (which is the caught and displayed like 1/0 would be in a normal shell). But stdlib pdm is for sure not designed for that, and I am struggling even imaging a python shell that does. I guess dynamically changing the recursion limit for each exec is a decent approach?