A way of monitoring the max stack depth reached

Ideas
21

Recursion limit serves two roles. First, it is a guard against stack overflow. Unfortunately, there is no standard way to determine the size of the stack, how much stack is available, and how much is needed for the next C call. So this is left for the end user.

Second, it is a guard against programming errors causing infinite recursions (they are common for beginners). Normal program does not have too deep recursion. If it exceeds any reasonable limit, there is a good chance that there is a bug in your program.

22

Unfortunately, there is no standard way to determine the size of the stack, how much stack is available, and how much is needed for the next C call. So this is left for the end user.

But the behaviour seems to be different for genuine recursion versus import chains. If I set a high recursion limit I can get a function to recurse, say, a million times. But there seems to be a different mechanism for import chains - very high recursion limits are ignored, and it raises a RecursionError anyway after a chain of 1,249 imports. I tried to get it to stack overflow in this case and it couldn’t. Do you see what I mean?

Maybe it would be useful to document this behaviour at least, if it is intentional.

23

RecursionError is somewhat of a misnomer. The problem is that the arbitrary limit to the depth of the call stack is reached, which is most commonly due to recursion, but in the OP’s case it’s due to too many nested imports.

24

I want to point that an import chain is recursion: the import machinery recursively calls itself, if only indirectly.

2 Likes
25

Ah, yes, there is also Py_C_RECURSION_LIMIT. It is not documented and not configurable, so I forgot about it. The only way to change it is to modify the CPython source code. It limits the total recursion depth for the compiler. It does not distinguish Python frames from C frames, so it limits you even if the real stack consumption is small.

This is a part of a complex problem. We have no solution.

3 Likes
26

Indeed it should have been call CallDepthLimitError as its not recursion that triggers it. I do know that but fell into the trap of use “recursion” as a call depth on the imports.

27

Ok! Yes I see the 10,000 limit here.

To me, the behaviour seems reasonable to document at least. I’ve raised an issue here which suggests adding some docs, which I’m happy to do assuming the maintainers agree.

28

Good news. I’ve come up with a solution that resolves any RecursionError resulted from nested imports by iteratively retrying failing imports in reverse order, and if any import incurs a RecursionError while being retried in reverse order, the failing nested imports will keep stacking up and getting retried in reverse order until the original import succeeds:

def bottom_up_import(orig_import=__import__):
    def _import(*args, **kwargs):
        nonlocal depth
        to_import.append((args, kwargs))
        depth += 1
        try:
            if depth > 1:
                module = orig_import(*args, **kwargs)
                to_import.pop()
            else:
                while to_import:
                    args, kwargs = arguments = to_import[-1]
                    try:
                        module = orig_import(*args, **kwargs)
                        to_import.pop()
                    except RecursionError:
                        if to_import[-1] == arguments:
                            raise # we've got a real recursion error
        finally:
            depth -= 1
        return module
    depth = 0
    to_import = []
    return _import

import builtins
builtins.__import__ = bottom_up_import()

so that with an import that nests 10000 levels deep (using a custom InspectLoader to avoid having to create 10000 actual modules):

import sys
from importlib.util import spec_from_loader
from importlib.abc import InspectLoader

class FooLoader(InspectLoader):
    def __init__(self, number):
        self.number = number

    def get_source(self, fullname):
        if self.number == 0:
            return 'total = 0'
        return f'''\
import foo_{self.number - 1} as foo
total = foo.total + {self.number}
'''

class FooFinder:
    def find_spec(self, fullname, path=None, target=None):
        prefix, _, number = fullname.partition('_')
        if prefix == 'foo':
            return spec_from_loader(fullname, FooLoader(int(number)))

sys.meta_path.insert(0, FooFinder())

import foo_10000 as foo
print(foo.total)

This outputs (basically 1 + 2 + 3 + … + 10000):

50005000

Demo here

This works as long as none of your modules modifies import machineries that affect the nested imports.

29

Interesting approach! I hadn’t thought to do it programmatically, thanks!