Has sys.setrecursionlimit() behaviour changed in Python 3.12b?

Hi all,
today I tried to understand a test failure I’m seeing trying out python-rapidjson under Python 3.12b4.

One particular test (see tests/test_circular.py#L45-L70 [link removed to honor the “new user limitation”…]) uses sys.setrecursionlimit() to assert that the underlying C++ implementation (that uses Py_EnterRecursiveCall(), see for example properly raises a RecursionError.

Up to Python 3.11, it does work as expected, while on 3.12 it does not, or more precisely I do not seem able to control the recursion depth: it is like the recursion limit is fixed, and whatever value I pass to sys.setrecursionlimit() does not influence that.

I looked around to find news about changes related to either sys.setrecursionlimit() or Py_EnterRecursiveCall(), but found nothing. So I poked into the source of stdlib json module, and seeing it very similar to the rapidjson code, I tried to replicate the “problem” with that, and indeed I see a similar result.

Under Python 3.11 I see this:

$ python3.11
Python 3.11.4 (main, Jun  7 2023, 10:13:09) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> d = {}
>>> for x in range(10_000):
...   d = {'k': d}
... 
>>> import json
>>> foo = json.dumps(d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while encoding a JSON object
>>> import sys
>>> sys.setrecursionlimit(100_000)
>>> foo = json.dumps(d)
>>> 

With Python 3.12b4 I get this instead:

$ python3.12
Python 3.12.0b4 (main, Jul 13 2023, 14:24:02) [GCC 13.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> d = {}
>>> for x in range(10_000):
...   d = {'k': d}
... 
>>> import json
>>> foo = json.dumps(d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.12/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while encoding a JSON object
>>> import sys
>>> sys.setrecursionlimit(100_000)
>>> foo = json.dumps(d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.12/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while encoding a JSON object
>>> sys.setrecursionlimit(10_000_000)
>>> foo = json.dumps(d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.12/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while encoding a JSON object
>>> 

Am I missing something, or effectively the recursion constraint has been changed in some way?

Thanks in advance for any clarification.

Thanks for testing with the betas! I think this is worth opening an issue on the CPython repo for, would you mind doing so? Possibly related to GH-91079: Decouple C stack overflow checks from Python recursion checks. by markshannon · Pull Request #96510 · python/cpython · GitHub

Edit: confirmed it bisects to that PR, opened an issue here: Regression in 3.12 beta in json.dump deeply nested dict · Issue #107263 · python/cpython · GitHub

1 Like

Thanks so much!