Hi all,
today I tried to understand a test failure I’m seeing trying out python-rapidjson under Python 3.12b4.
One particular test (see tests/test_circular.py#L45-L70 [link removed to honor the “new user limitation”…]) uses sys.setrecursionlimit() to assert that the underlying C++ implementation (that uses Py_EnterRecursiveCall(), see for example properly raises a RecursionError.
Up to Python 3.11, it does work as expected, while on 3.12 it does not, or more precisely I do not seem able to control the recursion depth: it is like the recursion limit is fixed, and whatever value I pass to sys.setrecursionlimit() does not influence that.
I looked around to find news about changes related to either sys.setrecursionlimit() or Py_EnterRecursiveCall(), but found nothing. So I poked into the source of stdlib json module, and seeing it very similar to the rapidjson code, I tried to replicate the “problem” with that, and indeed I see a similar result.
Under Python 3.11 I see this:
$ python3.11
Python 3.11.4 (main, Jun 7 2023, 10:13:09) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> d = {}
>>> for x in range(10_000):
... d = {'k': d}
...
>>> import json
>>> foo = json.dumps(d)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.11/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/json/encoder.py", line 200, in encode
chunks = self.iterencode(o, _one_shot=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/json/encoder.py", line 258, in iterencode
return _iterencode(o, 0)
^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while encoding a JSON object
>>> import sys
>>> sys.setrecursionlimit(100_000)
>>> foo = json.dumps(d)
>>>
With Python 3.12b4 I get this instead:
$ python3.12
Python 3.12.0b4 (main, Jul 13 2023, 14:24:02) [GCC 13.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> d = {}
>>> for x in range(10_000):
... d = {'k': d}
...
>>> import json
>>> foo = json.dumps(d)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.12/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/json/encoder.py", line 200, in encode
chunks = self.iterencode(o, _one_shot=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/json/encoder.py", line 258, in iterencode
return _iterencode(o, 0)
^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while encoding a JSON object
>>> import sys
>>> sys.setrecursionlimit(100_000)
>>> foo = json.dumps(d)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.12/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/json/encoder.py", line 200, in encode
chunks = self.iterencode(o, _one_shot=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/json/encoder.py", line 258, in iterencode
return _iterencode(o, 0)
^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while encoding a JSON object
>>> sys.setrecursionlimit(10_000_000)
>>> foo = json.dumps(d)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.12/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/json/encoder.py", line 200, in encode
chunks = self.iterencode(o, _one_shot=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/json/encoder.py", line 258, in iterencode
return _iterencode(o, 0)
^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while encoding a JSON object
>>>
Am I missing something, or effectively the recursion constraint has been changed in some way?
Thanks in advance for any clarification.