Non-frame object returned when walking the frame stack in Python 3.11

Hi there. I would like to ask for some help with an issue related to frame stack walking with Python 3.11 that was reported to us by PyPI. The following traceback

AttributeError: 'code' object has no attribute 'f_code'
  File "ddtrace/internal/", line 73, in run
  File "ddtrace/profiling/collector/", line 42, in periodic
    for events in self.collect():
  File "ddtrace/profiling/collector/stack.pyx", line 492, in ddtrace.profiling.collector.stack.StackCollector.collect
  File "ddtrace/profiling/collector/stack.pyx", line 353, in ddtrace.profiling.collector.stack.stack_collect
  File "ddtrace/profiling/collector/_traceback.pyx", line 44, in ddtrace.profiling.collector._traceback.pyframe_to_frames
  File "ddtrace/profiling/collector/_traceback.pyx", line 55, in ddtrace.profiling.collector._traceback.pyframe_to_frames

is being generated by the Datadog Python profiler when attempting to sample the current frame stack of each running thread. The stack_collect method uses the following helper to collect the running threads and create a mapping between thread ID and top frame (among other details).

cdef collect_threads(thread_id_ignore_list, thread_time, thread_span_links) with gil:
    cdef dict current_exceptions = {}

    IF UNAME_SYSNAME != "Windows" and PY_MAJOR_VERSION >= 3 and PY_MINOR_VERSION >= 7:
        cdef PyInterpreterState* interp
        cdef PyThreadState* tstate
        cdef _PyErr_StackItem* exc_info
        cdef PyThread_type_lock lmutex = _PyRuntime.interpreters.mutex
        cdef PyObject* exc_type
        cdef PyObject* exc_tb
        cdef dict running_threads = {}

        # This is an internal lock but we do need it.
        # See
        if PyThread_acquire_lock(lmutex, WAIT_LOCK) == PY_LOCK_ACQUIRED:
            # Do not try to do anything fancy here:
            # Even calling print() will deadlock the program has it will try
            # to lock the GIL and somehow touching this mutex.
                interp = PyInterpreterState_Head()

                while interp:
                    tstate = PyInterpreterState_ThreadHead(interp)
                    while tstate:
                        # The frame can be NULL
                        # Python 3.11 moved PyFrameObject to internal C API and cannot be directly accessed from tstate
                        IF PY_MINOR_VERSION >= 11:
                            frame = PyThreadState_GetFrame(tstate)
                            if frame:
                                running_threads[tstate.thread_id] = <object>frame
                            exc_info = _PyErr_GetTopmostException(tstate)
                            if exc_info and exc_info.exc_value and <object> exc_info.exc_value is not None:
                                # Python 3.11 removed exc_type, exc_traceback from exception representations,
                                # can instead derive exc_type and exc_traceback from remaining exc_value field
                                exc_type = Py_TYPE(exc_info.exc_value)
                                exc_tb = PyException_GetTraceback(exc_info.exc_value)
                                if exc_tb:
                                    current_exceptions[tstate.thread_id] = (<object>exc_type, <object>exc_tb)
                            frame = tstate.frame
                            if frame:
                                running_threads[tstate.thread_id] = <object>frame
                            exc_info = _PyErr_GetTopmostException(tstate)
                            if exc_info and exc_info.exc_type and exc_info.exc_traceback:
                                current_exceptions[tstate.thread_id] = (<object>exc_info.exc_type, <object>exc_info.exc_traceback)
                        tstate = PyThreadState_Next(tstate)

                    interp = PyInterpreterState_Next(interp)
        cdef dict running_threads = <dict>_PyThread_CurrentFrames()

        # Now that we own the ref via <dict> casting, we can safely decrease the default refcount
        # so we don't leak the object

    cdef dict cpu_times = thread_time(running_threads.keys())

    return tuple(
            thread_span_links.get_active_span_from_thread_id(pthread_id) if thread_span_links else None,
        for (pthread_id, native_thread_id), cpu_time in cpu_times.items()
        if pthread_id not in thread_id_ignore_list

The top frames thus collected are passed to the pyframe_to_frames helper, which performs the stack unwinding from each top frame passed to it

cpdef pyframe_to_frames(frame, max_nframes):
    """Convert a Python frame to a list of frames.

    :param frame: The frame object to serialize.
    :param max_nframes: The maximum number of frames to return.
    :return: The serialized frames and the number of frames present in the original traceback."""
    frames = []
    nframes = 0
    while frame is not None:
        nframes += 1
        if len(frames) < max_nframes:
            code = frame.f_code
            lineno = 0 if frame.f_lineno is None else frame.f_lineno
            frames.append((code.co_filename, lineno, code.co_name, _extract_class_name(frame)))
        frame = frame.f_back
    return frames, nframes

The line code = frame.f_code is where we seem to get a Python object that is not a FrameType object on some occasion, which is then responsible for the traceback shown above.

The full sources are available from GitHub - DataDog/dd-trace-py: Datadog Python APM Client (the two relevant Cython sources featured in the traceback are dd-trace-py/stack.pyx at 1.x · DataDog/dd-trace-py · GitHub and dd-trace-py/_traceback.pyx at 1.x · DataDog/dd-trace-py · GitHub)

We have double-checked all our reference counting and decremented them whenever the C API we used returned a strong reference. We haven’t been able to spot any other potential issues with our coding. In particular, we cannot see how the frame variable in the pyframe_to_frames helper could ever be something other than a FrameType (or None). Note that the traceback above is one instance of similar tracebacks, where instead of 'code' we would have some other random Python objects (e.g. 'Project' from SQLAlchemy etc…).

Assuming that we have not overlooked anything in our code, this has left us with essentially two possible conclusion:

  • this is an issue with Cython support for CPython 3.11
  • this is an issue with CPython 3.11 in the way that frame objects are created lazily.

We’re trying to reproduce this issue locally to narrow down which C API call might be returning the random object that we see in the traceback, but so far we haven’t been able to. I’m hoping that perhaps a core dev can think of something that could potentially cause this issue to appear when using certain C API functions around frame stack unwinding, while we try to gather more insight into it.