Getting Fatal _PyInterpreterState_GET error running basic extension type in debug build

Python Help
1

Summary: the standard python debug build (3.11.7 on macOS both arm64 and x86_64) fails to run the simplest newtype extension example provided in the python docs and gives the following error when run:

Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: unknown

NOTE: I have create a tiny github project to illustrate this error case.

Here’s the write up if you don’t want to jump to the GitHub project:

The newtypes tutorial has as its most basic initial example a python type extension which we will put in a file custommodule.c as follows:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

typedef struct {
    PyObject_HEAD
    /* Type-specific fields go here. */
} CustomObject;

static PyTypeObject CustomType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom.Custom",
    .tp_doc = PyDoc_STR("Custom objects"),
    .tp_basicsize = sizeof(CustomObject),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_new = PyType_GenericNew,
};

static PyModuleDef custommodule = {
    PyModuleDef_HEAD_INIT,
    .m_name = "custom",
    .m_doc = "Example module that creates an extension type.",
    .m_size = -1,
};

PyMODINIT_FUNC
PyInit_custom(void)
{
    PyObject *m;
    if (PyType_Ready(&CustomType) < 0)
        return NULL;

    m = PyModule_Create(&custommodule);
    if (m == NULL)
        return NULL;

    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) {
        Py_DECREF(&CustomType);
        Py_DECREF(m);
        return NULL;
    }

    return m;
}

This imports and runs without issues in a normal (non-debug) python build but if it is imported in a standard debug-build python compiled as follows:

CWD=`pwd`
BUILD=${CWD}/build
DOWNLOADS=${BUILD}/downloads
PYTHON=${CWD}/python # local destination
PYTHON_URL=https://www.python.org/ftp/python/3.11.7/Python-3.11.7.tar.xz
PYTHON_ARCHIVE=`basename ${PYTHON_URL}`
PYTHON_FOLDER=`basename -s .tar.xz ${PYTHON_ARCHIVE}`

mkdir -p ${PYTHON} && \
	mkdir -p ${DOWNLOADS} && \
	cd ${DOWNLOADS} && \
	wget ${PYTHON_URL} && \
	tar xvf ${PYTHON_ARCHIVE} && \
	cd ${PYTHON_FOLDER} && \
	mkdir debug && \
	cd debug && \
	../configure --prefix ${PYTHON} \
				 --with-pydebug \
				 --enable-shared \
				 --disable-test-modules \
				 --without-static-libpython \
				 --with-ensurepip=no && \
	make EXTRA_CFLAGS="-DPy_DEBUG" && \
	make install

It will fail and give the following error:

Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: unknown

The traceback was given (Apple system reporting) as:

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	    0x7ff8113181e2 __pthread_kill + 10
1   libsystem_pthread.dylib       	    0x7ff81134fee6 pthread_kill + 263
2   libsystem_c.dylib             	    0x7ff811276b45 abort + 123
3   libpython3.11d.dylib          	       0x10c077afd fatal_error_exit + 13
4   libpython3.11d.dylib          	       0x10c0779bb fatal_error + 1019
5   libpython3.11d.dylib          	       0x10c0772e4 _Py_FatalErrorFunc + 52
6   libpython3.11d.dylib          	       0x10c015380 _Py_FatalError_TstateNULL + 16
7   libpython3.11d.dylib          	       0x10bf4a66a new_dict + 298
8   libpython3.11d.dylib          	       0x10bf7ac61 PyType_Ready + 177
9   custom.cpython-311-darwin.so  	       0x10b025da4 PyInit_custom + 20 (custommodule.c:30)
10  Python                        	       0x10b3f200b _imp_create_dynamic + 801
11  Python                        	       0x10b32fd91 cfunction_vectorcall_FASTCALL + 89
12  Python                        	       0x10b3c285e _PyEval_EvalFrameDefault + 54618
13  Python                        	       0x10b3c5082 _PyEval_Vector + 92
14  Python                        	       0x10b2edb81 object_vacall + 264
15  Python                        	       0x10b2eda1f PyObject_CallMethodObjArgs + 226
16  Python                        	       0x10b3ee9e0 PyImport_ImportModuleLevelObject + 1312
17  Python                        	       0x10b3bcb56 _PyEval_EvalFrameDefault + 30802
18  Python                        	       0x10b3b47a5 PyEval_EvalCode + 191
19  Python                        	       0x10b40d5ec run_eval_code_obj + 72
20  Python                        	       0x10b40d57c run_mod + 96
21  Python                        	       0x10b40f527 PyRun_StringFlags + 100
22  Python                        	       0x10b40f48b PyRun_SimpleStringFlags + 69
23  Python                        	       0x10b427838 pymain_run_command + 134
24  Python                        	       0x10b427305 Py_RunMain + 302
25  Python                        	       0x10b428416 Py_BytesMain + 42
26  dyld                          	    0x7ff810ff641f start + 1903

What is the cause of this error and how does one resolve it? It looked it failed at PyType_Ready…

2

OK, I found the answer: the error was caused by inadvertently using non-debug python to run a script which imported the extension compiled against a debug python build.