Hi,
I modified Python 3.8 to make the PyObject structure ABI compatible between Python release and debug builds. You can just use python-debug (debug build) as a drop-in replacement of python (release build), it “just works” (on most platforms, ex: Linux).
I fixed Python 3.10 to make add support of the limited C API to Python debug build (Py_REF_DEBUG): Py_INCREF() and Py_DECREF() are implemented as opaque function calls in this case. Moreover, PyObject_INIT() is now implemented with an opaque function call: _Py_NewReference() is no longer called directly as the ABI level.
Previously, the C API exposed many implementation details at the ABI level (through inlined functions and macros):
- private
_Py_RefTotalvariable (Py_REF_DEBUGmacro of debug build) - private
_Py_NewReference()and_Py_ForgetReference() - private
_Py_tracemalloc_configvariable - private
_Py_inc_count()and_Py_dec_count()functions ofCOUNT_ALLOCspecial build (removed in Python 3.9)
In Python 3.12, with the implementation of immortal objects (PEP 683), Py_INCREF() and Py_DECREF() implementation became more complex than PyObject.ob_refcnt++ and PyObject.ob_refcnt--.
Extract of Py_INCREF() implementation (simplified code):
struct _object {
_PyObject_HEAD_EXTRA
union {
Py_ssize_t ob_refcnt;
#if SIZEOF_VOID_P > 4
PY_UINT32_T ob_refcnt_split[2];
#endif
};
PyTypeObject *ob_type;
};
static inline int _Py_IsImmortal(PyObject *op)
{
#if SIZEOF_VOID_P > 4
return _Py_CAST(PY_INT32_T, op->ob_refcnt) < 0;
#else
return op->ob_refcnt == _Py_IMMORTAL_REFCNT;
#endif
}
static inline void Py_INCREF(PyObject *op)
{
#if SIZEOF_VOID_P > 4
// Portable saturated add, branching on the carry flag and set low bits
PY_UINT32_T cur_refcnt = op->ob_refcnt_split[PY_BIG_ENDIAN];
PY_UINT32_T new_refcnt = cur_refcnt + 1;
if (new_refcnt == 0) {
return;
}
op->ob_refcnt_split[PY_BIG_ENDIAN] = new_refcnt;
#else
// Explicitly check immortality against the immortal value
if (_Py_IsImmortal(op)) {
return;
}
op->ob_refcnt++;
#endif
_Py_INCREF_STAT_INC();
#ifdef Py_REF_DEBUG
_Py_INC_REFTOTAL();
#endif
}
Link to full implementation:
By the way, the current implementation causes new C compiler warnings/errors.
While PEP 683 says that the stable ABI is not affected (C extensions built with Python 3.11 continue to work), IMO it’s now time to consider converting Py_INCREF() and Py_DECREF() to opaque function calls in limited C API version 3.12 and newer. I propose: issue and PR.
Guido opened a similar discussion at capi-workgroup/problems.
In the long term, I even consider doing a similar change for the regular C API, as HPy does, to hide even more implementation details and slowly bend the regular C API to the limited C API and the stable ABI. But that’s out of my short term scope. Now I only consider changing the limited C API version 3.12.
Victor
The limited API has changed enough over time that promise is well broken, and it’s somewhere around 3.6-3.7 that things start becoming usable again (if you’re careful, and test thoroughly on multiple versions on multiple platforms, by which time you may as well have just built without it).