PEP 703 poses some issues regarding the stable ABI. I’ve described these issue below along with a short proposal.
--disable-gil builds of CPython 3.13 will not be able to load existing wheels that target the stable ABI. However, we can make wheels that target the 3.13+ stable ABI work with both the default (with GIL) and
--disable-gil builds going forward.
CPython C API extensions can be compiled for either the stable ABI or version-specific ABI. Targeting the stable ABI allows a single wheel to work with multiple versions of CPython. To target the stable ABI, C API extensions must limit themselves to the “limited API,” a subset of the Python C API that is marked as stable. Most extensions do not use the stable ABI; they build wheels for specific versions of Python (i.e., the version-specific ABI). These wheels are compatible across “patch” releases e.g., 3.10.0, 3.10.1, etc.), but not across minor (feature) releases of Python (e.g., 3.10 vs. 3.11).
There is not a single “stable ABI” but rather a stable ABI for each minimum Python version starting with Python 3.2. New Python releases may add new functions to the stable ABI. To take advantage of these new functions, extensions using the stable ABI must target a specific minimum version. For example, as of the time of this writing, the latest cryptography wheels target cp37-abi3 so they work with Python 3.7 and newer releases. Additionally, ABI stability depends on the platform (OS, architecture, libc implementation).
Common reasons for targeting the stable ABI include:
- Reducing the workload to build and test wheels. For example, the cryptography package builds 22 wheels (10 for CPython, 12 for PyPy) for each release. Without the stable ABI, this would increase to 72 wheels to support Python 3.7-3.12.
- Ensuring extensions are immediately available when a new version of CPython is released.
- Some extensions do not use the Python C API at all, but have platform specific native libraries accessed via cffi. These extensions may be labeled as if they used the stable ABI because they are compatible across Python versions but not across platforms.
Common reasons for targeting the version-specific ABI include:
- The stable ABI may not expose sufficient functionality for the extension.
- The stable ABI may reduce extension performance due to use of function calls instead of macros/inline functions for some operations. For example, Py_INCREF and Py_DECREF are non-inlineable function calls when targeting the 3.12 stable ABI or newer.
PEP 703 requires changing the PyObject reference count fields for
--disable-gil builds. This means that existing wheels that target the stable ABI will not work with
--disable-gil builds because they include inlined code (from macros or inline functions) that directly access these fields.
However, it’s possible to make wheels that target the Python 3.13 stable ABI work with both the default and
--disable-gil builds going forward. We can do this by ensuring that all reference counting operations use function calls instead of macros or inline functions. The Python 3.12 stable ABI has taken a step in this direction: Py_INCREF and Py_DECREF already use the function calls _Py_IncRef and _Py_DecRef when targeting the Python 3.12+ stable ABI.
Additionally, the Py_SIZE / Py_SET_SIZE macros would need to use function calls since the offset of the ob_size field would differ between
--disable-gil and the default build.
Add the following functions to the 3.13 stable ABI as “ABI only” functions:
- Py_ssize_t _Py_Refcnt(PyObject * ob);
- void _Py_SetRefcnt(PyObject *ob, Py_ssize_t refcnt);
- int _Py_IsImmortal(PyObject *ob);
- Py_ssize_t _Py_Size(PyObject *ob);
- void _Py_SetSize(PyObject *ob, Py_ssize_t size);
Note that the names begin with underscores because they are not intended to be called directly, even when targeting the limited API. Instead, they should be called via the corresponding standard macros. This is the pattern used by functions like _Py_IncRef, _Py_DecRef, and _Py_Dealloc. See functions marked
abi_only=True in https://github.com/python/cpython/blob/main/Misc/stable_abi.toml.
Change the following macros to use function calls when targeting Py_LIMITED_API >= 3.13:
- Py_REFCNT to use _Py_Refcnt
- Py_SET_REFCNT to use _Py_SetRefcnt
- Py_SIZE to use _Py_Size
- Py_SET_SIZE to use _Py_SetSize
Add the following macro to the public limited API for Py_LIMITED_API >= 3.13:
(This is already exposed by PyO3, which is one of the most common ways developers rely on the stable ABI)
Add (and document) a new field “abi_disable_gil” to sys.implementation to reliably differentiate
--disable-gil builds from the default build in CPython. The field
sys.implementation.abi_disable_gil is set to
--disable-gil builds and
Do we want to make Py_TYPE and Py_SET_TYPE use function calls? This is not strictly necessary since we can ensure that the
ob_type field is at the same offset for both
--disable-gil and the default builds, but may avoid some future issues.
Extensions that target the older versions of the stable ABI will continue to work with the default build of CPython 3.13+, but not
--disable-gil builds of CPython. If an extension author using the stable ABI does not wish to support
--disable-gil builds, they do not need to change anything.
Extension authors using the stable ABI that wish to support both older versions of CPython (e.g., 3.7+) and
--disable-gil builds will need to build two wheels per platform. For example, a future version of the cryptography package might provide:
Note that the default (with GIL) build of Python 3.13 could use either wheel.
Extensions that do not use the stable ABI (e.g., NumPy), will need separate wheels for the 3.13 default build and the 3.13
--disable-gil build. For example, a future NumPy release for 3.13 might have the filenames and platform tags:
- numpy-1.27.0-cp313-cp313-win_amd64.whl (default build)
- numpy-1.27.0-cp313-cp313t-win_amd64.whl (
This is unchanged from PEP 703, other than the use of “t” (for threaded) instead of “n” (for nogil) as suggested here.
- pypa/packaging: When computing platform tags for
--disable-gilbuilds (i.e., when
sys.implementation.abi_disable_gilis True), cpython_tags() should exclude abi3 builds that target versions before 3.13.
- pip: pip vendors the “packaging” package and would need to update their copy
- poetry: poetry makes use of “packaging” to compute tags
- hatch: hatch makes use of “packaging” to compute tags
- PyPI: No changes necessary. It already supports different ABI tags (see colesbury-c-extension · PyPI)
- setuptools: No changes necessary.