Hi,
tl; dr: Would you be ok to start using the limited C API in Python stdlib?
I propose to convert some stdlib C extensions to the limited C API to eat our own dog food, embrace and test this API. I would like to promote the limited C API and the stable ABI: using the limited C API ourself is a first step. Using private C API functions and the internal C API should be the exception, not the default in Python stdlib; see also C API: My plan to clarify private vs public functions in Python 3.13 . Performance remains a good reason to use the internal C API, but performance is not strictly needed for the 115 stdlib extensions (104 extensions + 11 test extensions).
The stable ABI makes the distribution of package binaries easier. For example, binaries are already available before the new Python is being released! It makes newer Python usable since the first day of its release, because it’s simply the same binary for all Python versions. (One binary per platform+architecture is still needed.)
In 2009, Martin v. Löwis created a subset of the C API called the “Limited C API” in PEP 384 – Defining a Stable ABI used when the Py_LIMITED_API macro is defined (it can be set to a Python version). If you limit your code to this API and build your C extension in a specific way, you can use the “stable ABI” which works on Python 3.2 and newer without having to rebuild Python.
The stable ABI usage remains low: around 551 PyPI packages provide abi3 binaries. The two most popular users are PySide and cryptography (4.37K packages and 16.9K repositories depend on cryptography). Last year, Paul Moore (@pf_moore) ran a search on all PyPI packages providing binaries built for the stable ABI. See also the interesting discussion Let’s get rid of the stable ABI, but keep the limited API which discuss how and why it’s being used currently (for now, there is no plan to remove the stable ABI).
In my experience, a practical problem of the limited of the C API and the stable ABI is that it’s not used by Python itself, and so issues are discovered too late (missing important function, broken implementation, regression, etc.). Petr Viktorin (@encukou) made a great progress in Python 3.10 with PEP 652 – Maintaining the Stable ABI which added Misc/stable_abi.toml, test_stable_abi_types, and way more cool stuff.
I propose to start eating our own dog food by converting some stdlib C extensions to the limited C API. In some cases (a few extensions at least), the change is just about adding a single line at the top of the file!
#define Py_LIMITED_API 0x030d0000
I did some tests and so far, I saw the two following issues.
Function missing in the limited C API like PySys_Audit(). We should look at missing functions on a case by case basis to see if it’s worth it to add the function, or if using the Python API in C (PyImport_ImportModule(), PyObject_GetAttrString(), PyObject_CallFunction(), …) can be acceptable (performance?).
Performance regression: most efficient code is hidden in the internal C API, the limited C API can be slower. My PR for _statistics made _statistics._normal_dist_inv_cdf() made the function 2x slower because of the usage of the legacy METH_VARARGS calling convention (instead of more recent METH_FASTCALL which avoids creating a temporary tuple to pass positional arguments, and a temporary dict to pass keyword arguments). There is room for improvement, Argument Clinic and the limited C API can be enhance to get better performance.
Many C extensions can be converted to the limited C API without any change, or only with minor changes, and no significant impact on performance. Moreover, many C extensions are not performance sensitive. For example, the pwd and grp extensions which just expose C functions in Python, I don’t expect them to be part of hot code in any application.
In 2020, I already tried to convert some stdlib C extensions to the limited C API, but I was blocked by multiple issues:
- Argument Clinic didn’t support the limited C API
- The limited C API missed some multiple important features
Good news: Argument Clinic just got support for the limited C API! And the important features were added to the limited C API in the meanwhile! See C API: Convert a few stdlib extensions to the limited C API (PEP 384) for the details.
A lot of work has been put over last years in converting extensions static types to heap types. This work is non trivial and is a starting point for the conversion to the limited C API, since this API doesn’t support static types.
Victor
