Expanding Da Woods’ idea:
Currently, for checking whether an extension module is ABI-compatible, CPython relies on:
- an ABI tag in the extension filename (for example, in NumPy’s
_umath_linalg.cpython-313t-x86_64-linux-gnu.so
), and/or - installing extension files in interpreter-specific
site-packages
directories.
This places a rather heavy responsibility on build and install tools.
I don’t think we should fully take away that responsibility, but I’d like to add an additional version check inside CPython to:
- make it safer/easier to create lightweight build/install tools;
- position CPython as the authoritative source on which ABI- and version-related information is relevant, which build settings
affect the ABI, and which settings are compatible with a given interpreter; - make the info available to the CPython runtime for future use cases. For example, if we add “support windows” for the stable ABI, we could raise deprecation warnings when it’s time to rebuild an extension;
- (to answer the “why now?”): take some pressure off build tools to support free-threaded stable ABI correctly.
For the record: Python 2 had a similar (but simpler) check, PYTHON_API_VERSION
, which is long unused.
To be clear, I don’t want to detect any possible ABI incompatibility; the check should guard against some simple mistakes like extension authors defining a wrong flag or users copying an existing file to where it doesn’t belong.
Proposal: add a new module slot, Py_mod_abi
, whose value would point to a static struct that would carry info about an extension’s ABI and would be checked at runtime (i.e. CPython raises ImportError
if it doesn’t match).
It would come with a convenience macro, so typical usage would be:
// declares the variable `abi_info`, similar to PyDoc_STRVAR
PyABIInfo_VAR(abi_info);
static PyModuleDef_Slot my_slots[] = {
...
{Py_mod_abi, &abi_info},
};
For advanced use cases (and non-C languages), the structure used would be fully opaque (and versioned).
Also, setting any field to 0
would disable the corresponding check (so you can e.g. experiment with ABI that’s compatible across multiple builds).
The initial version can be minimal:
typedef struct PyABIInfo {
uint16_t abiinfo_version;
uint16_t flags;
uint32_t build_version;
uint32_t abi_version;
} PyABIInfo;
abiinfo_version
: version of the structure- 0: skip all checking
- 1: use the initial version of the spec
- larger values: reserved for future extensions
flags
:- 1 bit: using the limited API?
- 2 bits: free-threaded only / non-FT only / supports both
- 2 bits:
Py_TRACE_REFS
/ no-tracerefs / no check - 1 bit: using the internal API?
build_version
: full hex version of the CPython headers used to build the extensionabi_version
: thePy_LIMITED_API
version, if defined
To make this “default”, the new slot would be mandatory in any new module-defining API (such as PEP 793, if it’s accepted), with an easy “opt out”, e.g. {Py_mod_abi, &zero}
.