Module slot for checking ABI compatibility

Expanding Da Woods’ idea:

Currently, for checking whether an extension module is ABI-compatible, CPython relies on:

  • an ABI tag in the extension filename (for example, in NumPy’s
    _umath_linalg.cpython-313t-x86_64-linux-gnu.so), and/or
  • installing extension files in interpreter-specific site-packages
    directories.

This places a rather heavy responsibility on build and install tools.
I don’t think we should fully take away that responsibility, but I’d like to add an additional version check inside CPython to:

  • make it safer/easier to create lightweight build/install tools;
  • position CPython as the authoritative source on which ABI- and version-related information is relevant, which build settings
    affect the ABI, and which settings are compatible with a given interpreter;
  • make the info available to the CPython runtime for future use cases. For example, if we add “support windows” for the stable ABI, we could raise deprecation warnings when it’s time to rebuild an extension;
  • (to answer the “why now?”): take some pressure off build tools to support free-threaded stable ABI correctly.

For the record: Python 2 had a similar (but simpler) check, PYTHON_API_VERSION, which is long unused.

To be clear, I don’t want to detect any possible ABI incompatibility; the check should guard against some simple mistakes like extension authors defining a wrong flag or users copying an existing file to where it doesn’t belong.


Proposal: add a new module slot, Py_mod_abi, whose value would point to a static struct that would carry info about an extension’s ABI and would be checked at runtime (i.e. CPython raises ImportError if it doesn’t match).

It would come with a convenience macro, so typical usage would be:

// declares the variable `abi_info`, similar to PyDoc_STRVAR
PyABIInfo_VAR(abi_info);

static PyModuleDef_Slot my_slots[] = {
    ...
    {Py_mod_abi, &abi_info},
};

For advanced use cases (and non-C languages), the structure used would be fully opaque (and versioned).
Also, setting any field to 0 would disable the corresponding check (so you can e.g. experiment with ABI that’s compatible across multiple builds).
The initial version can be minimal:

typedef struct PyABIInfo {
    uint16_t abiinfo_version;
    uint16_t flags;
    uint32_t build_version;
    uint32_t abi_version;
} PyABIInfo;
  • abiinfo_version: version of the structure
    • 0: skip all checking
    • 1: use the initial version of the spec
    • larger values: reserved for future extensions
  • flags:
    • 1 bit: using the limited API?
    • 2 bits: free-threaded only / non-FT only / supports both
    • 2 bits: Py_TRACE_REFS / no-tracerefs / no check
    • 1 bit: using the internal API?
  • build_version: full hex version of the CPython headers used to build the extension
  • abi_version: the Py_LIMITED_API version, if defined

To make this “default”, the new slot would be mandatory in any new module-defining API (such as PEP 793, if it’s accepted), with an easy “opt out”, e.g. {Py_mod_abi, &zero}.

3 Likes

So if a extension module is built under a high version interpreter, can it be imported by an old interpreter even if the module itself is fully compatible with the old one. It seems the old interpreter is unable to parse the newly-added slot?

When I initially raised this it was mainly motivated by freethreading on Windows, where selecting the right header files seems to be a real disaster. I think freethreading/non-freethreading is the main thing this interface would need to cover, and anything else is just a bonus.

The added complication here is that Python.h arranges for the extension module to be automatically linked to the Python library using some #pragma magic. That meant that you ended up with (e.g.) freethreading Python trying to load a non-freethreading module, but the module was also somehow linked to non-freethreading Python and was trying to call the functions in that dll.

So my one warning here is that your new slot would need to be robust to that enormous mess.

(Initially I thought I’d come up with a clever way to add a similar check to Cython without special interpreter support but ultimately it didn’t work; I think because of the linking problems above).

Yes – as with any new feature, you can use this once you don’t build for interpreters without the feature.
You could check the Python version and hide the slot from old interpreters, but that’s probably too much effort for any gain.
(Or, if PEP 793 is accepted for the same Python version, you can include it only for the new hook. Similar with optional slots.)

Sounds like this proposal would prevent this? A freethreading Python trying to load a module with “non-FT only” in PyABIInfo.flags would fail.

Do you have a reproducer I could use to test this?

Install Python 3.13 from the Windows installer with including the freethreading build:

mod.c:

#include <Python.h>

static struct PyModuleDef module = {
    .m_base = PyModuleDef_HEAD_INIT,
    .m_name = "mod",
    .m_size = 0,  // non-negative
    .m_slots = NULL,
};

PyMODINIT_FUNC
PyInit_mod(void)
{
    return PyModuleDef_Init(&module);
}

setup.py:

from setuptools import Extension, setup

setup(
    ext_modules=[
        Extension(
            name="mod",
            sources=["mod.c"],
        ),
    ]
)

With the freethreading Python, run python3.13t.exe setup.py build_ext -if

Start python3.13t and import mod - it’ll crash.

You can load mod.cp313t-win_amd64.pyd in the Dependencies tool to see that it’s linked to the non-freethreading Python library:

1 Like

I understand that in some way it is neat to piggyback on module slots for this, but I suggest considering a separate mechanism which would be executed as the very first thing before loading anything else. Communicating an extension’s ABI to the runtime is a special thing:

  • Ideally you do not want to execute any ABI calls if the expected ABI does not match
  • Eventually in the future CPython could support multiple ABIs, and the choice of the ABI must be done as the first thing. The module slots and whole module initialization contract is part of the ABI.

In C, it can be just a macro, that would generate the boilerplate code: static struct with the right values, and some MyModule_GetPythonABI exported function, which Python would lookup and call if it is available. The contract would be that no ABI calls should be made in MyModule_GetPythonABI and the convenience macro would lead people to avoid doing that.

I want slots to be that. From PEP 793:

We expect [the hook] to export a static constant, or one of several constants chosen depending on, for example, Py_Version. Dynamic behaviour should generally happen in the Py_mod_create and Py_mod_exec functions.

Yes, but that passes PyObject* to the initial entry point, which invites for accessing its fields or passing it to some API calls, otherwise there would be no use for that parameter?

Yes, ideally you wouldn’t use it. I added it for hacks, hotfixes, and specialized loaders.
Also note that the slot I’m proposing isn’t meant to catch any ABI incompatibility – it’s meant to guard against common mistakes.

Sounds like we might need a new export hook in the future! Or a different filename tag at least.