Extension module ABI compatibility check

There’s a number of things that affect the compatibility between an extension module and the Python interpreter. Just listing a few:

  • Free-threading
  • debug/no-debug
  • Py_TRACE_REFS
  • Python version

Because of the way the Windows headers are distributed, it’s currently pretty easy to build an extension module with the “free-threading” filename tag, but built again non-freethreaded header. This obviously immediately crashes. The other incompatibilities are less easy to do by mistake, but may still be worthwhile to check.

I think I’ve managed to hack something together in Cython that checks many of these parameters and raises an exception if they don’t match without manipulating Python objects in directly C (i.e. it only uses bits of the ABI which don’t change in the different build modes). The idea being that users are able to receive an ImportError telling them that their extension doesn’t match the interpreter rather than a mystery crash.

However, I was wondering if the Python interpreter could provide an interface to do this check that would be more robust than my hacked-together one. It’d probably want to be extensible so something like:

int Py_ABI_Check(int major_version, int minor_version, unsigned long flags);

where flags is built up by something like:

Py_ABI_FLAGS = 0 |
#if Py_DEBUG
    1 |
#endif
#if Py_GIL_DISABLED
    2 |
#endif
#if Py_TRACE_REFS
    4
#endif
// etc

The idea is that you could write

PyObject *PyInit_module_name(void) {
    if (Py_ABI_Check(PY_MAJOR_VERSION, PY_MINOR_VERSION, Py_ABI_FLAGS) < 0) {
        return NULL;
    }
    return &moduledef;
}

and get a sensible (and cheap) error message if the interpreter doesn’t match. Obviously it wouldn’t be compulsory for extension authors to do this, but it might be a useful tool to have.

This would be nice to do, but it isn’t straightforward. How much time do you want to spend on it?
Off the top of my head:

  • At least on Linux, debug/no-debug use the same ABI.
  • The stable ABI is compatible with several versions.
  • In C, the details could be gathered automatically by a macro (so the call would be just if (Py_ABI_CHECK() < 0)), but we probably want a “real” function as well. If a future version of the function would need more arguments, we’d need a new version of the function.

Also take a look at PyModule_Create implementation for some prior art. (That predates my involvement with CPython; AFAIK it was used before the ABI was version-specific).

Ideally not too much :slight_smile: . I was mainly proposing to diagnose some common cases rather than every possible combination.

That’s good to know - thanks.

The issue that inspired this thought is probably more of an issue combining setuptools with the Windows distribution. So possibly this proposal is a little premature for something that may well be fixed elsewhere.