Stable ABI/Limited API for free-threaded builds

Core Development C API
1

Hello,
I’ve been looking into stable ABI for free-threading builds for the past few months. I think I have plans for all the pieces, but I don’t think I can assemble them for 3.14.
I now need to take a break for a month, so I’ll share where I am. Sorry for the lack of background info. If you would like to work on any of this on in April, don’t ask for my permission.

The general plan is to have 3 variants of the stable ABI abi3 (non-free-threaded, same as existing), “abi3t” (free-threaded, same API as abi3), and “abi4” (supports both, but requires API changes).

The API change needed for an ABI that supports both builds (and for fixing the main issue in stable ABI itself) is making the PyObject struct opaque. IMO, on the CPython side, we can hide the structs if the user defined Py_OPAQUE_PYOBJECT.
Before that’s practical, we need a way to define modules without a static PyModuleDef; I want to add a new module export hook that returns a slots array – see my proposal and early proof-of-concept branch. (And ideally, introduce new slots in the same release, so we don’t need support 2 versions of slots in the new export hook.)

To support stable ABI of 3.13 and below, we’ll need to finish replacing accessor macros by exported functions (these are left: Py_SET_TYPE, Py_SIZE, Py_SET_SIZE). Extensions need to call the functions where available but fall back to direct member access. Sam proposed to use weak symbols for this; I think it would be easier – and more cross-platform friendly – to put function pointers in a capsule called e.g. sys._abi_compat, and add a static function to transparently import that on first use. See my rough notes.

We’ll need to define support windows, so we can gradually phase out past versions of the stable ABI (so we don’t wait for a “Python 4” break point). That might be something like 10 years (for the ABI surface – the behaviour behind the interface is subject to PEP 387).

I’d like to add a module slot that indicates the version of Python used to build the extension, the Py_LIMITED_API value used (if any), and some flags (Py_OPAQUE_PYOBJECT as above; Py_BUILD_CORE). This would be checked for compatibility, so we no longer rely only on wheel/.soname tags, and so we can add deprecation warnings for upcoming incompatibilities.

It will be necessary to define .so tags and wheel tags.

And we’ll need tests that this all works.

7 Likes
2

I don’t think this is worthwhile.[1] It’s not inherently bad or wrong, but it’s effort that isn’t worth our time compared to the better options.

If the change is just part of a new ABI, we can actually change runtime interactions. For example, abi4 can make them opaque and be more compatible at runtime as a result (between free-threaded and not).

If the change isn’t part of an ABI, this is just a diagnostic tool for a developer to run their build with and see what breaks. Which is fine, but doesn’t actually provide any benefit over running your build with abi4 and seeing what breaks.

It may be that adding the preprocessor option that breaks your own build but doesn’t affect the build output is easier to add (so we can do it first in less time), but that’s only true if we then go on to change the ABI in the same way as the variable worked. Otherwise, it’s just as (subtly) incompatible, and devs still have to do a test build with the actual option. Plus it creates more code that we really shouldn’t have to maintain.

Everything else proposed sounds like a good way forward, so thanks for doing the planning/thinking work to get to this stage. Enjoy your time off!

  1. I feel like I’m always pushing back on new compile-time macro options… ↩︎

1 Like