Stable ABI/Limited API for free-threaded builds

Core Development C API
1

Hello,
I’ve been looking into stable ABI for free-threading builds for the past few months. I think I have plans for all the pieces, but I don’t think I can assemble them for 3.14.
I now need to take a break for a month, so I’ll share where I am. Sorry for the lack of background info. If you would like to work on any of this on in April, don’t ask for my permission.

The general plan is to have 3 variants of the stable ABI abi3 (non-free-threaded, same as existing), “abi3t” (free-threaded, same API as abi3), and “abi4” (supports both, but requires API changes).

The API change needed for an ABI that supports both builds (and for fixing the main issue in stable ABI itself) is making the PyObject struct opaque. IMO, on the CPython side, we can hide the structs if the user defined Py_OPAQUE_PYOBJECT.
Before that’s practical, we need a way to define modules without a static PyModuleDef; I want to add a new module export hook that returns a slots array – see my proposal and early proof-of-concept branch. (And ideally, introduce new slots in the same release, so we don’t need support 2 versions of slots in the new export hook.)

To support stable ABI of 3.13 and below, we’ll need to finish replacing accessor macros by exported functions (these are left: Py_SET_TYPE, Py_SIZE, Py_SET_SIZE). Extensions need to call the functions where available but fall back to direct member access. Sam proposed to use weak symbols for this; I think it would be easier – and more cross-platform friendly – to put function pointers in a capsule called e.g. sys._abi_compat, and add a static function to transparently import that on first use. See my rough notes.

We’ll need to define support windows, so we can gradually phase out past versions of the stable ABI (so we don’t wait for a “Python 4” break point). That might be something like 10 years (for the ABI surface – the behaviour behind the interface is subject to PEP 387).

I’d like to add a module slot that indicates the version of Python used to build the extension, the Py_LIMITED_API value used (if any), and some flags (Py_OPAQUE_PYOBJECT as above; Py_BUILD_CORE). This would be checked for compatibility, so we no longer rely only on wheel/.soname tags, and so we can add deprecation warnings for upcoming incompatibilities.

It will be necessary to define .so tags and wheel tags.

And we’ll need tests that this all works.

9 Likes
2

I don’t think this is worthwhile.[1] It’s not inherently bad or wrong, but it’s effort that isn’t worth our time compared to the better options.

If the change is just part of a new ABI, we can actually change runtime interactions. For example, abi4 can make them opaque and be more compatible at runtime as a result (between free-threaded and not).

If the change isn’t part of an ABI, this is just a diagnostic tool for a developer to run their build with and see what breaks. Which is fine, but doesn’t actually provide any benefit over running your build with abi4 and seeing what breaks.

It may be that adding the preprocessor option that breaks your own build but doesn’t affect the build output is easier to add (so we can do it first in less time), but that’s only true if we then go on to change the ABI in the same way as the variable worked. Otherwise, it’s just as (subtly) incompatible, and devs still have to do a test build with the actual option. Plus it creates more code that we really shouldn’t have to maintain.

Everything else proposed sounds like a good way forward, so thanks for doing the planning/thinking work to get to this stage. Enjoy your time off!

  1. I feel like I’m always pushing back on new compile-time macro options… ↩︎

2 Likes
4

How do you opt in to the “new” unified ABI?
I’m less and less convinced to g with the abi4 theme, but if we do that, it would be reasonable to set Py_LIMITED_API to 0x04.......
If we go with what I wrote here, you’d #define Py_OPAQUE_PYOBJECT (an API-focused name). Then instead of abi4 the result could use the wheel tag abi3.abi3t.

5

This is a non-response, but I agree those are the options. I’m not a huge fan of abi3t (because I want the t to be temporary, and this would make it permanent and we’d eventually drop abi3 (if nogil is merged)), but it is cheaper and easier than doing abi4, and more consistent than a specific flag.

The other aspect is our test matrix, which ought to include all combinations of build flags we offer. So keeping it limited to an entire “profile” (i.e. abi3 etc.) rather than a set of independent options means we can actually keep up with our own offering.

FWIW, I see abi4 as the only chance we’ve really got of avoiding/significantly delaying a Python 4, but that does mean we should design it properly (with many of the “api-revolution” ideas) so that we have a good buffer to allow runtime changes underneath the API. Calling it abi3t and doing the same thing is just lying to everyone, and calling it abi3t and doing the bare minimum is only delaying the pain by a couple of years.

However, there seems to be too much in flux right now to design a forward-compatible abi4 anyway. So I guess we’ll be forced into doing the bare minimum, in which case I’d strongly prefer it to actually be the minimum, and not something that adds (more) significant burdens on us.

4 Likes
6

That means new slots and support windows are out for now. I’ll write up a separate motivation for them later :‍)

I’m still leaning toward a new version-checking slot, especially if it would mean that extension “.so” filenames can keep using just abi3 for abi3t & abi3+abi3t.

From your post, I can’t tell whether you think there are other ways of doing things more minimally than what I proposed. If there are, let me know.

It seems that abi3t is the best name for this, in a world where abi3 and this need to coexist for a while. When phase out the t everywhere else and stop supporting the unadorned abi3, it’ll be just another naming wart for abi4 to fix.

2 Likes
7

Probably, but I don’t think it’s a good idea.

It’s a shame we can’t use abi3.1, but yeah, I agree with this.

1 Like
8

Published now: PEP 793 – PyModExport: A new entry point for C extension modules.

2 Likes