Stable ABI/Limited API for free-threaded builds

Core Development C API
1

Hello,
I’ve been looking into stable ABI for free-threading builds for the past few months. I think I have plans for all the pieces, but I don’t think I can assemble them for 3.14.
I now need to take a break for a month, so I’ll share where I am. Sorry for the lack of background info. If you would like to work on any of this on in April, don’t ask for my permission.

The general plan is to have 3 variants of the stable ABI abi3 (non-free-threaded, same as existing), “abi3t” (free-threaded, same API as abi3), and “abi4” (supports both, but requires API changes).

The API change needed for an ABI that supports both builds (and for fixing the main issue in stable ABI itself) is making the PyObject struct opaque. IMO, on the CPython side, we can hide the structs if the user defined Py_OPAQUE_PYOBJECT.
Before that’s practical, we need a way to define modules without a static PyModuleDef; I want to add a new module export hook that returns a slots array – see my proposal and early proof-of-concept branch. (And ideally, introduce new slots in the same release, so we don’t need support 2 versions of slots in the new export hook.)

To support stable ABI of 3.13 and below, we’ll need to finish replacing accessor macros by exported functions (these are left: Py_SET_TYPE, Py_SIZE, Py_SET_SIZE). Extensions need to call the functions where available but fall back to direct member access. Sam proposed to use weak symbols for this; I think it would be easier – and more cross-platform friendly – to put function pointers in a capsule called e.g. sys._abi_compat, and add a static function to transparently import that on first use. See my rough notes.

We’ll need to define support windows, so we can gradually phase out past versions of the stable ABI (so we don’t wait for a “Python 4” break point). That might be something like 10 years (for the ABI surface – the behaviour behind the interface is subject to PEP 387).

I’d like to add a module slot that indicates the version of Python used to build the extension, the Py_LIMITED_API value used (if any), and some flags (Py_OPAQUE_PYOBJECT as above; Py_BUILD_CORE). This would be checked for compatibility, so we no longer rely only on wheel/.soname tags, and so we can add deprecation warnings for upcoming incompatibilities.

It will be necessary to define .so tags and wheel tags.

And we’ll need tests that this all works.

12 Likes
2

I don’t think this is worthwhile.[1] It’s not inherently bad or wrong, but it’s effort that isn’t worth our time compared to the better options.

If the change is just part of a new ABI, we can actually change runtime interactions. For example, abi4 can make them opaque and be more compatible at runtime as a result (between free-threaded and not).

If the change isn’t part of an ABI, this is just a diagnostic tool for a developer to run their build with and see what breaks. Which is fine, but doesn’t actually provide any benefit over running your build with abi4 and seeing what breaks.

It may be that adding the preprocessor option that breaks your own build but doesn’t affect the build output is easier to add (so we can do it first in less time), but that’s only true if we then go on to change the ABI in the same way as the variable worked. Otherwise, it’s just as (subtly) incompatible, and devs still have to do a test build with the actual option. Plus it creates more code that we really shouldn’t have to maintain.

Everything else proposed sounds like a good way forward, so thanks for doing the planning/thinking work to get to this stage. Enjoy your time off!

  1. I feel like I’m always pushing back on new compile-time macro options… ↩︎

2 Likes
4

How do you opt in to the “new” unified ABI?
I’m less and less convinced to g with the abi4 theme, but if we do that, it would be reasonable to set Py_LIMITED_API to 0x04.......
If we go with what I wrote here, you’d #define Py_OPAQUE_PYOBJECT (an API-focused name). Then instead of abi4 the result could use the wheel tag abi3.abi3t.

5

This is a non-response, but I agree those are the options. I’m not a huge fan of abi3t (because I want the t to be temporary, and this would make it permanent and we’d eventually drop abi3 (if nogil is merged)), but it is cheaper and easier than doing abi4, and more consistent than a specific flag.

The other aspect is our test matrix, which ought to include all combinations of build flags we offer. So keeping it limited to an entire “profile” (i.e. abi3 etc.) rather than a set of independent options means we can actually keep up with our own offering.

FWIW, I see abi4 as the only chance we’ve really got of avoiding/significantly delaying a Python 4, but that does mean we should design it properly (with many of the “api-revolution” ideas) so that we have a good buffer to allow runtime changes underneath the API. Calling it abi3t and doing the same thing is just lying to everyone, and calling it abi3t and doing the bare minimum is only delaying the pain by a couple of years.

However, there seems to be too much in flux right now to design a forward-compatible abi4 anyway. So I guess we’ll be forced into doing the bare minimum, in which case I’d strongly prefer it to actually be the minimum, and not something that adds (more) significant burdens on us.

4 Likes
6

That means new slots and support windows are out for now. I’ll write up a separate motivation for them later :‍)

I’m still leaning toward a new version-checking slot, especially if it would mean that extension “.so” filenames can keep using just abi3 for abi3t & abi3+abi3t.

From your post, I can’t tell whether you think there are other ways of doing things more minimally than what I proposed. If there are, let me know.

It seems that abi3t is the best name for this, in a world where abi3 and this need to coexist for a while. When phase out the t everywhere else and stop supporting the unadorned abi3, it’ll be just another naming wart for abi4 to fix.

2 Likes
7

Probably, but I don’t think it’s a good idea.

It’s a shame we can’t use abi3.1, but yeah, I agree with this.

1 Like
8

Published now: PEP 793 – PyModExport: A new entry point for C extension modules.

2 Likes
9

Here’s an update on my current thinking & progress, formatted as rough notes.
If you’d like more details on any of the notes, ask “why not do Y instead”, or if you’d like to spend some time/effort on some of these, ask away! :‍)

:triangular_ruler::crystal_ball: My current plan

i.e. what should be in 3.15:

ABI & wheel tags

  • abi3 wheel tag – the existing stable ABI, non-free-threaded only
  • abi3t – new stable ABI for free-threaded builds. As a first step, this would be 3.15+ only.
  • abi3.abi3t – compatible with either (a standard use of compressed tag sets)
  • Later (orthogonal to free-threading):
    • abi3.15 – like abi3, but versioned so it can go EOL
    • (No abi4 planned, to match the “rolling deprecations, no Python 4.0” versioning of CPython itself.)

API & configuration macros

  • existing Py_LIMITED_API=0x03yy0000 – use limited API; compiles to abi3 or abi3t depending on Py_GIL_DISABLED
  • new Py_OPAQUE_PYOBJECT – make PyObject opaque & remove a few additional APIs. Requires Py_LIMITED_API; compiles to abi3.abi3t instead.

API removal

If Py_OPAQUE_PYOBJECT is defined:

  • PyObject is made opaque
  • PyModuleDef is removed. See PEP 793 for replacement (currently waiting for C API WG voting)
  • Py_SET_TYPE, Py_SIZE, Py_SET_SIZE are also removed
    • (limited API is not stable; we can “just” remove stuff from 3.15)
    • Can be added back (in 3.15 or later). Can even made available for older stable ABI – see Sam’s notes (weak linkage) and mine (capsule in sys)

:puzzle_piece: The pieces that need to fit together

PEP 793 - PyModExport

This is the current blocker. If not accepted, the strategy for everything else would need to change.
(Unfortunately, C API WG voting is slow…)

Py_OPAQUE_PYOBJECT

Should be a straightforward addition if PEP 793 is accepted.
A bit hacky, but practical – see this thread.

Testing infrastructure

We currently don’t actually test ABI compatibility across CPython versions. I need to spend a few weeks setting something up.

Build tools

[blocked on PEP 793 & Py_OPAQUE_PYOBJECT; help wanted]

Setuptools, Meson, etc. etc. will need to grow flags that define Py_OPAQUE_PYOBJECT&c. and generate the right wheel tags.
IMO; the time to kick off that discussion will be after PEP 793 is in.

Also:

  • packaging.cpython_tags will become complicated, and might need help from CPython’s side. (AFAIK, nearly all build tools vendor/use/mimic packaging.)
  • For CPython we’ll need .so/.pyd filename tags; IMO this should be designed together with the packaging parts.
    (Early hacks can use bare .so.)

Version-checking slot

[somewhat orthogonal]

See idea thread: To no longer rely (fully) on build/install tools’s “safety” mechanisms, add a module slot that “bake in” configuration (build Python version, Py_LIMITED_API, Py_OPAQUE_PYOBJECT, Py_BUILD_CORE), and is checked at runtime.

New API

[help welcome]

PyMutex and PyCriticalSection are not part of the limited API; they probably need to be added to allow useful stable-ABI extensions with Py_MOD_GIL_NOT_USED.

The best way forward seems to be that the stable ABI reserves a bit more space than necessary for these, to allow future expansion.

ABI support windows

[orthogonal; help welcome]

Plan to deprecate and remove abi3 (in a decade or so); define abi3.15, abi3.16 etc. with limited lifetimes.

Compatibility hack for 3.14 & below

[stretch goal]

With PEP 793, it should be possible to take advantage of ABI stability to extend abi3.abi3t to Python 3.14 & below: a header-only library would define a PyInit that calls PyModExport and fills and returns either “Py3_14_PyModuleDef” or “Py3_14t_PyModuleDef” – both of which it would inline from the 3.14 headers, complete with the PyObject part.

3 Likes
10

Is the intention that abi3.15 and abi3 are equivalent (i.e. wherever one is used, the other could be)? To put it another way, will an abi3 wheel satisfy an abi3.15 constraint, and/or vice versa?

Is the intention to have an abi3.x for each Python version 3.x, or is the 15 entirely coincidental, and we’ll have abi bumps only when needed, rather than having an abi per version?

2 Likes
11

The intention is currently to have an abi3.x for each Python version.
It could also be every N versions. Or the version could be only in “runtime” metadata, rather than the wheel tags & filenames.

The intention is that they’re equivalent, but abi3 is “stuck” at being equivalent to abi3.15, and EOL’d together with abi3.15.
(Two names for the same thing allow updating the “naming scheme” independently from the ABI itself.)

1 Like
12

So abi3.15 (and 3.16, 3.17, etc) won’t be a stable ABI, then? Will we retire the idea of a stable ABI, or will there be some other approach for users who want to build extensions that support multiple Python versions? (Or more likely, am I missing something important here?)

13

We have one? It’s cp3x.

IMHO, a cross-runtime ABI shouldn’t use CPython’s version numbers. We should just be bumping it to 4, 5, etc., and if that’s too on the nose, it’s probably a sign that the stable ABI isn’t stable enough.

3 Likes
14

Yes, but…
There’s some value in the version as “stable ABI 3.17 as compiled by CPython 3.19” – imagine that 3.19 fixed an issue that causes an incompatibility with 3.23, in a way that’s still compatible with 3.17.
Yes, needing this is probably a sign that the stable ABI isn’t stable enough – and with the other improvements, it hopefully won’t be needed. But it’d be good to have some insurance against lack of foresight.

(And, as we bike-shed the next-to-last, somewhat orthogonal :puzzle_piece: piece: do the others look like good next steps?)

15

When “Stable ABI” was introduced, it was supposed to be stable until Python 4.0 – that is, for a decade or so.

abi3.15 could also be stable for a decade (or more), but not forever (which is the current meaning of “until Python 4”.)

2 Likes
16

So we won’t be introducing a new version every Python release? In which case, Steve’s question applies - why make it look like a Python version? We could just as easily use abi4 instead of abi3.15, and work up from there.

3 Likes
17

Py_LIMITED_API currently expects a Python version…

What is the role of Requires-Python in all this? My practice, if the
minimum version I want to support is 3.x, is that I set Py_LIMITED_API
and Requires-Python accordingly. I would then want to set the tag to
abi3.x and the tag would just be a way of determining the version of
Python needed without having to read METADATA.

18

It affects informs packaging tools when they’re about to install a package that no longer supports an older version of Python (e.g. example==2.0 doesn’t support 3.9, hence it “requires Python 3.10”), so that they can find an earlier release of the package. After install, it plays no role at all.

1 Like
19

We can, with each Python release supporting a range of ABI versions.

1 Like
20

Requires-Python is packaging metadata, for e.g. PyPI or pip; Py_LIMITED_API is a preprocessor macro for the C compiler. A build backend should generally keep them in sync, when building a stable ABI extension.

21

If it’s orthogonal to free-threading, could we end up with compressed tag sets such as abi3.15.abi3.15t? (is that synctatically valid?)