Stable ABI/Limited API for free-threaded builds

Here’s an update on my current thinking & progress, formatted as rough notes.
If you’d like more details on any of the notes, ask “why not do Y instead”, or if you’d like to spend some time/effort on some of these, ask away! :‍)

:triangular_ruler::crystal_ball: My current plan

i.e. what should be in 3.15:

ABI & wheel tags

  • abi3 wheel tag – the existing stable ABI, non-free-threaded only
  • abi3t – new stable ABI for free-threaded builds. As a first step, this would be 3.15+ only.
  • abi3.abi3t – compatible with either (a standard use of compressed tag sets)
  • Later (orthogonal to free-threading):
    • abi3.15 – like abi3, but versioned so it can go EOL
    • (No abi4 planned, to match the “rolling deprecations, no Python 4.0” versioning of CPython itself.)

API & configuration macros

  • existing Py_LIMITED_API=0x03yy0000 – use limited API; compiles to abi3 or abi3t depending on Py_GIL_DISABLED
  • new Py_OPAQUE_PYOBJECT – make PyObject opaque & remove a few additional APIs. Requires Py_LIMITED_API; compiles to abi3.abi3t instead.

API removal

If Py_OPAQUE_PYOBJECT is defined:

  • PyObject is made opaque
  • PyModuleDef is removed. See PEP 793 for replacement (currently waiting for C API WG voting)
  • Py_SET_TYPE, Py_SIZE, Py_SET_SIZE are also removed
    • (limited API is not stable; we can “just” remove stuff from 3.15)
    • Can be added back (in 3.15 or later). Can even made available for older stable ABI – see Sam’s notes (weak linkage) and mine (capsule in sys)

:puzzle_piece: The pieces that need to fit together

PEP 793 - PyModExport

This is the current blocker. If not accepted, the strategy for everything else would need to change.
(Unfortunately, C API WG voting is slow…)

Py_OPAQUE_PYOBJECT

Should be a straightforward addition if PEP 793 is accepted.
A bit hacky, but practical – see this thread.

Testing infrastructure

We currently don’t actually test ABI compatibility across CPython versions. I need to spend a few weeks setting something up.

Build tools

[blocked on PEP 793 & Py_OPAQUE_PYOBJECT; help wanted]

Setuptools, Meson, etc. etc. will need to grow flags that define Py_OPAQUE_PYOBJECT&c. and generate the right wheel tags.
IMO; the time to kick off that discussion will be after PEP 793 is in.

Also:

  • packaging.cpython_tags will become complicated, and might need help from CPython’s side. (AFAIK, nearly all build tools vendor/use/mimic packaging.)
  • For CPython we’ll need .so/.pyd filename tags; IMO this should be designed together with the packaging parts.
    (Early hacks can use bare .so.)

Version-checking slot

[somewhat orthogonal]

See idea thread: To no longer rely (fully) on build/install tools’s “safety” mechanisms, add a module slot that “bake in” configuration (build Python version, Py_LIMITED_API, Py_OPAQUE_PYOBJECT, Py_BUILD_CORE), and is checked at runtime.

New API

[help welcome]

PyMutex and PyCriticalSection are not part of the limited API; they probably need to be added to allow useful stable-ABI extensions with Py_MOD_GIL_NOT_USED.

The best way forward seems to be that the stable ABI reserves a bit more space than necessary for these, to allow future expansion.

ABI support windows

[orthogonal; help welcome]

Plan to deprecate and remove abi3 (in a decade or so); define abi3.15, abi3.16 etc. with limited lifetimes.

Compatibility hack for 3.14 & below

[stretch goal]

With PEP 793, it should be possible to take advantage of ABI stability to extend abi3.abi3t to Python 3.14 & below: a header-only library would define a PyInit that calls PyModExport and fills and returns either “Py3_14_PyModuleDef” or “Py3_14t_PyModuleDef” – both of which it would inline from the 3.14 headers, complete with the PyObject part.

3 Likes