Stable ABI/Limited API for free-threaded builds

Core Development C API
49

This was also a common opinion when I asked around at EuroPython.

And making the ABI backwards-compatible is not as important as I thought. One can always build extra wheels with an older Python.
Also, an API change seems less problematic than I thought. Since extensions opt in to new versions of limited API, it can be done there.

So, my updated current plan is option 2 above:

Version 3.15 of the limited API makes PyObject opaque

The following API is removed:

  • PyObject_HEAD
  • _PyObject_EXTRA_INIT
  • PyObject_HEAD_INIT
  • PyObject_VAR_HEAD
  • struct _object (i.e. PyObject)
  • struct PyVarObject
  • PyModuleDef_Base (made opaque)
  • PyModuleDef_HEAD_INIT
  • PyModuleDef (made opaque)
  • Py_SIZE (can be added back as a function)
  • Py_SET_TYPE (ditto)
  • Py_SET_SIZE (ditto)

(You can test the effects in current main, by defining _Py_OPAQUE_PYOBJECT before including <Python.h>. Note that this macro will be either removed or renamed for 3.15.0.)

Module creation API is left in, but becomes unusable in practice (since
you can’t create the input for these functions):

  • PyModuleDef_Init
  • PyModule_Create, PyModule_Create2
  • PyModule_FromDefAndSpec, PyModule_FromDefAndSpec2

New export hook

To allow defining modules with opaque PyObject, we add the PyModExport
hook from PEP 793 (now submitted to the SC).

Wheel tags

There are two main options:

  1. Wheels that use the new version of the ABI use the abi3.abi3t ABI tag.

    This adds an ugly new tag.

  2. Tools learn that cp315 and above + ABI tag abi3 is compatible with free-threading.

    This avoids a new tag, but is much less obvious.

For the long term: tools that don’t support 3.14 any more may only use option 2, and ignore abi3t.

Runtime ABI checks

It still makes sense to me that build/install tools are responsible for
not putting incompatible extensions in site-packages – the metadata they can use is much richer.
But, we can provide some checks for when the tools get it wrong (or you’re not
using a tool that follows the latest spect).

  • Add an ABI checking slot is added, and made mandatory if PyModExport is used (that is: for stable ABI 3.15+).
    This will prevent new extensions (stable ABI 3.15+) from being loaded in incompatible interpreters.

  • In free-threaded builds, PyModuleDef_Init should reject extensions compiled for the pre-free-threading ABI.
    (PyModuleDef_Init is also called from PyModule_Create* and PyModule_FromDefAndSpec*`, so this covers single-phase init too.)

    I’m testing a prototype that looks at a certain bit pattern which appears in PyModuleDef of free-threaded builds but not in PyModuleDef_HEAD_INIT of the current abi3.
    This is fragile, and the check might need to be removed as PyObject internals change further. But, hopefully it can stay in for the transition period toward free-threading.

What’s missing is preventing existing free-threading extensions (cp314t) being loaded in gil-ful builds of Python, but, that’s out of scope of this thread.

Sunsetting; ABI support windows

The current stable ABI (3.14 & below) will, necessarily, become less useful when free-threading builds become default (phase III of the rollout), and unusable after the GIL builds are removed entirely.

We might also plan to regularly deprecate older versions of stable ABI after some time.
With the ABI checking slot, we can start emitting deprecation warnings as soon as we agree on a policy; there’s no rush to start the discussions now.

New API

[help welcome]

Add APIs like PyMutex and PyCriticalSection to stable ABI, so extensions with Py_MOD_GIL_NOT_USED are practical.

7 Likes