Stable ABI/Limited API for free-threaded builds

Core Development C API
42

You’re already on top of most of the details, it’s just the labelling and the wheel/so tags we’re debating.

So my complete proposal is that we define abi4 with all the technical changes needed to support free-threaded builds and future forwards+backwards compatible expansion, implement it in 3.15 in all build configurations, freeze abi3 and only keep it in in build configurations where it works (i.e. not free-threaded), marked as deprecated and discouraged until the release we have to completely drop it.[1] Defining Py_LIMITED_API omits all non-abi4 members from the headers, with essential fallbacks[2] for Py_LIMITED_API<0x04000000. Existing abi3 releases continue to work everywhere they would work already, and new releases can be made using abi3 by package developers who refuse to update their code, and those who will update can fix compiler errors until they fit into abi4 (perhaps involving their own #if Py_LIMITED_API+0<0x04000000 checks, depending on which versions they care to support).

I’ll admit, this only really makes sense if we think free-threaded is inevitable. The SC sure sounds that way, so I think it’s responsible for us to assume it will be and we have to provide a reasonable way for package developers to handle it - hence, people who want to support it have to make code changes anyway (and for far more complicated problems than compiler errors!), and so migrating them to abi4 is a simplifying step. But if those code changes aren’t needed (because we give up on free-threading) then migrating to abi4 is pure pain, and I wouldn’t support it.

In any case, I don’t support compiler flags or ABI flags that indicate “I never rely on the members or shape of PyObject”. Either it’s allowed by the ABI or it isn’t, and if it isn’t allowed then it should just be a new ABI so we can properly change what we need to change to enable it. We can even require use of a new module export with a new ABI, which greatly simplifies that whole exercise[3].

  1. Because we’re actually changing PyObject, not merely making it opaque. ↩︎

  2. Such as ob_refcnt and friends. ↩︎

  3. To me, the challenge there is the migration process, not the technicalities of how it looks/works. No matter how perfect or imperfect, people won’t want to change to it, so we need to offer a better reason than “we assumed you wrote bad code before”. ↩︎

1 Like
43

Sounds like bikeshedding to me! Yay!
If PEP 793 is accepted, I’ll

  • add a private macro (_Py_OPAQUE_PYOBJECT) for easier experimentation, with a promise to remove it well before 3.15 RC, and
  • summarize the options & open a thread specifically for the tags & names.

Does that sound fair?

44

I can’t really argue with a private macro, but I still think it’s an unnecessary burden. If there’s experimentation you want to do, you’ve already got a fork with that change, and if you’re trying to distribute the work to the package maintainers who install our alphas/betas then I think we can be a bit more decisive by then (and instead ask them to use something that won’t be removed, but might be modified based on feedback).

I don’t think we need a new thread, unless everyone’s muted this one because it’s just been us two for the last 50 messages. The core question is do we want a new free-threading-specific Stable ABI, or a new Stable ABI that works for all build configurations (noting that neither will be compatible with the existing ABI). It still sounds like you’re committed to the former and me to the latter.

The only reason I care about abi3t vs abi4 is because of that core question, and if everyone says they want a non-freethreaded ABI and a freethreaded ABI, then I’m not going to argue about the name further. But I’ll be surprised if people want to formalise two ABIs for the rest of time rather than only having a single ABI for 3.15+.[1]

  1. Not counting the then legacy abi3 that stays as long as it can. ↩︎

2 Likes
45

+1 for a single ABI longterm[1], even if it means some transition pain (the plans here already provide ample mitigation for that pain IMO; i.e. roll cut over to new ABI with 3.x, keep the old abi3 around for 2-3 years, then drop it).

image
image256×185 1.36 KB

You’re too hard on yourself. :wink:

  1. assuming freethreading stays, which seems like a foregone conclusion though ↩︎

6 Likes
46

Same from me. I don’t build packages using the C API/ABI myself, but I believe that a single stable ABI (with a well defined versioning scheme) is the best approach for packaging such projects.

2 Likes
47

I want both. One can be API-compatible; the latter can’t[1].

In my proposal, in the long term there’s one single ABI, called abi3.abi3t. (Which can in the long long term be shortened to abi3t, or abi4 if we make a breaking change. Or we can just call it abi4 now, if the spelling is important.)

There might also people who do not care about supporting free-threading, and are OK with the runtime warning that the GIL was enabled (which is the default that happens if you don’t add Py_MOD_GIL_NOT_USED to a module).

Does your proposal support Py_LIMITED_API=0x030f0000? (Or the other way: are those users locked out of other 3.15 additions, like PySys_GetAttr)
Does your proposal support adding new features in 3.16? (Would the cpp flag for cp315-abi4 be Py_LIMITED_API=0x040f0000?)
Or do we need forward-compatible slots in 3.15?

  1. (practically) ↩︎

48

The problem here is these are two different groups of people. Package maintainers choose whether or not to impose the runtime warning on application developers, and we’re the ones in the middle who have to decide whether that’s okay or not.

I mean, I guess we can keep adding stuff to abi3 after we deprecate it, if that’s what we need/want to do. But people can only use those new things by making source code changes, which goes against your argument that people won’t have to make source code changes.

No, because I really want to stop the idea that the stable ABI changes with each release. Hence the forward-compatible slots, so that new interfaces can be added that way (which moves knowledge of the slots into extension compile time, so it can be aware of the latest available ones, but implementation belongs to the runtime, and so it can safely reject slots it doesn’t know about - callers just have to have their own fallbacks for when a new feature is missing, which might be to raise an error, but hopefully will be to just use a slower/alternate codepath).

If/when we one day need abi4_1, then you’d use Py_LIMITED_API=0x04010000 to select it. But I’m hoping that never comes.

So yes, any implementation that supports abi4 would need the forward-compatible slots. They don’t have to be available in abi3 though (but of course are available to anyone not using the limited API).

49

This was also a common opinion when I asked around at EuroPython.

And making the ABI backwards-compatible is not as important as I thought. One can always build extra wheels with an older Python.
Also, an API change seems less problematic than I thought. Since extensions opt in to new versions of limited API, it can be done there.

So, my updated current plan is option 2 above:

Version 3.15 of the limited API makes PyObject opaque

The following API is removed:

  • PyObject_HEAD
  • _PyObject_EXTRA_INIT
  • PyObject_HEAD_INIT
  • PyObject_VAR_HEAD
  • struct _object (i.e. PyObject)
  • struct PyVarObject
  • PyModuleDef_Base (made opaque)
  • PyModuleDef_HEAD_INIT
  • PyModuleDef (made opaque)
  • Py_SIZE (can be added back as a function)
  • Py_SET_TYPE (ditto)
  • Py_SET_SIZE (ditto)

(You can test the effects in current main, by defining _Py_OPAQUE_PYOBJECT before including <Python.h>. Note that this macro will be either removed or renamed for 3.15.0.)

Module creation API is left in, but becomes unusable in practice (since
you can’t create the input for these functions):

  • PyModuleDef_Init
  • PyModule_Create, PyModule_Create2
  • PyModule_FromDefAndSpec, PyModule_FromDefAndSpec2

New export hook

To allow defining modules with opaque PyObject, we add the PyModExport
hook from PEP 793 (now submitted to the SC).

Wheel tags

There are two main options:

  1. Wheels that use the new version of the ABI use the abi3.abi3t ABI tag.

    This adds an ugly new tag.

  2. Tools learn that cp315 and above + ABI tag abi3 is compatible with free-threading.

    This avoids a new tag, but is much less obvious.

For the long term: tools that don’t support 3.14 any more may only use option 2, and ignore abi3t.

Runtime ABI checks

It still makes sense to me that build/install tools are responsible for
not putting incompatible extensions in site-packages – the metadata they can use is much richer.
But, we can provide some checks for when the tools get it wrong (or you’re not
using a tool that follows the latest spect).

  • Add an ABI checking slot is added, and made mandatory if PyModExport is used (that is: for stable ABI 3.15+).
    This will prevent new extensions (stable ABI 3.15+) from being loaded in incompatible interpreters.

  • In free-threaded builds, PyModuleDef_Init should reject extensions compiled for the pre-free-threading ABI.
    (PyModuleDef_Init is also called from PyModule_Create* and PyModule_FromDefAndSpec*`, so this covers single-phase init too.)

    I’m testing a prototype that looks at a certain bit pattern which appears in PyModuleDef of free-threaded builds but not in PyModuleDef_HEAD_INIT of the current abi3.
    This is fragile, and the check might need to be removed as PyObject internals change further. But, hopefully it can stay in for the transition period toward free-threading.

What’s missing is preventing existing free-threading extensions (cp314t) being loaded in gil-ful builds of Python, but, that’s out of scope of this thread.

Sunsetting; ABI support windows

The current stable ABI (3.14 & below) will, necessarily, become less useful when free-threading builds become default (phase III of the rollout), and unusable after the GIL builds are removed entirely.

We might also plan to regularly deprecate older versions of stable ABI after some time.
With the ABI checking slot, we can start emitting deprecation warnings as soon as we agree on a policy; there’s no rush to start the discussions now.

New API

[help welcome]

Add APIs like PyMutex and PyCriticalSection to stable ABI, so extensions with Py_MOD_GIL_NOT_USED are practical.

7 Likes
Limited API & free-threaded build
PEP 803: Stable ABI for Free-Threaded Builds
50

This is my favorite version of the proposal so far.

I think either naming scheme can be made to work, but the second option, designating that cp315-abi3 is compatible with free threading, seems much nicer to me:

  • It only requires a small change to tooling similar to the change we made two years ago.
  • The failure mode for old tooling (like pip) is not so bad: they may refuse to load cp315-abi3 packages in 3.15t builds, but they will still load them in non-free-threaded 3.15 Python builds.
7 Likes
51

Late reply but has anything been done here? Python 3.14.2 is generally available. You can build Python 3.14.2 with –disable-gil. However, if you rely on awscrt it will not build due to this issue. Is there some way to make this work?

52

Python 3.14 free-threaded builds don’t support the limited API. Hopefully 3.15 will support a version of the limited API that is compatible with both builds.

For now, extensions that hard-code a need for the limited API in their builds will need to disable that for the free-threaded build.

You should perhaps try the suggestion here.

1 Like
53

Yes, I’ve seen that. The change for 3.14 there is not yet merged.