PEP 803: Stable ABI for Free-Threaded Builds

[steve.dower]

@encukou

@steve.dower

Are extension developers risking deadlocks by using the actual macros in a build that still has the GIL?

Even in the free-threading build, a FT-incompatible library will re-enable the GIL. So, any deadlock risk would be for both builds.

So this means we can assume that a FT-migrated extension is always going to work correctly even with the GIL enabled? Since even on FT it may get the GIL, and so it will be thoroughly tested under both?^[1]

---

1. I mean, obviously they won’t be. But from the POV of what we officially endorse we’re allowed to assume extensions “behave correctly”. ↩︎

[ngoldbaum (Nathan Goldbaum)]

@steve.dower

So this means we can assume that a FT-migrated extension is always going to work correctly even with the GIL enabled? Since even on FT it may get the GIL, and so it will be thoroughly tested under both?

As your footnote implies: generally we set PYTHON_GIL=0 in CI for extensions that do not yet support running without the GIL and pytest-run-parallel has a check for the GIL being enabled at runtime during tests. I don’t think there’s been much systematic testing of the free-threaded build with the GIL enabled.

That said, PyMutex has the property:

While blocked, the thread will temporarily detach the thread state if one exists.

This means the GIL is released while a thread is blocked on acquiring a PyMutex, so deadlocks with the GIL are impossible.

That means native locks are really the only source of deadlocks. But then in order to use a native lock correctly and avoid deadlocks with global synchronization events in the interpreter (anything needing a stop-the-world pause, like a GC pass), users need to detach from the runtime before acquiring the lock. Cython and PyO3 both have wrappers around native lock types to ease this dance. Even if the GIL is disabled, it’s possible to hit deadlocks in exactly the same places as one might deadlock with the GIL.

Our team has been on the front lines of getting extensions working on the free-threaded build and I’m not aware of any reports of deadlocks when the GIL is enabled at runtime.

But also I think that most people will, in practice, simply force-disable the GIL in situations where it’s re-enabled at runtime due to a recalcitrant dependency that doesn’t yet support the free-threaded build. It doesn’t make much sense to run multithreaded code on the free-threaded build with the GIL enabled: it will likely scale worse than on the GIL-enabled build.

[da-woods]

@ngoldbaum

I don’t think there’s been much systematic testing of the free-threaded build with the GIL enabled.

I accidentally did some on Python 3.14t (because it turned out the rule to set PYTHON_GIL=0 was too specific). No bugs to report. Now that I’ve spotted it though I don’t plan to do any more.

[nas (Neil Schemenauer)]

@encukou

However, if the new knob is added, then it wouldn’t be valid to use Py_LIMITED_API+Py_GIL_DISABLED without the new knob.

I think that’s an argument to have a separate flag. You can give a more descriptive error in this case. In general I prefer to have meaningful boolean flags rather than either overloading other flags to mean multiple things (even though logically you might never set one and not the other), or having combinations of flags replace a single well-named flag that does the same thing.

[encukou (Petr Viktorin)]

@rgommers

However, reusing Py_GIL_DISABLED to enable the new abi3t which will also support the GIL-enabled build and produce wheels tagged abi3.abi3t, will be cumbersome. If a package author makes source-level changes to support abi3t, then they can no longer build under a GIL-enabled 3.15 interpreter without disabling the Limited API completely. Which will require annoying fiddling with default build flags.

I think this is fine.
For local development: Building with a given Python with Py_LIMITED_API will create a Stable ABI extension that works with that Python.
When building for others, we can expect that you can install a free-threaded Python build.

Note that any source-level changes should be compatible with GIL-enabled builds.

@rgommers

Related, testing with abi3t on a GIL-enabled interpreter, say in CI, would require building against a free-threaded interpreter, and then in a separate job install it for a GIL-enabled interpreter - not all that simple.

But that is exactly what you should do if you’re distributing a single wheel compatible with several interpreters.
A separate knob would make it easier to do the wrong thing…

@rgommers

Note: in the above I am assuming that it’s a bad idea to define Py_GIL_DISABLED when building under a GIL-enabled interpreter, because that define controls a lot of other things, both inside and outside CPython. I hope you didn’t have that in mind?

Let’s allow this. On a GIL-enabled 3.15+, you’ll be able to define Py_GIL_DISABLED together with Py_LIMITED_API.
It already works on Windows, and if abi3.abi3t is to work, the flag can’t affect anything the Stable ABI.

Note that the plan is to remove the GIL-enabled eventually. If/when that happens, an extra knob would become redundant.

[colesbury (Sam Gross)]

I agree with Neil and Ralf - I think you should have a separate flag to control the ABI target. It seems messy and confusing to me that Py_GIL_DISABLED is something provided by the Python.h in some contexts and something the user defines to configure Python.h in other contexts. We don’t attempt those sorts of shenanigans with the limited API version vs. the Python version (users set Py_LIMITED_API, they don’t overwrite Py_VERSION_HEX).

[encukou (Petr Viktorin)]

Alright. How do people feel about this then:

• Py_TARGET_ABI3=<version>: Compile for abi3 of the given version.
• Py_TARGET_ABI3T=<version>: Compile for abi3t of the given version.

For both:

• if Py_LIMITED_API is set, it’ll be checked (needs to be the same or lower)
• if Py_LIMITED_API is unset, it’ll be set to the given version

And Py_TARGET_ABI3T will also set _Py_OPAQUE_PYOBJECT.

Users can set Py_TARGET_ABI3T without Py_TARGET_ABI3, which currently won’t do anything in the CPython headers, but if we hit a case where we can’t provide unified ABI for GIL&FT builds, this will be the way to select that. Correspondingly, build tools may treat abi3 and abi3t as separate ABIs, but combinable into abi3.abi3t.

Conceptually, Py_LIMITED_API becomes more of an implementation detail of how CPython exposes (or not) the API that’s compatible with a given target ABI. Same as _Py_OPAQUE_PYOBJECT, but with a public name for backcompat.

[steve.dower]

I lean towards your original proposal - Py_LIMITED_API set to opt into the limited API/ABI, and Py_GIL_DISABLED to opt into the free-threaded ABI.

I get Sam’s issue with Py_GIL_DISABLED being embedded into pyconfig.h (and only overridable on Windows, I believe?), but that choice isn’t going anywhere,^[1] and unless the new variable is somehow going to override (or early-error on conflict with) Py_GIL_DISABLED, I don’t see how adding more knobs makes things any easier.

From a quick look at the headers, it seems like building with Py_LIMITED_API, _Py_OPAQUE_PYOBJECT and not Py_GIL_DISABLED but labelling it abi3t should work? But is that a path we want to support? I’d rather not - I’d prefer to just say you need to have Py_GIL_DISABLED set if you plan to call it abi3t, and then anyone working on the header files has one fewer condition to worry about getting right.^[2]

But at most, if we do have an extra knob, I’d prefer Py_LIMITED_API_T and make it override (or break on conflict with) Py_GIL_DISABLED and Py_LIMITED_API.^[3] So we basically have a flag that checks all the conditions we want, but doesn’t actually appear again in the headers.

---

1. Though there are discussions about overlapping installations that haven’t gotten into CPython header files yet, but probably should. ↩︎

2. Specifically, “does this inline function work for free-threaded even though I’m putting it in a GIL-only preprocessor block” ↩︎

3. And _Py_OPAQUE_PYOBJECT, though I’d also happily see that be rolled into Py_LIMITED_API && Py_GIL_DISABLED, and possibly || Py_OMIT_LEGACY_API ↩︎

[encukou (Petr Viktorin)]

@steve.dower

From a quick look at the headers, it seems like building with Py_LIMITED_API, _Py_OPAQUE_PYOBJECT and not Py_GIL_DISABLED but labelling it abi3t should work? But is that a path we want to support? I’d rather not

I agree: yes that should work, and no I don’t think we shouldn’t support it.

@steve.dower

So we basically have a flag that checks all the conditions we want, but doesn’t actually appear again in the headers.

Right, that would be a good thing
My Py_TARGET_ABI3 & Py_TARGET_ABI3T proposal is a generalization that will do this for Py_LIMITED_API as well once GIL builds are gone.

@steve.dower

But at most, if we do have an extra knob, I’d prefer Py_LIMITED_API_T and make it override (or break on conflict with) Py_GIL_DISABLED and Py_LIMITED_API.

OK, one more iteration then:

• Py_LIMITED_API=<version>: Compile for abi3 of the given version.
• Py_TARGET_ABI3T=<version>: Compile for abi3t of the given version.

If Py_LIMITED_API=v and Py_GIL_DISABLED is set, it automatically sets Py_TARGET_ABI3T=v.
If Py_TARGET_ABI3T=v is set, it automatically sets Py_LIMITED_API=v (unless it’s already set to a higher version). And it defines _Py_OPAQUE_PYOBJECT.

This is:

• same as my earlier proposal, except Py_TARGET_ABI3 is merged with (or kept as) Py_LIMITED_API.
• same as your proposal (modulo naming), except the _T macro is versioned. This means that if/when GIL builds go away, Py_TARGET_ABI3T will be the only knob to set^[1].

Naming-wise: the packaging thread showed that Py_LIMITED_API isn’t a great name for the “target ABI” knob. With my proposal, in the long-term GIL-less possible future, it’ll only remain for backwards compatibility.

---

1. except maybe PEP 743’s Py_OMIT_LEGACY_API ↩︎

[encukou (Petr Viktorin)]

I’ve updated the PEP and started a new discussion thread: PEP 803, round two – “abi3t”: Stable ABI for Free-Threaded Builds