PEP 803: Stable ABI for Free-Threaded Builds

[encukou (Petr Viktorin)]

@pf_moore

And in my view, that breakage should come with a version bump.

And would you prefer the bump to come at the beginning of the transition period, or at the end? :‍)

@pf_moore

I don’t know what the proposal is for platform tags for free-threaded Python builds. Has anyone made a formal proposal on this yet? I have no feel for whether free threaded CPython 3.15 should accept cp315 wheels, or conversely whether CPython 3.15 with the GIL should accept cp315t wheels.

It doesn’t help that cp315 can be both a Python tag or an ABI tag.
Would a table like this help?
CPython builds & the wheel tags that can be loaded on them:

	cp314-cp314	cp314-cp314t	cp314-abi3	cp315-cp315	cp315-cp315t	cp315-abi3
CPython 3.14 (GIL-enabled)	yes	no	yes	no	no	no
CPython 3.14 (free-threaded)	no	yes	no	no	no	no
CPython 3.15 (GIL-enabled)	no	no	yes	yes	no	yes
CPython 3.15 (free-threaded)	no	no	no	no	yes	yes
Summary	(3.14, GIL)	(3.14, FT)	(3.14+,GIL)	(3.15, GIL)	(3.15, FT)	(3.15+, any)

Or as a list:

• CPython 3.14 (GIL) accepts ABI tags cp314 and abi3
• CPython 3.14 (FT) accepts ABI tag cp314t
• CPython 3.15 (GIL) accepts ABI tags cp315 and abi3
• CPython 3.15 (FT) accepts ABI tag cp315t, and abi3 if the Python tag is cp315+. Which is where this runs into the issue you mention next, right?

@pf_moore

And it’s important to note that generally, tags are considered separately, so “it depends on whether it’s using abi3” isn’t a helpful answer here.

Alright, that sounds like a good reason for an abi3t ABI tag. (Which would always be used as abi3.abi3t, since abi3t doesn’t expose the parts that would be incompatible with abi3.)

---

I intend to open a discussion in Packaging after the next PEP update, after this thread gets quiet.

[encukou (Petr Viktorin)]

@encukou

Alright, that sounds like a good reason for an abi3t ABI tag.

I guess that would also make things less confusing for users – make it clearer that we will have this:

@Monarch

Ideally, I’d like to see:

• abi3 continues to provide forward compatibility with non-free-threaded Python
• abi4 introduced to cover both free-threaded and GIL-enabled builds, with the same forward compatibility guarantee

except it’d be abi3.abi3t instead of abi4 – even explicitly compatible with both the regular ol’ builds and the t ones.

[pf_moore (Paul Moore)]

@encukou

And would you prefer the bump to come at the beginning of the transition period, or at the end? :‍)

That’s a good question, that I don’t have an opinion about

@encukou

It doesn’t help that cp315 can be both a Python tag or an ABI tag.

And even more so that cp315 as a Python tag is the only possibility, whereas when it’s an ABI tag it can be cp315 or cp315t (as well as d, m and u suffixes).

[pf_moore (Paul Moore)]

The problem is less in matching tags (that’s simple - compare the tags on the wheel to the list of supported tags for the interpreter, and if there’s an exact match in there, you’re OK) than in computing the list of valid tag sets for a given interpreter. That’s an implementation detail in the packaging library (which is in effect the de facto standard here). The reason it’s that way is precisely because CPython provides no specification of this, and if an outcome of this discussion is that the core does start documenting it, then that’ll be a great result.

You might want to take a look at packaging/src/packaging/tags.py at main · pypa/packaging · GitHub, which is the code that computes the tag sets for a given interpreter. That’s the code that would need to change based on the results of this discussion.

To answer Phil’s question, based on what packaging currently says, 3.15 will support cp39-abi3 only in the GIL-based version (see _abi3_applies in the packaging source).

@encukou

Alright, that sounds like a good reason for an abi3t ABI tag. (Which would always be used as abi3.abi3t, since abi3t doesn’t expose the parts that would be incompatible with abi3.)

I think the point I made here was wrong, and I don’t think it constitutes a case for an abi3t tag. I was getting very confused over when you meant Python tags, when you meant ABI tags, and when you meant actual interpreters. Sorry.

When calculating the list of supported tags, the behaviour is described in a comment (see cpython_tags in the source I linked above) as:

The tags consist of:

• cp<python_version>-<abi>-<platform>
• cp<python_version>-abi3-<platform>
• cp<python_version>-none-<platform>
• cp<less than python_version>-abi3-<platform> # Older Python versions down to 3.2.

That’s modified by the _abi3_applies function which currently seems to detemine whether any abi3 tag sets are generated, based on whether it’s a FT or GIL build of Python doing the calculation.

This is all something that could change (in a backward compatible way!) if necessary. But it’s already complicated and not at all how people’s intuition works, so it needs to be changed with care.

[Jelle (Jelle Zijlstra)]

@pf_moore

I don’t think that these days anyone associates “abi3” with “Python 3”

For what it’s worth, I do associate it with Python 3. (But I understand that using “abi4” for free-threading is a reasonable choice.)

[brettcannon]

@pf_moore

The problem is less in matching tags (that’s simple - compare the tags on the wheel to the list of supported tags for the interpreter, and if there’s an exact match in there, you’re OK) than in computing the list of valid tag sets for a given interpreter.

And just to be very crisp about how all of this works (since I wrote most of the code ):

from packaging import tags

tag_set = tags.parse_tag("cp315-abi3.abi3t-win_amd64")
# Compressed tags generate all combinations, so `tag_set` is
# `{Tag('cp315', 'abi3', 'win_amd64'), Tag('cp315', 'abi3t', 'win_amd64')`.
for tag in tags.sys_tags():
    if tag in tag_set:
        return True  # The tag is compatible.
else:
    return False # Won't work.

Change the if to check all the tags for all available wheels for a project release and you get the algorithm used to choose the “best” wheel to install.

[emmatyping (Emma Smith)]

Thank you for drafting this PEP Petr! I am excited to see a path to a stable ABI for free-threaded.

@encukou

And would you prefer the bump to come at the beginning of the transition period, or at the end? :‍)

I know this was originally asked of Paul, but I think it is important to have the (eventually) incompatible, future ABI available as early as possible into the transition to free-threading being the default. The community needs time to test and adopt these changes.

@encukou

For PEP 803, an important premise is that building an extension twice (or rather, a “small” number of times), for different ABIs, is not an issue for extension authors.
If that premise is wrong, let’s do things differently.

Cython and nanobind, two of the more popular binding generators, only relatively recently gained support for targeting the stable ABI. I expect with changes to the limited API and the ABI, these projects and others like them will need significant work to be made compatible with the new API/ABI and also keep compatibility with generating bindings for older versions. So while “building one more wheel” is generally not that hard, I expect it will be a lot harder and take a fair bit of time to get changes into the hands of many extension authors.

@Jelle

@pf_moore

I don’t think that these days anyone associates “abi3” with “Python 3”

For what it’s worth, I do associate it with Python 3. (But I understand that using “abi4” for free-threading is a reasonable choice.)

I agree with Jelle here, I also associate “abi3” with “any Python 3.x”, and I think the Python documentation on the stable ABI pretty much says the current stable ABI applies to any (later) Python 3.x release:

To enable this, Python provides a Stable ABI: a set of symbols that will remain ABI-compatible across Python 3.x versions.

Furthermore, PEP 425 (compatibility tags) states that “abi3” means “the Stable ABI”:

The CPython stable ABI is abi3 as in the shared library suffix.

Taken together, it reads to me that “abi3” means “works across Python 3 versions.” I think it would be a major departure from what is specified and documented to change the compatibility story of what “abi3” means (aka, there is some future Python 3 version where an abi3 wheel won’t work).

@encukou

@philthompson

Things would be easier (but not impossible) if that ABI
supported the existing ABI for a couple of years or so.

It should be possible to make a single ABI compatible with 3.11+ (non-free-threaded) and 3.15+ (all builds).

Personally, I’d be happy to do it – please convince the SC that it’s a good idea! Currently it seems the complexity in testing and supporting this is not worth it.
It could be done as an “add-on” to PEP 803 – basically, extending the compatibility to lower versions.

I absolutely would love if this were feasible! I don’t know enough of the nitty gritty ABI details to know whether or not it is, but I think having a single ABI across existing supported versions would be a huge step to making migration to free-threaded Python more palatable to the community. One of the biggest complaints of 2->3 from my recollection was it was hard to be compatible with both versions and incrementally move to supporting 3.x from a big Python 2.x codebase. That became a lot easier as features were added to Python 3 like the u"" prefix and I think if we make it more or less drop in for users to move to the new ABI (and incrementally adopt the new limited API), people will be happy to do so. 3.10 is EOL when 3.15 is released, so timing wise it also seems like things line up nicely for this plan.

[encukou (Petr Viktorin)]

@pf_moore

• cp<python_version>-<abi>-<platform>
• cp<python_version>-abi3-<platform>
• cp<python_version>-none-<platform>
• cp<less than python_version>-abi3-<platform> # Older Python versions down to 3.2.

So with the current PEP, the last point would change to:

• cp<less than python_version>-abi3-<platform> # Older Python versions down to 3.15 for free-threaded builds or 3.2 for non-free-threaded ones.

With an additional abi3t tag, an extra point would be added instead (for free-threaded builds only):

• cp<less than python_version>-abi3t-<platform> # Older Python versions down to 3.15

@emmatyping

Cython and nanobind, two of the more popular binding generators, only relatively recently gained support for targeting the stable ABI. I expect with changes to the limited API and the ABI, these projects and others like them will need significant work to be made compatible with the new API/ABI and also keep compatibility with generating bindings for older versions.

Yeah. On the other hand, once Cython and nanobind are adapted, the other projects don’t need to do anything except put a new build in the matrix.

@emmatyping

I absolutely would love if [free-threaded ABI supporting older versions] were feasible! I don’t know enough of the nitty gritty ABI details to know whether or not it is

The nitty gritty details for the best implementation I currently know are:

• Py_OPAQUE_OBJECT as a user-settable knob for limited API 3.14 and below
• A shim for PyInit_* that calls PEP 793’s PyModExport and adapts the result into one of exisitng ABIs. (An ugly but self-contained hack.)
• A few shims for things that became non-static functions after the targeted version (probably the most complicated thing^[1])
• Adding a abi3t wheel tag, rather than relying on cp315-abi3

@emmatyping

One of the biggest complaints of 2->3 from my recollection was it was hard to be compatible with both versions and incrementally move to supporting 3.x from a big Python 2.x codebase.

That shouldn’t be a problem here – build & install one artifact for abi3 (say, 3.10+ w/ GIL) and one for the new ABI (for any 3.15+) from the same source, and interpreters compatible with both (3.15+ GIL) can mix and match.

@emmatyping

3.10 is EOL when 3.15 is released, so timing wise it also seems like things line up nicely for this plan.

I picked “3.11“ because it lines up like that :‍) We can extend the compatibility to 3.14, then 3.13, and so on as far as feasible.

---

1. this could be the platform-specific solution as Sam originally proposed (weak linking for ELF, I forget the Windows mechanism); or a capsule in the sys module ↩︎