And would you prefer the bump to come at the beginning of the transition period, or at the end? :)
It doesn’t help that cp315 can be both a Python tag or an ABI tag.
Would a table like this help?
CPython builds & the wheel tags that can be loaded on them:
cp314-cp314
cp314-cp314t
cp314-abi3
cp315-cp315
cp315-cp315t
cp315-abi3
CPython 3.14 (GIL-enabled)
yes
no
yes
no
no
no
CPython 3.14 (free-threaded)
no
yes
no
no
no
no
CPython 3.15 (GIL-enabled)
no
no
yes
yes
no
yes
CPython 3.15 (free-threaded)
no
no
no
no
yes
yes
Summary
(3.14, GIL)
(3.14, FT)
(3.14+,GIL)
(3.15, GIL)
(3.15, FT)
(3.15+, any)
Or as a list:
CPython 3.14 (GIL) accepts ABI tags cp314 and abi3
CPython 3.14 (FT) accepts ABI tag cp314t
CPython 3.15 (GIL) accepts ABI tags cp315 and abi3
CPython 3.15 (FT) accepts ABI tag cp315t, and abi3if the Python tag is cp315+. Which is where this runs into the issue you mention next, right?
Alright, that sounds like a good reason for an abi3t ABI tag. (Which would always be used as abi3.abi3t, since abi3t doesn’t expose the parts that would be incompatible with abi3.)
I intend to open a discussion in Packaging after the next PEP update, after this thread gets quiet.
That’s a good question, that I don’t have an opinion about
And even more so that cp315 as a Python tag is the only possibility, whereas when it’s an ABI tag it can be cp315 or cp315t (as well as d, m and u suffixes).
The problem is less in matching tags (that’s simple - compare the tags on the wheel to the list of supported tags for the interpreter, and if there’s an exact match in there, you’re OK) than in computing the list of valid tag sets for a given interpreter. That’s an implementation detail in the packaging library (which is in effect the de facto standard here). The reason it’s that way is precisely because CPython provides no specification of this, and if an outcome of this discussion is that the core does start documenting it, then that’ll be a great result.
To answer Phil’s question, based on what packagingcurrently says, 3.15 will support cp39-abi3 only in the GIL-based version (see _abi3_applies in the packaging source).
I think the point I made here was wrong, and I don’t think it constitutes a case for an abi3t tag. I was getting very confused over when you meant Python tags, when you meant ABI tags, and when you meant actual interpreters. Sorry.
When calculating the list of supported tags, the behaviour is described in a comment (see cpython_tags in the source I linked above) as:
The tags consist of:
cp<python_version>-<abi>-<platform>
cp<python_version>-abi3-<platform>
cp<python_version>-none-<platform>
cp<less than python_version>-abi3-<platform> # Older Python versions down to 3.2.
That’s modified by the _abi3_applies function which currently seems to detemine whether anyabi3 tag sets are generated, based on whether it’s a FT or GIL build of Python doing the calculation.
This is all something that could change (in a backward compatible way!) if necessary. But it’s already complicated and not at all how people’s intuition works, so it needs to be changed with care.
And just to be very crisp about how all of this works (since I wrote most of the code ):
from packaging import tags
tag_set = tags.parse_tag("cp315-abi3.abi3t-win_amd64")
# Compressed tags generate all combinations, so `tag_set` is
# `{Tag('cp315', 'abi3', 'win_amd64'), Tag('cp315', 'abi3t', 'win_amd64')`.
for tag in tags.sys_tags():
if tag in tag_set:
return True # The tag is compatible.
else:
return False # Won't work.
Change the if to check all the tags for all available wheels for a project release and you get the algorithm used to choose the “best” wheel to install.
Thank you for drafting this PEP Petr! I am excited to see a path to a stable ABI for free-threaded.
I know this was originally asked of Paul, but I think it is important to have the (eventually) incompatible, future ABI available as early as possible into the transition to free-threading being the default. The community needs time to test and adopt these changes.
Cython and nanobind, two of the more popular binding generators, only relatively recently gained support for targeting the stable ABI. I expect with changes to the limited API and the ABI, these projects and others like them will need significant work to be made compatible with the new API/ABI and also keep compatibility with generating bindings for older versions. So while “building one more wheel” is generally not that hard, I expect it will be a lot harder and take a fair bit of time to get changes into the hands of many extension authors.
I agree with Jelle here, I also associate “abi3” with “any Python 3.x”, and I think the Python documentation on the stable ABI pretty much says the current stable ABI applies to any (later) Python 3.x release:
To enable this, Python provides a Stable ABI: a set of symbols that will remain ABI-compatible across Python 3.x versions.
The CPython stable ABI is abi3 as in the shared library suffix.
Taken together, it reads to me that “abi3” means “works across Python 3 versions.” I think it would be a major departure from what is specified and documented to change the compatibility story of what “abi3” means (aka, there is some future Python 3 version where an abi3 wheel won’t work).
I absolutely would love if this were feasible! I don’t know enough of the nitty gritty ABI details to know whether or not it is, but I think having a single ABI across existing supported versions would be a huge step to making migration to free-threaded Python more palatable to the community. One of the biggest complaints of 2->3 from my recollection was it was hard to be compatible with both versions and incrementally move to supporting 3.x from a big Python 2.x codebase. That became a lot easier as features were added to Python 3 like the u"" prefix and I think if we make it more or less drop in for users to move to the new ABI (and incrementally adopt the new limited API), people will be happy to do so. 3.10 is EOL when 3.15 is released, so timing wise it also seems like things line up nicely for this plan.
So with the current PEP, the last point would change to:
cp<less than python_version>-abi3-<platform> # Older Python versions down to 3.15 for free-threaded builds or 3.2 for non-free-threaded ones.
With an additional abi3t tag, an extra point would be added instead (for free-threaded builds only):
cp<less than python_version>-abi3t-<platform> # Older Python versions down to 3.15
Yeah. On the other hand, once Cython and nanobind are adapted, the other projects don’t need to do anything except put a new build in the matrix.
The nitty gritty details for the best implementation I currently know are:
Py_OPAQUE_OBJECT as a user-settable knob for limited API 3.14 and below
A shim for PyInit_* that calls PEP 793’s PyModExport and adapts the result into one of exisitng ABIs. (An ugly but self-contained hack.)
A few shims for things that became non-static functions after the targeted version (probably the most complicated thing[1])
Adding a abi3t wheel tag, rather than relying on cp315-abi3
That shouldn’t be a problem here – build & install one artifact for abi3 (say, 3.10+ w/ GIL) and one for the new ABI (for any 3.15+) from the same source, and interpreters compatible with both (3.15+ GIL) can mix and match.
I picked “3.11“ because it lines up like that :) We can extend the compatibility to 3.14, then 3.13, and so on as far as feasible.
this could be the platform-specific solution as Sam originally proposed (weak linking for ELF, I forget the Windows mechanism); or a capsule in the sys module ↩︎
Like @vstinner, I would prefer making Py_SIZE() and Py_SET_SIZE() regular (opaque) functions rather than removing them. I would prefer to avoid API breakages when possible.
Then again, why not make the release that has official support built in for free threading Python 4? (I.e., you no longer need to get or build a separate binary.)
Perhaps. I upvoted the comment nevertheless, because despite the scars from the 2->3 transition (with the broadly accepted conclusion “let’s never do that again”), I think it actually is a reasonable idea to use v4 as “just” an indicator for what major version numbers are intended to do: signal some relevant change, like, say, the promised-to-be-stable-during-v3 ABI changing incompatibly.
The issue with cp315-abi3 is that it prevents us from adding an ABI that’s also compatible with 3.14t builds. The abi3.abi3t tag allows that (in some future version, if not now in 3.15). It’s also is more explicit, so I’m now leaning towards that.
As for abi3t vs. abi4, that seems to be a battle of opinions. I’ll go for abi3t as I feel that it better describes the ABI – the same as abi3, except with changes necessary to support the free-threaded build.
I think the simpler the better. For an outsider, abi4 is easy to understand: it’s incompatible with, and supersedes, abi3. abi3t is… what?
(and, of course, cp315-abi3 is even worse as you need expert knowledge to understand that it doesn’t represent the same kind of incremental evolution as cp314-abi3 did)
It’s the same kind of evolution, but a (much) bigger step. There already were some instances of new versions of Limited API removing some functions – just nothing as important as PyObject.
Why couldn’t/shouldn’t/wouldn’t we effectively alias abi3.abi3t as abi4?
I agree that “abi4” should not automatically imply “Python 4”[1], and we should break that association with messaging and documentation whether we adopt “abi4” now or not.
With the rejection of PEP 2026, I think “Python 3” is more of a brand now than a version number. If free-threading isn’t the Big Thing that pushes us to “Python 4” (and I don’t think it is), then what would be? And if there’s nothing, then we don’t want the “Python 3” brand to forever hamstring future ABI evolution.
Although, in the gilectomy days I did think that the “Python 4” moniker would be required for true free-threading. ↩︎
Existing tools still need abi3 there to install for the GIL-enabled builds. So for the short term, the alternatives are abi3 alone [^1], abi3.abi3t, or abi3.abi4. The first option prevents us from adding compatibility with 3.14t later; the latter two are cosmetic.
But there’s also the filename tags, which the current PEP doesn’t touch. Changing those to to abi4 would be an extra technical change (and a naming bikeshed: the files are still compatible with abi3 so they shouldn’t be just renamed, and we don’t have compressed tag sets here so we’d need to invent something like abi3-abi4.)
From a “branding” perspective, it would look weird if a abi3.abi4 (or abi4 only) wheel contained only .abi3.so files. Hopefully, an abi3.abi3t wheel with only .abi3.so files looks compatible enough.
IMO, the only thing that’s hamstrung is the naming :)
Or abi2025/abi25 (I know people at the sprint are aware of my preference here for CalVer, but seems most of this thread are not here, so I’ll put it up again). More likely abi2026 TBH, but that’s the idea.
One ABI for both forks (GIL/no GIL) and legacy abi3 continuing on the GIL branch, and as well as free-threaded support we can also clean up the other things we dislike (non-opaque PyObject, borrowed refs, [lack of] error results, …).
People migrating to free-threaded can either do a quick migration to non-stable ABI or a bigger migration to abi2025, but after migration their binaries (not just sources) work for all variants of 3.15+.
And eventually we can retire abi3 and its borrowed references, whether free-threaded merges into default or not, and we’ll have a better ABI with a versioning scheme that doesn’t scare everyone into thinking we’re going to redefine strings on them…
):