I’d like to know your thoughts on PEP 803 – a proposal for stable ABI for Free-Threaded Builds.
The main packaging-related points:
Wheels tagged with the new ABI tagabi3t would be compatible with free-threaded builds (subject to Python tags like now, e.g. cp316-abi3t wouldn’t be compatible with 3.15)
C API extensions built with Py_LIMITED_API=0x030f0000 (3.15) and above would be compatible with free-threaded builds, as well as non-free-threaded ones. Wheels containing such extensions should get the compressed tag cp315-abi3.abi3t to reflect this. (Build tools for native extensions generally want to provide a knob to define Py_LIMITED_API.)
Extension filenames should be unchanged: *.abi3.so, *.pyd, etc.
We’ll clarify explicitly that installers[1] are responsible for not putting incompatible extensions on Python’s import paths. (Packaging metadata is rich; CPython doesn’t have it and so it can’t do compatibility checks well. We will do better than in 3.14, but only to detect common mistakes (like outdated or misconfigured tools).
Please check Rejected ideas if you have a “why not?” question :)
Note that there’s a competing proposal in PEP 809, which adds a abi2026 wheel tag, to be generated when compiling with Py_LIMITED_API=0x03ff_2026.
It is possible to start preliminary testing of this in 3.15.0a2 with:
define _Py_OPAQUE_PYOBJECT (internal, testing only, will removed in final releases of 3.15)
don’t use API that’s getting removed (contact me for porting advice or example/test modules)
due to a bug, free-threading builds of 3.15.0a2 don’t enable the GIL for updated modules properly
This stood out to me. I can’t quite decide whether I like this decision, so here are some thoughts/pros/cons. To motivate this, all PEP 803 says is “Changing this while keeping compatibility with GIL-enabled builds would be an unnecessary technical change.”; this comment on the main PEP 803 thread says a bit more but I’m not sure what it says is correct (“the files are still compatible with abi3 so they shouldn’t be just renamed”); it’s just a new ABI with changes at the source level for PEP 793, there’s no such thing as “compatible with abi3” I think, that’s putting things in reverse. Instead, both abi3 and the new ABI are compatible with a given CPython version yes or no.
A potential upside of keeping .abi3 is that it might require a couple fewer changes in tooling, however, most build systems (e.g., meson, cmake) as well as abi.stable_abi_suffix in build-details.json will derive the value from sysconfig.get_config_var('EXT_SUFFIX') & co. anyway, so zero changes may be necessary either way.[1]
A downside is that it becomes harder to tell whether an extension module is valid or not. From current experience (e.g, with relying on cp3xx.so vs. abi3.so during debugging, or the annoyance of Windows currently not using .abi3.pyd) it can be quite annoying to not be able to distinguish what was actually built.
It may also be harder for build backends; if the build system produces .abi3.so extension modules, the build backend may rely on that to do validation, derive its own wheel tags from it, etc. E.g., here is an example for meson-python checking file extensions here.
Thinking about it from the source level and taking into account what backwards compatibility says, many libraries will have to support both old and new stable ABIs because they will support older Python versions for several more years. If I’d want to add support for abi3t to a library, I think I’d need:
Support in the build backend first.
It should either no longer auto-switch from the limited API to the default CPython-specific one when run under a free-threaded interpreter, or gain a new config setting to let the package author tell the backend to do so. The latter is probably needed for backwards compat.
Allow targeting abi3t when building with a default (with-GIL) 3.15 interpreter. This needs to be opt-in, because the build backend cannot know whether the package author has made the changes required by PEP 793.
Build backends may not know what value was used for Py_LIMITED_API (e.g., for setuptools + cmake, it’s usually hidden in CMakeLists.txt; setuptools only has a py_limited_api boolean keyword in Extension that controls the filename, nothing else) - another reason for adding a config setting to let the package author select between abi3 and abi3.abi3t wheel tags.
Binding generators (e.g., Cython, nanobind) with support for using the limited API should be updated for PEP 793.
Cython is already in progress I see (cython#7276) - nice!
Then port all extension modules in the package to support both ABIs.
This will happen one by one. Is there an easy way to detect which ones have been ported? I believe that this is easy by design because -DPy_LIMITED_API=0x030f0000 will error out at compile time for not-yet-ported modules. @encukou the PEP doesn’t say so explicitly I think, can you confirm that this is the case? The “discontinued” in the table in compatibility overview might be saying the opposite - that’s unclear to me.
Should wheels tagged with cp315-abi3 be prevented from uploading to PyPI? And what are installers supposed to do with it, treat it as compatible or incompatible with a 3.15t interpreter? EDIT: sorry, the table with checkmarks and crosses does make it clear; let me think some more based on that.
Installers cannot really know this. The responsibility is for build backends I’d think to produce valid wheels, and installers just have to assume that. If the .abi3.so extension was changed, they could validate that at least - but I’m not sure that that’s a useful request either.
PEP 809 says nothing about filename extensions either way. I assume the intent there is to change it to .abi2026.so@steve.dower?
This depends a bit on the answer to my questions further down, it’s not quite clear to me if it’s still possible to build the old-style abi3 with `-DPy_LIMITED_API=0x030f0000. ↩︎
This part suggests to me that the PEP 803 ABI is not compatible with the previous Stable ABI, contrary to what the PEP suggests:
(Implementation note: A message will be printed before raising the exception, because extensions that attempt to handle an exception using incompatible ABI will likely crash and lose the exception’s message.)
Therefore I agree with @rgommers that keeping the abi3 tag is misleading and error-prone.
To be clear, I’m really not sure either way yet what is better here. It’s nontrivial to reason about, given there are so many tools involved and they may all have only partial information about what the build is targeting.
That “discontinued”, which in the clarification below it says it really means “discouraged” (i.e., still allowed everywhere? if so, till when?) is bothering me, and is a problem for determining the rollout strategy here. There’s a range of strategies possible, I’ll sketch the two most extreme ones:
Strategy 1:
cp315-abi3 is legacy but supported until with-GIL interpreters disappear completely
Build backends should aim to support both, continue defaulting to cp315-abi3 for backwards compatibility and add opt-in flags for package authors to ask for cp315-abi3.abi3t.
Strategy 2:
cp315-abi3 is unsupported; it may continue to work locally for a while with old tools only
PyPI should refuse uploads and new versions of installers should start erroring out when they see this tag.
Build backends should start unconditionally producing abi3.abi3t when cp315 is targeted. They don’t need new opt-in flags.
For either of those strategies:
Binding generators can start producing PEP 793-compatible code for >=3.15, as cython#7333 does. That is fully backwards compatible
Build systems may produce .abi3.so filename extensions, or be required to produce .abi3t ones. If the latter, they will need an interface for the build backend to tell them that abi3t is targeted, because they cannot derive this information from anything else.
Strategy 1 is more work for build backends and results in a worse UX long-term because of the need of opt-in flags. E.g., setuptools only has a bdist-wheel setting for cp3x (example in test case), so would need to start accepting cp3x-abi3t for that setting or add a new use_abi3t = True one.
Strategy 1 is slightly more compatible. It’s unclear if that really matters, because it will not be useful for package authors to produce cp315-abi3 any time soon, they’re still at around cp310-abi3 now and can produce cp314-abi3 wheels even when building with a 3.15 interpreter, so there’s no pressure for package authors to adopt PEP 793 asap. Disclaimer: that “can produce cp314-abi3 wheels on 3.15” is currently not 100% true: scikit-build-core, setuptools, maturin and setuptools-rust can all do it, but meson-python can’t yet - it only has a use-limited-api= true/false setting, so on 3.15 the user gets cp315-abi3. That should be improved soon anyway though, so I wouldn’t worry about that one.
Adding more build config settings and passing them around between tools is a pain, so I’d personally lean towards (a) Strategy 2, and (b) only use .abi3.so extensions. That will result in the least amount of work and the cleanest UX for build backends/systems.
It’s possible to build a cp314-abi3.abi3t extenstion – one compatible with 3.14 (both free-threaded build and default). The technical side of that is not terribly hard; the hard parts are:
making it convenient and safe for general extensions
testing it (as CPython’s test suite doesn’t involve other CPython versions than the one being tested)
So, this best suited to an external project (like pythoncapi-compat), it’s probably out of scope for CPython’s C API, and it’s definitely out of scope for this PEP. But I don’t want to close that door entirely.
See also: previous discussion.
Yes:
with the default (non-free-threaded) build.
The stable ABI is versioned, and the 3.15 stable ABI is not compatible with 3.14.
No change there from Stable ABI in 3.14 on 3.13. The new thing is compatibility with free-threaded builds.
Yes – as you found out, relying on filenames never worked, if you care about Windows at all.
So, the PEP focuses on wheel metadata and runtime checks, and keeps the filename.
Yes, depending on which Python versions they choose to support. That doesn’t change.
What does change is that one wheel module can now target “3.15+ regular and 3.15+ free-threading” at the same time.
Yes.
(As in: it’s a bug if that doesn’t work; sadly it’s not guaranteed by construction. Again, no change there.)
Yes, I simplified my summary for this post too much.
CPython core relies on build & install tools cooperating to avoid utting incompatible extensions on import paths. They’re expected to do that via wheel metadata. (Of course, Conda or DEB or RPM metadata should work just as well – they start from the trivial baseline of building for a specific CPython build, and add to that, as they’ve always done.)
Yes – Stable ABI 3.n+1 is not compatible with Stable ABI 3.n. That’s been the case since Python 3.3.
Also, Stable ABI 3.n is compatible with the default build of 3.n+1. That’s also “always” been the case; but the default build clause is much more relevant since 3.13.
For example, cp39-abi3 isn’t compatible with free-threading. The part you mentioned adds a better check for that.
(BTW, that part is already in main – it’s rather uncontroversial, applies to older versions, and can be removed easily at any time.)
The “discontinued” tag, cp315-abi3, means an extension that’s compatible with free-threading builds, but is marked an incompatible. IOW, the tag is safe to use, works like before, but is now unnecessarily limited.
I’d recommend this strategy:
Build backends: whenever they’d generate cp315-abi3, build backends should generate cp315-abi3.abi3t instead. If/until they don’t, the wheels they build will work as before, but they’ll be tagged as incompatible with free-threading.
Installers should recognize the abi3t tag as compatible with free-threading builds.
Note that abi3.abi3t is a compressed tag set; installers that don’t understand it (yet) should treat it as abi3
The issue here is an existing one: they can’t derive a value for Py_LIMITED_API, or determine it if the user sets it on their own somewhere lower in the stack, e.g. with a #define in the source.
The required interface here is the same as what’s needed for syncing the Py_LIMITED_API value for the compiler with the generated cp3XX-abi3 tag.
Ah, that’s great. If you could add this clarification to the PEP text near those two tables, then I think I’m happy with everything you propose. Adding the recommendation for what build backends should do will also be helpful.
I’m very confused by this statement. Maybe I’m confusing ABI vs API, or misunderstanding “stable”, but I thought that “since Python 3.3” until now, we’ve only had one stable ABI, abi3. So comments about the compatibility of different stable versions makes no sense to me.
Also, “Stable ABI 3.n+1 is not compatible with Stable ABI 3.n” isn’t what I’d call a “stable” ABI - quite the opposite, it seems pretty unstable - assuming that the “3.n” here matches the Python version.
If in fact, the “n” in “Stable ABI 3.n” is unrelated to the “n” in “Python 3.n”, in the sense that there won’t be a stable ABI 3.n for each Python version 3.n, then I think it’s very misleading to number them the same. I’d rather see the next stable ABI version after abi3 be abi4, or maybe abi3.1.
Again, there’s no “always been the case” here - we’ve only had abi3 until now, so it’s hard to understand what this statement is intended to mean (beyond “if you use the stable ABI, you’ll always be compatible with the default build”, which clearly won’t be true at the point where free threading becomes the default build).
Beyond that, it seems to be saying that the “stable build” will only be guaranteed compatible with two Python versions (i.e., for two years). That again doesn’t sound very “stable” to me…
I understand that the underlying problem here is that with the way that free threading is developing, ABI stability is hard (and maybe even impossible) to guarantee, but I’d rather that if that is the case, we’re open about it. For example, saying that abi3 is compatible with all GIL-based builds, as it has always been, but for free threaded builds there won’t be a stable ABI until development has settled down (and become the default), at which point we will introduce a stable free-threaded ABI, which will provide long-term stability guarantees the same way that abi3 has done until now.
Stable ABI is compatible with the version it’s compiled against and all later versions, it is not necessarily compatible with older versions than the one it was compiled against. Otherwise you wouldn’t be able to add APIs to the stable ABI.
Well, you thought wrong :(
The Stable ABI is versioned, and it grew new (incompatible) additions in every single CPython version since it was introduced.
Pardon the simplification, that was meant in a different context. I could instead say:
Stable ABI 3.n is compatible with the default build of 3.n+k, for all k.
That’s been the case since 3.2 (except some bugs along the way).
Yes. If and when free threading becomes the default, we’ll need to either break PEP 384’s promise or release Python 4.
Thank you @encukou for opening up the discussion for this.
I get the gist of the idea which is very nice and the table helps a lot but one question I can’t shake for now after reading it a few times.
Can’t we just call both Stable API and Stable ABI and switch from the “limited” before anything? I’m not sure if I am reading things correctly because different words are coming in and out.
Obviously, there is a relationship between having a stable ABI and to make it work there is a breaking/new API to use it so (which is the limited API I guess). As discussed above there is some terminology that is overloaded with prior insider knowledge so I just want to get it correctly to avoid my own misunderstanding leaking into the discussion.
Another orthogonal question is that if all 3.n+k is going to be compatible with 3.n, this tells me that everything will be compatible with 3.15 and forward via transitive property.
So from that I speculate that from 3.15 onwards the ABI is going to be backwards compatible with 3.15. Is this a fair deduction or too strong? Because it sounds like a very ambitious goal.
I don’t think I’ve ever heard that mentioned anywhere else - and without a way to express “the Python version this was compiled against”, I don’t think the existing wheel tag system has a way to ensure that constraint is respected. Or is the intention that the python tag expresses this in conjunction with the abi tag? So cp311-abi3 in effect means “ABI 3, built on CPython 3.11”[1]?
I’ll note, just for clarity, that I am not involved in any projects that produceabi3 wheels, so I have no real opinion on this from the producer POV. And as a wheel consumer, I trust that everything will simply continue to work seamlessly, so there’s nothing much to say there. My interest is solely as a tool maintainer, where I need to understand what’s going on when I interpret platform tags, and it’s from that point of view that I find the situation confusing[2]
I always assumed it meant “abi3, but there are other non-ABI reasons this requires CPython 3.11 or later” ↩︎
Apparently I was also confused by the current state of affairs, but at least that seems to have been a relatively benign misunderstanding until now… ↩︎
My (new) understanding is that ABI version 3.n+k can add new symbols compared to 3.n, but cannot remove symbols, or change the meaning of existing symbols. This seems very achievable (and indeed is what abi3 does right now, but without an explicit version number beyond “the version of Python this was compiled with”).
It means abi3, but targeting the version of abi3 as of Python 3.11. So you can’t use things that were added in newer versions of the stable ABI in newer Python versions. But you can still use the newer Python version to build an abi3 wheel targeting an older version of the stable ABI. So, to make that concrete, it’s perfectly fine to build a cp311-abi3 wheel on Python 3.14. Or even without using Python at all (as Maturin can do).
I also get confused with the terminology (and apologies if I am about to
demonstrate another level of confusion) but I think the terms should be
“limited API” and “stable ABI”.
The limited API is a sub-set of the full API and like that can have
elements added, deprecated and removed. The stable ABI can only have
elements added (when there are corresponding additions to the limited
API). However when elements are removed from the limited API their
implementations remain in the stable ABI so that extensions built
against an older limited API continue to work.
Yea, you have to combine the Python tag with the ABI tag like you suggested, unless you happen to be compiling against the first CPython that had ABI3… 3.3? I think.
Thank you that clarifies the stability part but then other than the version numbers what is the relationship between the API and the ABI. I would assume that that ABI won’t change without the API. Let’s talk over an example because there are not enough words and abbreviations are getting overloaded. Let’s say we are at Python 3.35 we have functions, (I am completely making stuff up, so I’ll use something that is not Python, say struct, and made up functions)
Now I am a wheeler(patent pending) who wants to use this new addition of command and my package supports the versions 3.33 up to 3.36. Now Stable ABI says nothing happened to PyStruct so we are cool. But API has moved
When can I use safely this new function, or should I wait until 3.36 becomes my lowest supported version?
Thanks for clarifying. IMO, splitting the details of the ABI in use across two separate tags like that is a terrible interface, but that’s off-topic so I’ll say nothing more here on that matter.
Other than to ask why PEP 803 doesn’t take the opportunity to fix this mistake and use abi3.15 (more generally, abi3.n) as a new ABI tag. I haven’t yet looked closely at PEP 809, but is the key difference there that it does move the ABI version into the tag (while simultaneously switching to a calver-based versioning scheme)?
Maybe I missed it, but I don’t see a rejected idea covering “why not include the full ABI version in the wheel abi tag, rather than splitting it between the abi and python tags?”
I think I am not equipped enough to understand your point. I don’t see how you can remove something from the API but it keeps living in the ABI after compilation. I am under the impression that we are talking about C API And C ABI is that right? Like the size or the nature of the PyObject (as this PEP mentions) or some field ordering is modified of a container etc.
If I can be of an annoyance, do you have an example in mind so I can understand better?