PEP 803: Stable ABI for Free-Threaded Builds (packaging thread)

[encukou (Petr Viktorin)]

One thing that’s noted in a Rejected Idea but has influenced the design:

It’s possible to build a cp314-abi3.abi3t extenstion – one compatible with 3.14 (both free-threaded build and default). The technical side of that is not terribly hard; the hard parts are:

• making it convenient and safe for general extensions
• testing it (as CPython’s test suite doesn’t involve other CPython versions than the one being tested)

So, this best suited to an external project (like pythoncapi-compat), it’s probably out of scope for CPython’s C API, and it’s definitely out of scope for this PEP. But I don’t want to close that door entirely.
See also: previous discussion.

@rgommers

Instead, both abi3 and the new ABI are compatible with a given CPython version yes or no.

Yes:

• with the default (non-free-threaded) build.
• The stable ABI is versioned, and the 3.15 stable ABI is not compatible with 3.14.

No change there from Stable ABI in 3.14 on 3.13. The new thing is compatibility with free-threaded builds.

@rgommers

A downside is that it becomes harder to tell whether an extension module is valid or not. From current experience (e.g, with relying on cp3xx.so vs. abi3.so during debugging, or the annoyance of Windows currently not using .abi3.pyd) it can be quite annoying to not be able to distinguish what was actually built.

Yes – as you found out, relying on filenames never worked, if you care about Windows at all.
So, the PEP focuses on wheel metadata and runtime checks, and keeps the filename.

@rgommers

many libraries will have to support both old and new stable ABIs because they will support older Python versions for several more years.

Yes, depending on which Python versions they choose to support. That doesn’t change.
What does change is that one wheel module can now target “3.15+ regular and 3.15+ free-threading” at the same time.

@rgommers

I believe that [detecting which modules have been ported] is easy by design because -DPy_LIMITED_API=0x030f0000 will error out at compile time for not-yet-ported modules. @encukou the PEP doesn’t say so explicitly I think, can you confirm that this is the case?

Yes.
(As in: it’s a bug if that doesn’t work; sadly it’s not guaranteed by construction. Again, no change there.)

@rgommers

Installers cannot really know [whether an extension is compatible]. The responsibility is for build backends I’d think to produce valid wheels, and installers just have to assume that. If the .abi3.so extension was changed, they could validate that at least - but I’m not sure that that’s a useful request either.

Yes, I simplified my summary for this post too much.
CPython core relies on build & install tools cooperating to avoid utting incompatible extensions on import paths. They’re expected to do that via wheel metadata. (Of course, Conda or DEB or RPM metadata should work just as well – they start from the trivial baseline of building for a specific CPython build, and add to that, as they’ve always done.)

@pitrou

This part suggests to me that the PEP 803 ABI is not compatible with the previous Stable ABI, contrary to what the PEP suggests:

Yes – Stable ABI 3.n+1 is not compatible with Stable ABI 3.n. That’s been the case since Python 3.3.

Also, Stable ABI 3.n is compatible with the default build of 3.n+1. That’s also “always” been the case; but the default build clause is much more relevant since 3.13.
For example, cp39-abi3 isn’t compatible with free-threading. The part you mentioned adds a better check for that.
(BTW, that part is already in main – it’s rather uncontroversial, applies to older versions, and can be removed easily at any time.)

@rgommers

That “discontinued”, which in the clarification below it says it really means “discouraged” (i.e., still allowed everywhere? if so, till when?) is bothering me

The “discontinued” tag, cp315-abi3, means an extension that’s compatible with free-threading builds, but is marked an incompatible. IOW, the tag is safe to use, works like before, but is now unnecessarily limited.

I’d recommend this strategy:

• Build backends: whenever they’d generate cp315-abi3, build backends should generate cp315-abi3.abi3t instead. If/until they don’t, the wheels they build will work as before, but they’ll be tagged as incompatible with free-threading.
• Installers should recognize the abi3t tag as compatible with free-threading builds.
  Note that abi3.abi3t is a compressed tag set; installers that don’t understand it (yet) should treat it as abi3

@rgommers

Build systems may produce .abi3.so filename extensions, or be required to produce .abi3t ones. If the latter, they will need an interface for the build backend to tell them that abi3t is targeted, because they cannot derive this information from anything else.

The issue here is an existing one: they can’t derive a value for Py_LIMITED_API, or determine it if the user sets it on their own somewhere lower in the stack, e.g. with a #define in the source.
The required interface here is the same as what’s needed for syncing the Py_LIMITED_API value for the compiler with the generated cp3XX-abi3 tag.