PEP 803: Stable ABI for Free-Threaded Builds (packaging thread)

[encukou (Petr Viktorin)]

@ngoldbaum

The NPY_TARGET_VERSIONmacro works just like Py_LIMITED_API in that it allows consumers to target multiple NumPy versions grater than some minimum supported version.

“Minimum target version” is a lovely name.

@pf_moore

Sadly, that’s just not how build tools work currently. […]

[footnote] I just looked, and it appears that setuptools still mostly just gives you an option to say “this code uses the limited API”, plus a way to set preprocessor symbols for the compile, and otherwise leaves it to you to write the correct code

Yeah. Get options from the config file and feed them to the compiler. That’s how it should work.

There’s a mismatch between what’s in the setuptools config file and what CPython accepts (and also what’s in the user’s brain). We should fix that. And it looks like most of the fix should be on the CPython side.

@pf_moore

@encukou

I think that the interface that build tools should give to extension authors should ask questions like […]

How would you like to build the extension?

• compatible with Python 3.__
• also compatible with later Python versions?: y/n
• also compatible with free-threaded builds?: y/n

What you’re describing feels more like a C extension template generator than a build backend.

That’s not my intent.
Here’s a strawman proposal, as something concrete (with silly names):

• pyproject.toml’s [c] section contains min_target_python = "3.13": Setuptools takes that, sets Py_MIN_TARGET_PYTHON to 3.13 for the compiler, and generates a cp3_13 tag
• pyproject.toml’s [c] section contains forward_compatible = true: Setuptools sets Py_FORWARD_COMPATIBLE, and a wheel tag that indicates 3.14+ compatibilty ^[1].
• pyproject.toml’s [c] section contains free_threading_compatible = true: Setuptools sets Py_FREE_THREADING_COMPATIBLE, and a wheel tag for that.

And if the user calls Py_UseAThreeFourteenFeature() that doesn’t work with the settings, the compiler complains that it’s not available. That error can’t really be clear on its own, but at least the docs for Py_UseAThreeFourteenFeature mention the required min_target_python.

---

@steve.dower

As to “two different numbers that increase in lockstep”, I agree, but the answer is to not increase them in lockstep That’s what is meant by “that association should be broken.”

That gives us PEP 809 (one of them merges five +1’s into a +5), or the 2000-era PYTHON_API_VERSION. I know your preference on that one

(I’m being silly here.)

---

@ilayn

The terminology here is very strange when stable or limited words are attached to it (for my understanding) if I have to put my C programming hat on.

Yes. We should rebrand, and avoid the terms “API” and “ABI”.

(Here I’m being serious.)

@ilayn

I thought, maybe naively, There will be a certain subset of Python C API, this includes the actual functions and data structures we are using such as PyObject and so on will be frozen.

Not only that, but also the ABI is going to be frozen, that means if we compile something against a future version of Python, it will be ABI compatible meaning that since we are using a subset of the C API those functions will communicate using the same binary ordering sizes etc. So if there is an object defined this is always going to allocate same number of bytes, with same structure etc. So we can move one library compiled with one version of Python and use it in another if we stick to this particular subset of the API. So I don’t need to compile the same library 5 times if I am supporting 5 versions of Python (in the future). But no such lunch with a discount apparently.

That is the case. If you use the 3.13 Stable ABI, the functions and the struct memory layouts your extension uses are frozen, and compatible with 3.13, 3.14, 3.15, 3.16, and so on until Python 4we remove non-free-threaded builds.

To use the 3.13 Stable ABI, your source needs to restrict itself to only using the functions and structs that have this forward-capability guarantee. That subset is called the 3.13 Limited API.

(Does that make sense? I’ll expand on that now.)

If your source is compatible with the 3.13 Limited API, it’ll also be compatible with:

• the “full” CPython 3.13-only API (since Limited API is a subset of that)
• most likely, the 3.14 LImited API (since there have been no removals, or very few ones that touched some old niche functions)

So, you can compile your “3.13+” compatible source into a “3.14+” compatible extension, or a “3.13-only” extension. Both might get some performance improvements over the “3.13+” extension – some function might be able to avoid a backwards-compatibility hack, or otherwise use a faster implementation.

There’s compile-time option for choosing the desired “minimum target version” and the “forward compatibility? y/n”. The option’s name is… extremely confusing.
Let’s name it Py_MIN_TARGET_VERSION instead.

---

1. not being concrete here: options for a short tag currently have bagage ;‍) ↩︎

[rgommers (Ralf Gommers)]

@encukou

@pf_moore

Sadly, that’s just not how build tools work currently. […]

[footnote] I just looked, and it appears that setuptools still mostly just gives you an option to say “this code uses the limited API”, plus a way to set preprocessor symbols for the compile, and otherwise leaves it to you to write the correct code

Yeah. Get options from the config file and feed them to the compiler. That’s how it should work.

There’s a mismatch between what’s in the setuptools config file and what CPython accepts (and also what’s in the user’s brain). We should fix that. And it looks like most of the fix should be on the CPython side.

Actually, this is pretty much exactly how build backends/systems already work (except for setuptools). For example:

• CMake’s FindPython module has Development.SABIModule which you set to a value like 3.8 and that will add the relevant C/C++ -DPy_LIMITED+API=... define and add .abi3 to the filename.
• Meson’s python_install.extension_module works the same, it has a keyword limited_api : '3.8'.
• scikit-build-core has a config system where you set wheel.py-api = "cp38".
• maturin autodetects how you configure PyO3 with features = ["abi3-py38"]
• meson-python currently has only a boolean tool.meson-python.limited-api setting, building on Meson’s limited_api keyword (we’ll extend that to take a string value like '3.8').

It’d be great for setuptools to improve its API, but let’s not let its current state drive decision here. It’d be easy enough to add such a feature to setuptools as well if someone has energy for that.

As a head up: our team just started building a standalone test suite of small test packages using different build backends and binding generators and targeting the stable ABI (old and new), with the dual purpose of helping PEPs 803/809 along and providing an overview of how things currently work. We should be ready to share that within the next couple of weeks I expect.

@encukou I’m not quite sure what you mean with “it looks like most of the fix should be on the CPython side”. I think given the level of confusion the documentation could be improved, but the functionality is fine?

[encukou (Petr Viktorin)]

@rgommers

I’m not quite sure what you mean with “it looks like most of the fix should be on the CPython side”. I think given the level of confusion the documentation could be improved, but the functionality is fine?

Yes. The functionality is fine, we need a rebranding.

[rgommers (Ralf Gommers)]

I’d be a lot more convinced that it needs a rebranding if people who actually use the stable ABI are confused enough to make that worth it. My impression is that this is an inherently hard topic^[1], and effort spent on this topic would be more effectively focused on improving documentation and the setuptools UX^[2], rather than adding to the confusion with a name change.

---

1. E.g., NumPy’s naming is better but there’s endless confusion there as well ↩︎

2. or moving projects away from setuptools ↩︎

[pf_moore (Paul Moore)]

@rgommers

Actually, this is pretty much exactly how build backends/systems already work (except for setuptools)

Ah, thanks. Looks like my knowledge of the state of the art in compiling C extensions is more out of date than I thought it was

This is off-topic, but it would be really helpful if the packaging user guide were more up to date regarding options for building extensions. There’s a bunch of options shown for pure Python packages, but you get the distinct impression that setuptools is still the only viable approach for native code. Could we not do something similar to the pure Python version, and have a simple “Hello, world” style C extension, with a route through the process that shows how to configure for the backends you mentioned?

[rgommers (Ralf Gommers)]

@pf_moore

This is off-topic, but it would be really helpful if the packaging user guide were more up to date regarding options for building extensions.

Agreed, I think it’s well past time to do that. Before we dive into that, should we start a new thread - this has the potential to turn into a longer discussion. Not sure about the etiquette here. Maybe you can repost your suggestion in a fresh thread and I can write a longer reply there?

[pf_moore (Paul Moore)]

Done: Adding extension module examples to the packaging user guide

[h-vetinari]

@steve.dower

Basically, PEP 809 codifies the entire discussion we’re having in a way that it can be applied in the future. 803 just tries to do it once.

This is very reminiscent of the manylinux2014 vs “perennial manylinux” discussions to me (and hence my preference for the latter).

[mgorny (Michał Górny)]

I’ve been playing with PEP 803 / Python 3.15 alpha a bit as part of GitHub - Quansight-Labs/stable-abi-testing: Projects to test the upcoming the Python stable ABI changes and I have to say I like it very much. Most importantly, the changes I needed to to do to the tools were pretty minimal.

For build backend part, it was just adding an option to output abi3.abi3t wheels. My solution was quite hacky since Python 3.15 alphas require explicit -D_Py_OPAQUE_PYOBJECT but my understanding is that the way PEP 803 proposes it, we’ll just need to look at limited API Python version.

And for pip, it was just changing the abi3 logic to use abi3t instead, for freethreading Python 3.15+.

[encukou (Petr Viktorin)]

I’ve updated the PEP. There’s now an explicit, separate abi3t wheel tag; you can build abi3 or abi3.abi3t wheels (or abi3t only but there’s not much benefit).

There’s no change (relative to the previous PEP) in recommendations for installers.
Build tools need to define Py_TARGET_ABI3T, which is to abi3t as Py_LIMITED_API is to abi3. (Or “just” use Py_LIMITED_API with headers for free-threaded Python.) On the plus side, existing build tools that set Py_LIMITED_API to the current CPython version will continue to work (producing abi3 extensions – stable ABI for GIL-ful Python only).