Python ABIs and PEP 703

Core Development
1

PEP 703 poses some issues regarding the stable ABI. I’ve described these issue below along with a short proposal.

tl;dr

The --disable-gil builds of CPython 3.13 will not be able to load existing wheels that target the stable ABI. However, we can make wheels that target the 3.13+ stable ABI work with both the default (with GIL) and --disable-gil builds going forward.

Background

CPython C API extensions can be compiled for either the stable ABI or version-specific ABI. Targeting the stable ABI allows a single wheel to work with multiple versions of CPython. To target the stable ABI, C API extensions must limit themselves to the “limited API,” a subset of the Python C API that is marked as stable. Most extensions do not use the stable ABI; they build wheels for specific versions of Python (i.e., the version-specific ABI). These wheels are compatible across “patch” releases e.g., 3.10.0, 3.10.1, etc.), but not across minor (feature) releases of Python (e.g., 3.10 vs. 3.11).

There is not a single “stable ABI” but rather a stable ABI for each minimum Python version starting with Python 3.2. New Python releases may add new functions to the stable ABI. To take advantage of these new functions, extensions using the stable ABI must target a specific minimum version. For example, as of the time of this writing, the latest cryptography wheels target cp37-abi3 so they work with Python 3.7 and newer releases. Additionally, ABI stability depends on the platform (OS, architecture, libc implementation).

Common reasons for targeting the stable ABI include:

  • Reducing the workload to build and test wheels. For example, the cryptography package builds 22 wheels (10 for CPython, 12 for PyPy) for each release. Without the stable ABI, this would increase to 72 wheels to support Python 3.7-3.12.
  • Ensuring extensions are immediately available when a new version of CPython is released.
  • Some extensions do not use the Python C API at all, but have platform specific native libraries accessed via cffi. These extensions may be labeled as if they used the stable ABI because they are compatible across Python versions but not across platforms.

Common reasons for targeting the version-specific ABI include:

  • The stable ABI may not expose sufficient functionality for the extension.
  • The stable ABI may reduce extension performance due to use of function calls instead of macros/inline functions for some operations. For example, Py_INCREF and Py_DECREF are non-inlineable function calls when targeting the 3.12 stable ABI or newer.

PEP 703 ABI Challenges

PEP 703 requires changing the PyObject reference count fields for --disable-gil builds. This means that existing wheels that target the stable ABI will not work with --disable-gil builds because they include inlined code (from macros or inline functions) that directly access these fields.

However, it’s possible to make wheels that target the Python 3.13 stable ABI work with both the default and --disable-gil builds going forward. We can do this by ensuring that all reference counting operations use function calls instead of macros or inline functions. The Python 3.12 stable ABI has taken a step in this direction: Py_INCREF and Py_DECREF already use the function calls _Py_IncRef and _Py_DecRef when targeting the Python 3.12+ stable ABI.

Additionally, the Py_SIZE / Py_SET_SIZE macros would need to use function calls since the offset of the ob_size field would differ between --disable-gil and the default build.

Proposal

Add the following functions to the 3.13 stable ABI as “ABI only” functions:

  • Py_ssize_t _Py_Refcnt(PyObject * ob);
  • void _Py_SetRefcnt(PyObject *ob, Py_ssize_t refcnt);
  • int _Py_IsImmortal(PyObject *ob);
  • Py_ssize_t _Py_Size(PyObject *ob);
  • void _Py_SetSize(PyObject *ob, Py_ssize_t size);

Note that the names begin with underscores because they are not intended to be called directly, even when targeting the limited API. Instead, they should be called via the corresponding standard macros. This is the pattern used by functions like _Py_IncRef, _Py_DecRef, and _Py_Dealloc. See functions marked abi_only=True in https://github.com/python/cpython/blob/main/Misc/stable_abi.toml.

Change the following macros to use function calls when targeting Py_LIMITED_API >= 3.13:

  • Py_REFCNT to use _Py_Refcnt
  • Py_SET_REFCNT to use _Py_SetRefcnt
  • Py_SIZE to use _Py_Size
  • Py_SET_SIZE to use _Py_SetSize

This effectively an extension of the work done in gh-105387 by @vstinner and discussed previously.

Add the following macro to the public limited API for Py_LIMITED_API >= 3.13:

  • Py_IS_IMMORTAL(ob)

(This is already exposed by PyO3, which is one of the most common ways developers rely on the stable ABI)

Add (and document) a new field “abi_disable_gil” to sys.implementation to reliably differentiate --disable-gil builds from the default build in CPython. The field sys.implementation.abi_disable_gil is set to True in --disable-gil builds and False otherwise.

Open Questions

Do we want to make Py_TYPE and Py_SET_TYPE use function calls? This is not strictly necessary since we can ensure that the ob_type field is at the same offset for both --disable-gil and the default builds, but may avoid some future issues.

What about the default (with GIL) build of CPython?

Extensions that target the older versions of the stable ABI will continue to work with the default build of CPython 3.13+, but not --disable-gil builds of CPython. If an extension author using the stable ABI does not wish to support --disable-gil builds, they do not need to change anything.

Extension authors using the stable ABI that wish to support both older versions of CPython (e.g., 3.7+) and --disable-gil builds will need to build two wheels per platform. For example, a future version of the cryptography package might provide:

  • cryptography-42.0.0-cp37-abi3-win_amd64.whl
  • cryptography-42.0.0-cp313-abi3-win_amd64.whl

Note that the default (with GIL) build of Python 3.13 could use either wheel.

What about extensions that do not use the stable ABI?

Extensions that do not use the stable ABI (e.g., NumPy), will need separate wheels for the 3.13 default build and the 3.13 --disable-gil build. For example, a future NumPy release for 3.13 might have the filenames and platform tags:

  • numpy-1.27.0-cp313-cp313-win_amd64.whl (default build)
  • numpy-1.27.0-cp313-cp313t-win_amd64.whl (--disable-gil build)

This is unchanged from PEP 703, other than the use of “t” (for threaded) instead of “n” (for nogil) as suggested here.

Packaging and Tooling Changes

  • pypa/packaging: When computing platform tags for --disable-gil builds (i.e., when sys.implementation.abi_disable_gil is True), cpython_tags() should exclude abi3 builds that target versions before 3.13.
  • pip: pip vendors the “packaging” package and would need to update their copy
  • poetry: poetry makes use of “packaging” to compute tags
  • hatch: hatch makes use of “packaging” to compute tags
  • PyPI: No changes necessary. It already supports different ABI tags (see colesbury-c-extension · PyPI)
  • setuptools: No changes necessary.
8 Likes
2

IMO the nogil project is likely to fail if we don’t offer a “simple” solution to distribute C extensions working on regular (GIL) Python build and nogil Pyhon build. The stable ABI sounds like a good solution since it already exists and partially solves the use case. But as you explained, the current ABI still has some flaws: it still “leaks” some implementation details which make it incompatible with nogil build :frowning:

IMO the stable ABI is not widely adopted simply because it’s not the default in distutils, setuptools, poetry, etc. Switch the default to stable ABI, and it’s likely that it will become way more widely used :slight_smile:

What I don’t get is how can a C extension maintainer support nogil and Python 3.12 with the stable ABI? I don’t think that a single binary fits all use cases, since nogil requires calling new functions at the ABI level, functions only available on Python 3.13. I suppose that on PyPI, for each platform/architecture, two binaries are needed:

  • Binary for Python >= 3.13: compatible with nogil
  • Binary for Python <= 3.12: incompatible with nogil

What happens with a stable ABI compiled on Python 3.12 is used on Python 3.13 nogile build? Crash? Is it slower and may crash if run for days? Or is the behavior just undefined?

Can packaging tooling be updated to handle “nogil” and ship 2 binaries on PyPI instead of 1?

Honestly, the nogil change sounds big enough to motivate creating a new abi4 version. Currently, the stable ABI is called abi3 since Python 3.2 (2011). Since abi3, many missing features were discovered in the limited C API and the stable ABI. So when you say “abi3”, just the name doesn’t say anything about which Python versions are supported, as Sam explained, you must also specify the target version of the limited C API. Hopefully, the version is part of the full wheel binary package filename.

Not only abi4 would add features needed by nogil, but also avoid checking the version to know what’s available or missing. Well, I expect that Python 3.14 will add again new functions to the stable ABI. But abi4 would mean “compatible with nogil build”… and also “not compatible with Python 3.12 and older”.

That’s maybe what people expected by “removing the GIL will require Python 4 (incompatible major version)”. People just misunderstood that only the stable ABI major version should not change, not the Python version :wink:

Obviously, abi3 will still be supported by Python 3.13: supporting it is free (already implemented). Support for Python 3.12 and older stays. For there is nothing like Python 2 to Python 3 “only way only” migration. Here the migration towards abi4 can be done incrementally, on demand when supporting nogil build is “needed”. And if my previous paragraph makes sense, we can easily ship abi3 and abi4 binary wheels of the same C extension to support all Python versions.

Py_SIZE() function/macro has issues: see Py_SIZE is not useful · Issue #10 · capi-workgroup/problems · GitHub

We should consider fixing it before adding it to a stable ABI.

We should expose it as a macro and a function for programming languages and use cases unable to use macros. For example, the vim text editor only uses dlsym() to load libpython symbols, it cannot access macros, even if it’s written in C.

Note: I would prefer Py_IsImmortal() name :wink:

3

I haven’t read this through yet, but I strongly suggest cross-posting this to the “Packaging” category, as there will be a lot of people affected by this who might only read that category.

2 Likes
4

IMO the nogil project is likely to fail if we don’t offer a “simple” solution to distribute C extensions working on regular (GIL) Python build and nogil Pyhon build.

I do not think having to build an extra wheel will cause the nogil project to fail, even if it adds some inconvenience for package maintainers.

However, I am concerned that tying PEP 703 implementation to other issues like fixing the C API in general or requiring the stable ABI will cause the project to fail.

What I don’t get is how can a C extension maintainer support nogil and Python 3.12 with the stable ABI?

This is discussed in second paragraph of this section. They would need to build an additional wheel per platform.

What happens with a stable ABI compiled on Python 3.12 is used on Python 3.13 nogil build?

It would not load.

Honestly, the nogil change sounds big enough to motivate creating a new abi4 version

I don’t think calling these ABI changes abi4 solves any problems; it only introduces problems. The proposal adds a few functions to the stable ABI whose functionality already exists (as direct field access.). The proposed ABI changes to cp13-abi3 keep the ABI compatible with older Python versions.

1 Like
5

Will I be able to build and install --disable-gil next to --no-disable-gil in parallel?

6

As a packager of popular packages with abi3 wheels (who is very excited about the glorious nogil future)-- my view is that as long as:

a) This is a one time thing (i.e., there’s abi3 wheels targeting minimum version through 3.12, and then 3.13+)
b) pip correctly selects between them (including older pip versions)

Then we’re probably ok. We cannot afford to do a build for every single Python version, but we can do 2x versions.

This is purely from a “package maintainer burden” perspective, I’m not thinking through the other considerations here.

2 Likes
7

I think the answer is “only to the same extent that you can install multiple versions of Python next to each other”. There is not a plan for a unified installers.

8

Fair enough. It’s actually pretty easy to do so for pythonX.Y on systems I care about, much harder for pythonX.Y.Z (e.g. you need to use different installation directories when installing from source). gil vs no-gil installations of the same pythonX.Y will likely also require different installation directories and can’t live side-by-side.

9

4 posts were split to a new topic: Naming the Python binary when --disable-gil is set

10

Depends on what you mean by “in parallel”. You could do a “fat” wheel where you put two .so/.dll files next to each other for each build as long as we make sure the file names we search for in Python itself won’t clash. Otherwise it’s like installing any other package unless you want to change the import system.

Pip’s logic uses packging.tags which calculates the list of acceptable wheel tags. The code specifically related to the stable ABI is covered in:

So old versions of pip today will consider any abi3 usage for the version of Python being installed for and older as being compatible. If some magical point in time where abi3 wheel tag means something different for Python 3.13 than all previous versions then that will only work with new pip versions. If you change the ABI version (i.e., abi4) then those wheel files will simply be ignored by old pip versions.

1 Like
11

Just to give more information on the use cases you need to cover:

I maintain a stable-ABI wheel which we distribute to our customers, so they can run on anything from Python 3.7 up. (I know 3.7 is now EOL, but it is still supported by some of our supported Linux distros, and there isn’t a lot of advantage in changing to 3.8+).

I am currently working on switching to use Python 3.11 to build this wheel. I do not expect to be able to use Python 3.13 for at least two years (possibly quite a bit longer).

I don’t actually care about no-GIL builds at the moment, which I currently regard as very experimental, so my requirements are:

  • It must be possible to load a stable-ABI .so which targets python 3.7+ into a “with GIL” build of Python 3.13+.
  • Pip must successfully install a cp37-abi3 wheel into a with-GIL build of Python 3.13+.
  • Pip must refuse to install a cp37-abi3 wheel into a “no GIL” build of Python 3.13+
  • There must be a plan for how I can support both GIL and no-GIL builds of Python in the future. This would either involve building two wheels (GIL and no-GIL or cp37/cp313), or building a “fat wheel” which contains both shared objects, and loads one or the other conditionally.

Other thoughts:

It is a real shame that when the stable ABI was defined, it wasn’t done properly, with all interactions with the python run-time done via function calls and completely opaque pointers. However, until we get the Software Development Time Machine™ working, it’s too late for that.

If we are going to have to make the change to make incref and decref be via function calls, I think we should take the opportunity to make everything else be via function calls too, so that we don’t have to go through this ever again.

2 Likes
12

This are where I would like API/ABI to go:

  • Stable ABI should have first-class support for non-C languages, which can only access exported symbols, not macros or inline functions.
  • Stable ABI should be implementable by non-CPython implementations, which might not use refcounting or immortality.

I plan to discuss this at the sprint & come up with a high-level direction proposal. It might turn out we don’t want to go this way, in which case the rest of this post is moot, but if those sound like good ideas, read on :‍)

Some rules of thumb follow. Of course Py_REFCNT & co. are very special and can bend these, but only if necessary.

  • All API functions should be exported as proper symbols (perhaps in addition to macros/inlines for C)
  • All functions should be able to signal failure. (This allows us to deprecate them cleanly – with a runtime warning that -Wall may turn into an exception.)

With that, I’d change your proposal to a add bunch of fully public functions:

  • PyObject_Refcnt: returns SOME_HUGE_VALUE if refcounting does not apply, returns -1 with an exception set on error (even though CPython will currently never raise here)
  • int PyObject_SetRefcnt: returns 1 on success, 0 if it did nothing (refcounting does not apply), -1 with an exception set on error. (Users must handle the 0 case, but AFAIK this function is mainly used for happy fast paths where you can switch to the slow path)
  • PyObject_IsImmortal: returns 0 for non-immortal (incl. if the implementation doesn’t use immortality), 1 for immortal, and -1 with an exception set on error
  • PyObject_GetObSize: returns -1 & sets an exception on error. (I prefer the weird-sounding ObSize to avoid confusion with PyObject_Size which calls __len__.)
  • int PyObject_SetObSize: returns -1 & sets an exception on error

In version-specific builds, these can be inline functions with assert(result != -1), so compilers can remove users’ error handling.
(And yes, I know, users will forget to add the error handling if it does nothing. That’s a bug to be fixed in their code.)

Note how PyObject_SetRefcnt turns what a C programmer would expect to be a compile-time error into a runtime one. This is what stable ABI should generally do. (We can of course also add a compile-time deprecations/errors later, for users that happen to recompile.)
Note that Py_INCREF cannot raise. A moving GC could have to make it “pin” the object (possibly with high overhead, which is OK for stable ABI). But I can’t think of a similar hack for PyObject_SetRefcnt, so let’s allow it to raise.

(That was all assuming replacements for Py_REFCNT and Py_SET_REFCNT are necessary for stable ABI extensions. Perhaps we can not include them at all?0

IMO, one point is missing from the proposal: PyObject should become opaque, so e.g. all uses of obj->ob_refcnt need to switch to Py_REFCNT(). That’s the biggest pain point in the stable ABI, and if we’re doing any kind of compatibility break we should include this one.

Like Victor, I would like to call the new ABI abi4. But here’s the twist: we can make abi4 extensions compatible with abi3, as long as they don’t use the newly added functions like Py_REFCNT/PyObject_Refcnt!

Hers’s the situation I’d like to get:

  • Which CPython can use an extension with the given ABI:

    3.8-abi3 3.12-abi3 3.13-abi3 3.8-abi4 3.12-abi4 3.13-abi4
    CPy 3.8 ✓ ✘ ✘ ✓* ✘ ✘
    CPy 3.12 ✓ ✓ ✘ ✓* ✓* ✘
    CPy 3.13 ✓ ✓ ✓ ✓ ✓ ✓
    CPy 3.13-nogil ✘ ✘ ✘** ✓ ✓ ✓

    *) packaging tools will need to rename the extension file on wheel install, so that the old Python recognizes it
    **) cannot work as it possibly uses ob_refcnt directly

  • Which CPython can compile an extension with the given ABI:

    3.8-abi3 3.12-abi3 3.13-abi3 3.8-abi4 3.12-abi4 3.13-abi4
    CPy 3.8 ✓ ✘ ✘ ✘ ✘ ✘
    CPy 3.12 ✓ ✓ ✘ ✘ ✘ ✘
    CPy 3.13 ✓ ✓ ✓ ✓ ✓ ✓
    CPy 3.13-nogil ✓ ✓ ✓ ✓ ✓ ✓

Eventually we drop the abi3 series (and force everyone off ob_refcnt), but that’s a separate discussion.

And for clarity:

Yes. The only remaining things that aren’t function calls are the refcount, ob_type, and ob_size.
(The other non-opaque structs are the interop-related Py_buffer and various “blueprint” structs like PyType_Spec – of which one, PyModuleDef, is a PyObject which will cause some trouble for abi4.)

5 Likes
13

I don’t think we should worry too much about whether old versions of pip work properly for the experimental, --disable-gil build of CPython 3.13. As it is, old versions of pip frequently do not work with new versions of Python. For example, pip==23.1.1 and older (from just 5 months ago) will break if installed in CPython 3.13 (missing pkgutil.ImpImporter).

  1. 3.13-abi3 doesn’t use ob_refcnt directly. That was changed to use a function call in 3.12. If we can make abi4 work with GIL and --disable-gil then we can probably do the same for 3.13-abi3.
  2. You don’t state it explicitly, but I guess the idea is that refcounting in abi4 could use the older Py_IncRef function call, which is available in CPython 3.8.

Ooof, I’m not sure how to address PyModuleDef.

1 Like
14

Pip’s policy is that you should always upgrade to the latest version. I think it’s fine if an older version breaks for newer Python. But I’m less sure that silently installing the wrong package is acceptable. People do use older versions of pip, and loud breakages aren’t the same as silent errors. But we (pip) don’t have a specific policy on this.

I don’t personally have a strong opinion on this, but I do think that if we want people to try out the free-threaded builds, having to debug ABI compatibility issues like this is likely to be rather off-putting for them.

2 Likes
15

As a package maintainer, I beg of folks: pip silently doing the wrong thing creates a flood of issues for packages like pyca/cryptography.

5 Likes
16

Py_INCREF doesn’t use ob_refcnt, but the field is exposed and can still be used directly. Disallowing that (along with ob_type/ob_base/ob_size, and stuff like PyObject_HEAD and anything else that needs PyObject struct size) is the necessary API break. (edit: API break, not just ABI)

Well, should stop being a PyObject, but the details might be off-topic in this thread.

1 Like
17

I think this is a critical missing piece, so I’d appreciate any suggestions on how to handle it.

1 Like
18

Do you suggest that users would check PyObject_Refcnt and take different code paths depending on whether reference counting applies? Is this meant to just distinguish immortal objects, or distinguish whether the underlying runtime uses reference counting?

Do you actually need and want to distinguish those situations? Could the API be made generic such that the user does not have to worry about and cannot abuse implementation details, such as immortal objects, or in general which memory management strategy the runtime uses?

19

Let’s first ask the question: what is the use case for PyObject_Refcnt in third-party (non-CPython) code? The only one I can think about is to check whether an object is shared or not, and in that case PyObject_RefCnt(x) == K (where K may be 1 or 2 depending on the context) is sufficient and will continue to work ok.

2 Likes
20

What is that useful for?

I assume this is all about 3rd party code, I don’t see a reason why CPython would internally do a call (to a non inline function) for something like this. In general, public API for extending (C)Python should be just something else than internal APIs used in CPython, they have different purposes, different guarantees, different lifespans, mixing the two is IMO source of some of the issues.