One time ABI breakage

Core Development
1

Abstract

I’d like to propose that we have a one time break in the stable ABI,
such that extensions can be compiled into one of two builds:

  1. Support versions up to 3.15
  2. Support version 3.12 (3.14 for 32 bit builds) and later.

With this break, it will not be possible to compile a single binary that
supports both 3.11 and 3.16, or both 3.13 and 3.16 for 32 bit builds.

Having this break will allow significant improvements in maintainability
and performance of CPython.

Motivation

Maintainability

Having the freedom to adjust the PyObject header means that
we can homogenize the code across 64 bit (little and big endian)
and 32 bit platforms, reduce the differences between the default
and free-threading structs, and simplify GC and allocator code by
storing necessary information in the object header.

Specifically this gets us:

  • Reduced code size
  • Better testing
  • More flexibility in the implementation
  • Reduced diff between default and free-threading implementations

Performance

Simple refactoring should produce a 1-2% speedup, by speeding up checks for GC attributes, object layout, etc.
On top of that, the main performance benefit will be in allowing better data structures and algorithms for allocation, reference counting and garbage collection.
I don’t know much this will gain us, but there is lots of room for improvement in the those areas of the CPython code base. My guess would be 5-10%.

Some of these improvements can be done without the ABI breakage, but not as well or as cleanly.

Why is a break necessary? Why does the stable ABI need to change?

Traditionally, Py_TYPE(), Py_DECREF() and PY_INCREF() were implemented as macros, meaning that the ABI depended on the layout of PyObject, which could never be changed.

Since 3.12, Py_DECREF and PY_INCREF have been implemented as functions in the stable ABI. With builds of C extensions that support 3.12 and later only, the implementation can change the layout of the PyObject header.

Requiring that we support 3.12 does limit the changes that can be made, specifically we cannot change the size of the object header.
That is fine for 64 bit builds as there is spare space in the object header, but 32 bit builds may need the object header to grow.
Consequently the break has to be at 3.14 for 32 bit builds.

Proof of concept

As a test of the utility of this proposal, I refactored just the GC code to use bits in the ob_flags field of the header, instead of tagging pointers.

The resulting code is easier to understand and less error prone. Quantitatively, it reduces the code size by 100 lines and produces a consistent 1% speedup across the 5 platforms (all 64 bit) tested on.

13 Likes
2

There are several clear benefits to an ABI break. One change I’d personally love to see is allowing for a concurrent garbage collection in free-threading. That would likely require introducing barriers in incref/decref.

However, an ABI break would have widespread ramifications for the community. A lot of projects build abi3 wheels to minimize their build matrix and rely on the stability of the ABI to ease usage across Python versions. This has been exceedingly beneficial in ensuring libraries are compatible with new Python releases and enabled more testing of beta versions of Python, which has been invaluable.

I think before we consider an ABI break, we should have a clear list of potential changes we may want to make, and from that a clear migration plan that addresses issues caused by the breakage.

15 Likes
3

I believe that abi3t enables support for both the gil-enabled and free-threaded build by removing any definition of both layout and size of the object header from the stable ABI. This probably gives you the freedom to make the simplifications you want to the free-threaded build today.

My understanding is that distributors are being encouraged to build abi3t wheels for 3.15+ so that a single stable abi wheel supports everything going forward from that point.

Maybe with that in mind, an ABI break which drops support for anything older than 3.15 will have relatively little impact (for projects with active maintenance) because the new wheels will somewhat hide the breakage.

However, I also got the impression that the hope is that one day the free-threaded build might be the only build, so maybe it’s good enough to put the clean code on the free-threaded build and accept some cruft in the gil-enabled build until it’s phased out?

8 Likes
4

For projects using bindings generators like PyO3 I think that’s the guidance that makes sense.

It’s less clear-cut for projects that use Cython or Nanobind (both don’t yet or can’t yet support abi3t).

It’s also less clear-cut for projects that are written in C directly without a bindings generator. Typically those are older projects that implement static types. It would be a signficant amount of C API work to transition projects like that to use heap types. Not to mention the more mechanical changes needed to support abi3t like adopting the PySlot and PyModExport mechanisms.

1 Like
5

I’d hope (but can’t promise) that Cython probably would by the time that this happened. I doubt that Mark is trying to sneak this into 3.15 (or at least I doubt that he’ll be successful).

That presumably wouldn’t be a problem because they’d be compiling against the version-specific ABI anyway which would directly use the new object format. I’d really hope that wouldn’t go away (and don’t see why it should).

I’d see the likely issues as:

  • updating packaging tooling.
  • The open question of if abi3t lets extensions be fast enough to be practically useful
  • What promises ABI3 made about compatibility and how much notice you’d have to give to break them - i.e. I think everyone knows that it can’t last “forever” but “next Python version” might feel quite soon to come people.
6

Ah, sorry. I made a mistake: adopting abi3 requires adopting heap types. So that’s most of the pain and going from abi3 to abi3t isn’t so bad. Others following along can see your comment in the astropy issue about this.

7

Rather than doing it once, and realising in 5, 10 or 15 years that something else in the ABI needs to change, why not go with PEP 809, which handles this in a future-proof manner?

1 Like
8

Another alternative is to drop abi3, and require version-specific builds for projects that can’t switch to abi3t.
(The practicality of that would depend on how fast we can make abi3t more usable, and how fast free-threading is becoming the default/only option.)

4 Likes
9

Regarding nanobind, see also: PEP 803, round two – “abi3t”: Stable ABI for Free-Threaded Builds - #28 by wjakob
My personal preference would be to focus on abi3t with the intention of (eventually) dropping abi3, given the reality that development resources are not infinite.

1 Like
10

This is the same possibility that occurred to me as well. abi3t is only defined for 3.15 and later, so projects using it necessarily aren’t exposed to the old layout.

Adding a new ABI name that differs from both abi3 and abi3t would be incredibly awkward to explain, and it’s hard to see how redefining abi3 without introducing a new name could work in practice from a wheel distribution perspective.

11

I’ll be a bit pedantic here, just to be clear about the terms being used.

Text wall incoming

I don’t think the term ABI break, is the right one to use, in order to describe what’s happening here. In the Linux world, the term ABI generally refers to the Linux Standard Base, which also happens to include a specification for versioning library APIs via data within an ELF section.

The largest reason why these ABIs matter, in my opinion, is that they describe how machine code and assembler programs interact with other machine code, within the same process. It also describes how a process “image,” or executable, which is represented as an ELF file, is to be laid out within memory, and eventually executed. The LSB also includes an ELF specification, I believe.

The likely reason as to why the term “ABI break” is used in order to describe version changes of a given library, likely has something to do with the conflation of the LSB’s specification of library versioning, and the idea of library, or API versioning, itself.

I think the primary reason why anyone should actually care, is that it is important to distinguish that ABIs primarily refer to inter-program interactions, and interactions between the program image itself, and the operating system. It generally does not refer to application APIs, or any intra-program interactions that are in a higher abstraction level than assembly language. I also consider a program using a shared library, as an intra-program interaction, since technically, a shared library is a separate, non-executable image.

I believe that failing to distinguish between ABIs, and APIs, will (and have) lead to many misunderstandings.

I would consider this an API break.

Edit: I consider the changing of function parameters to be an API change, even if it technically breaks compatibility at the machine level. I think that the idea that the C API is also being changed, is what matters more here, when it comes to distinction. I’d consider the changing of a given library’s calling standard to be an ABI break; and, you can also change a calling standard, without changing the C API, which to me, matters more in cases of binary compatibility.

Many web APIs for instance, don’t interface with each other outside of ASCII characters, and they usually interact via JSON. It would be odd to call a switch of parameters, an “ABI” break, in those instances, since most don’t consider ASCII text to be binary in a non human-readable sense.

That’s my two cents.

12

You are misunderstanding and ABI is correct here. The issue is the ability to compile extension modules that work across multiple versions of Python. These can depend on the binary layout of the Python object and this is what is being proposed to change.

6 Likes
13

Hi Mark, I think it’s confusing if there are multiple ABIs for regular Python and one for free-threaded Python. If there was to be such a split, then it would make more sense to stop supporting abi3 after 3.15 and adopt abi3t to target both free-threaded and regular builds for 3.15+. Whether the extension declares itself to be compatible with free-threading is unrelated, so incompatible extensions could still ship via abi3t. To me, the main appeal of these ABIs is to cut down insane build matrices with multiple GBs of wheels per release. Having multiple redundant ABI conventions in flight harms that goal.

14

It’s more that abi3t excludes the PyObject layout (and that’s the only difference from abi3 :slight_smile: )

AFAIK, the key difference is that in Mark’s proposed post-break ABI you can still use the PyObject layout, i.e. you can define types in the traditional way.

1 Like
15

This isn’t directly related to the OP’s question but something that has been on my mind regarding the stable ABI. While developing/maintaining nanobind, I have often struggled with this ABI quite a bit. I wish to solve the wheel distribution problem but without paying a major tax on efficiency. This is hard in the stable ABI because

  • Functions/macros that normally inline turn into shared library calls that cause register spilling on the caller’s side. This reduces the performance of important pieces of code, like dispatching calls from one language to the other.
  • Accessing Python object internals (e.g. tuples) requires additional API calls.
  • We generally want to target the lowest possible stable ABI version, but this version then tends to lack important features as part of the API/ABI surface. Working around those limitations has a cost.
  • For nanobind, the stable ABI became usable starting in 3.12. Users must still ship per-minor-version wheels for 3.10 and 3.11.
  • The API around some tricky aspects of Python is rapidly evolving. For example, there is now an upcoming “interpreter views” feature in 3.15 that seems really great. I’d really like to use it but can’t since nanobind targets the 3.12 stable ABI.
  • The latency from adding a stable ABI feature to being broadly available is effectively infinite from a planning perspective (~5 years).

This may read like a complaint arguing that things should be designed differently. It is not—I find the stable ABI a great solution for many tasks. The points above are IMO unavoidable practical realities that result from shipping the stable ABI as part of Python itself, and binding libraries needing advanced features.

There was an attempt at another interesting distribution model, which was to ship an alternative stable ABI as a Python module (“HPy”). Sadly it seems to not be developed anymore.

There is yet another distribution model: binding tools can simply ship their own stable ABI. I plan to explore this possibility in an upcoming version of nanobind (3.x). In this case, there is a “nanobind-abi” package with all the low level and performance critical bits (distributed by me on PyPI, with unstable ABI wheels per Python version). User-built extensions delegate work to this package, and their own API surface becomes tiny. They can be shipped as stable ABI wheel targeting the lowest non-EOL Python version (e.g. 3.10).

With this plan, nanobind would become less dependent on details of the stable ABI. It will use abi3t to ship wheels for free-threaded Python, but its stable ABI usage will be limited to basic functionality–reference counting, dict lookups, etc. It removes some of the pressure (For example, Python 3.15’s abi3t made PyVarObject unusable, and as a consequence nanobind cannot target it. Such issues would just disappear.)

5 Likes
16

In principle I think PEP 509 would help with this (at least if you’re prepared to treat the features as optional and have fallback code if they’re not). Presumably at some runtime cost.

The nanobind-abi idea sounds like a good one. Obviously at the cost of shifting a bunch of maintenance to yourself.

1 Like
17

I’m happy to (figuratively) throw 32-bit under the bus… but would love to see details spelled out in more detail.

The OP feels like a solid outline. I’d love to see what header fields are getting moved; the refactoring PoC; details on whether header fields can be moved again later; etc.

18

sip · PyPI has been doing that for quite a while. The big advantage the approach has over the main stable ABI is, as you say, it can evolve at the same speed as the binding generator itself.

We’re well and truly off topic for this particular thread, though.

1 Like
19

I’m worried that people are treating the free-threading build as some sort of Python 4.
The idea seems to be that we can make any breaks we want, and some how pretend we aren’t breaking things as we haven’t broken the default build. Yet, at the same time say that the default build will somehow be “phased out”

I really don’t like crippling development of the default build for some hypothetical future

6 Likes
20

I primarily wanted to highlight that I believe you’re already free to land all the proposed optimisations on the free-threaded build. This would presumably make it easier to do full quantitative measurements of the complete implementation to evaluate the value of performing an ABI break on the GIL-enabled build to land the same improvement.

My words about not bothering to land the changes on the GIL-enabled build are written purely from understanding of the PEP 703 and 779 acceptance texts which both discuss a “phase 3” of free-threading rollout where the default is flipped, and also a likely intention to drop the separate GIL-enabled build at some point thereafter. I would not like to suggest that the free-threaded build is a Python 4 where we can break whatever we please.

I’m not sure anyone has currently indicated how many years we are from phase 3 (2? 5? 10?). If we’re still many years from that transition then that of course weighs positively toward the cost/benefit tradeoffs of making an ABI break on the GIL-enabled build.

As for performing the break, as a bindings generator PyO3 would have fewer configurations to implement and test if we moved towards deprecating abi3, which I believe would have the effect of the ABI break you want.