Let's get rid of the stable ABI, but keep the limited API

the issue is that stable ABI promised stability while it did not really solve the reasons for instability: the need to evolve Python internals, because the ABI still exposes internal details.

This can go on and on in circles. The real solution is, IMHO, to have a different API (and really stable ABI) for extensions that abstracts and hides the implementation details and is different to what CPython uses to implement itself. How did Java or C# and others manage to innovate so much in last decades while keeping binary compatibility with existing extensions all that time?


I’m confused. Stable ABI or limited API? How can the stable ABI have only just become good enough? The whole point is that it hasn’t changed… I’m guessing that other parts of the ecosystem (tooling, or the limited API, maybe?) have made it easier to build ABI3 binaries. That’s good - the stable ABI isn’t useful if people can’t use it easily - but it’s a different issue.

I don’t think we should remove the stable ABI. I think it’s clear from this discussion that it adds value for extension authors. But I am in favour of @tiran’s suggestion:

I think that if the design of the stable ABI is limiting the evolution of Python, because it exposes too many internal details, then it should change. Improving performance and other internal enhancements are important work. They just shouldn’t happen without considering the impact on extension authors. So we have to balance - but “the stable ABI will never change” goes too far in one direction, just as “let’s get rid of the stable ABI” goes too far in the other direction.

Fully agreed on that. The key point is that we stabilized something that wasn’t yet ready for the very very long haul (and that’s OK!).

I didn’t state how long a putative abi4 should stay untouched, just that abi3 isn’t future-ready enough to not be changed anymore, ever.

Stable ABI and limited API are intertwined. The stable ABI is stable, but not fixed. Minor releases of Python can add new features to the stable ABI. Therefore I see the stable ABI as an extension of the limited API.

An extension that uses limited API of Python 3.11 will also compiled with any Python version >= 3.11, < 4.0. An extension module compiled with limited API and stable ABI of Python 3.11 can be imported by Python >=3.11, < 4.0.


I assume that this is meant to be about the current stable ABI, but I want to note that in general this is doable and if there is ever some ABI4, this could/should be its goal. AFAIK, the JNI (java native interface) ABI hasn’t been broken since Java 6 or so and in the meanwhile Java added, for example, compressed object pointers or several new GC algorithms.

My hope/dream is that we will sit down as a group and figure out what kind of C ABI/API we would want if we were to start from scratch (I know HPy has been held up as that API, but I don’t think we have made that as a statement instead of an idea).

I (and I think most of the people involved with HPy) would love to sit down as a group too. How do we make this happen?


this list appears to be incomplete – or at least it is missing some of my packages – https://pypi.org/simple/ukkonen/ is one example https://pypi.org/simple/editdistance-s/ is another

EDIT: oh, didn’t realize this would end up at the end of the thread – just wanted to voice my concern in that abi3 finally made it it reasonable to release C extension wheels on pypi and to go back to the old “need to rebuild everything every time cpython releases” would be quite painful

1 Like

Weird. I used a derived table, maybe the code I used to populate it was broken :frowning: I’ve switched to an alternative DB schema that doesn’t need the separate table, so hopefully it was a one-off. Sorry about that.

Updated version (330 projects, 12523 files). Hopefully right this time!

Project listing
Project N
eclipse-zenoh-nightly 1385
polars 755
advent-of-code 413
py-randomprime 307
anki 281
tcod 263
graphlib2 251
cryptography 248
glean-sdk 241
betfair-data 176
miguel-lib 161
routrie 136
rjsonnet 125
hdbcli 118
rfiletype 118
clvm-tools-rs 116
rjmespath 103
solders 103
fat-macho 98
rjieba 89
linguars 82
ensmallen 81
murmurhash2 80
pyffmpeg 76
rbcl 76
deltalake 72
pycryptodome 72
pyheck 72
iced-x86 71
pycryptodomex 71
qcustomplot-pyside2 70
pyoxigraph 68
dss-python 66
crfs 64
pyqt5 64
iota-client-python 60
polars-u64-idx 55
bcrypt 52
pyside2 52
nsmblib 51
reasonable 50
qscintilla 48
onigurumacffi 47
pyqtchart 44
qecstruct 44
watchfiles 44
memprocfs 42
narrow-down 42
nionui-tool 42
metadata-guardian 41
pillow-heif 41
pyside6 41
shiboken6 41
nionswift-tool 40
pyqt3d 40
pyqtdatavisualization 40
pyqtpurchasing 40
shiboken2 40
protobuf 39
portmod 38
bcl 37
nh3 37
graspologic-native 36
iota-client 36
opencv-contrib-python 36
opencv-contrib-python-headless 36
opencv-contrib-python-rolling 36
opencv-python 36
opencv-python-headless 36
opencv-python-rolling 36
adblock 35
clvm-rs 35
jpeglib 35
pyobjc-framework-addressbook 35
pyobjc-framework-authenticationservices 35
pyobjc-framework-avfoundation 35
pyobjc-framework-avkit 35
pyobjc-framework-cfnetwork 35
pyobjc-framework-contacts 35
pyobjc-framework-contactsui 35
pyobjc-framework-coreaudiokit 35
pyobjc-framework-corebluetooth 35
pyobjc-framework-coredata 35
pyobjc-framework-corelocation 35
pyobjc-framework-coremediaio 35
pyobjc-framework-coreml 35
pyobjc-framework-coreservices 35
pyobjc-framework-corespotlight 35
pyobjc-framework-corewlan 35
pyobjc-framework-cryptotokenkit 35
pyobjc-framework-discrecording 35
pyobjc-framework-externalaccessory 35
pyobjc-framework-fsevents 35
pyobjc-framework-gamecenter 35
pyobjc-framework-gamekit 35
pyobjc-framework-gameplaykit 35
pyobjc-framework-imagecapturecore 35
pyobjc-framework-imserviceplugin 35
pyobjc-framework-inputmethodkit 35
pyobjc-framework-intents 35
pyobjc-framework-mapkit 35
pyobjc-framework-mediatoolbox 35
pyobjc-framework-metal 35
pyobjc-framework-metalkit 35
pyobjc-framework-modelio 35
pyobjc-framework-multipeerconnectivity 35
pyobjc-framework-network 35
pyobjc-framework-networkextension 35
pyobjc-framework-notificationcenter 35
pyobjc-framework-oslog 35
pyobjc-framework-photos 35
pyobjc-framework-photosui 35
pyobjc-framework-pushkit 35
pyobjc-framework-safariservices 35
pyobjc-framework-scenekit 35
pyobjc-framework-screensaver 35
pyobjc-framework-scriptingbridge 35
pyobjc-framework-securityinterface 35
pyobjc-framework-speech 35
pyobjc-framework-storekit 35
pyobjc-framework-syncservices 35
pyobjc-framework-systemconfiguration 35
pyobjc-framework-systemextensions 35
pyobjc-framework-usernotifications 35
pyobjc-framework-videotoolbox 35
pyobjc-framework-vision 35
pyobjc-framework-webkit 35
cao-lang 33
opencv-contrib-python-headless-rolling 33
opencv-python-headless-rolling 33
pyobjc-framework-accessibility 32
pyobjc-framework-automaticassessmentconfiguration 32
pyobjc-framework-classkit 32
pyobjc-framework-coremidi 32
pyobjc-framework-gamecontroller 32
pyobjc-framework-metalperformanceshaders 32
pyobjc-framework-passkit 32
pyobjc-framework-replaykit 32
pyobjc-framework-virtualization 32
babycat 30
chia-rs 30
fastbloom-rs 30
moteus-pi3hat 30
num-dual 30
pyqt6 30
pyqtwebengine 30
rust-regex 30
si-units 30
brotlicffi 29
sip 29
based58 28
bqskitrs 28
pantsbuild-pants 28
leechcorepyc 26
libtcod-cffi 26
skytools 26
prql-python 25
tdl 25
entab 24
pyfast 24
scalib 24
table-five 24
openlineage-sql 23
chdimage 22
sha512-crypt 22
akinator-py 21
mclbn256 21
pyqir-evaluator 21
pyqir-generator 21
pyqir-parser 21
wiring-rs 21
feos 20
ligo-skymap 20
milagro-bls-binding 20
peak-engines 20
primetools 20
pyqec 20
tornado 20
zarena 20
argon2-cffi-bindings 19
datafusion 19
libimagequant 19
pynacl 19
tangram 19
geo-rasterize 18
gstools-core 18
ncollpyde 18
nexxt 18
pyqt6-3d 18
pyqt6-networkauth 18
sdbus 18
cramjam 17
argon2-cffi 16
fugle-trade-core 16
pyzsf 16
dbt-extractor 15
pac-synth 15
pybip39 15
pygamemode 15
pyqt6-qscintilla 15
ryaml 15
scalene 15
pyobjc-framework-spritekit 14
snakefusion 14
umarkdown 14
pyqt6-charts 13
pyqt6-datavisualization 13
clarabel 12
eclipse-zenoh 12
fluvio 12
iota-wallet 12
morfeusz2 12
peppi-py 12
pyplanetarium 12
pyqt6-webengine 12
sunpy 12
tikv-client 12
temporalio 11
ddx-python 10
dualnum 10
h3ronpy 10
minilsap 10
monotrail 10
qsrs 10
redvox-native 10
whitebox-workflows 10
gaoya 9
gint 9
libertem-dectris 9
libsass 9
light-curve-python 9
mdf-iter 9
mitmproxy-wireguard 9
pep272-encryption 9
pygeohash-fast 9
ukkonen 9
brotlipy 8
btclib-libsecp256k1 8
fast-histogram 8
pyqtnetworkauth 8
dockerfile 7
hi-tension 7
marginpy 7
modelfox 7
pyhyphen 7
pyside6-addons 7
pyside6-essentials 7
setuptools-golang-examples 7
sf-heu 7
gosdt 6
momba-engine 6
nano-qmflows 6
python-bsonjs 6
raylib 6
regexp-sar 6
secure-context 6
ukkonen-rs 6
editdistance-s 5
ffilupa 5
protobufrex 5
pybundletool 5
rdp-rust 5
roaring-landmask 5
ceresdb-client 4
gdp 4
hicuml2v2p4p0-vae 4
lightstep-streaming 4
lz4-flex-py 4
mbf-bam 4
pnumpy 4
pydia2 4
pyffmpeg-bin 4
pyobjc-framework-qtkit 4
pyqtads 4
pyradamsa 4
pywpsrpc 4
pyxel 4
qcs-sdk-python 4
reddit-decider 4
virxerlu-rlib 4
adx-arrow 3
css-inline 3
diffdart 3
hicuml2v2p4p0-internal 3
iterframes 3
polars-lts-cpu 3
pybddisasm 3
pyobjc-framework-coremedia 3
pyobjc-framework-coretext 3
pyobjc-framework-security 3
pyomexmeta 3
quizdown 3
rambenchmark 3
raptorq 3
refl1d 3
rtea 3
safelife 3
sonicparanoid 3
synapse-auto-compressor 3
truckle 3
alouette 2
flashtextr 2
mwa-hyperbeam 2
pygfried 2
pyopenls9 2
pyroexr 2
qiniu-sdk-alpha 2
redb 2
replay-memory 2
ruuid 2
twmap 2
xaynet-sdk-python 2
yelp-cheetah 2
advent-of-code-2017-day-1 1
aymara 1
cryptacular 1
fast-jieba 1
hashers 1
hyo2-qc 1
ms-toollib 1
pahmm 1
pifacecam 1
procmaps 1
pybuild 1
pyrbtree 1
rust-demangle 1
sknrf-core 1
synapse-compress-state 1
ultrametric-matrix-tools 1
xxh3 1

For the pyobjc entries in this list: I use the limited ABI in a number of framework bindings to reduce the amount of files I upload to PyPI.

Using the limited ABI has no other advantages for me because the actual Python<->Objective-C bridge (pyobjc-core) cannot use the limited ABI. Because of this my build proces still builds all sub projects for all supported Python versions (times 1 OS and 1 CPU “architecture”) and just ends up with fewer wheels than I’d get by not using the limited ABI where possible.

A low priority goal for me is to move all bindings to the limited ABI, although I might end up with binary wheels that don’t contain C extensions at all, just shared libraries that don’t use the Python API at all.

Meta: you can wrap long tables etc. like that in a <details> tag, so the rest of us don’t have to scroll multiple screenfuls :wink:

<summary>List of modules</summary>

put the list of modules here


Thanks, I didn’t know about that!

Sadly it disables Markdown table formatting :slightly_frowning_face: I’ve used <details><pre> to hide the table, the formatting’s not as nice but it’s reasonable.

1 Like

This is an obscure detail about Markdown/Commonmark HTML blocks: the blank line after <summary>...</summary>, and the one before <details>, are important. Try with them?


The pyobjc and opencv projects illustrate what I think is an important use case for the Stable ABI: wrappers/bindings for projects written in other languages. Besides these, think of applications (think of a project like Blender, if it didn’t have the resources to maintain and bundle its own build of Python).

I’ll share some general thoughts about these. I only have anecdotal evidence for these thoughts, but hopefully they make sense:

  • with these kinds of projects, there’s a bigger chance that they don’t release on PyPI.
  • making these bindings easy to maintain is pretty important for Python to keep its status as a “glue language”
  • if Python support becomes a burden, these kinds of projects have the option to simply drop Python and move on
  • these projects frequently use binding generators (think: pybind11, PyO3, SWIG, JPype, and of course pyobjc). But established binding generators are quite complex, and thus it’s hard to add stable ABI support to them. And they also have some unique issues which are only now getting addressed – e.g. it’s hard to wrap classes from a foreign type system without a custom metaclass.
  • bindings to compiled languages don’t benefit too much from “faster CPython”-style improvements, and typically aren’t too sensitive to e.g. refcounting becoming slower

cffi-based projects is a good example of this – cffi generates abi3-compatible shared objects by default

1 Like

pyo3 also supports abi3 wheels: Building and distribution - PyO3 user guide


The second sentence is the key issue for not supporting the stable ABI in PyObjC: that ABI does not provide everything I need, and to be honest it probably shouldn’t.

For PyObjC I access some internals where the correct change is not to add accessors to the stable ABI, but to provide a better feature. PEP 447 is an attempt to do this for one example of this (mostly pining for the fjords due to lack of time on my end).

Another example is that I subclass PyUnicode_Type in C to add a new slot to the C struct (that isn’t a PyObject*). The root cause there is that C API doesn’t have a way to implement a type that can be used wherever the builtin str type is accepted.

Other than that I can probably transition to the limited API, although this is a significant amount of work and something I haven’t tried beyond translating some simple types from static types to heap types. What doesn’t help beyond the amount of work is that I don’t like the limited API for defining new types (but that’s for a different discussion)

BTW. The same is true for sub interpreter support: Other than [subinterpreters] Make the PyGILState API compatible with subinterpreters · Issue #59956 · python/cpython · GitHub supporting sub interpreters is mostly just a lot of grunt work (and some non-trivial design work because two sub interpreters could interact through the Objective-C side of things).

1 Like

I found the following just-appeared paper today: “An empirical study of the Python/C API on evolution and bug patterns”. It builds on an earlier publication " The Python/C API: Evolution, Usage Statistics, and Bug Patterns". I’m still in the process of studying the papers in full detail, but they appear quite relevant. The analysis code (and a pdf of the second paper) is in this repository. If somebody is interested and cannot get access to the PDFs, feel free to mail me.

Just to chime in as another package maintainer who’s using abi3 to reduce the size of his wheel build matrix: it’s a key part of being able to support more platforms than I’d otherwise be able to, and has saved me a handful of times from suddenly needing to test against a new CPython version.

That being said, I completely understand the need to make abi3-incompatible changes. As such, I think the proposal strikes a nice balance:

A minor downside to that is of course that abiN stops matching pythonN, but I’m not sure that was ever formally guaranteed to begin with. And a minor upside to that downside is that it gets over the emotional hump of numbers being important (versions should be cheap!)


Three possibilities:

  1. A representative attends a sprint for the other group.
  2. HPy rep attends/presents at the language summit at PyCon US.
  3. Schedule a meeting with the SC.