Let's get rid of the stable ABI, but keep the limited API

[steve-s (Stepan Sindelar)]

@h-vetinari

Without taking sides, it’s probably also frustrating to try to evolve python (a very important task, I think we can agree - the alternative is stasis and eventual decay), when so much that ended up being covered by an old promise that keeps getting extended makes that nigh-impossible.

the issue is that stable ABI promised stability while it did not really solve the reasons for instability: the need to evolve Python internals, because the ABI still exposes internal details.

This can go on and on in circles. The real solution is, IMHO, to have a different API (and really stable ABI) for extensions that abstracts and hides the implementation details and is different to what CPython uses to implement itself. How did Java or C# and others manage to innovate so much in last decades while keeping binary compatibility with existing extensions all that time?

[pf_moore (Paul Moore)]

@wjakob

ABI3 has just become good enough for many projects. For my projects, Python 3.12 will be the first release where the limited ABI is finally complete enough so that I can make the switch and avoid the fragility of so having many builds.

I’m confused. Stable ABI or limited API? How can the stable ABI have only just become good enough? The whole point is that it hasn’t changed… I’m guessing that other parts of the ecosystem (tooling, or the limited API, maybe?) have made it easier to build ABI3 binaries. That’s good - the stable ABI isn’t useful if people can’t use it easily - but it’s a different issue.

I don’t think we should remove the stable ABI. I think it’s clear from this discussion that it adds value for extension authors. But I am in favour of @tiran’s suggestion:

@tiran

We could consider to introduce a new stable ABI that would be valid for another 5 to 10 years.

I think that if the design of the stable ABI is limiting the evolution of Python, because it exposes too many internal details, then it should change. Improving performance and other internal enhancements are important work. They just shouldn’t happen without considering the impact on extension authors. So we have to balance - but “the stable ABI will never change” goes too far in one direction, just as “let’s get rid of the stable ABI” goes too far in the other direction.

[h-vetinari]

@steve-s

the issue is that stable ABI promised stability while it did not really solve the reasons for instability: the need to evolve Python internals, because the ABI still exposes internal details.

Fully agreed on that. The key point is that we stabilized something that wasn’t yet ready for the very very long haul (and that’s OK!).

@steve-s

The real solution is, IMHO, to have a different API (and really stable ABI) for extensions that abstracts and hides the implementation details and is different to what CPython uses to implement itself.

I didn’t state how long a putative abi4 should stay untouched, just that abi3 isn’t future-ready enough to not be changed anymore, ever.

[tiran (Christian Heimes)]

@pf_moore

I’m confused. Stable ABI or limited API? How can the stable ABI have only just become good enough? The whole point is that it hasn’t changed…

Stable ABI and limited API are intertwined. The stable ABI is stable, but not fixed. Minor releases of Python can add new features to the stable ABI. Therefore I see the stable ABI as an extension of the limited API.

An extension that uses limited API of Python 3.11 will also compiled with any Python version >= 3.11, < 4.0. An extension module compiled with limited API and stable ABI of Python 3.11 can be imported by Python >=3.11, < 4.0.

[steve-s (Stepan Sindelar)]

@pf_moore

but “the stable ABI will never change” goes too far in one direction

I assume that this is meant to be about the current stable ABI, but I want to note that in general this is doable and if there is ever some ABI4, this could/should be its goal. AFAIK, the JNI (java native interface) ABI hasn’t been broken since Java 6 or so and in the meanwhile Java added, for example, compressed object pointers or several new GC algorithms.

[hodgestar (Simon Cross)]

My hope/dream is that we will sit down as a group and figure out what kind of C ABI/API we would want if we were to start from scratch (I know HPy has been held up as that API, but I don’t think we have made that as a statement instead of an idea).

I (and I think most of the people involved with HPy) would love to sit down as a group too. How do we make this happen?

[asottile (Anthony Sottile)]

this list appears to be incomplete – or at least it is missing some of my packages – https://pypi.org/simple/ukkonen/ is one example https://pypi.org/simple/editdistance-s/ is another

EDIT: oh, didn’t realize this would end up at the end of the thread – just wanted to voice my concern in that abi3 finally made it it reasonable to release C extension wheels on pypi and to go back to the old “need to rebuild everything every time cpython releases” would be quite painful

[pf_moore (Paul Moore)]

@asottile

this list appears to be incomplete

Weird. I used a derived table, maybe the code I used to populate it was broken I’ve switched to an alternative DB schema that doesn’t need the separate table, so hopefully it was a one-off. Sorry about that.

[pf_moore (Paul Moore)]

Updated version (330 projects, 12523 files). Hopefully right this time!

Project listing

Project	N
eclipse-zenoh-nightly	1385
polars	755
advent-of-code	413
py-randomprime	307
anki	281
tcod	263
graphlib2	251
cryptography	248
glean-sdk	241
betfair-data	176
miguel-lib	161
routrie	136
rjsonnet	125
hdbcli	118
rfiletype	118
clvm-tools-rs	116
rjmespath	103
solders	103
fat-macho	98
rjieba	89
linguars	82
ensmallen	81
murmurhash2	80
pyffmpeg	76
rbcl	76
deltalake	72
pycryptodome	72
pyheck	72
iced-x86	71
pycryptodomex	71
qcustomplot-pyside2	70
pyoxigraph	68
dss-python	66
crfs	64
pyqt5	64
iota-client-python	60
polars-u64-idx	55
bcrypt	52
pyside2	52
nsmblib	51
reasonable	50
qscintilla	48
onigurumacffi	47
pyqtchart	44
qecstruct	44
watchfiles	44
memprocfs	42
narrow-down	42
nionui-tool	42
metadata-guardian	41
pillow-heif	41
pyside6	41
shiboken6	41
nionswift-tool	40
pyqt3d	40
pyqtdatavisualization	40
pyqtpurchasing	40
shiboken2	40
protobuf	39
portmod	38
bcl	37
nh3	37
graspologic-native	36
iota-client	36
opencv-contrib-python	36
opencv-contrib-python-headless	36
opencv-contrib-python-rolling	36
opencv-python	36
opencv-python-headless	36
opencv-python-rolling	36
adblock	35
clvm-rs	35
jpeglib	35
pyobjc-framework-addressbook	35
pyobjc-framework-authenticationservices	35
pyobjc-framework-avfoundation	35
pyobjc-framework-avkit	35
pyobjc-framework-cfnetwork	35
pyobjc-framework-contacts	35
pyobjc-framework-contactsui	35
pyobjc-framework-coreaudiokit	35
pyobjc-framework-corebluetooth	35
pyobjc-framework-coredata	35
pyobjc-framework-corelocation	35
pyobjc-framework-coremediaio	35
pyobjc-framework-coreml	35
pyobjc-framework-coreservices	35
pyobjc-framework-corespotlight	35
pyobjc-framework-corewlan	35
pyobjc-framework-cryptotokenkit	35
pyobjc-framework-discrecording	35
pyobjc-framework-externalaccessory	35
pyobjc-framework-fsevents	35
pyobjc-framework-gamecenter	35
pyobjc-framework-gamekit	35
pyobjc-framework-gameplaykit	35
pyobjc-framework-imagecapturecore	35
pyobjc-framework-imserviceplugin	35
pyobjc-framework-inputmethodkit	35
pyobjc-framework-intents	35
pyobjc-framework-mapkit	35
pyobjc-framework-mediatoolbox	35
pyobjc-framework-metal	35
pyobjc-framework-metalkit	35
pyobjc-framework-modelio	35
pyobjc-framework-multipeerconnectivity	35
pyobjc-framework-network	35
pyobjc-framework-networkextension	35
pyobjc-framework-notificationcenter	35
pyobjc-framework-oslog	35
pyobjc-framework-photos	35
pyobjc-framework-photosui	35
pyobjc-framework-pushkit	35
pyobjc-framework-safariservices	35
pyobjc-framework-scenekit	35
pyobjc-framework-screensaver	35
pyobjc-framework-scriptingbridge	35
pyobjc-framework-securityinterface	35
pyobjc-framework-speech	35
pyobjc-framework-storekit	35
pyobjc-framework-syncservices	35
pyobjc-framework-systemconfiguration	35
pyobjc-framework-systemextensions	35
pyobjc-framework-usernotifications	35
pyobjc-framework-videotoolbox	35
pyobjc-framework-vision	35
pyobjc-framework-webkit	35
cao-lang	33
opencv-contrib-python-headless-rolling	33
opencv-python-headless-rolling	33
pyobjc-framework-accessibility	32
pyobjc-framework-automaticassessmentconfiguration	32
pyobjc-framework-classkit	32
pyobjc-framework-coremidi	32
pyobjc-framework-gamecontroller	32
pyobjc-framework-metalperformanceshaders	32
pyobjc-framework-passkit	32
pyobjc-framework-replaykit	32
pyobjc-framework-virtualization	32
babycat	30
chia-rs	30
fastbloom-rs	30
moteus-pi3hat	30
num-dual	30
pyqt6	30
pyqtwebengine	30
rust-regex	30
si-units	30
brotlicffi	29
sip	29
based58	28
bqskitrs	28
pantsbuild-pants	28
leechcorepyc	26
libtcod-cffi	26
skytools	26
prql-python	25
tdl	25
entab	24
pyfast	24
scalib	24
table-five	24
openlineage-sql	23
chdimage	22
sha512-crypt	22
akinator-py	21
mclbn256	21
pyqir-evaluator	21
pyqir-generator	21
pyqir-parser	21
wiring-rs	21
feos	20
ligo-skymap	20
milagro-bls-binding	20
peak-engines	20
primetools	20
pyqec	20
tornado	20
zarena	20
argon2-cffi-bindings	19
datafusion	19
libimagequant	19
pynacl	19
tangram	19
geo-rasterize	18
gstools-core	18
ncollpyde	18
nexxt	18
pyqt6-3d	18
pyqt6-networkauth	18
sdbus	18
cramjam	17
argon2-cffi	16
fugle-trade-core	16
pyzsf	16
dbt-extractor	15
pac-synth	15
pybip39	15
pygamemode	15
pyqt6-qscintilla	15
ryaml	15
scalene	15
pyobjc-framework-spritekit	14
snakefusion	14
umarkdown	14
pyqt6-charts	13
pyqt6-datavisualization	13
clarabel	12
eclipse-zenoh	12
fluvio	12
iota-wallet	12
morfeusz2	12
peppi-py	12
pyplanetarium	12
pyqt6-webengine	12
sunpy	12
tikv-client	12
temporalio	11
ddx-python	10
dualnum	10
h3ronpy	10
minilsap	10
monotrail	10
qsrs	10
redvox-native	10
whitebox-workflows	10
gaoya	9
gint	9
libertem-dectris	9
libsass	9
light-curve-python	9
mdf-iter	9
mitmproxy-wireguard	9
pep272-encryption	9
pygeohash-fast	9
ukkonen	9
brotlipy	8
btclib-libsecp256k1	8
fast-histogram	8
pyqtnetworkauth	8
dockerfile	7
hi-tension	7
marginpy	7
modelfox	7
pyhyphen	7
pyside6-addons	7
pyside6-essentials	7
setuptools-golang-examples	7
sf-heu	7
gosdt	6
momba-engine	6
nano-qmflows	6
python-bsonjs	6
raylib	6
regexp-sar	6
secure-context	6
ukkonen-rs	6
editdistance-s	5
ffilupa	5
protobufrex	5
pybundletool	5
rdp-rust	5
roaring-landmask	5
ceresdb-client	4
gdp	4
hicuml2v2p4p0-vae	4
lightstep-streaming	4
lz4-flex-py	4
mbf-bam	4
pnumpy	4
pydia2	4
pyffmpeg-bin	4
pyobjc-framework-qtkit	4
pyqtads	4
pyradamsa	4
pywpsrpc	4
pyxel	4
qcs-sdk-python	4
reddit-decider	4
virxerlu-rlib	4
adx-arrow	3
css-inline	3
diffdart	3
hicuml2v2p4p0-internal	3
iterframes	3
polars-lts-cpu	3
pybddisasm	3
pyobjc-framework-coremedia	3
pyobjc-framework-coretext	3
pyobjc-framework-security	3
pyomexmeta	3
quizdown	3
rambenchmark	3
raptorq	3
refl1d	3
rtea	3
safelife	3
sonicparanoid	3
synapse-auto-compressor	3
truckle	3
alouette	2
flashtextr	2
mwa-hyperbeam	2
pygfried	2
pyopenls9	2
pyroexr	2
qiniu-sdk-alpha	2
redb	2
replay-memory	2
ruuid	2
twmap	2
xaynet-sdk-python	2
yelp-cheetah	2
advent-of-code-2017-day-1	1
aymara	1
cryptacular	1
fast-jieba	1
hashers	1
hyo2-qc	1
ms-toollib	1
pahmm	1
pifacecam	1
procmaps	1
pybuild	1
pyrbtree	1
rust-demangle	1
sknrf-core	1
synapse-compress-state	1
ultrametric-matrix-tools	1
xxh3	1

[ronaldoussoren]

For the pyobjc entries in this list: I use the limited ABI in a number of framework bindings to reduce the amount of files I upload to PyPI.

Using the limited ABI has no other advantages for me because the actual Python<->Objective-C bridge (pyobjc-core) cannot use the limited ABI. Because of this my build proces still builds all sub projects for all supported Python versions (times 1 OS and 1 CPU “architecture”) and just ends up with fewer wheels than I’d get by not using the limited ABI where possible.

A low priority goal for me is to move all bindings to the limited ABI, although I might end up with binary wheels that don’t contain C extensions at all, just shared libraries that don’t use the Python API at all.

[erlendaasland (Erlend E. Aasland)]

Meta: you can wrap long tables etc. like that in a <details> tag, so the rest of us don’t have to scroll multiple screenfuls

<details>
<summary>List of modules</summary>

put the list of modules here

</details>

[pf_moore (Paul Moore)]

Thanks, I didn’t know about that!

Sadly it disables Markdown table formatting I’ve used <details><pre> to hide the table, the formatting’s not as nice but it’s reasonable.

[encukou (Petr Viktorin)]

This is an obscure detail about Markdown/Commonmark HTML blocks: the blank line after <summary>...</summary>, and the one before <details>, are important. Try with them?

[encukou (Petr Viktorin)]

The pyobjc and opencv projects illustrate what I think is an important use case for the Stable ABI: wrappers/bindings for projects written in other languages. Besides these, think of applications (think of a project like Blender, if it didn’t have the resources to maintain and bundle its own build of Python).

I’ll share some general thoughts about these. I only have anecdotal evidence for these thoughts, but hopefully they make sense:

• with these kinds of projects, there’s a bigger chance that they don’t release on PyPI.
• making these bindings easy to maintain is pretty important for Python to keep its status as a “glue language”
• if Python support becomes a burden, these kinds of projects have the option to simply drop Python and move on
• these projects frequently use binding generators (think: pybind11, PyO3, SWIG, JPype, and of course pyobjc). But established binding generators are quite complex, and thus it’s hard to add stable ABI support to them. And they also have some unique issues which are only now getting addressed – e.g. it’s hard to wrap classes from a foreign type system without a custom metaclass.
• bindings to compiled languages don’t benefit too much from “faster CPython”-style improvements, and typically aren’t too sensitive to e.g. refcounting becoming slower

[asottile (Anthony Sottile)]

cffi-based projects is a good example of this – cffi generates abi3-compatible shared objects by default

[alex_Gaynor]

pyo3 also supports abi3 wheels: Building and distribution - PyO3 user guide

[ronaldoussoren]

@encukou

But established binding generators are quite complex, and thus it’s hard to add stable ABI support to them. And they also have some unique issues which are only now getting addressed – e.g. it’s hard to wrap classes from a foreign type system without a custom metaclass .

The second sentence is the key issue for not supporting the stable ABI in PyObjC: that ABI does not provide everything I need, and to be honest it probably shouldn’t.

For PyObjC I access some internals where the correct change is not to add accessors to the stable ABI, but to provide a better feature. PEP 447 is an attempt to do this for one example of this (mostly pining for the fjords due to lack of time on my end).

Another example is that I subclass PyUnicode_Type in C to add a new slot to the C struct (that isn’t a PyObject*). The root cause there is that C API doesn’t have a way to implement a type that can be used wherever the builtin str type is accepted.

Other than that I can probably transition to the limited API, although this is a significant amount of work and something I haven’t tried beyond translating some simple types from static types to heap types. What doesn’t help beyond the amount of work is that I don’t like the limited API for defining new types (but that’s for a different discussion)

BTW. The same is true for sub interpreter support: Other than [subinterpreters] Make the PyGILState API compatible with subinterpreters · Issue #59956 · python/cpython · GitHub supporting sub interpreters is mostly just a lot of grunt work (and some non-trivial design work because two sub interpreters could interact through the Objective-C side of things).

[cfbolz (Carl Friedrich Bolz-Tereick)]

I found the following just-appeared paper today: “An empirical study of the Python/C API on evolution and bug patterns”. It builds on an earlier publication " The Python/C API: Evolution, Usage Statistics, and Bug Patterns". I’m still in the process of studying the papers in full detail, but they appear quite relevant. The analysis code (and a pdf of the second paper) is in this repository. If somebody is interested and cannot get access to the PDFs, feel free to mail me.

[woodruffw (William Woodruff)]

Just to chime in as another package maintainer who’s using abi3 to reduce the size of his wheel build matrix: it’s a key part of being able to support more platforms than I’d otherwise be able to, and has saved me a handful of times from suddenly needing to test against a new CPython version.

That being said, I completely understand the need to make abi3-incompatible changes. As such, I think the proposal strikes a nice balance:

@encukou

Both issues you talk about could be solved by a one-time compatibility break (an abi4) rather than removing stable ABI entirely. Specifically: to make those changes, the remaining runtime structs (PyObject & PyVarObject) would need to become opaque.

A minor downside to that is of course that abiN stops matching pythonN, but I’m not sure that was ever formally guaranteed to begin with. And a minor upside to that downside is that it gets over the emotional hump of numbers being important (versions should be cheap!)

[brettcannon]

@hodgestar

I (and I think most of the people involved with HPy) would love to sit down as a group too. How do we make this happen?

Three possibilities:

1. A representative attends a sprint for the other group.
2. HPy rep attends/presents at the language summit at PyCon US.
3. Schedule a meeting with the SC.