Use the limited C API for some of our stdlib C extensions

Core Development
1

Hi,

tl; dr: Would you be ok to start using the limited C API in Python stdlib?

I propose to convert some stdlib C extensions to the limited C API to eat our own dog food, embrace and test this API. I would like to promote the limited C API and the stable ABI: using the limited C API ourself is a first step. Using private C API functions and the internal C API should be the exception, not the default in Python stdlib; see also C API: My plan to clarify private vs public functions in Python 3.13 . Performance remains a good reason to use the internal C API, but performance is not strictly needed for the 115 stdlib extensions (104 extensions + 11 test extensions).

The stable ABI makes the distribution of package binaries easier. For example, binaries are already available before the new Python is being released! It makes newer Python usable since the first day of its release, because it’s simply the same binary for all Python versions. (One binary per platform+architecture is still needed.)

In 2009, Martin v. Löwis created a subset of the C API called the “Limited C API” in PEP 384 – Defining a Stable ABI used when the Py_LIMITED_API macro is defined (it can be set to a Python version). If you limit your code to this API and build your C extension in a specific way, you can use the “stable ABI” which works on Python 3.2 and newer without having to rebuild Python.

The stable ABI usage remains low: around 551 PyPI packages provide abi3 binaries. The two most popular users are PySide and cryptography (4.37K packages and 16.9K repositories depend on cryptography). Last year, Paul Moore (@pf_moore) ran a search on all PyPI packages providing binaries built for the stable ABI. See also the interesting discussion Let’s get rid of the stable ABI, but keep the limited API which discuss how and why it’s being used currently (for now, there is no plan to remove the stable ABI).

In my experience, a practical problem of the limited of the C API and the stable ABI is that it’s not used by Python itself, and so issues are discovered too late (missing important function, broken implementation, regression, etc.). Petr Viktorin (@encukou) made a great progress in Python 3.10 with PEP 652 – Maintaining the Stable ABI which added Misc/stable_abi.toml, test_stable_abi_types, and way more cool stuff.

I propose to start eating our own dog food by converting some stdlib C extensions to the limited C API. In some cases (a few extensions at least), the change is just about adding a single line at the top of the file!

#define Py_LIMITED_API 0x030d0000

I did some tests and so far, I saw the two following issues.

Function missing in the limited C API like PySys_Audit(). We should look at missing functions on a case by case basis to see if it’s worth it to add the function, or if using the Python API in C (PyImport_ImportModule(), PyObject_GetAttrString(), PyObject_CallFunction(), …) can be acceptable (performance?).

Performance regression: most efficient code is hidden in the internal C API, the limited C API can be slower. My PR for _statistics made _statistics._normal_dist_inv_cdf() made the function 2x slower because of the usage of the legacy METH_VARARGS calling convention (instead of more recent METH_FASTCALL which avoids creating a temporary tuple to pass positional arguments, and a temporary dict to pass keyword arguments). There is room for improvement, Argument Clinic and the limited C API can be enhance to get better performance.

Many C extensions can be converted to the limited C API without any change, or only with minor changes, and no significant impact on performance. Moreover, many C extensions are not performance sensitive. For example, the pwd and grp extensions which just expose C functions in Python, I don’t expect them to be part of hot code in any application.

In 2020, I already tried to convert some stdlib C extensions to the limited C API, but I was blocked by multiple issues:

  • Argument Clinic didn’t support the limited C API
  • The limited C API missed some multiple important features

Good news: Argument Clinic just got support for the limited C API! And the important features were added to the limited C API in the meanwhile! See C API: Convert a few stdlib extensions to the limited C API (PEP 384) for the details.

A lot of work has been put over last years in converting extensions static types to heap types. This work is non trivial and is a starting point for the conversion to the limited C API, since this API doesn’t support static types.

Victor

9 Likes
2

Running this query on Release 2023-08-03 · sethmlarson/pypi-data · GitHub finds 551 packages with abi3 wheels:

$ sqlite3 'pypi.db' "SELECT DISTINCT package_name FROM wheels WHERE abi = 'abi3';"
acquire-imaging
adblock
adrt
advent-of-code-2017-day-1
advent-of-code
adx-arrow
aioquic
akinator-py
alouette
anchorpy-core
anki
argon2-cffi-bindings
arrow-json
arthseg
aspeak
atomic-counter
atomicx
autocorrect-py
automerge
awscrt
axio-cli
aymara
babycat
base2048
based58
bcl
bcrypt
belinda
betfair-data
biodivine-aeon
bliss-audio
bobzillapypi
bqskitrs
br-navigator
brotlicffi
brotlipy
can-message-data-generator
cao-lang
capcruncher-tools
casa-formats-io
ceresdb-client
cev-metrics
chdimage
chidori
chinillaclvm-rs
chinillaclvm-tools-rs
clarabel
cloudproof-aesgcm
cloudproof-anonymization
cloudproof-ecies
cloudproof-fpe
clvm-tools-rs
clvm-rs
componentize-py
concept-x-converter
contamxpy
cosmian-kms
cover-crypt
cozo-embedded
cpr-gym
crfs
cryptography
css-inline
cssfinder-backend-rust
curl-cffi
cvldoc-parser
dantzig
databend
datafusion
dbscan
dbt-extractor
ddx-python
deltalake
depthai-viewer
devsim
didppy
diro-py
dockerfile
dora-ros2-bridge
dora-rs
dss-python-backend
dualnum
eclipse-zenoh
eclipse-zenoh-flow
eclipse-zenoh-nightly
edit-anki
editdistance-s
ensmallen
entab
epdx
evtx
ezkl-lib
ezkl
fast-alphashape
fast-geodist
fast-histogram
fast-query-parsers
fastbloom-rs
fastcrc
fasttextaug
fat-macho
feos
finance-structs
findex
flashtextr
fluvio
frida
fugle-trade-core
fusion-blossom
gaoya
gdp
gef-file-to-map
genomedata
geo-rasterize
geopolars
getdaft
gint
gl-client-py
gl-client
glaredb
glazari-test
glean-sdk
gosdt
gpu-tracking
gpu-tracking-python
gpu-tracking-app
graph-mate
graphlib2
graspologic-native
gstools-core
guasca
h3ronpy
hashers
hat-drivers
hat-event
hat-sbs
hdbcli
hi-tension
hicuml2v2p4p0-vae
hicuml2v2p4p0-internal
hidefix
histongram
home-assistant-chip-core
httparse
hydro-deploy
hyperparameter
hyperqueue
iced-x86
ichika
igraph
imrc
inflatox
iota-client-python
iterframes
jcan
jpeglib
jxml
klvm-tools-rs
klvm-rs
kurbopy
lakeapi2sql
lebai-sdk
leechcorepyc
leidenalg
libertem-asi-tpx3
libertem-dectris
libimagequant
libredr
libsass
libsass-bin
light-curve
lightstep-streaming
ligo-skymap
linguars
linkspace
lively-tk
llm-rs
llm-rs-metal
llm-rs-opencl
ltp-extension
lummao
lz4-flex-py
lzallright
maptide
marginpy
matrix-http-rendezvous-synapse
matrix-synapse
mcalf
mdf-iter
memflow
memprocfs
merlon
metadata-guardian
metlo-python-agent-bindings-common
metricspace
mft
miguel-lib
milagro-bls-binding
mini-groove
minijinja
minilsap
minimappers2
minishogilib
mitmproxy-wireguard
mitmproxy-rs
modelfox
momba-engine
monotrail
morfeusz2
moteus-pi3hat
mqt-core
mrzerocore
ms-toollib
mulder
murmurhash2
mwps
nano-qmflows
nanolsap
narrow-down
ncollpyde
netifaces2
nexxt
ngrok
nh3
nionswift-tool
nionui-tool
nonebot-adapter-walleq
nsmblib
num-dual
nutils-poly
object-store-python
onigurumacffi
opencv-contrib-cuda-python
opencv-contrib-python
opencv-contrib-python-headless
opencv-contrib-python-rolling
opencv-contrib-python-headless-rolling
opencv-python
opencv-python-headless
opencv-python-headless-rolling
opencv-python-rolling
opendal
openlineage-sql
pac-synth
pahmm
pathway
payment-order-renderer
peak-engines
pep272-encryption
pep440-rs
pep508-rs
peppi-py
perpetuo
pgpq
pifacecam
placeholder
pnumpy
polars
polars-lts-cpu
polars-u64-idx
portmod
pqp
prelude-parser
primetools
procmaps
protobuf
protobufrex
protosaurus
prql-python
psutil
psychtoolbox
py-horned-owl
py-industrial-robots
py-near-primitives
py-randomprime
pyattimo
pybip39
pybuild
pybundletool
pycryptodome
pycryptodomex
pydds
pydia2
pydracula
pyextrasafe
pyfast
pyffmpeg-bin
pygamemode
pygeohash-fast
pygfried
pyguess
pyhaloxml
pyheck
pyhyphen
pykeyset
pylance
pynacl
pyobjc-framework-accessibility
pyobjc-framework-addressbook
pyobjc-framework-automaticassessmentconfiguration
pyobjc-framework-authenticationservices
pyobjc-framework-avfoundation
pyobjc-framework-avrouting
pyobjc-framework-avkit
pyobjc-framework-backgroundassets
pyobjc-framework-cfnetwork
pyobjc-framework-classkit
pyobjc-framework-contacts
pyobjc-framework-contactsui
pyobjc-framework-coreaudiokit
pyobjc-framework-corebluetooth
pyobjc-framework-coredata
pyobjc-framework-corelocation
pyobjc-framework-coreml
pyobjc-framework-coremediaio
pyobjc-framework-corespotlight
pyobjc-framework-corewlan
pyobjc-framework-coreservices
pyobjc-framework-coremidi
pyobjc-framework-cryptotokenkit
pyobjc-framework-discrecording
pyobjc-framework-externalaccessory
pyobjc-framework-extensionkit
pyobjc-framework-gamecontroller
pyobjc-framework-gamecenter
pyobjc-framework-fsevents
pyobjc-framework-imagecapturecore
pyobjc-framework-gameplaykit
pyobjc-framework-gamekit
pyobjc-framework-healthkit
pyobjc-framework-imserviceplugin
pyobjc-framework-inputmethodkit
pyobjc-framework-intents
pyobjc-framework-iobluetooth
pyobjc-framework-mapkit
pyobjc-framework-mediatoolbox
pyobjc-framework-metal
pyobjc-framework-metalkit
pyobjc-framework-metalperformanceshaders
pyobjc-framework-metalfx
pyobjc-framework-multipeerconnectivity
pyobjc-framework-modelio
pyobjc-framework-notificationcenter
pyobjc-framework-networkextension
pyobjc-framework-network
pyobjc-framework-oslog
pyobjc-framework-passkit
pyobjc-framework-photos
pyobjc-framework-pushkit
pyobjc-framework-photosui
pyobjc-framework-qtkit
pyobjc-framework-safariservices
pyobjc-framework-replaykit
pyobjc-framework-scenekit
pyobjc-framework-screensaver
pyobjc-framework-safetykit
pyobjc-framework-scriptingbridge
pyobjc-framework-securityinterface
pyobjc-framework-sharedwithyou
pyobjc-framework-sharedwithyoucore
pyobjc-framework-storekit
pyobjc-framework-speech
pyobjc-framework-systemconfiguration
pyobjc-framework-syncservices
pyobjc-framework-systemextensions
pyobjc-framework-usernotifications
pyobjc-framework-videotoolbox
pyobjc-framework-virtualization
pyobjc-framework-webkit
pyobjc-framework-vision
pyonear
pyooz
pyopenls9
pyoxigraph
pypcode
pyperscan
pypicorom
pyplanetarium
pyqec
pyqir
pyqrlew
pyqt3d
pyqt5
pyqt6-3d
pyqt6
pyqt6-charts
pyqt6-datavisualization
pyqt6-networkauth
pyqt6-qscintilla
pyqt6-webengine
pyqtads
pyqtchart
pyqtdatavisualization
pyqtnetworkauth
pyqtpurchasing
pyqtwebengine
pyradamsa
pyrage
pyrbtree
pyroexr
pyside2
pyside6
pyside6-essentials
pyside6-addons
pyside6-qtads
pystval
python-bsonjs
pytinysoundfont
pytokei
pywaterflood
pyxel
pyxet
pyzsf
qcustomplot-pyside2
qecstruct
qecp
qh3
qiniu-sdk-alpha
qiskit-terra
qscintilla
qsharp-python
qsrs
quizdown
rambenchmark
raptorq
ratio-genetic-py
raysql
rbcl
rbloom
rdp-rust
reasonable
redb
reddit-decider
redvox-native
regexp-sar
regress
replay-memory
rerun-sdk
rfiletype
ril
rjieba
rjmespath
rjsonnet
rlviser-py
roaring-landmask
rocketsim
routrie
rpi-derive-key
rqrcode
rs-tabler
rs2py
rtea
rubelp
ruspy
rust-demangle
rust-neighborlist
rust-regex
rustgrok
rustquant
rustregression
ruuid
ryaml
safelife
scalib
sdbus
secure-context
self-limiters
semantic-text-splitter
sequence-align
setuptools-golang-examples
sgx-pck-extension
sha512-crypt
shiboken6
shiboken2
si-units
sigalign
singlestoredb
sip
snakefusion
solders
solpos
spiff-element-units
ssmv
steptools
stralgo
stretchable
substring-match
sudokutils
sunpy
symagen
symdiff
table-fifth
table-five
tangram
taos-ws-py
tcod
teapy
temporalio
temporalio-arta
text-correction-utils
texy
thumbor
tikv-client
timeblok-py
timescope
tongsuopy
tongsuopy-crayon
tornado
transports
tree-sitter-pymanifest
tree-sitter-requirements
treefarms
tsv2py
turnip-text
twmap
typst
ukkonen
ukkonen-rs
ultibi
ultrametric-matrix-tools
umarkdown
unblob-native
unique-with-tol
uplift-kit
uuidt
viazoom
vidyut
village-temporalio
virxerlu-rlib
vkgdr
vl-convert-python
vlmc
voronoiville
vybe-solana-helpers
walt-vpn
wasmpy
watchfiles
watchgraf
whitebox-workflows
windows-control
wiring-rs
wonnx
xaynet-sdk-python
xmodits-py
xpmir-rust
xxh3
yelp-cheetah
ytsaurus-yson
ytsaurus-rpc-driver
zarena
zenithta
1 Like
3

I like this proposal both to EOODF and to debug problems with the limited API early. Doing so also provides a good example for other extension module authors, and maybe even makes a potential future splitting up of the stdlib easier [1].

Is there a PEP or other document about the criteria for what goes in the limited C API or not? I’m not looking for a list, but just a set of general principles that can be applied when asking what should get added, or even what should get removed?

  1. because PyPI stdlib extension modules could publish abi3 binaries ↩︎

4 Likes
4

Well, the limited C API should be… limited. I considered adding PyObject_CallOneArg(), but IMO it’s not worth it: existing PyObject_CallFunctionObjArgs() can be used instead (no major perf difference expected, whereas PyObject_CallNoArgs() has a nice optimization for stack memorty usage).

In the past, we removed functions which were broken or private from the limited C API:

  • 3.11:

    • PyUnicode_CHECK_INTERNED() – was broken
    • PyWeakref_GET_OBJECT() – was broken
    • PyMarshal_WriteLongToFile() – use FILE*
    • PyMarshal_WriteObjectToFile() – use FILE*
    • PyMarshal_ReadObjectFromString() – use FILE*
    • PyMarshal_WriteObjectToString() – use FILE*

  • 3.10:

    • PyOS_ReadlineFunctionPointer() – use FILE*

  • 3.9:

    • PyFPE_START_PROTECT(), PyFPE_END_PROTECT()
    • PyThreadState_DeleteCurrent()
    • _Py_CheckRecursionLimit
    • _Py_NewReference()
    • _Py_ForgetReference()
    • _PyTraceMalloc_NewReference()
    • _Py_GetRefTotal()
    • The trashcan mechanism which never worked in the limited C API.
    • PyTrash_UNWIND_LEVEL
    • Py_TRASHCAN_BEGIN_CONDITION – trashcan never worked in the limited C API
    • Py_TRASHCAN_BEGIN
    • Py_TRASHCAN_END
    • Py_TRASHCAN_SAFE_BEGIN
    • Py_TRASHCAN_SAFE_END

Recently (Python 3.13), I added PyLong_AsInt() to the limited C API. To convert some C extensions to the limited C API, I would like to add the following functions (see PRs attached to the issue):

  • PyMem_RawMalloc()
  • PySys_Audit()
  • PyInterpreterState_IsMain() – new function

History of main Limited C API changes:

  • 3.12: Py_INCREF() and Py_DECREF() are now implemented as opaque function calls (!) in the limited C API version 3.12 and newer (see the discussion).

  • 3.11: for limited C API version 3.11 and newer, <stdlib.h>, <stdio.h>, <errno.h> and <string.h> are no longer included by <Python.h>.

  • 3.10: if Python is built in debug mode (Py_DEBUG), the limited C API is now supported, and Py_INCREF() and Py_DECREF() are implemented as opaque function calls.

  • 3.9:

    • PyObject_INIT() and PyObject_INIT_VAR() are implemented as opaque function calls
    • fix Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() (was broken before), implement them as opaque function calls.
3 Likes
5

I like the idea of eating our own dog food. I’m also the author of several of those packages that use abi3 wheels, so I have a strong interst in the limited API becoming better :slight_smile:

I’m also sympathetic to the people who will say, “eating our own dog food isn’t a good enough reason to lose performance”, so I think it would be a very good outcome of this process if, wherever we identify areas for improvement by eating our own dog food, we make the dog food taste better.

8 Likes
6

I worry that this is going to add too much churn for the team. What happened to “if it isn’t broken, don’t fix it”? It’s fine of course to experiment without committing PRs – like when you found that _statistics was significantly slowed down. But in general I worry that this is just going to create a stream of PRs that have low priority, that few people care to review, and that will increase everybody’s frustration (not just yours) with how hard it is to get people to review PRs.

“Eat your own dogfood” is a fine idea, and I think it’s great to apply it to new modules. Just like we sometimes add annotations to new code, despite our general reluctance to add annotations to existing code (especially stdlib code). I feel the same ought to apply here: let’s not try to “fix” existing modules, because they aren’t broken, and ultimately there is no reason for the stdlib to use the limited API.

Related, what’s the point of making Argument Clinit support the Limited API? I thought it was an internal tool that we’re not maintaining for 3rd parties. Was there a discussion about a change of course somewhere?

3 Likes
7

It’s still internal. I think the support is simply necessary for this idea to work.

8

The cart seems to be going before the horse then. The work on AC was already done before the desirability of using the Limited API in stdlib modules was discussed.

1 Like
9

Some people expressed interest to use Argument Clinic outside CPython. It generates a signature useful for inspect.signature() introspection. It generates efficient code to parse function arguments. It would be nice to make use usable outside Python internals, but using the internal C API outside Python is not a good idea: the limited C API is a better target.

The work started in 2018 (without explicitly adding #define Py_LIMITED_API) and discussions started in 2020:

10

Victor, I’m sorry, but your long list of issues, proposals and commits just leaves me confused. What are you trying to argue exactly?

Most of those don’t seem to be about AC (Argument Clinic) at all. Looking at some samples, what is the point of making stdlib modules PEP 384 compatible? The argument seems to be “dogfooding is good” and possibly that stdlib modules are used widely as “example code” so best practices should be followed? Those aren’t technical reasons though – IMO this smells like technical solutions for social problems.

Compare this to the changes to switch modules to multi-phase init (PEP 489). Those are also largely mechanical changes that excite few people (and sometimes are having trouble getting reviewers). But the difference is that those serve a clear technical goal: to be able to safely import those modules in subinterpreters (PEP 684). Therefore I support that work. In comparison, I cannot support work to make stdlib modules use the Limited API, or work to make Argument Clinit emit code that supports it. It feels like churn, which I find unnecessarily risky (like that infamous change by Eric Raymond, over 20 years ago, to change the entire stdlib to use some newly introduced string operation that introduced some subtle bugs).

11

I feel that it is not fair to use more efficient and convenient private API while the authors of third-party extensions are left with slower and more limited API. And some things cannot be performed with only public API at all. Of course we cannot add all experimental and unstable functions to the public API, but after some period of use and stabilization we can provide public versions, even if with several checks and limitations.

For positional-only parameters the user can use METH_VARARGS and PyArg_ParseTuple(), PyArg_UnpackTuple() or simply PyTuple_Size() + PyTuple_GetItem(). Or they can use more efficient METH_FASTCALL, and then they do not need tuple unpacking API, but they do not have a convenient function like PyArg_ParseTuple() – converting every argument individually is more flexible and efficient, but more verbose.

But for keyword-only parameters the user only have PyArg_ParseTupleAndKeywords() which is very slow (it creates a new Python string object for every parameter name every time the function is called). Nothing can be used with METH_FASTCALL|METH_KEYWORDS. And METH_METHOD can only combine with METH_FASTCALL.

If we do not use the limited C API, we cannot know what is needed to be included. I do not think that we should make significant number of extensions to use the limited C API. I hope Victor did not mean this. But 5-10 small extensions can be used to ensure that the limited C API is adequate to modern standards and be an example for third-party extensions.

As for Argument Clinic, first, it needs to support the limited C API because most of extension modules use it, so it needs to support it if we want to use the limited C API in these extensions. Second, Argument Clinic is a complex piece of software, and it became even more complex after adding many features by different people. I suppose that nobody completely understand it (I do not). It needs to evolve if we do not want to be left without a maintainer. It is not completely tested, the main test is its use in the CPython code, but it does not cover all possible combinations. Adding support of the limited C API and testing it in different modes helps to rewrite it in more uniform and general way. I found several minor bugs in the current code during this rewriting.

5 Likes
12

Most of the slow public API is just the fast private API with checks and limitations. It’ll never be fast enough for those who complain about it.

I don’t see any issue with us using internal APIs, since we’re internal! If we’re migrating our extension modules to the limited API because we’re planning to split them out of the stdlib and distribute them independently then I’m fine with it, because that (a) implies the module doesn’t depend on our internals (except for perf) and (b) gets us to a more maintainable core.

Simply expanding the number of APIs we have to support in case someone out there cares about a few microseconds and not having to recompile is us taking on the wrong burden, IMHO. We’d be better focused on finding ways to preserve public API compatibility in all our changes (which, anyway, is the practical impact of adding things to the limited API).

3 Likes
13

I think stdlib should use limited API, but compiled for performance. Costly stuff (checks, limitations, slower implementations that preserve ABI compatibility) should only be present on special/debug builds.

Currently, the options to control what the API compiles to are rather crude. Py_LIMITED_API both limits the API (hides some definitions), and makes extensions compile to the slower stable ABI.

2 Likes
14

I think that there is some misunderstanding about my remarks on performance. What I mean that stdlib extensions which would have worse performance using limited C API will not be converted to the limited C API, especially the extensions known to be useful for performance (like the _statistics extension).

For example, the _stat, _scproxy, _testmultiphase and _uuid and extensions don’t use any internal C API and they don’t use Argument Clinic. Using the limited C API on them has no impact performance.

I propose to start by converting the _stat extension: see my PR. The change just adds a single line, that’s all:

diff --git a/Modules/_stat.c b/Modules/_stat.c
index 6cea26175d..230acd2b00 100644
--- a/Modules/_stat.c
+++ b/Modules/_stat.c
@@ -11,6 +11,9 @@
  *
  */
 
+// Need limited C API version 3.13 for PyModule_Add() on Windows
+#define Py_LIMITED_API 0x030d0000
+
 #include "Python.h"
 
 #ifdef __cplusplus
15

The technical issue is that the limited C API is badly tested by Python itself. In the past, we introduced bugs which were only discovered too late, after a release. Testing means building a whole C extension with the limited C API, but also execute the code by running tests on it. Currently, the xxlimited module is tested but it’s quite artificial and only covers a small part of the API.

Moreover, there are some functions which are missing in the limited C API which is non-obvious when you only consider trivial C extensions. Converting some stdlib extensions help to discover missing functions, that’s how I come up with proposing to add PySys_Audit(), PyMem_RawMalloc() and PyInterpreterState_IsMain() functions.

16

One issue that I have is that it’s not easy to distinguish the limited C API from the “regular” C API. In Python 3.12, many private functions are accessible in the regular C API, whereas they are not part of the limited C API.

Do you use the internal C API on purpose, or by mistake? It’s not always obvious.

That’s why I’m working on cleaning the C API to try to more clearly separate the “clean API” aka “Limited C API” (without implementation details, portable, stable) and the “dirty API” aka “regular C API” or “internal C API” (internal C API with implementation details, structure members, private functions, no backward compatibility).

The separation between the “regular API” and the “internal C API” is blurry in CPython (code of stdlib extensions). Sometimes you can get some internal C API from a header file without knowing that it’s an internal C API.

17

FWIW, I don’t think it’s a good idea to rush adding functions to the limited API only because a stdlib extension uses them. That way we’ll miss chances to learn from any past mistakes, and perhaps encourage use cases and assumptions that we want to drop in the future: PyMem_RawMalloc is like PyMem_Malloc but you don’t need to hold the GIL; PyInterpreterState_IsMain assumes there is a privileged main interpreter.

18

All in all, a good discussion. I recommend not hurrying with actions, this may be important to some folks, but it doesn’t seem urgent to anyone. Let’s wait until we’ve had some C API discussions among those core devs who make it to Brno in October.

Separately, do we have documentation somewhere of all the various C API layers? I feel it would maybe belong in Extending and Embedding the Python Interpreter — Python 3.11.5 documentation but I don’t see anything in the index that seems to cover this topic.

I’ve found at least the following categories – did I miss or misrepresent any?

  • Internal API – Lives in Include/internal/pycore_*.h, requires defining Py_BUILD_CORE before including. Is typically included by adding Include/internal to the C compiler’s include search path.
  • CPython API – Some kind of public API that’s specific to CPython. Lives in Include/cpython. Don’t include these directly, you get them automatically when you include the corresponding file in Include unless you define Py_LIMITED_API. (Most of these are automatically included by Python.h, but a few aren’t.)
  • Limited API – What you get from #include "Python.h" if you first define Py_LIMITED_API. This specifically excludes everything in Include/cpython.
  • Stable ABI (note “B” for “Binary”) – ABI that CPython promises to be stable across releases (there must be an expiration but it’s unclear what that is if we never move to Python 4.x). It consists of the Limited API. There are versions you can select by setting Py_LIMITED_API to specific values. See also PEP 384 and PEP 652.
  • Private API – Legacy APIs that are currently exported by the limited and/or CPython API but are not intended for public use nor have guarantees about ABI stability. These can be recognized by a name starting with _ (usually _Py) and being undocumented, but occasionally an apparent private API is treated as a public API because we have found that it’s commonly used by 3rd party packages and we don’t want to break them.
  • Unstable API – A recent development, see PEP 689. These are public APIs that are useful for some category of 3rd party tools (like debuggers, profilers) but that for various reasons we don’t want to have to guarantee across minor versions. Their name must start with PyUnstable_. Users of these APIs may have to rewrite their code for each new minor release (though we will try not to change these unnecessarily).
5 Likes
19

Yes, we have user docs and dev docs.

Spot on, except what you call internal and private are pretty much the same thing.
The details that separate them (whether someone considers them “legacy”, whether they’re documented, whether they’re behind Py_BUILD_CORE…) aren’t aligned with each other, so the two don’t work well as separate categories. BTW, lately Victor has been moving everything from “private” to “internal” (using your terms), so if his plan somehow works there won’t be any more of these distinctions.
Anyway, this category also includes implementation details that need to be exposed to the compiler or linker but shouldn’t be used directly, e.g. a function that implements the slow path of the DECREF macro.

20

The current count appears to 648.
That’s 0.138% of all PyPI projects (466520 at the moment) or
around 5.6% of all projects which publish binary wheel files on PyPI (11528 at the moment).

The stable ABI has been around for around 14 years now and pick up by the community is very very slow. Shouldn’t this tell us something ?

Before spending more time and effort on making stdlib extensions use the limited API (which then results in the extension being compatible to the stable ABI), I think we ought to ask ourselves whether this additional dimension of complexity in the Python C API is really something we want to keep maintaining for many many years to come.

You keep saying that the Python C API is hard to evolve due to the backwards compatibility requirements. The stable ABI / limited API makes this even harder, so I don’t follow your reasoning.

Given the low adoption rate, it would make much more sense, to drop the stable ABI / limited API and put the freed effort into the compatibility tooling you have started.

Projects could then still use older APIs with newer Python versions, so wouldn’t need to be updated with every new Python version. The only change would be to recompile the packages for new Python versions, but that’s easily done using cbuildwheels, while at the same time removing a lot of the build headaches.

And the core team could focus on evolving the C API for the better, with more freedom and a working backwards compatibility plan.

3 Likes