Python ABIs and PEP 703

Core Development
27

Building on @encukou’s comments, here’s a modification of the original proposal. (Don’t focus too much on naming, we can work out those details later.)

Here’s a prototype: Prototype ABI4 with Python 3.7+ compatibility · colesbury/cpython@2c1d02a · GitHub

  • We add a new “abi4” with targets “cp37-abi4”, …, “cp313-abi4”.
  • C extensions can target by “abi4” by defining Py_ABI_VERSION_4 along with the usual Py_LIMITED_API=0x30...
  • The --disable-gil builds can load “abi4” wheels, but not “abi3” wheels. So “cp37-abi4” wheels are compatible with Python 3.7+, including --disable-gil builds.
  • PyObject and PyVarObject become opaque structs in abi4.
  • PyObject_HEAD is still needed for extensions to define their own types. In abi4, it reserves enough space for both abi3 PyObject headers and --disable-gil PyObject headers. This means 24 or 32 bytes on 32-bit or 64-bit platforms. Note that extra space in the header is fine.
  • PyModuleDef is a bit different. It reserves exactly the space for an abi3 PyObject header. (It can’t be bigger because older versions of CPython still need to access the fields.) At the API level, PyModuleDef_Init no longer guarantees that it returns “def cast to PyObject*”. Old versions of CPython, of course, still do this as does the default build of CPython 3.13. The --disable-gil builds return a simple wrapper.

The implementations of Py_TYPE and similar require special care when targeting pre-3.13 versions of Python and ABI 4. (See this file).

  • Extensions must call the Py_Type function in 3.13, but that won’t be available in 3.12 and older. When loaded on those versions, the extension must access the field directly.
  • On most platforms, we handle this by making Py_Type a weak symbol. Windows doesn’t support weak symbols, so we use GetProcAddress instead to determine if Py_Type is available.
  • There’s a dispatcher mechanism so that we only have to look up the symbol on the first invocation.

Risks and weird stuff

  • Wheels can be named like cryptography-42.0.0-cp37-abi4-[platform].whl but the actual shared libraries will still need to be named with the .abi3.so extension to be loadable by older versions of Python.
  • pip (or more specifically pypa/packaging) will needed to disallow abi3 wheels from --disable-gil builds and allow abi4 wheels for Python 3.7+.
3 Likes