Python ABIs and PEP 703

[encukou (Petr Viktorin)]

@colesbury

• We add a new “abi4” with targets “cp37-abi4”, …, “cp313-abi4”.
• C extensions can target by “abi4” by defining Py_ABI_VERSION_4 along with the usual Py_LIMITED_API=0x30...

Sounds good for PEP 703. We can tweak the versioning & defines in another PEP.

@colesbury

PyObject_HEAD is still needed for extensions to define their own types. In abi4, it reserves enough space for both abi3 PyObject headers and --disable-gil PyObject headers. This means 24 or 32 bytes on 32-bit or 64-bit platforms. Note that extra space in the header is fine.

Makes sense.
I’d say that PyObject_HEAD is still available, rather than needed. PEP 697 makes it possible to avoid it, but that’s only available in Python 3.12+, which is too late if we want abi4 in 3.13.
But that gives us a clearer way to remove PyObject_HEAD eventually – hopefully by the time we need to increase the object head size. (We’ll probably want a gradual deprecation mechanism instead of planning for abi5).
If we can get in an even better way, like Grand Unified Python Object Layout, all the better :‍)

@colesbury

Extensions must call the Py_Type function in 3.13, but that won’t be available in 3.12 and older. When loaded on those versions, the extension must access the field directly.

Use PyObject_Type instead of Py_Type :‍)

For the others, could we get away with removing them from the limited API entirely? They’re questionable, anyway:

• set type (dangerous; in the safe cases can it be replaced by Python API – assignment to __class__)
• get/set refcount (CPython-specific)
• get/set size (ill-defined for the general PyVarObject; if individual types like PyTuple expose length information they should do so by other API – like PyObject_Size a.k.a. __len__)

Forward API compatibility (i.e. older extensions working on abi4 with just recompilation) is nice, but it might not be worth requiring weak symbols. Especially since…

@colesbury

PyObject and PyVarObject become opaque structs in abi4.

This looks like a more serious break of forward API compatibility.
To define a type with no extra C instance state, you currently set PyType_Spec.basicsize to sizeof(PyObject). I haven’t checked, but I expect a lot of extensions do it.
We got lucky here: you instead can set basicsize to 0, meaning “inherit”, which was only documented and tested in 3.12 but happens to work on older versions as well.