Python ABIs and PEP 703

[encukou (Petr Viktorin)]

@guido

Interesting. The folks with whom I’ve discussed this so far all seemed to assume that you can’t expect to compile a wheel with Python 3.N+1 and expect it to be linkable with Python 3.N, even if you select the Limited API version 3.N – the latter is more of a source compatibility guard (you shouldn’t expect to build with 3.N-1 at all), but because of tooling (e.g. compiler or C runtime) you should build your future-proof wheels with exactly 3.N.

How much evidence do you have that this in fact works? Does it work for Windows and Mac too?

I spent the weekend away from computers, but kept thinking about how to best research this… And when I returned, the first place I looked had the answer:

As far as I can see, cryptography builds its py3.7-abi3 wheels using Python 3.11.
So, yes, it works in practice, on Mac, Windows, manylinux and musllinux.
Not sure what more is there to research. But there are still questions:

• Should we (continue to) support this? (IMO: yes. At least for the initial rollout of abi4 – so that projects that build a single set of wheels can both keep 3.8+ compatibility and support nogil. And for the future: arguably this isn’t that difficult to support, and it allows us to “retroactively” apply improvements/optimizations in some cases.)
• Should we test this better? (Not a real question. AFAIK we currently don’t have any CI checks that use multiple Python versions. If we want start testing any kind of stable ABI, this should be included.)

@steve.dower

In practice, [using old Py_LIMITED_API] hasn’t – e.g. abi3 extensions built with 3.11+ use functions for reference counting, even if running on ever reliably worked (at least no more reliably than using the regular API and omitting the tag from the extension module), and so really you would build with the oldest version you want to support. I thought this was due to errors/mistakes/slackness on our side, but apparently it’s always been the intended design.

Do you have a recent (3.8+) example? I would indeed consider that an error/mistake/slackness.