Python ABIs and PEP 703

[guido (Guido van Rossum)]

@encukou

Motivation for a stable ABI in general, right?

No, I was not asking for that.

I was asking for the motivation to support building wheels with a newer CPython version that are meant to work (also) with an older version, since that’s not how the ABI has worked so far. I had assumed we were okay with a one-time “barrier” – the idea that if you want your wheel to support ABI4 (which is needed to support no-GIL builds) you have to rebuild for 3.13 (and with an appropriate #define), and you’d be good for N future releases (where N >= 5 or so, TBD).

But it seems that the motivation for your proposal is simply that you think it can be done (and supposedly it would be friendlier for package maintainers). And that’s fine.

@encukou

But for projects where the Python bindings are just a convenience, you get a choice between adding loops (and support for multiple Python configurations) to a buildsystem like CMake or Cargo, distributing a specific vendored version of CPython, or dropping the Python bindings.

I’m not sure I fully understand this scenario. Is it like vim, which can dynamically load one of several different CPython versions and then offers it an API for manipulating vim internals? But in most cases, wouldn’t such an app just pick a CPython version to target?

Or are you thinking of frameworks like Tensorflow or PyTorch, which are presented as Python packages, but really contain a very large core of C++ code with Python linked in as the UI?

Thanks for the clarifications on PyObject_HEAD, I have no further questions there!

Regarding “too late for 3.12+”, I realize I was confused by the shorthand “3.12+” – I interpreted it as the state of the main branch after 3.12 was released (a notation we used in sys.version in the past IIRC), but I understand you meant “things that should be compatible with 3.12 and later”. Which is very different. And now I understand the “too late” part.

[encukou (Petr Viktorin)]

@guido

I was asking for the motivation to support building wheels with a newer CPython version that are meant to work (also) with an older version, since that’s not how the ABI has worked so far.

Hm, there’s a point of confusion. I always thought it has worked that way.
It’s best to set Py_LIMITED_API to the current CPython version, but if you compile with a later CPython, you get the same compatibility story.
Otherwise, what’s the point of the #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030A0000 guards?

I’m afraid I put in my interpretation when I wrote the docs for Py_LIMITED_API.

@guido

I’m not sure I fully understand this scenario. Is it like vim, which can dynamically load one of several different CPython versions and then offers it an API for manipulating vim internals? But in most cases, wouldn’t such an app just pick a CPython version to target?

Yes, like vim.
If vim picked a CPython version, it would need hope the user has that particular version installed (and upgrades it in lockstep with vim). Or vendor CPython to make sure users always gets the right version.

@steve.dower

FTR, I firmly believe any project that could conceivably vendor their own copy of CPython probably should vendor their own copy (and we should work on making embedding/vendoring CPython easier on all platforms).

Vendoring has advantages, but being a distributor of CPython is no small task. IMO it’s only suitable for relatively large projects.
I believe that it should be cheap and easy to tack Python bindings onto any kind of software :‍)

[guido (Guido van Rossum)]

@encukou

It’s best to set Py_LIMITED_API to the current CPython version, but if you compile with a later CPython, you get the same compatibility story.

Interesting. The folks with whom I’ve discussed this so far all seemed to assume that you can’t expect to compile a wheel with Python 3.N+1 and expect it to be linkable with Python 3.N, even if you select the Limited API version 3.N – the latter is more of a source compatibility guard (you shouldn’t expect to build with 3.N-1 at all), but because of tooling (e.g. compiler or C runtime) you should build your future-proof wheels with exactly 3.N.

How much evidence do you have that this in fact works? Does it work for Windows and Mac too?

[barry-scott]

@encukou

Hm, there’s a point of confusion. I always thought it has worked that way.
It’s best to set Py_LIMITED_API to the current CPython version, but if you compile with a later CPython, you get the same compatibility story.

You must set Py_LIMITED_API to the oldest version of python that your code is designed to work with. Then it will work with that version and all later versions regardless of which version you compile against.

As you say the guards prevent the extension having any reference to newer symbols that would prevent the dynamic linker from importing the extension.

[steve.dower]

@barry-scott

You must set Py_LIMITED_API to the oldest version of python that your code is designed to work with. Then it will work with that version and all later versions regardless of which version you compile against.

In practice, that hasn’t ever reliably worked (at least no more reliably than using the regular API and omitting the tag from the extension module), and so really you would build with the oldest version you want to support. I thought this was due to errors/mistakes/slackness on our side, but apparently it’s always been the intended design.

Basic extensions (those where the APIs haven’t changed) are likely to be fine. They’ve never been enough for me, so I’ve never really used the limited API to be sure how well it works. And more fundamental recent changes like how Py_INCREF and Py_DECREF work in certain versions would seem to reduce the chance of backwards compatibility (bearing in mind that these macros are embedded in your module, so if you build a limited API module with 3.12, you’ll get saturation checks in every version, but only in your module!).

So under the circumstances, there isn’t really any point in setting Py_LIMITED_API to a version earlier than the one you’re building with, which is the equivalent of setting it to 1 anyway. (I was corrected on this below)

[vstinner (Victor Stinner)]

@guido

The folks with whom I’ve discussed this so far all seemed to assume that you can’t expect to compile a wheel with Python 3.N+1 and expect it to be linkable with Python 3.N, even if you select the Limited API version 3.N – the latter is more of a source compatibility guard (you shouldn’t expect to build with 3.N-1 at all), but because of tooling (e.g. compiler or C runtime) you should build your future-proof wheels with exactly 3.N.

Some detour on dynamic linking. Until Python 3.7, C extensions were linked to libpython3.x.

Up to Python 3.7, when a C extension was built with a Python built with --enable-shared (small “python” program which loads the big libpython), the C extension was linked to libpython3.x. Problem: if you attempt to load this C extension in a “static python” (built without --enable-shared), the load pulled libpython and suddenly something weird happens: you have two Python in the same process, the “static python” and libpython.

Now (Python 3.8+), C extensions are no longer linked to libpython. When you load a C extension in a Python process, the dynamic linker looks for symbols such as PyLong_FromLong() in the current process: luckily, it’s available, and so it’s not needed to load libpython or anything else!

I expect that if you build a C extension on Python 3.13 using #define Py_LIMITED_API 0x03080000, it works on Python 3.8. But I didn’t try. At least, I don’t see any major reason why it wouldn’t work. Problems arise when you use more and more functions of the C API. Last years, I removed more and more implementation details from the limited C API. In the past, many implementation details, so the stable ABI was not as stable as you might expect.

I mean that the stable ABI starting at Python 3.12 (#define Py_LIMITED_API 0x030C0000), when Py_INCREF() becomes a function call, the promised ABI stability should be more real than before. In previous Python versions, honestly, I don’t have enough experience to see how good it worked or not. But well, PySide and cryptography were fine with it apparently

---

The devil is into the details. “C extensions are not linked to libpython” is true on Linux. But on Android, they must be linked to libpython. If I recall correctly, it’s also the case on AIX. And sadly, I also think that they must be linked to the libpython DLL on Windows. Maybe they are linked to the stable ABI libpython DLL (“libpython3.dll”) which is different, I’m not sure.

[vstinner (Victor Stinner)]

@steve.dower

So under the circumstances, there isn’t really any point in setting Py_LIMITED_API to a version earlier than the one you’re building with, which is the equivalent of setting it to 1 anyway.

#define Py_LIMITED_API 1 targets Python 3.2 ABI: oldest Python version supporting the stable ABI.

Example:

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03060000
PyAPI_FUNC(PyObject *) PyOS_FSPath(PyObject *path);
#endif

If the Py_LIMITED_API macro is set to 1, Py_LIMITED_API+0 >= 0x03060000 test is false and you don’t get PyOS_FSPath() function (added to limited C API version 3.6).

[itamaro (Itamar Oren)]

@vstinner

Now (Python 3.8+), C extensions are no longer linked to libpython

this has always been confusing to me…
see for example this comment (and that entire issue).
what is supposed to make it such that C extension no longer link to libpython*.so?
and if that was the case, are C extension expected to ignore unknown symbols during linking?

[barry-scott]

@itamaro

C extension expected to ignore unknown symbols during linking?

That is normal behaviour when linking, by not forcing libpython.so into the extension .so it avoids having two pythons in the same process.

When python imports the extension the symbols are resolved from those that are already accessible in the python process.

[itamaro (Itamar Oren)]

@barry-scott

That is normal behaviour when linking, by not forcing libpython.so into the extension .so it avoids having two pythons in the same process.

“linking against libpython.so” is not the same as “statically linking libpython (.a) into the extension .so”.
statically linking libpython into C extensions would be generally a bad idea (leading to bloated ELFs and multiple copies of python).
linking against libpython.so just means that the linker can check that symbols you’re using from libpython indeed exist in the libpython.so, so if you mess up something, it would be a link-time error, and not a runtime segfault.

[guido (Guido van Rossum)]

At this point my main response is continued confusion – clearly there are some deep technical issues here that few people fully understand. I suppose with some research it could all become clear. If someone feels compelled to move the discussion forward it would be nice if they could do some research on how this is supposed to work on various platforms, and perform some experiments to validate the theory.

Edit: And report back here, of course.

[barry-scott]

@itamaro

linking against libpython.so” is not the same as “statically linking libpython (.a) into the extension .so”.

No one would ever link libpython.a into an extension, the result cannot work.
Why did you think that was what was being suggested?

Indeed you go one to state the same. I am confused by your reply to my post.

[fonini (Pedro Fonini)]

@itamaro

statically linking libpython into C extensions would be generally a bad idea (leading to bloated ELFs and multiple copies of python).

This is not what Victor Stinner meant when he mentioned multiple copies of Python here:

@vstinner

Problem: if you attempt to load this C extension in a “static python” (built without --enable-shared), the load pulled libpython and suddenly something weird happens: you have two Python in the same process, the “static python” and libpython.

If you dynamically link your extension against libpython.so, and then load this extension from a python executable built without --enable-shared (which does not link against libpython.so), then you have two Pythons in the same process.

[encukou (Petr Viktorin)]

@guido

Interesting. The folks with whom I’ve discussed this so far all seemed to assume that you can’t expect to compile a wheel with Python 3.N+1 and expect it to be linkable with Python 3.N, even if you select the Limited API version 3.N – the latter is more of a source compatibility guard (you shouldn’t expect to build with 3.N-1 at all), but because of tooling (e.g. compiler or C runtime) you should build your future-proof wheels with exactly 3.N.

How much evidence do you have that this in fact works? Does it work for Windows and Mac too?

I spent the weekend away from computers, but kept thinking about how to best research this… And when I returned, the first place I looked had the answer:

As far as I can see, cryptography builds its py3.7-abi3 wheels using Python 3.11.
So, yes, it works in practice, on Mac, Windows, manylinux and musllinux.
Not sure what more is there to research. But there are still questions:

• Should we (continue to) support this? (IMO: yes. At least for the initial rollout of abi4 – so that projects that build a single set of wheels can both keep 3.8+ compatibility and support nogil. And for the future: arguably this isn’t that difficult to support, and it allows us to “retroactively” apply improvements/optimizations in some cases.)
• Should we test this better? (Not a real question. AFAIK we currently don’t have any CI checks that use multiple Python versions. If we want start testing any kind of stable ABI, this should be included.)

@steve.dower

In practice, [using old Py_LIMITED_API] hasn’t – e.g. abi3 extensions built with 3.11+ use functions for reference counting, even if running on ever reliably worked (at least no more reliably than using the regular API and omitting the tag from the extension module), and so really you would build with the oldest version you want to support. I thought this was due to errors/mistakes/slackness on our side, but apparently it’s always been the intended design.

Do you have a recent (3.8+) example? I would indeed consider that an error/mistake/slackness.

[steve.dower]

@encukou

Do you have a recent (3.8+) example? I would indeed consider that an error/mistake/slackness.

I don’t know where that quote came from, though I recognise parts of it from one of my earlier posts.

We were just arguing about changes to the reference counting macros for this release - why are you now saying that they’ve been functions for the last year? Victor only just proposed changing them all over. But the reference count saturation has gone into macros, so anything built with 3.12 targeting an earlier version is going to check for saturation when run on those earlier versions.

But yeah, most of my perception was developed around the 3.5 era when the functions available according to the headers, python3.dll and POSIX didn’t match up at all. We’ve made a lot of progress since then, but anyone who was trying to use it at that point will have hit issues and might have figured out that only using it for forward compatibility was the safest way.

(One big advantage on Windows at least is that linking to python3.dll instead of python3X.dll` can actually be loaded in other versions. Everything else is runtime-failure level of subtle.)

[encukou (Petr Viktorin)]

Oops, I somehow got unrelated text included in your quote? I blame Discourse, or my inability to use it.
I’ve crossed the extra part out. Sorry for the confusion!

@steve.dower

But the reference count saturation has gone into macros, so anything built with 3.12 targeting an earlier version is going to check for saturation when run on those earlier versions

Oh dear. Guess you’re right. I should have reviewed that PR carefully :‍/
I see you already did that but got turned down, or more likely, you and Victor were talking past each other.

@vstinner, @gpshead, @eelizondo, @eric.snow: for clarity: what’s your current understanding? Should an extension compiled on Python 3.12, with 3.11 Py_LIMITED_API, work with 3.11?

@steve.dower

But yeah, most of my perception was developed around the 3.5 era when the functions available according to the headers, python3.dll and POSIX didn’t match up at all.

Yup. That got a lot of checks and fixes around 3.10 (PEP 652: tests now load all stable ABI functions using ctypes, and on *nix macros/symbols are checked using a checker Pablo built earlier).

[danijar (Danijar Hafner)]

Was there a decision on this topic?

It would be great to know if there is currently a way to build a single wheel to support CPy 3.8+ and CPy 3.13t, and if not, whether that’s on the roadmap for 3.14.

[colesbury (Sam Gross)]

There’s currently no way to build a wheel that supports both CPython 3.8+ and 3.13t.

It’s something I’d like to see for 3.14, but other things keep taking priority.