Stable ABI/Limited API for free-threaded builds

[philthompson]

Yes, that is what my backend does. Therefore the version in the tag seems to be repeating existing information. Have I misunderstood?

@encukou

The intention is that they’re equivalent, but abi3 is “stuck” at being equivalent to abi3.15, and EOL’d together with abi3.15.

How does that affect existing abi3 extensions and my ability to create wheels for older Python versions?

[pf_moore (Paul Moore)]

@pitrou

If it’s orthogonal to free-threading, could we end up with compressed tag sets such as abi3.15.abi3.15t? (is that synctatically valid?)

It is, but it doesn’t mean what you want it to mean. And it demonstrates another reason why abi3.15 won’t work - wheel tags can’t have a . in them because it clashes with the compressed tag syntax.

[pitrou (Antoine Pitrou)]

@pf_moore

It is, but it doesn’t mean what you want it to mean.

Ok, but what does it mean then?

[encukou (Petr Viktorin)]

@pf_moore

wheel tags can’t have a . in them because it clashes with the compressed tag syntax.

Right, it would need to be spelled abi3_15. Sorry for the oversight.

@pitrou

Ok, but what does it mean then?

abi3.15.abi3.15t would mean (abi3 or 15 or abi3 or 15t), which is the same as (abi3) since the 15 and 15t wouldn’t match anything.
abi3_15.abi3_15t would mean (abi3_15 or abi3_15t).

@philthompson

the version in the tag seems to be repeating existing information. Have I misunderstood?

Yes, they repeat. Requires-Python >= 3.x version, Py_LIMITED_API, and the version in the filename an the wheel tag should generally all match, if you’re building a stable ABI wheel.
But, each one has its place:

Py_LIMITED_API (build-time) and the filename tag (run-time) are for core CPython; they should be useful even if you’re not using PyPA specs/tools.
The wheel tag is needed to get unique filenames for wheels.
That makes Requires-Python somewhat redundant here, but:

• it’s useful for pure-Python packages; it would be awkward to ban it for extensions.
• tools can use it as input.

@philthompson

How does [abi3_15] affect existing abi3 extensions and my ability to create wheels for older Python versions?

It should have no effect, except abi3 might eventually be incompatible with something like Python 3.25, so you’d need to switch to abi3_15 or later some time in the future.
What “some time” would be is the big question about the length of support windows. How often should people need to rebuild extensions?

[steve.dower]

@pitrou

If it’s orthogonal to free-threading, could we end up with compressed tag sets such as abi3.15.abi3.15t?

I would assume that as soon as we invent any new abi* at all, we make it compatible between free-threaded and not. That’s the big “problem” with abi3 right now, in that it’s impossible to preserve it and also enable free-threading (see the first few posts in this thread where it was discussed).

@encukou

(And, as we bike-shed the next-to-last, somewhat orthogonal piece: do the others look like good next steps?)

As I posted earlier, I think Py_OPAQUE_PYOBJECT is an unnecessary knob, and we should just rev the API completely.^[1] The changes behind Py_OPAQUE_PYOBJECT are fine, and should be in the next limited API, but no need for an orthogonal flag.

The way to get me on board with that is to make it explicitly about compile-time warnings/failures without producing different binaries. I suspect that’s the case already, but I want it explicit so that it’s clear (for future changes) that it must be safe to mix source units with/without the flag.

That flows nicely into supporting setting the flag immediately before including Python.h, which takes the build tools out of the equation (unless they want to be there). Users who want to progressively migrate their code can add it to their source code or build scripts, and since it doesn’t matter whether the final binary used the flag or not it doesn’t matter if the build tools ignore it.

---

I’d also like to have my forward-compatible slots idea considered, since this is the time it makes the most sense. It allows a stable API that can be extended over time without needing to re-version the entire thing or to force older users to fail. It ought to get us out of the place we’ve been for a few versions where 2-3 new “essential” functions are added to the stable ABI each release, making the older release unusable for some users, and avoid permanently leaking abstractions.^[2]

---

1. To abi4 and Py_LIMITED_API=0x04000000 by preference, decoupled from Python version, but that’s the contentious point you asked to avoid. ↩︎

2. In that it’s much easier to replace an implementation without breaking the interface (perhaps making the particular operation slow, depending on what changed, but without breaking the entire thing). ↩︎

[encukou (Petr Viktorin)]

@steve.dower

As I posted earlier, I think Py_OPAQUE_PYOBJECT is an unnecessary knob, and we should just rev the API completely.[1]

The changes behind Py_OPAQUE_PYOBJECT are fine, and should be in the next limited API, but no need for an orthogonal flag.

Yes, if we rev the API completely, Py_OPAQUE_PYOBJECT will be useless.
But I’m afraid of making these removals block people from using other 3.15 features.

@steve.dower

The way to get me on board with that is to make it explicitly about compile-time warnings/failures without producing different binaries.

Granted!
Py_OPAQUE_PYOBJECT does nothing but hide APIs that are incompatible between regular and free-threaded builds.
For the record, the preliminary list is:

• struct _object (making PyObject opaque)
• struct PyVarObject (making PyVarObject opaque)
• PyObject_HEAD
• _PyObject_EXTRA_INIT
• PyObject_HEAD_INIT
• PyObject_VAR_HEAD
• Py_SIZE (can be exported as a function instead)
• Py_SET_TYPE (can be exported as a function instead)
• Py_SET_SIZE (can be exported as a function instead)

---

@steve.dower

I’d also like to have my forward-compatible slots idea considered

I think your idea ties in nicely with my forward-compatible slots idea :‍)

If we do not rev the API completely, these can be added at any point.

[steve.dower]

@encukou

Setuptools, Meson, etc. etc. will need to grow flags that define Py_OPAQUE_PYOBJECT&c. and generate the right wheel tags.

This is the bit that made me think Py_OPAQUE_PYOBJECT had an ABI impact. If there’s no change in output, there’s no need for the build tools or wheel tags to be aware of it.

I’ll let others debate whether it’s even worth doing at that point. I’m inclined to think it’ll be basically free to implement/maintain on top of abi4, so I’m not so concerned, but I do think it’ll be massively underused and the cost of it should be calculated accordingly.

@encukou

I think your idea ties in nicely with my forward-compatible slots idea :‍)

Mine is the generalisation of yours, yes. Though structs that are not PyObjects and/or don’t have a PyTypeObject would need parallel APIs (I do think it makes sense to have those parallel APIs be almost identical though, rather than having inconsistent names and shapes).

[encukou (Petr Viktorin)]

@steve.dower

If there’s no change in output, there’s no need for the build tools or wheel tags to be aware of it.

They need to put the right tags in wheel & file names. I assume that they’ll want to set the compiler flags as well.

@steve.dower

I’m inclined to think it’ll be basically free to implement/maintain on top of abi4, so I’m not so concerned, but I do think it’ll be massively underused and the cost of it should be calculated accordingly.

Yes, it should be basically free on top of anything; it’s abi4 that’s the expensive part. Py_OPAQUE_PYOBJECT is a short list of APIs you can’t use, presented in a way the compiler understands.

[steve.dower]

@encukou

They need to put the right tags in wheel & file names.

The right tag would be cpXY or abi3, the same as today? So no change required here?

@encukou

I assume that they’ll want to set the compiler flags as well.

It should be settable in the source file:

#define Py_OPAQUE_PYOBJECT
#include <Python.h>
...

Being able to set it through anything other than an explicit “add additional defines” feature in a build backend is an exercise for build backends. It’s only a requirement for us to make it happen if we don’t have any other way to set the variable (such as by putting it in the source file).

[encukou (Petr Viktorin)]

@steve.dower

The right tag would be cpXY or abi3, the same as today?

Or abi3t.

@steve.dower

Being able to set it through anything other than an explicit “add additional defines” feature in a build backend is an exercise for build backends. It’s only a requirement for us to make it happen if we don’t have any other way to set the variable (such as by putting it in the source file).

Yes. You don’t need build tools or wheels at all for Py_OPAQUE_PYOBJECT.

[encukou (Petr Viktorin)]

Some alternatives I can see:

1. abi3t makes PyObject opaque.

   Pros:

   • The PyObject memory layout for free-threaded builds does not have to be frozen. (!)
   • abi3t would be compatible with non-free-threaded builds. (abi3.abi3t wheel tags would still be needed for 3.14 and below)
   • no Py_OPAQUE_PYOBJECT define needed: “just” build (with free-threaded Python) to get an extension compatible with 3.15+ & 3.15t+.

   Cons:

   • Not API-compatible. Stable-ABI extensions would require implementing the PEP-793 export hook (or different new API solving the same issue) to support free-threaded builds. This could slow down adoption.

2. The regular limited API makes PyObject opaque in 3.15

   Pros: as above, plus

   • no new abi3t tag to worry about

   Cons: as above, plus:

   • to use any 3.15 features, stable-ABI extenstions would need to implement the PEP-793 export hook (or different new API solving the same issue)

3. PyObject is not made opaque.

   Pros:

   • Simple to implement. (This is the “don’t worry about technical debt” option.)

   Cons:

   • No extension can be compatible with both regular & free-threaded builds.
   • No movement toward getting PyObject memory layout out of stable ABI.

---

Notice that 1 & 2:

• Both need PEP-793 (or similar)
• The differences between these and the “main” proposal are mainly in filename/wheel tags and guarantees/documentation. Putting the #ifdef Py_OPAQUE_PYOBJECT lines in main would be rather useful for experimenting with these, even if they’re made obsolete & removed before RC.

[steve.dower]

@encukou

abi3t makes PyObject opaque.

This sounds pretty doable. The con seems likely to be pretty unavoidable, but as long as people think they’re updating their code to work with free-threading rather than because we just felt like making them, I don’t think it’ll be too much of a problem.

@encukou

The regular limited API makes PyObject opaque in 3.15

Even if we did this, we can’t change the layout of PyObject until whenever we decide that limited API modules built with 3.14 are no longer supported. There isn’t an agreed upon definition of this deprecation timeline (that I’m aware of), so it seems likely that we’ll need a hard break migration at some point before it expires anyway. May as well be planning for that (and supporting it in parallel as much as we can) rather than trying to patch over design decisions from the past.