Stable ABI/Limited API for free-threaded builds

Core Development C API
22

Yes, that is what my backend does. Therefore the version in the tag seems to be repeating existing information. Have I misunderstood?

How does that affect existing abi3 extensions and my ability to create wheels for older Python versions?

23

It is, but it doesn’t mean what you want it to mean. And it demonstrates another reason why abi3.15 won’t work - wheel tags can’t have a . in them because it clashes with the compressed tag syntax.

24

Ok, but what does it mean then?

25

Right, it would need to be spelled abi3_15. Sorry for the oversight.

abi3.15.abi3.15t would mean (abi3 or 15 or abi3 or 15t), which is the same as (abi3) since the 15 and 15t wouldn’t match anything.
abi3_15.abi3_15t would mean (abi3_15 or abi3_15t).

Yes, they repeat. Requires-Python >= 3.x version, Py_LIMITED_API, and the version in the filename an the wheel tag should generally all match, if you’re building a stable ABI wheel.
But, each one has its place:

Py_LIMITED_API (build-time) and the filename tag (run-time) are for core CPython; they should be useful even if you’re not using PyPA specs/tools.
The wheel tag is needed to get unique filenames for wheels.
That makes Requires-Python somewhat redundant here, but:

  • it’s useful for pure-Python packages; it would be awkward to ban it for extensions.
  • tools can use it as input.

It should have no effect, except abi3 might eventually be incompatible with something like Python 3.25, so you’d need to switch to abi3_15 or later some time in the future.
What “some time” would be is the big question about the length of support windows. How often should people need to rebuild extensions?

26

I would assume that as soon as we invent any new abi* at all, we make it compatible between free-threaded and not. That’s the big “problem” with abi3 right now, in that it’s impossible to preserve it and also enable free-threading (see the first few posts in this thread where it was discussed).

As I posted earlier, I think Py_OPAQUE_PYOBJECT is an unnecessary knob, and we should just rev the API completely.[1] The changes behind Py_OPAQUE_PYOBJECT are fine, and should be in the next limited API, but no need for an orthogonal flag.

The way to get me on board with that is to make it explicitly about compile-time warnings/failures without producing different binaries. I suspect that’s the case already, but I want it explicit so that it’s clear (for future changes) that it must be safe to mix source units with/without the flag.

That flows nicely into supporting setting the flag immediately before including Python.h, which takes the build tools out of the equation (unless they want to be there). Users who want to progressively migrate their code can add it to their source code or build scripts, and since it doesn’t matter whether the final binary used the flag or not it doesn’t matter if the build tools ignore it.

I’d also like to have my forward-compatible slots idea considered, since this is the time it makes the most sense. It allows a stable API that can be extended over time without needing to re-version the entire thing or to force older users to fail. It ought to get us out of the place we’ve been for a few versions where 2-3 new “essential” functions are added to the stable ABI each release, making the older release unusable for some users, and avoid permanently leaking abstractions.[2]

  1. To abi4 and Py_LIMITED_API=0x04000000 by preference, decoupled from Python version, but that’s the contentious point you asked to avoid. ↩︎

  2. In that it’s much easier to replace an implementation without breaking the interface (perhaps making the particular operation slow, depending on what changed, but without breaking the entire thing). ↩︎

1 Like
27

Yes, if we rev the API completely, Py_OPAQUE_PYOBJECT will be useless.
But I’m afraid of making these removals block people from using other 3.15 features.

Granted!
Py_OPAQUE_PYOBJECT does nothing but hide APIs that are incompatible between regular and free-threaded builds.
For the record, the preliminary list is:

  • struct _object (making PyObject opaque)
  • struct PyVarObject (making PyVarObject opaque)
  • PyObject_HEAD
  • _PyObject_EXTRA_INIT
  • PyObject_HEAD_INIT
  • PyObject_VAR_HEAD
  • Py_SIZE (can be exported as a function instead)
  • Py_SET_TYPE (can be exported as a function instead)
  • Py_SET_SIZE (can be exported as a function instead)
  • [edited to add]:
    • PyModuleDef_Base
    • PyModuleDef_HEAD_INIT
    • PyModuleDef

I think your idea ties in nicely with my forward-compatible slots idea :‍)

If we do not rev the API completely, these can be added at any point.

28

This is the bit that made me think Py_OPAQUE_PYOBJECT had an ABI impact. If there’s no change in output, there’s no need for the build tools or wheel tags to be aware of it.

I’ll let others debate whether it’s even worth doing at that point. I’m inclined to think it’ll be basically free to implement/maintain on top of abi4, so I’m not so concerned, but I do think it’ll be massively underused and the cost of it should be calculated accordingly.

Mine is the generalisation of yours, yes. Though structs that are not PyObjects and/or don’t have a PyTypeObject would need parallel APIs (I do think it makes sense to have those parallel APIs be almost identical though, rather than having inconsistent names and shapes).

29

They need to put the right tags in wheel & file names. I assume that they’ll want to set the compiler flags as well.

Yes, it should be basically free on top of anything; it’s abi4 that’s the expensive part. Py_OPAQUE_PYOBJECT is a short list of APIs you can’t use, presented in a way the compiler understands.

30

The right tag would be cpXY or abi3, the same as today? So no change required here?

It should be settable in the source file:

#define Py_OPAQUE_PYOBJECT
#include <Python.h>
...

Being able to set it through anything other than an explicit “add additional defines” feature in a build backend is an exercise for build backends. It’s only a requirement for us to make it happen if we don’t have any other way to set the variable (such as by putting it in the source file).

31

Or abi3t.

Yes. You don’t need build tools or wheels at all for Py_OPAQUE_PYOBJECT.

32

Some alternatives I can see:

  1. abi3t makes PyObject opaque.

    Pros:

    • The PyObject memory layout for free-threaded builds does not have to be frozen. (!)
    • abi3t would be compatible with non-free-threaded builds. (abi3.abi3t wheel tags would still be needed for 3.14 and below)
    • no Py_OPAQUE_PYOBJECT define needed: “just” build (with free-threaded Python) to get an extension compatible with 3.15+ & 3.15t+.

    Cons:

    • Not API-compatible. Stable-ABI extensions would require implementing the PEP-793 export hook (or different new API solving the same issue) to support free-threaded builds. This could slow down adoption.

  2. The regular limited API makes PyObject opaque in 3.15

    Pros: as above, plus

    • no new abi3t tag to worry about

    Cons: as above, plus:

    • to use any 3.15 features, stable-ABI extenstions would need to implement the PEP-793 export hook (or different new API solving the same issue)

  3. PyObject is not made opaque.

    Pros:

    • Simple to implement. (This is the “don’t worry about technical debt” option.)

    Cons:

    • No extension can be compatible with both regular & free-threaded builds.
    • No movement toward getting PyObject memory layout out of stable ABI.

Notice that 1 & 2:

  • Both need PEP-793 (or similar)
  • The differences between these and the “main” proposal are mainly in filename/wheel tags and guarantees/documentation. Putting the #ifdef Py_OPAQUE_PYOBJECT lines in main would be rather useful for experimenting with these, even if they’re made obsolete & removed before RC.
33

This sounds pretty doable. The con seems likely to be pretty unavoidable, but as long as people think they’re updating their code to work with free-threading rather than because we just felt like making them, I don’t think it’ll be too much of a problem.

Even if we did this, we can’t change the layout of PyObject until whenever we decide that limited API modules built with 3.14 are no longer supported. There isn’t an agreed upon definition of this deprecation timeline (that I’m aware of), so it seems likely that we’ll need a hard break migration at some point before it expires anyway. May as well be planning for that (and supporting it in parallel as much as we can) rather than trying to patch over design decisions from the past.

1 Like
34

That hard break is called “Phase III”. No timeline, but it seems it could be pretty soon.

I agree about making the transition as painless as possible. That’s why I propose both API compatibility (no source changes → separate abi3 & abi3t builds) and ABI compatibility (PyModExport, opaque PyObject → abi3.abi3t builds).
We want extensions to switch to the latter (so that we can change PyObject in the future), but I think we need a window where we support both – both for extension authors to switch, and for CPython to react to feedback.

2 Likes
35

This seems like a good strategy to me. Based on experience with recent large-scale migration efforts - NumPy 2.0 and free-threading - a one year migration window is likely too tight. The inertia of a lot of projects having to do releases is very large, even if no source code changes are needed. With a two year or three year window, it should be possible to do it in a low-disruption manner though.

2 Likes
36

If “no source changes” is true, can we make the non-freethreaded build also support abi3t? That way only one build is required, at least for ~3.15 onwards, while builds made prior to 3.15 (and for versions prior to 3.15) continue to work on non-freethreaded builds for a while.

2 Likes
37

For extensions that do not use the PyObject layout, yes!
But for most extensions, avoiding the PyObject layout means a source change – see the PEP.

1 Like
38

Yeah, the end result is nearly the same, but I think there’s a difference in what package maintainers have to worry about.

Basically, if all 3.15 builds support abi3t, whether free-threaded or not, you only need to publish one “universal” package for 3.15 and later. Rather than having to build twice and publish two packages (how your proposal currently reads).

If you want universal builds for 3.14 and earlier and also 3.15 free-threaded and later, you need to do both abi3 and abi3t.

If you don’t want to do any builds at all, abi3 continues to work with non-freethreaded 3.15 and continues to not work with freethreaded.

So essentially, we can do a new ABI that works regardless of freethreading or not, it first works with 3.15, and we’ll keep the old ABI around as long as we can (which I suspect may not be long, but it’ll be longer than dropping it immediately). So set the transition point at a Python version, rather than at a build option.

(This should seem perfectly in line with my preference for calling it abi4 rather than 3t, and maybe my explanation makes more sense if you substitute 4 in there.)

2 Likes
39

My proposal calls that abi3.abi3t.
Yours would call that cp315-abi3, or abi4, if i understand correctly.
Implementation-wise, it’s about setting defaults, (dis)allowing options, and naming things. The options are open, but as far as I can see, they all need opaque PyObject (either with Py_OPAQUE_PYOBJECT or implicitly) and PEP 793.

40

Except it isn’t, really, because it’s not going to work with Python 3.14 or earlier. And having abi3 in the tag means that it should. So really, your proposal is cp315-abi3.abi3t, whereas mine is just cp315-abi3t (or cp315-abi4, since the t would imply free-threaded builds which aren’t a requirement under my proposal).[1]

The big problem we have is that abi3 is meant to be forward and backward compatible, but we gave up on that, and so in practice it’s just “cp3X or greater”. Adding a second ABI and claiming it’s interchangeable with the existing one (the meaning of abi3.abi3t) only really makes it more confusing. (What does cp314-abi3.abi3t mean, for example, when abi3t isn’t available there and abi3 is different from the runtimes where abi3t exists?)

Fiddling at the edges with minimal changes and new compile time options doesn’t solve the problem that we ultimately need a clean break, and need a compatible way to add new interfaces later on. Adding things that create complexity, can’t be easily removed, and don’t solve the problem will only make things worse.

  1. And cp315-abi3 would still be valid for non-freethreaded builds. ↩︎

1 Like
41

It means you use the limited API (abi3), treat PyObject as opaque (abi3.abi3t), and don’t require any 3.15+ APIs (cp314).
Which is hard to pull off, but possible.

Please be explicit about the “things” you mean.
I think that at this point i need to ask you for a more complete counter-proposal; this discussion is going in circles.