PEP 803: Stable ABI for Free-Threaded Builds

PEPs
1

Hello,
I’ve just published PEP 803 ­– Stable ABI for Free-Threaded Builds, based on latest post in the previous thread. Abstract:

Version 3.15 of the Stable ABI will be compatible with both free-threaded and GIL-enabled builds. To allow this, the PyObject internal structure and related APIs will be removed from version 3.15 of the Limited API, requiring migration to new API for common tasks like defining modules and most classes.

Now you can argue against a specific plan, rather than several possibilities!
Most alterantives from the previous thread are still possible though, so if you see something blocking you – especially in the PEP’s Rationale – do speak up.

My goal is that 3.15.0a1 should include a “MVP" of the CPython parts – enough that I can ask packaging tools to enable this, so we can start exposing it to users and getting feedback from real code.
That’s in a month, so if you want to speak up, please do so sooner. (Especially if you’re not going to the core dev sprint next week, before which there’s a SC deadline that might enable some development.)

Note that the PEP combines several pieces, some of which make sense individually. In particular, I’ve already merged in the Py_mod_abi slot.
(Of course, I’m ready to remove anything that turns out to have been a bad idea.)

8 Likes
2

This is really nice! I’m glad to see that you’re making a lot of progress on this.

Looking toward a future where NumPy decides to start shipping stable ABI wheels, I noticed just now there’s one spot NumPy uses Py_SIZE to get the size of a type to use for estimating memory usage:

Once PyObject is opaque, is there still a way to get at the total memory used by an object to do things like this?

To support the other changes in this PEP, NumPy would need to do a large-scale refactoring to avoid static types, so this is more of a theoretical concern, but maybe other extensions use Py_SIZE like this to get memory usage diagnostics.

3

I like the overall plan and the specification.

I would just suggest converting Py_SIZE() and Py_SET_SIZE() static inline functions to regular (opaque) functions, rather than removing them.

5 Likes
4

So essentially, we’re proposing to simply make ABI3 compiled pre-3.15 to refuse to load on 3.15 and later? So if your extension uses the Stable ABI, you’ll need two builds that are both tagged abi3 but the old one is py3-abi3 and the new one is cp315.cp316.cp317...-abi3 (until you stop supporting 3.14 and earlier and can just make the new one py3?[1]

  1. I’m 99% sure the compressed tag will be selected over py3, but admit I could be wrong about that. I think things are worse if I’m wrong, though… ↩︎

1 Like
5

If cp314-abi3 and cp315-abi3 are mutually unloadable, why not just call it abi4?

1 Like
6

I don’t think that’s correct? It’ll refuse to load in free-threaded builds of 3.15, but builds with the GIL should still work.

7

Okay then, replace 3.15 with 3.16 or whenever freethreaded becomes the default.

1 Like
8

Probably not 3.16, but even then, it may be the default but it will not be the only build. We’re talking about years and years of the ability for people to choose to use packages that require the older ABI by choosing not to use free-threaded Python. That’s why it’s not just abi4.

1 Like
9

Do we have a more concrete plan than https://peps.python.org/pep-0703/#python-build-modes then? (The “possible path” shown there is already clearly not going to happen, since “optional with env var + compatible ABI” is now known to not be possible, but I haven’t seen an officially approved alternative.)

1 Like
10

I think he’s referring to this: PEP 703 (Making the Global Interpreter Lock Optional in CPython) acceptance

In particular, the end of Phase III mentions the possibility of maybe phasing out the GIL eventually, but there’s no timeline.

1 Like
11

Sure, but this question and PEP goes directly to the open issue in PEP 779 – Criteria for supported status for free-threaded Python | peps.python.org, which was directed by the SC to be discussed for 3.15, but no specific design was mandated. That’s what we’re doing here (and Petr and I have largely been doing spread over a number of locations since before PEP 779 was accepted).

My primary concern with any of the proposals so far[1] is dividing the community for an unspecified length of time.[2] The choices largely come down to one of:

  • abi3 is broken at an arbitrarily time
  • abi3t exists until free-threading becomes default and then abi3 no longer exists
  • abi3 and abi4 exist in parallel, the latter supported by both builds, and abi3 no longer exists once free-threading becomes the default
  • free-threading has no stable ABI until it becomes the default (status quo, but already rejected by the SC)

I don’t particularly have specific desires about the contents of the new ABI[3] - virtually everything I’ve seen proposed has been pretty uncontroversial IMHO - it’s just that some of it is necessarily breaking the ABI and I think option 3 above manages that better than options 1 or 2.

  1. I haven’t had time to work up more than a sketch of a proposal, but my core criteria follows. ↩︎

  2. I originally wrote “permanently” knowing that’s not true, but it definitely flows better to use a touch of hyperbole and say that my primary concern is permanently dividing the community. ↩︎

  3. I have some ideas about future longevity, so we aren’t changing it every release like we have been with abi3, but those are totally separate from the free-threading issue, e.g. they could also be added to abi3 safely. ↩︎

6 Likes
12

Yeah, let’s do that now.
(I hoped to get rid of Py_SIZE eventually: PyType_GenericAlloc() and friends don’t need to store the size since the caller has it and can stash it in a field of their choosing, and that field is then more obviously private to the type – or made public, if the author wishes. But, removing friction in porting existing extensions is now more important than phasing out confusing designs.)

No, cp315-abi3 will load on CPython 3.16+. The Python tag has always been a lower bound.

Yes. That “possible path” is for non-stable ABI; a universal ABI is indeed not possible there at this point.

For PEP 803, an important premise is that building an extension twice (or rather, a “small” number of times), for different ABIs, is not an issue for extension authors.
If that premise is wrong, let’s do things differently.

After the Python 2→3 transition, we decided (for Python API), that there would not be a version 4, and instead any necessary breaking changes would be introduced gradually, with deprecation periods. Old code will stop working after enough warning is given.
That is also the policy for behaviour of the C API. Functions “behind“ the ABI can already start raising warnings and then stop working. A “stricter“ ABI means that you get “nice“ exceptions rather than memory corruption or “symbol not found“ errors that prevent loading the whole extension.

This PEP essentially does that for Stable ABI. There will be no abi4, and instead old versions of abi3 will be broken after some years of notice.

1 Like
13

Technically, the Python tag (like any other tag) is only accepted by environments that state they accept it. By convention, Python 3.14 states that it accepts cp313 wheels (this is implemented in the packaging library), but it’s not a requirement.

Steve is technically correct, although in practice cp315-abi3 should be fine.

I’d strongly advise bringing that up in the packaging category. My impression is that there are many projects where the number of versions they have to build is already problematic, and doubling that would definitely be an issue.

That seems to contradict the term “stable”, or at least the expectations people have of that term, though…?

6 Likes
14

Technically, the Python tag (like any other tag) is only accepted by
environments that state they accept it. By convention, Python 3.14
states that it accepts cp313 wheels (this is implemented in the
packaging library), but it’s not a requirement.

Steve is technically correct, although in practice cp315-abi3 should
be fine.

The bit I’m struggling with is understanding whether or not existing
cp39-abi3 (for example) wheels will be supported by Python v3.15???

I’d strongly advise bringing that up in the packaging category. My
impression is that there are many projects where the number of
versions they have to build is already problematic, and doubling that
would definitely be an issue.

Indeed. My own burden is 15 (related) projects on 5 platforms (counting
macOS Intel and ARM as one). The stable ABI makes this manageable. I
have no plans to support free-threaded builds until a stable ABI is
available. Things would be easier (but not impossible) if that ABI
supported the existing ABI for a couple of years or so.

3 Likes
15

The PEP should definitely be explicit on this IMHO.

3 Likes
16

Yes. But for free-threading, the stable ABI needs to be broken in some way.

Yes, but only in the regular (non-free-threaded) builds of 3.15+.

It should be possible to make a single ABI compatible with 3.11+ (non-free-threaded) and 3.15+ (all builds).[1] Personally, I’d be happy to do it – please convince the SC that it’s a good idea! Currently it seems the complexity in testing and supporting this is not worth it.
It could be done as an “add-on” to PEP 803 – basically, extending the compatibility to lower versions.

  1. ① Add Py_OPAQUE_OBJECT as a user-settable knob for limited API 3.14 and below; ② add a shim for PEP 793’s PyModExport (ugly but self-contained); ③ add a few shims for things that only became non-static functions in or 3.15; ④ add a abi3t wheel tag, rather than relying on cp315-abi3 ↩︎

2 Likes
17

I’m not very familiar with the stable ABI, but I do publish abi3 wheels (mostly because tools like pyo3 make it as easy as passing a flag) for the forward compatibility guarantee. Do abi3 wheels actually need to break? Ideally, I’d like to see:

  • abi3 continues to provide forward compatibility with non-free-threaded Python
  • abi4 introduced to cover both free-threaded and GIL-enabled builds, with the same forward compatibility guarantee
1 Like
18

I hadn’t realised that there would continue to be separate Python
builds.

2 Likes
19

To some extent that’s a question for tools (the packaging library in particular). There’s no standard that says “CPython 3.15 must support cp39-abi3 tagged wheels”. But yes, the PEP needs to be explicit about what it expects from tag semantics, and that’s not as simple as it looks.

And in my view, that breakage should come with a version bump. I don’t think that these days anyone associates “abi3” with “Python 3”, so there’s no implication that “abi4” would be in any way like the Python 2->3 transition.

I don’t know what the proposal is for platform tags for free-threaded Python builds. Has anyone made a formal proposal on this yet? I have no feel for whether free threaded CPython 3.15 should accept cp315 wheels, or conversely whether CPython 3.15 with the GIL should accept cp315t wheels. And it’s important to note that generally, tags are considered separately, so “it depends on whether it’s using abi3” isn’t a helpful answer here.

2 Likes
20

That’s a weird statement. Is it a packaging tool’s job to say whether CPython 3.15 is compatible with Stable ABI 3.9?