In short, it heavily references and borrows from PEP 803 (and endorses practically everything in that PEP), because most of the proposal is totally fine by me. What mine adds is a timeline/process for new Stable ABIs in the future, and an API for dynamic interface detection (which is going to be the big controversial part for sure, but we’ve discussed it with a few interested parties over the last couple days here and it seems to basically be a good idea with a lot of details to get comfortable with). There’s a description of the API and a reference implementation linked, so those are the bits to get acquainted with early - I don’t think we want a new Stable ABI without something along these lines, or else we’ll cripple our own ability to make incremental improvements.
I like some bits of the counter-proposal but not all of it.
I think defining a minimum support window is sensible and gets away from the perception that support is “forever”, which clearly isn’t sustainable.
I think that defining a way to try to dynamically load functions from future versions is potentially useful but only for some cases. It works well for functions dealing with primitive types of well-established typed (e.g. PyList_GetItemRef would be something that would have been easy to add with it). However some of the more useful APIs in recent memory have used types that are most useful as low-cost stack-allocated value types - for example Py_Buffer or critical sections or PyMutex. Obviously it’d be possible to get around that by making them heap-allocated but it does somewhat spoil them.
I’d be a bit worried about the rule that “the soonest a revision can be considered is every 5 years”.[1] It seems like it might lead to both
people having to rush proposals to get them in at the 5 year mark,
“political” games where people could push for (e.g.) abi2031 to be formalized with only minor changes as a way of blocking stuff they disagree with that might be coming to completion in 2032.
At least - that’s my reading of the proposal although I don’t thing that’s said absolutely explicitly ↩︎
The interfaces proposal is very similar to Py_buffer in that respect - it can be entirely stack allocated.
It wouldn’t really suit PyMutex, but as that is such a drastic change to how CPython is implemented, fully supporting it probably requires a new ABI (and you’ll note that the need to support free-threading is indeed what is motivating the need for a new ABI, so I think my position is pretty consistent here ).
Valid fears, but I expect a new ABI to require a new PEP, which means we’ll have the same processes that are already in place to manage this, including steering council veto if they believe that a proposal is rushed, or if the next ABI is insufficiently different from the previous to justify it. Even under my proposal, replacing abi2026 with abi2031 is a breaking change, so if we don’t really need to do it then we probably won’t.
Yeah, of course, I still need to get the PEP merged first. Literally just opened my laptop for the first time since leaving the sprints, which was only a few minutes after the earlier post.
However, it’s still a counter-PEP, which means they both really ought to be discussed in context of each other. So no need to be too strict on keeping the discussions separate - that’s just a way to make poor decisions. (That said, there’s definitely some areas in mine that will be off-topic for here, so as I say, the new thread is coming once the PEP is merged.)
(This may be more of a critique of PEP 697 than the current proposal, but I hope API ergonomics are on-topic here.)
If I understand correctly, “opaque PyObject” means that when implementing a custom type, I need to:
Define a token for the type using Py_tp_token;
Specify the size of the type’s private data using a negative basicsize in PyType_spec;
When I need to access the private data, call Py_TYPE, followed by PyType_GetBaseByToken, followed by PyObject_GetTypeData, followed by Py_XDECREF on the type object.
This PyType_GetBaseByToken dance seems to work but it feels heavyweight. It’s a bunch of extra boilerplate that virtually every extension type will need, and will need to call frequently. So it’d be nice to have a single API function PyObject_GetTypeDataByToken that encapsulates the whole process of obtaining type data from a PyObject *. Ideally, this function would also avoid the need to increment/decrement the type’s reference count.
A minor nit I also noticed is that PyType_GetBaseByToken requires the token pointer to be void * (rather than const void *.) It should, I think, be permissible for the token to be a pointer to a static constant.
It also occurs to me that, in a technical sense, it isn’t really necessary to look up the custom PyTypeObject - if I know that my custom type has tp_base equal to PyBaseObject_Type, then there could be a PyObject_GetTypeDataByBaseClass function to extract the type’s private data. (A possible disadvantage is that this bakes in assumptions about the memory layout - although it still gives more flexibility than the current stable ABI.)
it seems like this is the PEP together with PEP 809: Stable ABI for the Future we should be tracking more than PEP793 for now, since it is touching also other types that we use, and, potentially will impact the number of SciPy binaries we are currently providing, as of now around 60ish Release SciPy 1.16.2 · scipy/scipy · GitHub .
Performance isn’t great - Cython with the Python 3.15 Stable ABI is about 10% slower than Cython with the Python 3.14 Stable ABI (both running on Python 3.15). That’s obviously fairly extension-type-heavy code so other people may get better results.
I don’t see a good way to accomplish what you’re aiming for without opaque types but I wonder if they’re too expensive to be the long-term status quo.
I also wonder if there’s still value in the rejected “alternative stable ABI for freethreading” to help get back some performance. I realise that means more explanations to people who are already pretty confused about packaging.
Disclaimer: I’m not really a user of the Stable ABI (I’m just implementing it for other people to use) so maybe my opinion should be ignored and we should see what actual users think.
Need to have the GIL/attached thread state
I’ve opened an issue about this so Petr and others are aware. And maybe the discussion is best kept on that issue rather than here.
Cython has historically let you access non-PyObject attributes of extension types without needing the hold the GIL/have an attached thread state. I think that’s generally a reasonable thing to let users do. This is probably a fairly common pattern in extension modules in general. Opaque objects makes that harder to do (although I also think most people will find it easier to avoid than Cython currently does).
I’ve managed to implement this via a hack making some possibly-not-forward-compatible assumptions that object layout will always be the same. That may be a bad move.
None of this is really a claim that we shouldn’t do this PEP (or the similar competing one). The performance point is certainly something we may want to think about in future.
Yeah. That’s something to improve, but not necessarily in 3.15.
The alternative is, still, separate builds for 3.14t, 3.15t, 3.16t, and so on until desired performance is there – while keeping an (older) abi3 build for non-free-threaded CPython.
The “obvious” implementation[1] would mean freezing the “new” PyObject layout forever. Given that a “frozen” PyObject layout was the biggest pain point in Stable ABI, I don’t think it’s an option.
no changes in CPython, except allowing limited API with free-threaded builds ↩︎
After packaging and C API WG discussions, I have another possible take on this: add abi3t as a new variant of a stable ABI, so that you can still build cp3.15-abi3 (non-free-threaded only) extensions. Defining both Py_GIL_DISABLED and Py_LIMITED_API would turn on the API limitations (opaque PyObject) and enable building against abi3t.
What this would change is “just” how things are named and enabled, but, perhaps it’s a better way to think about this. What do you think?
I think this is a great change. Having distinct ABI tags between the existing ABI and the free threaded ABI will help avoid user confusion and make clear which ABI a library targets.
I’m very excited by PEP 803 and I hope it is accepted!
(As before, my position is that I write C code that works with this stuff but don’t actually do anything as useful as trying actually running that code in Python. So a lot of these problem are beyond my remit and I’m not the best person to comment)
I think Steve’s main complaint is that the Limited API[1] will have a very sudden and quite fundamental deprecation where PyObject_HEAD will suddenly stop working between 3.14 and 3.15 with basically no advanced notice.
I think it’s a fair complaint, and your solution of having two flags does solve it. It’s a pretty dramatic breakage for most extension modules otherwise.
On the other side, the new ABI is almost a subset of the old Stable ABI but not quite because you do need to create modules in a different way. So I also see the argument for calling it something different. Just giving it a new name might also clear up a lot of the packaging confusion.
I think there is a problem with this approach. I understand the motivation for the change: not breaking source compatibility on GIL-enabled builds, as @steve.dower points out in capi-workgroup#92-comment. And I agree that that’s a good and probably necessary goal.
However, reusing Py_GIL_DISABLED to enable the new abi3t which will also support the GIL-enabled build and produce wheels tagged abi3.abi3t, will be cumbersome. If a package author makes source-level changes to support abi3t, then they can no longer build under a GIL-enabled 3.15 interpreter without disabling the Limited API completely. Which will require annoying fiddling with default build flags.
Related, testing with abi3t on a GIL-enabled interpreter, say in CI, would require building against a free-threaded interpreter, and then in a separate job install it for a GIL-enabled interpreter - not all that simple.
It seems more user-friendly for the “enable abi3t” to be a separate define from Py_GIL_DISABLED. It doesn’t really matter what it is, but I’d think not coupling it to implementation details (like the -DPy_OPAQUE_PYOBJECT you had) but rather a separate knob (e.g., -DPy_LIMITED_API_TARGET_ABI3T).
Note: in the above I am assuming that it’s a bad idea to define Py_GIL_DISABLED when building under a GIL-enabled interpreter, because that define controls a lot of other things, both inside and outside CPython. I hope you didn’t have that in mind?
A separate knob is also not completely free, but much easier to deal with, and until that’s implemented in build systems/backends, any package author can add the extra define themselves pretty easily.
).