C API Working Group and plan to get a Python C API compatible with alternative Python implementations?

[paugier (Pierre Augier)]

Thanks a lot @steve-s for this nice and important post. It seems to me that the situation is very clear now, with 3 possibilities:

1. Status quo: the ecosystem waits and follows gradual and slow evolutions of the historical CPython C API.
2. HPy revival (new funding required): Numpy ported to HPy + HPy backend for Cython.
3. A new C API supported by CPython (but not in Python.h) inspired by HPy + something like pythoncapi-compat for usability on something like CPython >=3.10 + HPy like universal ABI.

I think it would be very valuable to have the points of view of CPython core devs (and of the C API WG) on this fact, or at least on the last possibility.

There were also two interesting points of view / pieces of information by Ralf Gommers (@rgommers) and Matti Picus (@mattip). I take the liberty to quote them here since it seems to me that they are highly relevant:

From Mailman 3 [Numpy-discussion] Re: Better compatibility of the Python scientific/data stack with fast Python interpreters - NumPy-Discussion - python.org

@rgommers

Having HPy or something like it will be very nice and lead ot long-term benefits. The problem with HPy seems to be more a social one at this point: if CPython core devs don’t want to adopt it but do their own “make the C API more opaque” strategy, then more effort on HPy isn’t going to help. If you’re going to dig into this more, I suggest trying to get a very good sense of what the CPython core dev team, and in particular its C API Workgroup, is thinking/planning. That will inform whether the right strategy is to help their efforts along, or work on HPy.

From Mailman 3 [pypy-dev] Re: [Numpy-discussion] Better compatibility of the Python scientific/data stack with fast Python interpreters - pypy-dev - python.org

@mattip

It does not appear there will be a PyPy 3.12. When NumPy drops support
for Python3.11 (Jan 2026), future versions based on the CPython C-API
will defacto no longer be relevant for PyPy3.11. However, if HPy
succeeds, there is a chance that a NumPy “universal” HPy wheel will
still work on PyPy 3.11, as the API will be abstracted away from a
direct connection with the CPython C-API. This is one of the
attractions of HPy, that puts it in a different category than Cython
or nanobind: universal HPy binaries can run on any python interpreter
or version that supports the abstracted API.

With the current level of active contributions to PyPy, I am not
convinced we can help making HPy a reality, even though it is a very
well thought out solution to once and for all detaching third party
libraries from a tight integration with the CPython C-API. I agree HPy
would be advantageous to both the CPython core team and package
maintainers. The yearly churn of packages needing to update for new
CPython C-API would be localized in the HPy layer for the new CPython
(similar to Cython, nanobind, and PyO3), and the CPython maintainers
would be free to change more aspects of the CPython internals that
unintentionally impact the external C-API (which differentiates HPy
from Cython, nanobind, and PyO3). But not all good ideas get to win
out and become the popular, goto solution. PyPy itself is an example
that, unfortunately.