Maybe the C API documentation should have a header, on each page, to mention recommended high-level projects such as Cython, nanobind, HPy, etc. Honestly, I would prefer that only these high-level projects would consume our C API, and that all others “C extensions” would only use these abstractions. It would reduce the number of impacted projects when the C API evolves. And it evolves at each Python (x.y) release: some functions are removed, some functions are added.
I don’t think that we need to have a preference between Cython and HPy. Users should make their own choices depending on their needs and use cases.
Cython and a few others are already mentions in two places:
Thanks for clarifying. I think it is fair to say that someone needs to (formally) present a convincing case to the SC that Python (the language) is in need of a new C API. They could include whatever justification they want - maybe emphasizing that a new C API allows for new optimizations in implementations (including the reference implementation), maybe that it would allow Python to be used in other ecosystems (.NET, JAVA, etc), maybe it could be a promise of reducing the energy footprint of Python, whatever.
The SC alone has jurisdiction on whether to accept this as a mandate, and also to make whatever plans that could be necessary to execute the mandate. For example, they could theoretically choose to seek outside funding for a working group, make helpful changes to the life cycle of Python minor versions or release cadence, provide the existing C API work group with new responsibilities or expecations, plan for a Python 4 with a new API (which might make sense for other reasons, like nogil or debug), or whatever else they choose.
It is not up to the C API working group, as it is currently organized, to presume to have the power to proclaim such a mandate, let alone do “what it takes”, whatever that is, to execute that mandate.
Seriously @steve.dower, could you drop the dismissive-aggressive tone here? You’re doing this on a regular basis and it’s certainly not a pleasant read.
(I might respond to the substance of your message but I wanted to get this across)
My first few attempts to explain that we are not responsible for numpy’s supposed inability to do anything fell on deaf ears. You’re welcome to clarify for the people who keep coming back with the same question that it was answered ages ago and they’re ignoring it.
I apologise for getting frustrated. It’s threads like this that made me wish I cared less about Python, so I could allow poor decisions to damage it. Unfortunately, I really want to keep those from happening, so I feel compelled to keep responding, even when it’s actively painful to me.
I understand the concern. However, looking at the currently active CPython contributors, many of whom are competent and experienced engineers (and knowing that such engineers also populate the SC), I think it is reasonable to rely on them from time to time too
I guess I should forget this last post even if it is a bit sad from my point of view since I triggered this thread and indeed insisted to get explicit and clear answers. No comment.
Anyway, I think that this thread was very useful to really understand the situation, which is very worrying regarding the long-term health of Python and of its ecosystem.
In its current state, the Python ecosystem is highly incompatible with any attempt to get fast and efficient Python implementations. This strongly affects PyPy and GraalPy and will certainly affect the Faster CPython project in few years. For example, if the ecosystem is not fixed, the most probable destiny of PyPy is to slowly die. This is not good news for Python.
Unfortunately, Python is organized in such a way that very few people really care about the facts described in the previous paragraph. In particular, this does not strongly affect CPython and projects like Numpy, scikits, Tensorflow or PyTorch.
Interestingly, CPython maintainers think that projects like Numpy should not directly use the CPython C API.
HPy (with its universal wheels) is a key project to improve the situation. If HPy does not get more care to be finalized and adopted by key actors (in particular Numpy and Cython), the situation won’t really improve in a reasonable time scale (something than less than ten years). Inversely, if HPy succeeds (with Numpy-HPy, Cython-HPy, PyO3-HPy, …), the situation would basically be fixed and the Python ecosystem would be in a much better state, in particular with universal wheels for much fundamental packages.
The scope of the C API working group is very narrow. It is “just” focused on the historical CPython C API and on its gradual and slow evolution.
There are no other concrete proposal alternative to HPy to fix the situation. The community should not wait for an “official” solution somehow supported by Python core developers, the C API working group or the SC.
A last note about something that stroke me: few messages discuss a possible deprecation/disappearance of the historical CPython C API. It seems to me that it is misleading to mention this possibility in this thread because it is not needed at all to fix the problematic situation and I don’t think anybody proposed that. I actually find it funny that some persons felt the need to mention troubles associated with such non existent project
Interesting way to summarize this problem! Thanks a lot. Then it’s only my problem and everything should be fine.
The fact that PyPy and GraalPy are still in 2025 quite unusable for anything needing the scientific Python stack is negligible. The bad news about PyPy’s state in 2025 should be unimportant for the Python community.
I’m not sure it’s very useful to talk about me. As I already wrote, I discuss about these problems to understand them and see if some money could not be raised to do some impactful work. It might be that the problems discussed in this thread are not actually so important for the future of Python. It might also well be that the overall problem is unsolvable but I’d like to check if there could be a nice solution. It might be that the solution involving HPy is actually not good or not feasible, for technical or human reasons.
But I feel that it should be discussed collectively.
I have to admit that I find a bit unfortunate to get such negative replies about my point of view and this thread (in few posts), in particular as if I asked anyone to work for free.
Interesting. I guess I should have written “Some CPython maintainers”… I’m not an English speaker and I thought “CPython maintainers” didn’t mean “All CPython maintainers”.
No, I do not have new questions. I think I already described issues and asked few explicit and clear questions. I also got many interesting answers. I don’t know why you ask this now. My last post was indeed not for questions but only to summarize what I learned and think now.
Anyway, I feel that I have to stop interacting on this thread because it starts to become useless and a bit toxic, at least for me. Thanks a lot to all the participants for this discussion.
Hi @paugier, I can understand your frustration, but I guess we just have to accept that there’s no clear strategy for the Python C API going forward, apart from making sure that the existing C API continues to evolve and improve over time.
Also note that there are different groups within the core devs, ranging from people who’d like trim down the C API a lot, or even replace it with something that’s mostly opaque, all the way to people who prefer to have a rich and comfortable native API, with special focus on speed and performance. Another dimension where people have different opinions is rate of change in the API / ABI.
IMO, this is actually a good setup, since we’ll likely get the best from both approaches over time. Whether this will be closer to Cython (which makes use of the rich API and often taps into internals to gain performance) or to something like HPy (which uses a more opaque high level approach) remains to be seen.
Projects using the C API should chime in to such discussions to provide real world feedback on where things should be heading.
On 12/05/2025 17:57, Marc-André Lemburg via Discussions on Python.org
wrote:
Hi @paugier, I can understand your frustration, but I guess we just
have to accept that there’s no clear strategy for the Python C API
going forward, apart from making sure that the existing C API
continues to evolve and improve over time.
Also note that there are different groups within the core devs,
ranging from people who’d like trim down the C API a lot, or even
replace it with something that’s mostly opaque, all the way to people
who prefer to have a rich and comfortable native API, with special
focus on speed and performance. Another dimension where people have
different opinions is rate of change in the API / ABI.
IMO, this is actually a good setup, since we’ll likely get the best
from both approaches over time. Whether this will be closer to Cython
(which makes use of the rich API and often taps into internals to gain
performance) or to something like HPy (which uses a more opaque high
level approach) remains to be seen.
Projects using the C API should chime in to such discussions to
provide real world feedback on where things should be heading.
As the author of a project that originally targeted Python v1.5, I’m
most interested in the long term commitment from the maintainers of a
(changing) API. In reality this means the Python core devs. I can’t see
myself choosing to target an API maintained by a 3rd party - even if it
was “better”.
As the author of an abstraction layer, I’d be surprised if any
abstraction layer was so complete that it wasn’t necessary to provide
the ability to access the underlying C API.
I apologize for my tone. Your posts seemed increasingly accusatory to me, and I failed both to assume good intention and to account for non-native English.
External projects like HPy are what can move quickly. If you want solutions in a few years, invest there.
CPython tries to be more stable, and thus it’s moving slowly. It’s hard to get a change in, because you need to convince a lot of people that it’s a good idea – both in general and in the specific details.
If an external project like HPy is successful, it’s a lot easier to show that its ideas are good.
Even if HPy ultimately fails for lack of maintenance, it has already shown that its ideas are generally possible to achieve. Notably, one of the things it shows is that it’s possible to build HPy on top of the current C API; one thing that’s unclear is how much do people (as in, contributors to various projects) want that.
If CPython exposed a HPy-like API, it would get little immediate benefit, since it still needs to support the existing API. If the motivation is “making Python implementations faster” – especially non-CPython ones – then HPy is again the project to invest in.
If the goal is to advance/sustain HPy, someone needs to help maintaining it and porting projects to it. And perhaps work on CPython side too, to make HPy easier to implement.
If their goal is for CPython to adopt HPy, their best bet would probably be to start writing down motivations and convincing people.
I would publish universal wheels based on HPy if it could be done with Cython. Even if the HPy wheels were a little slower it could still be possible to have the universal wheels to be used for the Python versions and implementations that we currently don’t provide any wheels for.
The blocker is Cython support though and I assume that applies to NumPy as well. From above it sounds like Cython support needs resource to make it work.
