Feedback on abstract for pyconfr? Universal Python extensions

Python Help
1

I’d like to submit an abstract for a short presentation at PyConFr. Since the subject is a bit complex, I thought I could ask feedback from people here.

The title would be Universal Python extensions: performance, compatibility, sustainability, and less CO₂.

and the text:

Python’s long-term relevance for numerical computing faces a long-term threat, not from
external competition alone, but from fundamental architectural decisions that prevent the
entire ecosystem from achieving better performance. While PyPy and GraalPy can make pure
Python code 4x faster than CPython, they face an existential crisis: the moment you
import NumPy or any package with C extensions, that performance advantage disappears
completely.

This isn’t just a technical curiosity. It’s putting alternative Python implementations in
genuine danger of end-of-life status.

The hard truth about Python’s performance future

The compatibility trap: Python’s C extension system creates an insurmountable barrier
between implementations. Popular packages like NumPy, Pandas, and scikit-learn are locked
into CPython-specific APIs, making faster interpreters effectively unusable for
real-world numerical work. Worse, the current C API also constrains CPython itself,
preventing future performance improvements that would break existing extensions.

HPy: promise and reality: The HPy project aimed to solve this with a universal C API,
showing impressive preliminary results with extensions that work across CPython, PyPy,
and GraalPy. However, the project has stalled without sufficient corporate and community
support, a critical opportunity slipping away.

The limited API alternative: CPython’s limited C API offers another path, but with
significant constraints and unclear long-term viability for the ecosystem’s needs.

What this talk proposes

Rather than accepting this situation, we need coordinated community action. This talk
argues for:

  • Community recognition: The Python community must acknowledge these performance
    limitations as a serious long-term challenge, not just a nice-to-have improvement.

  • Official support: A PEP process to formalize the need for universal C API solutions
    compatible with JIT Python compilers, with a dedicated working group to study possible
    solutions and coordinate actions across different ecosystem actors.

Why we need to act now

The stakes extend far beyond technical performance. Enabling more efficient Python
interpreters would have two transformative effects:

  • More Python: Performance-sensitive code could be written in pure Python with
    object-oriented patterns and small objects, rather than requiring native extensions,
    simplifying development and deployment.

  • Environmental impact: Given Python’s massive global usage, significant performance
    improvements would translate to measurable CO2 emission reductions across millions of
    applications worldwide.

Without community mobilization, we risk losing both alternative Python implementations
and Python’s competitive position in performance-critical domains. The technical
foundations exist but need coordinated effort to become reality.

Target Audience

This talk is for Python developers who care about the ecosystem’s long-term health.
Whether you’re a package maintainer or concerned user, you’ll understand the stakes and
learn how to contribute to solutions.

Come prepared to discuss hard truths about Python’s future and leave with concrete
actions for community engagement.

Of course, it’s quite short so there are some partly inexact statements. My goal with this potential presentation is to trigger discussions. For example, I’d like to better understand what can be done on this subject with the limited API.

Anyway, feedback would be appreciated.

2

It’s hard to judge the talk from this.

It seems that you want to both recruit people to work on the solution, and discuss what the solutions should be (and maybe even what the problems are).
It also seems that you want to tell volunteers what their priorities should be – and that’s not likely to work (unless you’re hiring).

Last time, the discussion took a wrong turn (for which I apologize again).
I don’t think arguing via text will work; would you be open for a real-time chat? I’d try to understand your concerns better, and suggest what you could do. If you’re at EuroPython next week, catch me there; if not DM me to schedule a call the week after that.

3

Do you have an example of a Python implementation of existing C Code that runs faster in PyPy
or GraalPy etc.?

It seems too good to be true that all the tools that compile Python (to the same assembly as C compiles to) like Nuitka, Mypyc and Codon have become unnecessary, and all we now need to do to get optimum performance in some applications is switch interpreters.

To be worth the audience’s time, the talk should be up to date with the latest efforts to make CPython faster, and provide a convincing example of code that cannot simply be redesigned (e.g. refactoring into two separate processes using the different interpreters), that does so much in pure Python that PyPy or GraalPy is a real benefit over CPython, but that also uses Numpy or SciPy for peformance elsewhere. With benchmarks proving this is can be a real bottleneck.

I’d like to see updated versions of the claimed performance improvements from the latest and best GraalPy and PyPy can do, to the same for CPython. Ideally including the best JIT, and maybe even free threading and GC-free Pythons. Not just the same old obsolete comparison between PyPy3.10 and a CPython 3.7.19 that’s been on PyPy’s website for years (and CPython 3.7.17 was the last security release before its EOL 2023, so I’d be interested where the PyPy team got 3.7.19 from Python Release Python 3.7.17 | Python.org)

The example problem doesn’t need to contain every micro-optimisation, but the whole thesis is about performance, so comparisons need to be against a sensible alternative well-optimised CPython system. No comparing single threaded code for an embarrassingly parallel problem, when a hugely powerful multi-core machine is available, like Mojo did.

I’m sure running Numpy etc. in PyPy and GraalPy would be great, but that’s the trade off users of those implementations picked, and not a problem the CPython ecosystem should be expected to solve.

If the PyPy or GraalPy devs want to build a compatibility layer, so that their users can import Numpy etc., then great. Also, I’m not claiming it’s more efficient than CPython (quite the opposite, in fact) but perhaps it’s useful for them to look at PythonNet? There are teething problems, but it is now possible to import Numpy in some kind of CPython embedded in .Net.

If universal extensions could easily be built using an alternative to scikit-build and maturin etc. or a plug-in for setuptools, then I think a lot of people would try it. I suspect ABI compatibility is a hard problem though.

4

Thanks for your feedback. I will try to use that to improve the abstract.

I should be clear that my plans are:

  1. Convince people that there is a fundamental issue (incompatibility of the ecosystem to any attempt to get a major performance boost for Python) and that something should be done at the level of the roots of the ecosystem.
  2. Better understand what can be done with the limited API (something sufficient to be able to obtain something like HPy universal wheels? With HPy like handles and ctx? What about HPy debug mode? With “typed functions” that could be used by Python JITs to avoid boxing/unboxing?)
  3. Get some kind of official acknowledgment that there is a problem and that a good and realistic solution has to be designed and made usable in a relatively short period of time (few years and not ten years).
  4. Potentially find funding to push a solution (which requires the official acknowledgment) and hire people.

This presentation will mostly be about point 1 and I wish to work on 2. before the talk to be able to say something reasonable.

Thanks a lot for the proposition of real-time chat! I won’t be at EuroPython so it will not be this time. I heard that there will be discussions on the CPython C API. I’m going to try to get information on what will be said. If I am brave enough, I may be able to contact you after my holidays. Proposing this talk will push me to do it!

1 Like
5

Yes, there are examples of such cases (for example Reducing the ecological impact of computing through education and Python compilers | Nature Astronomy, Reducing the ecological impact of computing through education and Python compilers - Archive ouverte HAL). However, it is usually due to problems in the C implementations because of course, one could improve them.

However, this is really not very interesting regarding this subject. We don’t require PyPy or GraalPy to be faster than C. They just need to be much faster than current CPython and not much slower than C.

Moreover, of course, many things will still have to be written in native languages! Having a much faster Python (PyPy, GraalPy or future CPython) and a compatible Python ecosystem just allows more things to be written in Python.

I am an early adopter of Pythran, which is one of these tools. I think I’m partly responsible of the fact that Pythran is now used in Scipy and scikit-image, so I know what you mean. However, I also know the limitations of such tools and I know that it would be very convenient to be able to avoid their usage for some cases, and in general to be able to put more pressure on the Python interpreter in some hot loops instead of always avoiding it.

We can for example quickly play with nbabel/py/bench_purepy_Point.py at main · paugier/nbabel · GitHub

cd nbabel/py
uv venv -p cpython-3.14.0b4-linux-x86_64-gnu .venv-cpy3.14
uv venv -p pypy .venv-pypy
.venv-pypy/bin/python bench_purepy_Point.py ../data/input64
10001 time steps run in 0:00:01.004117
$ /usr/bin/python3.11 bench_purepy_Point.py ../data/input64
10001 time steps run in 0:00:54.682014
PYTHON_JIT=0 .venv-cpy3.14/bin/python bench_purepy_Point.py ../data/input64
10001 time steps run in 0:00:45.776208
PYTHON_JIT=1 .venv-cpy3.14/bin/python bench_purepy_Point.py ../data/input64
10001 time steps run in 0:00:46.547922

The code in bench_purepy_Point.py is not that crazy neither specifically optimized for PyPy (for something like that, you can have a look at nbabel/py/bench_purepy_Particle.py at main · paugier/nbabel · GitHub).

CPython 3.14 even with its JIT is still much slower than performance oriented alternative implementations, which makes sense technically taken into account the methods used in these implementations (advanced GC algorithms, moving objects, unboxed native objects, meta tracing JIT, …).

Moreover, if the roots of the ecosystem are not made compatible with modern fast Python interpreters, CPython cannot go towards this direction without decreasing the performance in terms of interactions with extensions (which of course won’t happen!). CPython could change internally to be faster for pure Python but then it would have to use an emulation layer to emulate its own old implementation (something like PyPy cpyext). To be faster both for pure Python and for code using extensions, one needs to change the extensions. (Not all extensions need to be changed, with an emulation layer, CPython could still support different API.)

I guess I’m a little bit too pessimistic and that with a huge amount of money, CPython performance could be improved without decreasing the performance with extensions. However, one also has to consider the cost and the time for this plan. Unfortunately, I heard that the Faster CPython project was no longer supported by Microsoft.

Note that a new HPy-like C API without PyObject * is discussed as a need for the Faster CPython project in ideas/3.14 at main · faster-cpython/ideas · GitHub

It seems that this is indeed often a mainstream idea. The problems of PyPy and GraalPy (which are literally slowly killing PyPy) would only be PyPy and GraalPy problems. I claim that they are Python problems, that should be seriously taken into account by the Python community.

I think that Numpy (and other popular Python projects) should not be considered as the CPython ecosystem but the Python ecosystem. And that the Python community should consider its alternative interpreters as wealth that we should care about.

If PyPy finishes end-of-life in few years, I’m very pessimistic for other strategies to get a major performance boost for Python (when using the ecosystem of course).

More generally, the questions of the quasi-monopoly of CPython and the consideration to alternative Python implementations are important. It is really reasonable, even in a long-term perspective, that all tasks will be done with CPython and that alternative implementations have to follow - or emulate - CPython for all choices, in terms of language (normal - though not simple considering free-threading) but also for the implementation details (in particular reference counting)? I don’t think it is realistic to have CPython really optimized for everything.

Therefore, I think the pillars of our Python ecosystem should be “Python implementation agnostic” and that there should be a good technical solution to implement proper universal wheels.

These compatibility layers already exist. It has been shown that they are both expensive to maintain and a disaster for performance.

For that, HPy clearly showed that Universal wheels are possible. It already works.

2 Likes
6

The Nature article is paywalled, but thanks for the very thorough response.

7

The N-body problem is a nice example. I reran the code snippet, and the performance of the CPython versions was even worse for me - I got bored waiting for them to end.

Some of the other benchmarks in the repo also have greatly improved performance. If you had time in your talk I would like to here more why out of all those options, the way forward is to revamp PyPy/ GraalPy/ HPy, or rebuild extensions for those targets.

I’m not an expert in it, but I don’t think Python invented anything particularly special for simple C-extensions to CPython (that don’t need to accept pure Python callbacks, or manage the Python process’s GC etc.).

Simple extensions that Python can call out to, are just linux so or windows .dll dynamic shared objects/ libraries. There’s nothing to stop any other C based executable or library that matches the interface, from building against one and linking to it. I wonder what the problem was with PyPy and GraalPy. Java code can definitely call out to C++ libraries - loads of Android apps do so. So before throwing loads of effort at this, it would be good to ask the devs of those projects what the challenges were, or what is needed.

I’m much more interested in seeing the presentation now.

8

If you’re using ctypes (do PyPy and GraalPython support ctypes?) then you can call any standard .so/.dll. But when people talk about extensions, they mean code that uses the CPython “C API”, which was written very early in the lifetime of CPython, with portability most definitely not a goal (there wasn’t any implementation other than CPython at the time).

Since then, the C API has improved, but it’s a very slow process because the original C API exposed a lot of implementation details, and moving away from that in a way that maintains as much compatibility as possible, and provides migration paths when it isn’t, is just a really hard problem.

PyPy tried to emulate the C API, I believe, but performance was terrible (not surprisingly - there’s so much difference in the two implementations that the “translation layer” is significant).

Other APIs for building extensions exist, most notably HPy. The HPy project was created, I believe, with the goal of being a portable C API for Python. And as far as I know, it did a pretty good job at that. It supports CPython, PyPy and GraalPython, I think. But like many projects, it suffers from a lack of maintainer resource and investment, so it’s struggling. It’s not at all obvious how to address that problem. If I understand @paugier’s plan correctly, he’s looking to get community support and ownership for the idea that HPy could become the “official” C API for Python. But the idea on its own won’t be enough - we’ll need people to step up and offer to help, either maintaining and developing HPy, or working with the CPython core devs to develop a realistic plan for initially making HPy a complete alternative to the existing C API, and ultimately migrating all existing libraries to the HPy interface, phasing out the old C API. This is likely to be a multi-year (if not multi-decade) project, so it’ll need some serious commitment from the community - hence the need for publicising the idea.

@paugier raised the idea with the core devs previously (in the thread linked by @encukou). Reception was lukewarm, I’d say, with the core devs seeing this as something they aren’t in a position to take on. Personally, I agree with this view - the core devs are busy enough maintaining the existing C API, so resource for developing something new will likely have to come from elsewhere. So a reasonable next step is to try to get interest from the wider community - hence this thread. (@paugier I hope I’ve not misrepresented or misunderstood your intentions here, please feel free to correct me or clarify further if you think it’s needed).

2 Likes
9

This is the aspect I believe crosses the line into telling volunteers what their priorities should be, which is pretty much guaranteed not to be effective.

Folks are chipping away at the CPython API coupling problem for assorted reasons, but it’s far from clear that the next few years are going to be more significant on that front than the last 15 or so since the stable ABI was first defined.

It also indicates a belief that there is an available quickly deployable solution that we’ve just been collectively ignoring for the past decade and a half.

I do think a survey of the efforts to date, what has been achieved so far, and where these efforts still fall short would be a genuinely interesting talk. Trying to inject a sense of increased urgency is the aspect that seems more likely to come across as dismissive of the efforts of those already working in the area and risk not only being unproductive, but even being actively counter-productive.

2 Likes
10

Thanks for your nice explanations. I’d need to correct few things.

First, I don’t understand this notion of “ownership”.

More importantly, I think that an HPy-like API should become AN official supported C API so that it becomes possible to write packages like Numpy in a Python JIT-friendly way and produce something like HPy universal extensions.

There is absolutely no need for this project to deprecate the CPython C API, which is of course not an option!

Only few pillars of the ecosystems would need to be ported to this Python JIT-friendly C API so that they can provide universal wheels. Numpy, Matplotlib (which has already been ported to HPy by the way) and few projects already using the limited API would already be very good. Tools like Cython, Pythran or PyO3 would need to be able to use this API so that all packages based on these tools would be able to also upload universal wheels.

I also agree. My current thinking about this is that to get resource for such big project, one needs to be able to show that the Python community officially agrees that this issue is important for its future. If a group of experts is dedicated to evaluate the feasibility of the different technical options and produces a clear document, then people can try to gather funding.

I agree that it is a long term project, but I tend to be less pessimistic. There are two very positive points.

  • First, we have HPy and its developers. It’s not technically finalized. It is stalled without corporate or institutional support, but it basically works! Matplotlib and an old Numpy version have been ported. Universal extensions are impressive in practice.

  • A huge work has been done on the CPython side, with recent very interesting developments, in particular related to free-threading. Also CPython will need a JIT-friendly C API if it JIT starts to be efficient.

It seems to me that it’s the good time to put that together and to declare a common target: we could try to have in few years most pillars of the Python ecosystem implemented with Python JIT-friendly technologies.

1 Like
11

Conversely, I don’t understand what you mean by “an official supported C API”. Unless you mean that the core devs create and maintain such a thing, in which case isn’t that what your previous post (which didn’t get much support from the core devs) was about?

Getting the community to agree that we should have such an API won’t make developer resources appear out of nowhere, and “the community want this” won’t make the core devs more likely to work on such an API than they already are.

My point about “ownership” was that community members will need to offer to do the work, not just express support for the idea that someone else (presumably the core devs?) do the work.

I personally think that by framing the problem as “we need to add a new official API to CPython” you’re making it harder to get resource, precisely because it is going to be disruptive to the existing core dev work on the C API, as well as being politically more difficult. If, on the other hand, you were to advocate for funding to establish a sustainable basis for the HPy project, and to contribute HPy bindings to key ecosystem projects, I think you’d be much more likely to get support. It would only take one major company with an interest in HPy to fund such a project to make it viable.

If it’s an external project, like HPy, then I’m much more optimistic. What makes me pessimistic is the fact that you seem to want something more than a healthy, well-used HPy project. It’s the requirement of some sort of “official” status that I think is the problem here.

By the way - I don’t know if you are a HPy maintainer, but if not, do the HPy developers support your proposed initiative here? Because that seems like it needs to be the first step. And in fact, if you can announce in your PyConFR talk that the HPy developers are on board with the approach you’re peoposing, that would be a really useful starting point.

2 Likes
12

Something worth noting is that the CPython core devs interested in the C API and the HPy devs do talk to each other - it isn’t like HPy is completely divorced from the reference interpreter development process with no way to provide feedback on changes that would benefit the ecosystem as a whole.

It’s just that avoiding being tightly coupled to CPython is a large part of the point of HPy, so the areas where direct CPython contributions are mostly needed are when there are big performance gaps between the stable ABI and the full CPython API (the vector call API comes to mind as an example of a change along those lines, as well as the integer import/export API and the open proposal for a byteswriter API).

Browsing the PEP index makes me wonder if it might be worthwhile to define a dedicated “C API” topic (backfilling historical PEPs appropriately), so folks interested in this area can more easily find relevant PEPs.

3 Likes
13

I would be perfectly happy with just a healthy, well-used HPy project. However, I’m now quite convinced that a healthy, well-used HPy project cannot exist without being somehow “officially” recognized.

I guess you remember that HPy was supported by Oracle during few years, since it was their strategy for GraalPy. They stopped their support and I heard that it has to do with the total lack of interest of Python core devs. I understood that Oracle now has much less ambition and they just want to support the limited API with an emulation layer (which is now a bit easier). Of course, the potential global benefit for Python is much smaller.

Beside, PyPy is not a good shape, with very few maintainers and no corporate or institutional support.

So HPy is currently really stalled without clear direction.

One can read the point of view of a HPy and GraalPy maintainer here. @steve-s described two possible strategies for the future of HPy (fully external project or a new official Python VM API).

As usual about HPy stuff, reception by CPython core devs was clearly lukewarm.

I think there will be some discussions about Python APIs during EuroPython next week. The situation is clearly moving with free-threading, so we’ll see. For example CPython would need to support limited API extensions that can be used both with its free-threading build and its standard build, which looks like “Universal” extensions.

Regarding the needed “officiality”, I agree that it has to be clearly described.

A PEP could state that Python implementations should natively support a version of a universal JIT-friendly ABI (and the reference implementation would of course do it), with a working group dedicated to define the ABI and the corresponding API and decide how it should be done in practice (with HPy, another external project or directly with the limited API).

This does not mean any “ownership” of HPy by CPython. CPython would just need to natively support the universal ABI.

From the point of view of maintainers of projects that could use HPy, it would be a clear signal that it makes sense to invest on this technology.

With such PEP (or another clear indication of real interest by some CPython people), it would be much simpler to get funding and supports for the needed work.

Without such signal, nobody is going to invest on HPy.

Again, the HPy project is really too fundamental to be only supported by PyPy and GraalPy people. HPy showed that it is technically possible to have a JIT-friendly C API allowing universal ABI/extensions. If we think that it is useful for Python, this global project should be officially declared and launched. Considering the potential impact, funding, support and developer resources would come.

1 Like
14

You mentioned earlier that at one point there were patches that allowed NumPy to build against hpy. I’m not aware of a pull request for this or anyone actively working on the current version of NumPy.

Speaking as a NumPy maintainer, I think we’d be open to supporting hpy, assuming the code complexity increase is manageable. I also doubt it will happen without a new contributor coming along to push it forward. If NumPy shipped an hpy-compatible wheel, maybe that makes it easier for other projects to do the same, and to discover what issues there are outside of NumPy and matplotlib.

IMO, working on concrete ecosystem improvements along those lines would be a more productive use of your energies than trying to get people to do the hard work for you.

1 Like
15

HPy are attempting to port Numpy themselves. Or at least, they attempted it - the last commit was 2 years ago

16

This is also the kind of feedback that’s needed on the CPython side to guide limited API enhancements for cases where using HPy genuinely hurts runtime performance.

17

Indeed, it seems to me that we have a very interesting opportunity considering on the one side what has been done with HPy and on the other side the changes in CPython related to free-threading and performance (CPython might need in few years a JIT-friendly C API).

I don’t see why a HPy based solution would not be relatively “quickly deployable” (of course, it is about few years and of course, one needs money). I also note that I didn’t read strong arguments explaining why such solution would not be doable technically and good for the language.

I don’t mean that there are no technical and social difficulties with such path or that one cannot find good arguments against it. Just I don’t know them (which is quite normal since there are so many things that I don’t know :slightly_smiling_face:).

18

Numpy 1 has been ported to HPy (thanks @JamesParrott for the link). To be honest, I don’t think the problem of the Numpy C API has yet been tackled (and this is not a simple problem). But there were a lot of changes in C code with Numpy 2 so it seems that it would be easier to start the porting from zero…

Unfortunately, the HPy project is in a state for which such hobbyist individual contributions are rather useless. First, we need to evaluate if it worth it to relaunch the project, and then potentially to relaunch the project with a lot of communication and coordination, and with corporate or institutional funding.

In the past and current conditions, the project is not viable. We first need to change the conditions (general understanding of the usefulness, interest from CPython side, a PEP, …).

19

That may or may not be true, but the issue still remains, someone needs to commit to doing it. Discussions, community support, even “official” status are all very well, but none of that matters without someone willing to step up and do the work.

Who is that “someone”? It’s not the core developers - they already have plenty of work to do. It’s not the HPy team - they are struggling to keep HPy going at its current level. Are you planning on finding the necessary people to do this work (or doing it yourself)? If you are, that’s great and I’d support you (with words only, I have no free time to offer). If you’re not, though, who is?

This discussion feels like it’s turning into more wishful thinking than any form of practical plan. That’s a shame - I’d like to see something like this happen. But I’m not willing to support a proposal that doesn’t have a realistic plan for success, and I definitely don’t want to support something that simply loads more expectations onto the core team without offering any practical help. Which is where I feel we’re ending up :slightly_frowning_face:

20

I think @mattip is both involved in Numpy and HPy so I assume it was a conscious decision to keep the Numpy HPy port separate and not to merge it back at that stage.

1 Like