I would be perfectly happy with just a healthy, well-used HPy project. However, I’m now quite convinced that a healthy, well-used HPy project cannot exist without being somehow “officially” recognized.
I guess you remember that HPy was supported by Oracle during few years, since it was their strategy for GraalPy. They stopped their support and I heard that it has to do with the total lack of interest of Python core devs. I understood that Oracle now has much less ambition and they just want to support the limited API with an emulation layer (which is now a bit easier). Of course, the potential global benefit for Python is much smaller.
Beside, PyPy is not a good shape, with very few maintainers and no corporate or institutional support.
So HPy is currently really stalled without clear direction.
One can read the point of view of a HPy and GraalPy maintainer here. @steve-s described two possible strategies for the future of HPy (fully external project or a new official Python VM API).
As usual about HPy stuff, reception by CPython core devs was clearly lukewarm.
I think there will be some discussions about Python APIs during EuroPython next week. The situation is clearly moving with free-threading, so we’ll see. For example CPython would need to support limited API extensions that can be used both with its free-threading build and its standard build, which looks like “Universal” extensions.
Regarding the needed “officiality”, I agree that it has to be clearly described.
A PEP could state that Python implementations should natively support a version of a universal JIT-friendly ABI (and the reference implementation would of course do it), with a working group dedicated to define the ABI and the corresponding API and decide how it should be done in practice (with HPy, another external project or directly with the limited API).
This does not mean any “ownership” of HPy by CPython. CPython would just need to natively support the universal ABI.
From the point of view of maintainers of projects that could use HPy, it would be a clear signal that it makes sense to invest on this technology.
With such PEP (or another clear indication of real interest by some CPython people), it would be much simpler to get funding and supports for the needed work.
Without such signal, nobody is going to invest on HPy.
Again, the HPy project is really too fundamental to be only supported by PyPy and GraalPy people. HPy showed that it is technically possible to have a JIT-friendly C API allowing universal ABI/extensions. If we think that it is useful for Python, this global project should be officially declared and launched. Considering the potential impact, funding, support and developer resources would come.