C API Working Group and plan to get a Python C API compatible with alternative Python implementations?

Core Development C API
21

This isn’t a practical suggestion and is missing the point of the previous discussion in favour of trying to promote your early-prototype pet project.

Practically you’ll find that most non-trivial Python programs rely heavily on the Python object model. So just translating the logic to C doesn’t work - you have to reimplement (or just use) much of the Python runtime.

You’ll then find that most non-trivial Python program end up calling library code from many low-level languages. So transpiling to just one language isn’t practical because the full program is a mixture of Python, C, C++, Rust, etc. and you somehow have to interface with all of that.

5 Likes
22

On the subject of HPy:

Something that hasn’t been mentioned here (but was in the HPy authors’ original plan) is the possibility of the higher-level binding generators targeting HPy themselves (i.e. most extension authors aren’t actually writing Python C API code, they’re getting other tools to write C API code for them).

The HPy authors realized that and (to their credit) have prepared a prototype Cython fork with some level of HPy support, although I think it’s a fairly early prototype. I do still plan to try to help them merge it, although it’s been a while since I said that so it’s probably sounding like an increasingly hollow promise now.

The barriers I think are:

  • like everyone else, I have limited time and there’s plenty of other things to be working on,

  • I’m not sure if it’ll work with all of Cython - there’s definitely places where we want to generate C functions with defined signatures that don’t include a context as the first argument,

  • It’s a lot of work - on the basis that supporting the Limited API was a lot of work, and the Limited API is a lot closer to the existing full Python C API, and I think initial Limited API prototype that was committed to Cython was probably a little more complete,

  • It doesn’t actually get us any new platforms right now - with their C API emulation PyPy+Cython actually works quite well and GraalPython+Cython works a little less well but kind of works. So it may be making things better for them but it isn’t a dramatic improvement in where Cython modules can run.

  • It’s a bit unknown whether anyone will actually use it. Obviously this is a bit of a chicken-and-egg thing and that maybe Cython support would prove the advantages of HPy and everyone would be convinced. But practically you probably need to have more than one binding generator.

The point about the amount of work involved is probably the main, broader one - there are a bunch of large, complicated, established tools based on the current C API that are complicated to change so either they have to evolve (slowly) or they have to be replaced by other newer tools without the weight of backwards compatibility (but again, the replacement is likely to be slow).

7 Likes
23

What I’m saying is that “rewriting” or generating large amount of code is more practical now than it was 2 years ago.

The non-trivial nature of the said code comes from compatibility requirements from a developer’s point of view. Not end user point of view. A developer in this context could be someone invested in the python internals, tools like cython, pybind11 etc. End user is someone who says I want FastAPI to run faster on my machine.

A thin system call wrapper around the kernel (which already exists in many libcs) and reimplementation of the library code in static/transpilable python is what I believe in. All you need is a clear specification and machines are becoming very capable of spitting out an implementation.

People looking for alternatives to C-API may be interested in this line of thought and hence the comment.

24

Thanks a lot @steve-s for this nice and important post. It seems to me that the situation is very clear now, with 3 possibilities:

  1. Status quo: the ecosystem waits and follows gradual and slow evolutions of the historical CPython C API.
  2. HPy revival (new funding required): Numpy ported to HPy + HPy backend for Cython.
  3. A new C API supported by CPython (but not in Python.h) inspired by HPy + something like pythoncapi-compat for usability on something like CPython >=3.10 + HPy like universal ABI.

I think it would be very valuable to have the points of view of CPython core devs (and of the C API WG) on this fact, or at least on the last possibility.

There were also two interesting points of view / pieces of information by Ralf Gommers (@rgommers) and Matti Picus (@mattip). I take the liberty to quote them here since it seems to me that they are highly relevant:

From Mailman 3 [Numpy-discussion] Re: Better compatibility of the Python scientific/data stack with fast Python interpreters - NumPy-Discussion - python.org

From Mailman 3 [pypy-dev] Re: [Numpy-discussion] Better compatibility of the Python scientific/data stack with fast Python interpreters - pypy-dev - python.org

4 Likes
25

The problem with HPy seems to be more a social one at this point

There are a range of positions that the core devs can take towards HPy, ranging from least committal to most committal:

  1. We agree with the goals of HPy and wish them success (least committal)
  2. We will make decisions and take on work to increase the probability that HPy succeeds
  3. We will support the existing API and an unnamed new API that is inspired by HPy, giving extensions the option to target the latter officially
  4. We endorse HPy as a supported API for extensions and will consider the effect of CPython API evolution on HPy as an important stakeholder
  5. We endorse HPy as a recommended API for extensions and will follow the lead of HPy on API design
  6. We endorse HPy as the API for extensions and will freeze the stable C API once HPy has parity with the stable C API
  7. We take HPy into the CPython organization and dual-support CPython and HPy (most committal)

I guess we are somewhere in the neighborhood of 2/3 right now. There might be a dependency cycle between (1) Position of CPython on HPy → (2) Interest/funding in advancing HPy → (3) Better level of support of HPy → (4) Higher adoption of HPy in the ecosystem → (1)

5 Likes
26

Great summary, thank you. I’d say we (Python core developers) are between 1/2 for all projects that use the CPython C API, regardless of whether it’s an abstraction layer or not.

I think we’re unlikely to ever get to levels 4-7 for any project,[1] just because there isn’t really any difference between levels 4 and 7 (if we start giving out endorsements, we’re also effectively taking on responsibility for the sustainability of the project).

It’s unfortunate that some people see these levels as a minimum for “the project is okay to use”, but it’s still less effort for the core team to reeducate those people than to take on responsibility for every project that’s in this category (e.g. I’ve heard it said about pip, pipenv, twine, pytest, setuptools, and more).

Remember that the Python ecosystem is driven forward by a loose collective of volunteers, not coordinated by a single entity. We don’t motivate contributors by paying them - they work on what they find satisfaction in working on. So just because the volunteers for one project choose not to work on a different project doesn’t mean that the latter project is any less valuable or important. It just has a different set of volunteers.

  1. As a collective. Individual contributors can of course do what they like. ↩︎

5 Likes
27

I think there are also a bunch of levels “below” 1, covering various ways in which the core devs could be indifferent to, or actively against, HPy. IMO, it’s also important to emphasise that we’re not at any of those levels…

I think the point about a dependency cycle is correct, at least in terms of the idea that the more popular HPy becomes, the more support it will get from the core devs[1]. But I think that trying to boost adoption “artificially”, by giving it some sort of “endorsed by the CPython core developers” label, is unlikely to be the way of doing this.

  1. That’s what’s happened with Cython, for example - it’s a very popular binding generator, so whether API changes will adversely affect Cython tends to be an important consideration. ↩︎

5 Likes
28

I have to again reply on few things since unfortunately it would be useful to have more precise answers, in particular about the possibility that in some time CPython provides a new modern C API independent from the historical C API (that has to evolve slowly and relatively weakly). This new C API does not have to be HPy so your answers about the “positions that the core devs can take towards HPy” does not really help.

Your point of view about this possibility would be useful in particular to answer to @rgommers (see C API Working Group and plan to get a Python C API compatible with alternative Python implementations? - #24 by paugier), which is important to think about what should be done for Numpy.

I ask about this potential new Python C API because we see signs that it could be useful and that there might be some projects about this:

I should also mention GitHub - markshannon/New-C-API-for-Python: Design and discussion for the new C-API for Python and this idea from the Faster CPython project ideas/3.14 at main · faster-cpython/ideas · GitHub.

This potential new Python C API could also be somehow inspired by HPy, so it would be very interesting to have your point of view about “Future of HPy, Take 2” in C API Working Group and plan to get a Python C API compatible with alternative Python implementations? - #19 by steve-s

It is interesting to note that the Python code devs who replied in this page seem to consider HPy only as a consumer of the CPython C API (like pybind11 or Cython). This is indeed one of its nature (an implementation detail used only for CPython) but HPy is also something else. As clearly explained by @steve-s, it is also an attempt to design and implement a new modern C API for interactions between “the Python VM and 3rd party native extensions for Python”.

This second nature of HPy explains that one might think that the C API WG could be deeply interesting about HPy, because, especially after PEP 733, one could have thinked that the C API WG would also design and propose a new modern C API, i.e. that there would be two parallel and simultaneous efforts:

  • slow and gradual evolution of the historical CPython C API (what is currently done) and
  • work on a new modern Python C API,

and that the two APIs could live together for a very long time.

This is clearly this second nature of HPy (a new modern Python C API) which brings a particular difficulty that other projects like pybind11 or Cython do not have. HPy is in direct competition with a potential project that does not even exist: a new modern Python API designed and implemented by CPython core devs and the “official” C API WG. This project has not even been announced but the only fact that it could be started stops projects from investing on HPy. I think this is highly related to what @rgommers wrote in his email:

if CPython core devs don’t want to adopt it but do their own “make the C API more opaque” strategy, then more effort on HPy isn’t going to help.

Hence, I think that the C API WG should officially say something more about these problems, which seem to be in the scope of this WG.

I also like to react on something that @steve.dower mentions few time, for example here

This is of course partly true but paying people to work on something is also quite common, also in the Python area. For example, it seems to me that Microsoft and Meta are founding the Faster CPython project and free-threading, respectively. Knowing the current state of HPy, it seems to me that with a bit of money and few people paid on a new modern Python C API (HPy or something else, Numpy port, Cython backend, Meson build, etc.) during 2 years, we could have a Python ecosystem in a much better state than nowadays in a relatively short time (not ten years). And then motivation would come with the first nice results. In contrast to what you say, I think a clear direction indicated from above (in particular the C API WG) would help.

7 Likes
29

IMO, there is some miscommunication. It’s not a duty of C API WG to create new C API for a list of Python implementations.

Anyone can make a proposal of such API, can write a PEP or implement those in code on fork of CPython. Or try to make some group for this future work. But trying to force C API WG doing this doesn’t look like viable option.

30

I don’t think that anyone is trying to force anyone to do anything.

The Python community has worked itself into an awkward local minima regarding C API that no one is exactly happy with i.e it doesn’t work well for non-CPython but also CPython itself is apparently unsatisfied with it as well and keeps changing it. It takes some bigger picture thought to see how to potentially move on from here and @paugier is trying to have those bigger picture discussions.

5 Likes
31

Taking bigger picture into account is fully understandable. Maybe little more concrete proposals will be great. It’s better to be grounded, IMO. I’m sorry for my words if they look rude.

32

Note that I’d like to be clear that I do not try to force the C API WG to propose a new modern C API! I just think it would be useful for the whole Python community that the WG explicitly tells us if CPython is going to do it or not.

All answers would be fine but the community needs to know to be able to plan things and make good technical decisions.

Do you realize that after this long discussion, we are still not able to get clues to be able to discuss about the question: should Numpy be ported to HPy or wait for the new modern C API mentioned in ideas/3.14 at main · faster-cpython/ideas · GitHub ?

4 Likes
33

@steve-s described quite concrete proposals (in particular “Future of HPy, Take 2”) but unfortunately, we didn’t get the points of view of Python core devs specialist of the CPython C API.

34

Proposals can come from anyone. It’s extremely unlikely that the WG as a whole will propose something, when any one of its members can just do it themselves.
But, “adopt HPy” is not a proposal – it’s not clear what HPy is:

As it is now, HPy, is a third-party project and it can’t be adopted as-is. It has many “aspects” that would work better as separate proposals, Mohamed’s “range of positions” could be evaluated for each part.

For a rough first draft:

  • should we add individual APIs from HPy? (for calling objects, type definitions, deallocation, etc.)
    • should CPython adopt them wholesale, or make new designs to solve the same issues?
  • should we add a “context” argument to “all” functions (by adding new versions of “all” functions)?
    • should the context have HPy-compatible semantics? will that be enforced?
  • should CPython replace PyObject * arguments by opaque references? (again by adding new functions)
    • should those have HPy compatible semantics? will that be enforced?
  • should CPython then remove the non-HPy-friendly functions?
  • should CPython implement HPy’s “trace mode”?
  • should CPython devs maintain the hpy package on PyPI? ;‍)

The answers will range from “yes” through “not now” through “no, not in CPython”.
And if this list were compiled a few years ago, it would include a bunch of items that are already done now! (This ties back to the “invisible” support that Paul mentioned.)

2 Likes
35

Thanks, Petr. I was trying to emphasize the same thing.

IMO, near all people here will be happy to have great C API that can be used with different Python implementations. The question is what individual really can do for it since project is maintained by the group of volunteers. So speaking about concrete steps seems to be much more valuable.

Yes, I do. But this topic doesn’t look very clear and simple, does it? The maintainers of Numpy have to decide on their own, which approach is more useful for them.

36

It’s difficult to decide which approach is more useful when you have little clue about the evolution of the C API ecosystem in the future, and any transition is very costly. Which is why most projects are just waiting.

1 Like
37

Agreed. Some piece of clear documentation about future plans of CPython evolution would be useful, if this is a real possibility to have it.

1 Like
38

After sleeping on it, here are as clear statements of what I see as the future of the C API as I can make (and I think it’s non-controversial enough with the rest of the WG that they’ll agree, but check replies below for counterpoints):

Assume the C API will change and evolve, some parts slowly, some parts drastically, but always following the compatibility rules that CPython has always followed[1]

If projects want to wait for all the changes to occur, that’s up to them. It means they’ll never make a release, which would be a shame, but that’s the path they’ve chosen.

We recommend that projects avoid having to make too many changes by avoiding private/internal APIs, by sticking to abi3 APIs[2], and/or by using abstraction layers that can centralise the changes when they occur.

If projects don’t want to use an abstraction layer such as pythoncapi-compat/HPy/nanobind/pybind11/Cython/etc. because there might one day be another abstraction layer, or they are convinced that only our private APIs are “fast enough”, that’s up to them. It means they’ll never make a release, which would be a shame, but that’s the path they’ve chosen.

The C API WG is not aware of any proposals to completely reimplement the C API. This means we aren’t evaluating any proposals, and aren’t recommending any proposals to the Steering Council. This means the current C API is not going to suddenly disappear one day without warning, because the proposal has to be created first, and it has not yet been created.

If projects don’t want to use the current C API because one day there might be another C API, that’s up to them. It means they’ll never make a release, which would be a shame, but that’s the path they’ve chosen.

  1. That is, breaking changes only in x.y releases (not x.y.z) with deprecation, and no removals/semantic changes to abi3 ↩︎

  2. Even if you compile without the limited API enabled, you’re still not going to see changes to these APIs. ↩︎

5 Likes
39

If you switch to HPy (even partially), and HPy continues to be maintained (even minimally – to work with future CPython versions), and CPython introduces a new C API, then you will get the benefits of the new API “for free”.
So, if supporting other Python implementations is useful to your project, at least to the point where you can commit to help keeping HPy on life-support, I’d recommend switching to HPy. At least the parts of HPy where the switch can be easily reverted.
(If it’s not important, then, frankly, why should it be important for CPython devs?)

Incidentally, the part of HPy that’s probably most problematic for CPython – mass changes of function signatures – is rather easy to “revert”, with a bunch of #ifdefs to the current C API.

As Steve hinted just now, weeding out any (soft-)deprecated API that you’re still using should be step number zero; followed by avoiding non-Limited API.

As for plans of CPython evolution – well, I could tell you mine, but there’s no guarantee that they’ll be approved and implemented, or even that they’ll make sense when the PEP is fleshed out. You seek a crystal ball that doesn’t exist.

3 Likes
40

To put this (and Steve’s comment) into context, the SC sets the direction for the Python language and the CPython implementation, and the C API WG advises the SC on the subject of the CPython C API. But no-one has a specific responsibility for “a C API across multiple language runtimes”. The SC still technically has responsibility for the big picture in that area, but without some sort of proposal, the SC has nothing to work with.

If someone were to write a proposal along the lines of “Make HPy the native C API for CPython, rather than a compatibility library sitting on top of the CPython native API”, then that would be something concrete that the SC could take a view on (and which the C API WG would advise on). But it would need to answer questions like:

  • How will the current lack of HPy developer resource be addressed?
  • Over what timescale would the transition occur?
  • How would backward compatibility for existing users of the C API be handled?

These are all the questions that have been asked here, and that’s the point - it’s not the SC or the C API WG who would answer these questions, their responsibility is to decide if the answers provided are acceptable.

Whether numpy are willing to switch to the HPy API is going to be down to (among other things) whether the numpy maintainers have sufficient trust in the HPy maintainers’ plans for long-term stability of HPy. If those plans are “hope that the CPython devs will take over the API” then I’m not surprised the numpy maintainers are hesitant…

5 Likes