Use the limited C API for some of our stdlib C extensions

[vstinner (Victor Stinner)]

@guido

I think “if his plan works” is not sufficient – we also need to have clear guidelines or rules that prevent new “private” APIs from being introduced by accident. I don’t think many core devs have internalized the new rules yet, and continue to think of “private” and “internal” as referring to the same thing. The same is likely true for the majority of non-core contributors. I don’t know if we need new tooling, or a PEP clearly stating the rules, or just Victor policing API changes. But I think stating that they are pretty much the same is just perpetuating the confusion about where private/internal APIs should live and how they should be named.

There are still cases where adding a private _Py function is fine. For example, if it’s the implementation of a public C API: called by a macro or static inline function. But yeah, it should be the exception, not the rule Some tooling may help. So far, I used manually grep to discover these APIs. That’s how I landed on _PyObject_VisitManagedDict() added to Python 3.12.

@guido

I personally believe the problem is with the requirement that once 3.x.0 is released all 3rd party packages should be instantly available. This has never been the case and will continue to be an unreachable goal.

I don’t pretend to have a silver bullet solving all problems at once. I’m saying that converting more 3rd party extensions to the stable ABI will increase the number of C extensions usable in early stage of new Python versions.

@guido

I wish we could look for a solution in the area of making the building of binary wheels easier, or faster, or cheaper. This could take the form of a build farm to which one can submit build requests, or simpler GitHub Actions recipes, or something else.

A lot part of that is already automated by tooling by cibuildwheel. I don’t think that this part is the bottleneck. IMO the bottleneck is more that it’s a thankless work to do, it’s not exciting, and maintainers prefer to work on new features, or fix their own bugs, rather than following Python C API changes.

@guido

Packages are a different matter. I’m all for encouraging package maintainers to be prepared and start working on wheels once rc1 is out.

I don’t think that it’s the right move. Maintainers are doing their best effort and are unavailable for various reasons. It’s common that a maintainer of a critical dependency is away for months. I don’t think that putting more pressure on them “you have to fix my use case” (ex: support the new Python) is helping. By the way, as I wrote, my Fedora team is already doing exactly that, asking gently to ship wheel binary packages as soon as possible, with mixed results. And we already did that for a small number of projects (the ones we care the most about).

@guido

I still feel that the Stable ABI is a solution largely in search of a problem, and too much of the argument in favor of it feels based in wishful thinking.

Currently, each Python release introduces a various number of incompatible C API changes. Maintainers have to dedicate time at each Python release to make their code compatible. Sometimes, it takes one year and… then a new Python release introduces more incompatible C API. It doesn’t sound to be a pleasant work to do.

The deal here is that if you restrict your C extension to the limited C API and built it in a way to get a stable ABI binary wheel package, you will no longer have to do this maintenance work, and so you can use your time on other funnier tasks. Maintaining an open source dependency has to remain fun! Otherwise people just move away and abandon their project.

---

Well, I suppose that in practice they are still some issues time to time with the limited C API. But I expect that it’s way lower, since we have way stricter rules about the ABI compatibility.

Maybe what I say is just wrong and it doesn’t work as planned. But if we make it happen, it will be more pleasant for everybody.

I don’t think that all C extensions need bleeding edge performance. Many of them are just thin wrappers to another library, it’s a glue between Python and . The hot code in not the in C API glue code.

At least, with the limited C API you have the choice: either your use the limited C API, ship a stable ABI package and forget it. Or you can follow every C API change and adapt frequently your code to them.