Use the limited C API for some of our stdlib C extensions

Latest stable is NumPy 1.25.2 and doesn’t support Python 3.12, but they’ve uploaded 3.12 support as a pre-release: 1.26.0b1. So you need to do something like python3.12 -m pip install numpy --pre or python3.12 -m pip install numpy>=1.26.0b1.

Or:

'numpy; python_version<"3.12"',
'numpy>=1.26.0b1; python_version>="3.12"',

Or PIP_PRE=1 in the environment, or PIP_ONLY_BINARY=:all: / PIP_ONLY_BINARY=numpy (source).

So good news it’s available, but this pre-release method is less obvious and makes integration harder.

2 Likes

Yup, it’s fiiiinally usable for non-trivial extensions, and the word is getting out :‍)

Not really. Functions in the stable ABI can become no-ops, become very slow, start raising runtime deprecation warnings, start always raising exceptions, or even leak resources in extreme cases. The ABI guarantee is along the lines of the symbol staying around, so you don’t get linker errors, or the argument types and such not changing, so you don’t get data corruption.
Breaking as little as possible would be very nice of course, and PEP 387 applies. But stable ABI guarantees themselves are surprisingly weak.

It could start raising deprecation warnings, and then fail at runtime, just like Python code.
That said, as far as I know, there are ways to make most uses of existing slots work even with future redesigns.
And, many of the issues with slots – e.g. we can’t change their signatures – are shared with the non-limited API.

Well, we disagree there.
Anyway, let me know when stable ABI actually hinders your work (aside from the exposed PyObject* fields). In cases I’ve seen, the general backward compatibility policy (PEP-387) is where the pain comes from.

Yes. I’m not claiming to understand his work. But I don’t have the bandwidth to check it, so I’m left with trusting that it’s well thought out.
I did write some recent PEPs and documentation around this area, so I’m clarifying the intent and terminology used there. I’m not aware of a documented distinction between internal and private; they are used as synonyms.

No one? AFAIK the PEP that mentions it is 620, which is Withdrawn, even though it’s mostly completed.

What do you think is the proper amount of time before a new 3.x becomes a good default? A year or so?

The goal might be unreachable, but IMO it is something to strive for. And at least enable from our side.

2 Likes

That’s an interesting interpretation of a stable ABI :slight_smile:

If all the stable ABI provides is a guarantee that entry points continue to exist, but the functionality behind it may break anytime (subject to PEP 387), then it could just as well be implemented as a shim library interfacing to the full public C API and shipped as a separate extension on PyPI that you load in case needed.

Needless to say that such an interpretation pretty much goes against what regular users and authors of Python extensions using the stable ABI would expect, namely that these extensions “can be compiled once and work with multiple versions of Python.”

Where’s the benefit of saying “oh, you will be able to import this module in Python 3.20 without problems, but some parts may not work anymore” (and it’s not even clear which parts those are) ?

If we want to keep a stable ABI as a maintained feature in Python, we need to give more robust guarantees and also take the hit of making it harder to evolve the internals. I don’t think we can have both with the current design of the stable ABI.

A new and different approach which separates the ABI from the internals would be needed, something like e.g. the Windows COM interface or an indirection of most of the ABI via a per interpreter struct (similar to what Capsules provide for C extensions - PyCObjects in the old days). With such a solution we could have multiple stable ABIs together with support guarantees for a certain number of releases, but that’s a different story… :slight_smile:

1 Like

For Include/cpython, there was a poll here on Discourse[1]:

See also:


  1. it only received 3 votes, though, so it wasn’t exactly representative ↩︎

Historically it’s been months for many common 3rd party packages to become available. People have to decide when to adopt the new release based on their own requirements and dependencies. Usually few people running applications in production are in a hurry, and dependencies are a big part of the reason. (Another big part is the resources needed to test and migrate a production app – and the worry about bugs in any “point oh release”, which everybody would rather have someone else find first.)

Packages are a different matter. I’m all for encouraging package maintainers to be prepared and start working on wheels once rc1 is out. I’m unhappy about the pressure I am currently feeling to make it our fault if not every 3rd party package works on day one. I still feel that the Stable ABI is a solution largely in search of a problem, and too much of the argument in favor of it feels based in wishful thinking.

I wish we could look for a solution in the area of making the building of binary wheels easier, or faster, or cheaper. This could take the form of a build farm to which one can submit build requests, or simpler GitHub Actions recipes, or something else. (I’m not sure about the state of the art here, but I’m sure some folks will claim that conda-forge has solved this – however that probably doesn’t help the pip install foo crowd.)

3 Likes

Remarks on the Internal C API.

Are you talking about the directory name? Or do you mean that it’s unclear to you who uses which API for what purpose?

I think that it was Eric Snow who created the Include/internal/ directory to be able to move the PyInterpreterState structure there. I don’t recall the details.

By the way, the internal C API was created to solve an implementation problem. For example, the _Py_ID() API, which replaces the _Py_IdentifierAPI, goes deeper into internals. _Py_Identifier is a simple structure with 2 members: typedef struct _Py_Identifier { const char* string; Py_ssize_t index;} _Py_Identifier;. It can be inspected by its user. But _Py_ID(NAME) is another kind of beast, it looks for _PyRuntime.static_objects.singletons.strings.identifiers._py_NAME which is the new giant _PyRuntimeState structure which has… many members (most of them are generated by complex tools). See for example the pycore_global_strings.h header file. I don’t think that we want to expose such implementation details to users, nor users to rely on it. The list of “static strings” is changing frequently, the structure members offset changes often, so there is no such thing as ABI stability for that (which could be an issue even without the Stable ABI, for regular C extensions).

Another problem was the usage of atomic variables (pyatomic.h) and conditional variables (pycore_condvar.h). These header files caused many compiler errors when they were part of the public C API (ex: not compatible with C++). Having the ability to exclude them from the Public C API is a nice step towards a cleaner Public C API.

Python 3.13 evolves a lot since Python 2.7! For me, having a separated Internal C API made such work possible.

There are still cases where adding a private _Py function is fine. For example, if it’s the implementation of a public C API: called by a macro or static inline function. But yeah, it should be the exception, not the rule :slight_smile: Some tooling may help. So far, I used manually grep to discover these APIs. That’s how I landed on _PyObject_VisitManagedDict() added to Python 3.12.

I don’t pretend to have a silver bullet solving all problems at once. I’m saying that converting more 3rd party extensions to the stable ABI will increase the number of C extensions usable in early stage of new Python versions.

A lot part of that is already automated by tooling by cibuildwheel. I don’t think that this part is the bottleneck. IMO the bottleneck is more that it’s a thankless work to do, it’s not exciting, and maintainers prefer to work on new features, or fix their own bugs, rather than following Python C API changes.

I don’t think that it’s the right move. Maintainers are doing their best effort and are unavailable for various reasons. It’s common that a maintainer of a critical dependency is away for months. I don’t think that putting more pressure on them “you have to fix my use case” (ex: support the new Python) is helping. By the way, as I wrote, my Fedora team is already doing exactly that, asking gently to ship wheel binary packages as soon as possible, with mixed results. And we already did that for a small number of projects (the ones we care the most about).

Currently, each Python release introduces a various number of incompatible C API changes. Maintainers have to dedicate time at each Python release to make their code compatible. Sometimes, it takes one year and… then a new Python release introduces more incompatible C API. It doesn’t sound to be a pleasant work to do.

The deal here is that if you restrict your C extension to the limited C API and built it in a way to get a stable ABI binary wheel package, you will no longer have to do this maintenance work, and so you can use your time on other funnier tasks. Maintaining an open source dependency has to remain fun! Otherwise people just move away and abandon their project.


Well, I suppose that in practice they are still some issues time to time with the limited C API. But I expect that it’s way lower, since we have way stricter rules about the ABI compatibility.

Maybe what I say is just wrong and it doesn’t work as planned. But if we make it happen, it will be more pleasant for everybody.

I don’t think that all C extensions need bleeding edge performance. Many of them are just thin wrappers to another library, it’s a glue between Python and . The hot code in not the in C API glue code.

At least, with the limited C API you have the choice: either your use the limited C API, ship a stable ABI package and forget it. Or you can follow every C API change and adapt frequently your code to them.

1 Like

I am tired of this discussion. It doesn’t seem we’re reaching any kind of agreement. Let’s talk in person at Brno. Until then I will stop arguing.

Speaking from the perspective of a maintainer of downstream libraries there is one unnecessary thing that core Python does that contributes to the problem of users wanting all packages available from day one. The Python download page always defaults to suggesting the latest release of CPython even if that is only one day old:

Users who want to use Python along with various other packages will find it disappointing if they install the “default” version of Python but then many important packages are not available. Those users are not well served by being guided to install day-old releases of Python.

6 Likes

As @hugovk noted, there’s numpy 1.26.0b1 that supports 3.12 since about 3 weeks. Note that this is not an “average” new CPython release, because the removal of distutils has a large blast radius for libraries like numpy, scipy, etc. This by way of explanation why things are taking a while, despite people working on this with very high urgency.

4 Likes

Behaviour is covered by the same policy we have for Python code – PEP-387. It works pretty well: you can generally assume your code will continue working. It’s rare and discouraged for packagers to set “defensive” upper limits on the Python version.
I’d be all in for a stricter PEP-387. But I think ABI stability guarantees should stick to ABI.

Indeed, that’s a direction the stable ABI could evolve in. In fact, HPy essentially does this today.
But, I’d like to support stable ABI in the reference implementation of Python, and as far as I can see, it doesn’t hinder development much more than API stability guarantees.


Going back to vim: I’d love hear your thoughts on how desktop applications should handle Python scripting/plugins. IMO, we should have a lighter-weight way to do that than each such project becoming a redistributor of Python, and e.g. re-releasing each new Python security fix. (Ask the release managers how painful it is for CPython to bundle OpenSSL!)
Allowing users to use a Python they already have sounds like a good way to go for me. It’s not perfect yet of course, but it’s a good direction.


Oh. Thanks!
If it was me I’d write a PEP. I don’t really understand why one is not needed for such massive C-API reorganizations.

1 Like

Sorry that you feel the pressure, but AFAIK others want to improve this situation. A new release of Python should just work. Sure, it never was that way and maybe we’ll never get there, but that doesn’t mean we shouldn’t try.
Perhaps there’s something we can do to reduce the pressure on people who care about other things?

Yet, it has users that are very happy about it.

Neither does it help the homebrew install crowd, nor the [click here to Download Blender] crowd.
It’s not just building, and it’s not just wheels.

3 Likes

I agree with Victor here. It may not be practically possible to start working on wheels before the final Python release. One reason can be that dependencies (such as Numpy) are not ready. Another reason is that Python RCs are not available in most distribution channels (such as conda-forge, etc.). Actually, even the final Python may take several weeks to be packaged in those distribution channels.

You cannot really ask package maintainers to go out of their way and implement a different build or testing procedure for Python RCs (or betas) than the one they use for released Pythons.

And of course if the new Python version requires changes to the package’s source code to maintain compatibility, then the new wheels will lag even more than if a mere rebuild had been sufficient.

1 Like

I wish this would happen too, but work seems to have stalled since Feb 2022. While NumPy is an attractive target, perhaps it’s really too complicated and HPy should have started with a simpler project?

Right, but this effort is not going to go away by using the stable ABI.

Maintainers will still have to test their packages with the new Python release and fix any issues they find. Note that packages typically do include Python code as well, which doesn’t magically continue to work because the included C extension used the stable ABI :slight_smile:

FWIW, I don’t think it’s a good idea to tell users: hey, look, you can continue to use the packages released for Python 3.11 with Python 3.12, since the C extension uses the stable ABI, but without actually testing the package with 3.12.

Likewise, users should be made aware that packages they are installing with pip install may actually not be tested with the just-released new Python version.

Making it easier to port C extension packages to new Python releases sound like a much better plan, since then the effort for the maintainers is materially reduced and not just postponed.

Which is why I think effort on the core dev side is better spent on projects such as your compatibility tooling , rather than maintaining two variants of the same API.

I have already stated my opinion on this: the desktop application maintainers are in charge here. For Linux distributions, esp. the paid ones, the distribution companies should do this kind of packaging and relieve the maintainers from these tasks.

I know that handling OpenSSL upgrades is painful (we maintained a client-server product using Python and OpenSSL for many years), but that’s mostly due to the OpenSSL side of things, not so much because Python makes this difficult.

If there are many such desktop applications, perhaps the maintainers of these could join forces and create a distribution of Python which is geared towards making embedding easy and painless for them. I don’t think this is something the core dev team should be taking on.

This is an interesting point. People with experience know to pin their dependencies, test their apps automatically, wait for x.y.1 release before upgrading, but someone getting started may find the official download page and get the most recent release when the paint is still fresh on it. What do people think about reorganizing the page slightly so that the current version is above a very recent version?

1 Like

As a point of reference, the python stub that ships with Windows doesn’t switch to the latest release until there’s “broad community support” for it, which is a deliberately vague definition to allow us to judge the situation around each release (and avoid having people try and game it).

Generally it’s been switching over around 3.x.2.

4 Likes

I suspect it only shifts the goalposts. Right now, package maintainers are able to release compatible binary wheels as soon as Python RC1 is released (and made available to build tooling), so X.Y.0 already has some buffer. Making the default switch over later hurts increases the buffer (which there may be an argument for, but then I would say it’s easier to extend the RC period).

I think one problem here is that package maintainers aren’t aware RC releases are out simply because they have better things to do than keep up with Python pre-releases, and the first they are reminded of a new release is when users ask about support for it, which might be when X.Y.0 is out. My suggestion would be for us to be more proactive in notifying maintainers (I have an idea).

PS: I think this discussion is off-topic with respect to the limited C API usage in the standard library, and could be split off into a new thread

1 Like

I will not personally do that. I will prepare for the new release of CPython and have it tested through the alpha, beta etc stages in CI. I will not push out the release of my package claiming compatibility with CPython X.Y though until I can see the build complete and tests pass with the final release of CPython X.Y.0. The time at which I issue the release to support a new CPython version is always going to be after CPython issues its release. How long it takes depends on a bunch of factors because there are almost always some other things that need to be considered at the time even if everyone involved is not just busy with entirely different things.