As Guido pointed out, apparently, I’m really bad at sharing my agenda and rationale behind this work. I’m sorry about that Let me explain it from an user point of view.
There are multiple problems:
- When Python 3.12 will be shipped in 1 month, many C extensions will not be available at day 1 (2023-10-02).
- It’s expensive (complicated) to maintain more than one Python version in a Linux distribution. Most Linux distributions don’t, users are on their own if they want another Python version than the chosen one.
Python 3.12 vs C extensions
Python 3.12 is going to be shipped in 1 month:
3.12.0 final: Monday, 2023-10-02. Can you expect 100% of C extensions to be usable smoothly at day 1 of the release? According to previous releases, I can say: well… no!
At Red Hat, we are actively asking gently popular C extension maintainers to ship Python 3.12 binary wheel packages as soon as possible: issue Encourage upstreams to release Python 3.12 manylinux wheels before Fedora 39 final release. We did the same in the past for Python 3.9, 3.10 and 3.11.
By the way, I would like to say that it’s unpleasant that we are paid to asks volunteers to work for us to please our customers. But that’s how the open source community usually works: we pass user requests to upstream projects. We are not maintainers of these projects, we are providing changes to support new Python releases (we helped to fix Cython and many others projects to support Python 3.12), but we cannot do the last part, actually releasing a new version containing the fixes, and publish wheel packages.
For example, last week,
python3.12 -m pip install numpy still failed with:
ModuleNotFoundError: No module named 'distutils' (I don’t think that it has been fixed in the meanwhile). Same error when trying to install
matplotlib. Fedora 39 is scheduled to be released next month with Python 3.12. We do ship Fedora packages for
matplotlib (RPM), but users love to run
pip install anyway (directly or indirectly, for good or bad reasons), and then open bug report: “installing numpy doesn’t work!!!”.
Can users consider using Python 3.12 if
matplotlib are missing? Maybe yes, maybe not. From the Fedora point of view, it’s a big issue. In the past, we always got bug reports, users don’t understand why it’s not as usual when “pip install just worked”. Sometimes, bug report are about missing build dependencies, slower installation, or anything else.
Note: I have hope in the port of numpy to HPy HPy serves similar use cases, especially with its universal ABI.
Process or tooling to make things better?
I only named 2 popular C extensions, but the Python ecosystem is way wider than these two C extensions. Also, numpy is a healthy project with strong funding. If a project with strong funding and active development cannot be ready for Python 3.12 one month before the final release, how can you expect 100% of C extensions be ready for Python 3.12 final release?
In the past, I proposed PEP 608 – Coordinated Python release which required to hold a new Python release until the bare minimum ecosystem is functional, like pip and numpy. It was rejected because Python core devs cannot control the releases of pip or numpy, and apparently it’s better to ship a Python without these tools than not shipping Python. An optimistic person would say that it’s a chicken-egg problem, Python 3.12 should be released before numpy should consider supporting it, right?
I also proposed PEP 606 – Python Compatibility Version, a kind of “stable API” at the Python level, but it also got rejected. It’s common that pure Python projects are also unusable on newer Python, because of a bunch of minor changes, minor but enough to prevent using these projects.
Supporting more than one Python version
Few Linux distributions support more than one Python version because providing one binary package for C extensions is a huge maintenance burden. For example, if you want to support Python 3.6, 3.9 and 3.11 and you have packages for lxml, when there is yet another security vulnerability in the lxml “clean html” feature, you don’t have to build 1 new package but 3 new packages. Even if technically, it should be the same “source package”, building 3 packages requires more work than only one. It’s perfectly reasonable for Debian to say “we only support one Python version”.
RHEL8 distribution provides 4 Python versions: 3.6, 3.8, 3.9 and 3.11. Only Python 3.6 is fully supported (most packages), the other versions have the bare minimum packages that we can support.
RHEL8 lifecycle is about 10 to 15 years, so it has other concerns than Linux distributions with shorter lifecycle, like Fedora releases which are only supported for one year. Debian has a similar time scale, but it only supports one Python version.
Here comes the stable ABI!
If we manage to make the limited C API and the stable ABI looking more appealing, we can use it as a technical solution to this problem. It will not solve all problems that I listed (e.g. it will not solve social issues), but it should solve some of them.
For example, PySide and cryptography are shipped as stable ABI binaries. They don’t need any change to usable on Python 3.12. Yeah, I know, it’s hard to believe, Python got me used to suffer at each major release, right? I’m not a believer, I want to see it working for real:
$ python3.12 -m venv env
$ env/bin/python -m pip install cryptography
Using cached cryptography-41.0.3-cp37-abi3-manylinux_2_28_x86_64.whl.metadata (5.2 kB)
Successfully installed cffi-1.15.1 cryptography-41.0.3 pycparser-2.21
$ env/bin/python -c 'import cryptography.hazmat.bindings._rust as mod; print(mod)'
<module 'cryptography.hazmat.bindings._rust' from '/home/vstinner/env/lib64/python3.12/site-packages/cryptography/hazmat/bindings/_rust.abi3.so'>
Python 3.12 installs pure Python code but also a Rust extension built for the Python stable ABI known as abi3.
Note: it’s not perfect, cffi and pycparser don’t support the stable ABI, and so installing cryptography can fail if one of these two dependencies fail to build.
If we can help maintainers to move towards the limited C API, you can expect having more C extensions to be usable at day 1 of Python 3.13 release. Or it will done the usual way, users harassing maintainers to “just make it work” (without providing any technical or financial help, obviously), burnout of maintainers, and similar other cool stuffs
Python core devs can look at the limited C API, add missing functions, enhance the documentation, enhance it in general, and as I wrote, consider using it ourselves to see how users are struggling with using it. If we, developers of Python itself, don’t want to or cannot use the limited C API, why would other developers even give it a try?
There is hope!