Sorry for spreading this topic all of GitHub and Discourse, but discussion has dragged on for years on gh-122931: Allow stable API extensions to include a multiarch tuple in the filename by stefanor · Pull Request #122917 · python/cpython · GitHub and this thread was requested to bring the issue to a wider audience.
Are we willing to make a (somewhat) breaking change to 3.15, to avoid needing to support an extra binary extension filename for the long-term future?
The issue:
PEP-384 (stable ABI) was defined before PEP-3149 (tag extensions to .so files) but in practice was only implemented a couple of years after 3149. PEP-3149 anticipated PEP-384, but didn’t provide examples of it. And PEP-3149’s implementation didn’t cover the stable ABI, because it wasn’t something in use yet.
C extension naming doesn't take bitness into account · Issue #67169 · python/cpython · GitHub lead to cPython including platform tags (the multiarch architecture tuple) in C extension filenames, by default, using the tagging space allowed by PEP-3149. But this was not applied to the stable ABI. I think setuptools only gained support for it in 2016, and it took a few years after that to see wider use.
What problems does this cause?
Debian relies on these tagging mechanisms to allow multiple Python versions, architectures, and build types to share the same dist-packages library tree. Doing this made supporting newer Python releases in Debian much easier.
Multiple Python verision co-installability with the stable ABI is not an issue (by design).
Multiple build types were an issue, but we fixed that (bpo-28401), and soon the debug build gained the ability to import non-debug-built extensions (bpo-36722).
Multiple architectures continued to be an issue. This was less immediately obvious, because only packages that opt-into Multi-Arch co-installability ran into this problem.
Very few Python libraries needed to do this. But over time, as multi-arch use has risen, to currently around 132 multi-arch co-installable Python 3 binary packages, from a total of 1219 architecture-specific Python 3 binary packages. The primary use-case for this is to allow cross-compilation. And in general Debian has been working to improve the number of packages that we can cross-compile.
Debian has been carrying a patch since Python 3.13.0, to allow stable ABI extensions to include a platform tag in their filename, allowing this to work. The patch was forwarded (as #122917), but then got stuck in review for years, over performance concerns.
There was a desire to delay the change until an abi4 was required. We now have an abi4 (abi3t), but the change didn’t land yet, so we have decisions to make.
This issue does not affect other families of Linux distributions (to my knowledge), as none of them handle multiple architectures the same way. Debian’s Multi-Arch is much more powerful than other distributions mechanisms for handling multiple architectures. Of course it affects all Debian derivatives (Ubuntu, etc.).
The proposed change:
Merge gh-122931: Allow stable API extensions to include a multiarch tuple in the filename by stefanor · Pull Request #122917 · python/cpython · GitHub in its current form, and backport it to 3.15.
This resolves the issue for abi3t, supporting only .abi3t-${SOABI_PLATFORM}.so, not .abi3t.so. Platforms without such a platform tag see no change.
Some build tools already support abi3t.so and would need to be updated. I don’t think this is out of the question.
I would also like to see some progress on resolving the issue for abi3.so, but let’s not boil all the oceans at once…
See also:
- Tracking issue: Allow stable ABI extensions to include multiarch tuples in the filename · Issue #122931 · python/cpython · GitHub
- Update PEP 3149 to explicitly include the stable ABI · Issue #355 · python/steering-council · GitHub (closed, no update required)
- Update PEP-803 to use multiarch in abi3t · Issue #356 · python/steering-council · GitHub
- Defining a support policy on overlapping installations (which has had recent discussion on the topic)