Can we consider an uncoditional change of `abi3t.so` to `abi3t-SOABI_PLATFORM.so` for 3.15 (last-minute)

Packaging
1

Sorry for spreading this topic all of GitHub and Discourse, but discussion has dragged on for years on gh-122931: Include multiarch tuple in abi3t filenames by stefanor · Pull Request #122917 · python/cpython · GitHub and this thread was requested to bring the issue to a wider audience.

Are we willing to make a (somewhat) breaking change to 3.15, to avoid needing to support an extra binary extension filename for the long-term future?

The issue:

PEP-384 (stable ABI) was defined before PEP-3149 (tag extensions to .so files) but in practice was only implemented a couple of years after 3149. PEP-3149 anticipated PEP-384, but didn’t provide examples of it. And PEP-3149’s implementation didn’t cover the stable ABI, because it wasn’t something in use yet.

C extension naming doesn't take bitness into account · Issue #67169 · python/cpython · GitHub lead to cPython including platform tags (the multiarch architecture tuple) in C extension filenames, by default, using the tagging space allowed by PEP-3149. But this was not applied to the stable ABI. I think setuptools only gained support for it in 2016, and it took a few years after that to see wider use.

What problems does this cause?

Debian relies on these tagging mechanisms to allow multiple Python versions, architectures, and build types to share the same dist-packages library tree. Doing this made supporting newer Python releases in Debian much easier.

Multiple Python verision co-installability with the stable ABI is not an issue (by design).

Multiple build types were an issue, but we fixed that (bpo-28401), and soon the debug build gained the ability to import non-debug-built extensions (bpo-36722).

Multiple architectures continued to be an issue. This was less immediately obvious, because only packages that opt-into Multi-Arch co-installability ran into this problem.

Very few Python libraries needed to do this. But over time, as multi-arch use has risen, to currently around 132 multi-arch co-installable Python 3 binary packages, from a total of 1219 architecture-specific Python 3 binary packages. The primary use-case for this is to allow cross-compilation. And in general Debian has been working to improve the number of packages that we can cross-compile.

Debian has been carrying a patch since Python 3.13.0, to allow stable ABI extensions to include a platform tag in their filename, allowing this to work. The patch was forwarded (as #122917), but then got stuck in review for years, over performance concerns.

There was a desire to delay the change until an abi4 was required. We now have an abi4 (abi3t), but the change didn’t land yet, so we have decisions to make.

This issue does not affect other families of Linux distributions (to my knowledge), as none of them handle multiple architectures the same way. Debian’s Multi-Arch is much more powerful than other distributions mechanisms for handling multiple architectures. Of course it affects all Debian derivatives (Ubuntu, etc.).

The proposed change:

Merge gh-122931: Include multiarch tuple in abi3t filenames by stefanor · Pull Request #122917 · python/cpython · GitHub in its current form, and backport it to 3.15.

This resolves the issue for abi3t, supporting only .abi3t-${SOABI_PLATFORM}.so, not .abi3t.so. Platforms without such a platform tag see no change.

Some build tools already support abi3t.so and would need to be updated. I don’t think this is out of the question.

I would also like to see some progress on resolving the issue for abi3.so gh-122931: Allow stable abi3 API extensions to include a multiarch tuple in the filename by stefanor · Pull Request #152461 · python/cpython · GitHub, but let’s not boil all the oceans at once…

See also:

6 Likes
2

But it is broadly useful for anyone who wants to:

  • Do embedded (cross-architecture) development in a source tree, having built extensions for multiple architectures sitting next to each other.
  • Ship a multi-architecture compatible pile of Python, including binary extensions. macOS solves this with fat binaries, but that’s generally not the approach taken in the Linux world.

It’s way more convenient when you don’t have to worry about the wrong Python importing the wrong thing. You just point it at a tree, and it works.

2 Likes
3

I like this. For what it’s worth, the first time I built my module with the stable ABI and saw that the result was named hpk.abi3.so, I started looking for the bug in my build system that caused the x86_64-linux-gnu to be dropped.

To quote The Zen of Python,

Explicit is better than implicit.

2 Likes
4

I think it’s fine to change this while we’re in the beta period. I’d be opposed to doing it during the rc period, but doing it now is reasonable IMO.

5

I think it’d also be okay to change in the beta period - some build backends may need to do work and a new release, but that pain is limited and may be worth it, under one condition: this really needs a proper integration test in CPython, not just for extension names but for interleaved multi-architecture builds, with the test checking that if two builds have files with the same name, their content is identical (one job building for Linux x86-64 and aarch64 should be enough). Because this is the second release in a row where this issue has surfaced too late. For 3.14 it was build-details.json, now it’s filename extensions. For whatever packaging feature is next, this should be caught before merging the feature, not after support for it has been shipped in build systems and backends.

6 Likes
6
5 Likes
7

The steering council weighed in that they’d like EXTENSION_SUFFIXES to include both the bare abit3textension and the version with platform tags. Hugo also weighed in as release manager that it’s OK to change this in 3.15.

I don’t yet see a specific PR open implementing what Thomas outlined in the steering council issue. @stefanor, were you planing to take care of that before rc1? Does one of your open PRs already implement it?

1 Like
8

Didn’t the SC’s comment only address adding abi3t files (i.e. only for the brand new ABI)?

9

You’re right, their response only referred to abi3t.