tl;dr: Yeah, recognizing .abi3t.so in both builds and abi3.so only in non-free-threaded ones is probably a good move to make, but it’s more of a practical hack than conceptually the right thing to do.
The promise still holds for GIL-enabled builds.
For free-threaded builds, this is broken, needs to break now, or will break in a a free-threaded future. (Which of those? That depends on the exact wording of the promise just as much as on what PEP 803 or CPython do.)
Right. The PEP treats abi3 and abi3t as distinct – but, crucially, it allows a single installed extension to be compatible with both. This works for the actual 3.15 ABIs (they have a usable overlap), and for wheel tags where you can have abi3.abi3t.
The place where it doesn’t work is the filename extension. You can’t have both there.
Some possible solutions in this space:
- make the FT builds accept
*.abi3.so(what the PEP does). This reuses an extension that already exists; tooling that doesn’t care continues to work unchanged. - make the GIL build accept
*.abi3t.so(which you’re proposing, AFAIK). Conceptually, this is a lie just as much as the first option. - use just
*.so. Look ma, no lying! Also, Windows does this (with.pyd), with no ill effects.
If we’re avoiding lies and broken promises, the last solution seems best. But of course, that’s not what we’re doing – we’re trying to be most helpful to users.
Could we frame the discussion that way? Should be more actionable than complaints of brokenness.
In yet another instance of confusing terminology, 3.13t does have a (lowercase-s) stable ABI, and so does 3.14. An extension compiled for 3.13.0 will work on 3.13.4. It won’t work on 3.14 of course, but since it’s fixed and known, you can theoretically enumerate the differences between 3.13 and 3.14, and add runtime ifs for all those differences. You’d get an extension that works on both 3.13 and 3.14.
The same works for the (capital-S) Stable ABI. You can enumerate the difference between 3.14 abi3 and PEP-803 3.15 abi3t, and build an extension that you can correctly tag cp314-abi3.abi3t. As far as I know, this is not only possible but practical, especially for a project like PyO3 that would already generate wrappers for both ABIs.
The place where it’s not very practical to support this is CPython, so the PEP only talks about this as a possibility rather than a guarantee. It does try to keep this as a possibility, though – in response to Emma’s post from (much) earlier:
Do we make that use case harder, or do we make your use case harder? “Your” use case being Debian’s, as discussed on the SC thread.
Should we even support sharing a site-packages between interpreters? That’s perhaps something the SC should decide – the issue is perhaps bigger than this PEP. For example, we have #122917 which is blocked for performance reasons. Or the recent calls to disallow such “overlapping” installations entirely, even between debug/non-debug builds which have the same ABI. (Please chime in there – e.g. Greg’s post should be quite worrying for you.)
As for the “cryptography” case from the SC thread: note that the Stable ABI is, alas, only about ABI stability. There are other reasons why a cp38-abi3 wheel would be incompatible with Python 3.11 – what you can work around using if in Python, may become #ifdef in C. For a simple example: PyImport_Import has a stable ABI, put the module you import might be deprecated or removed.
There’s a question whether, in cryptography’s case, publishing a cp311-abi3 wheel means they can make their cp38-abi3 wheel incompatible with 3.11+ at runtime.
I haven’t asked either, but I’d bet cryptography maintainers would disagree with Debian here. And I doubt they test cp38-abi3 wheels on 3.14.
If we do support “overlapping” installations, we can do better. For example, would it be worth it to rewrite the import system to allow handle hundreds of possible filename tags efficiently, so you can be extremely specific there?
Anyway, all in all, recognizing .abi3t.so in both builds and abi3.so only in non-free-threaded ones is probably a good move to make, but it’s more of a practical hack than conceptually the right thing to do. I wish I had had more months to think about this problem.
I imagine it will complicate things a bit for build tools – but those need some changes anyway.
On the other hand, why wouldn’t Debian be fine building .abi3.so for GIL builds and individual .cpython-3XXt-*.so for free-threaded?
As for cp314-abi3.abi3t wheels (the ones you’d need specialized tooling to build), they can use “plain” .so filenames, with neither .abi3 nor .abi3t tag. So, that use case happens to still work.