PEP 803: Stable ABI for Free-Threaded Builds (packaging thread)

Packaging
2

Thanks for sharing @encukou!

This stood out to me. I can’t quite decide whether I like this decision, so here are some thoughts/pros/cons. To motivate this, all PEP 803 says is “Changing this while keeping compatibility with GIL-enabled builds would be an unnecessary technical change.”; this comment on the main PEP 803 thread says a bit more but I’m not sure what it says is correct (“the files are still compatible with abi3 so they shouldn’t be just renamed”); it’s just a new ABI with changes at the source level for PEP 793, there’s no such thing as “compatible with abi3” I think, that’s putting things in reverse. Instead, both abi3 and the new ABI are compatible with a given CPython version yes or no.

A potential upside of keeping .abi3 is that it might require a couple fewer changes in tooling, however, most build systems (e.g., meson, cmake) as well as abi.stable_abi_suffix in build-details.json will derive the value from sysconfig.get_config_var('EXT_SUFFIX') & co. anyway, so zero changes may be necessary either way.[1]

A downside is that it becomes harder to tell whether an extension module is valid or not. From current experience (e.g, with relying on cp3xx.so vs. abi3.so during debugging, or the annoyance of Windows currently not using .abi3.pyd) it can be quite annoying to not be able to distinguish what was actually built.

It may also be harder for build backends; if the build system produces .abi3.so extension modules, the build backend may rely on that to do validation, derive its own wheel tags from it, etc. E.g., here is an example for meson-python checking file extensions here.

Thinking about it from the source level and taking into account what backwards compatibility says, many libraries will have to support both old and new stable ABIs because they will support older Python versions for several more years. If I’d want to add support for abi3t to a library, I think I’d need:

  1. Support in the build backend first.
    • It should either no longer auto-switch from the limited API to the default CPython-specific one when run under a free-threaded interpreter, or gain a new config setting to let the package author tell the backend to do so. The latter is probably needed for backwards compat.
    • Allow targeting abi3t when building with a default (with-GIL) 3.15 interpreter. This needs to be opt-in, because the build backend cannot know whether the package author has made the changes required by PEP 793.
    • Build backends may not know what value was used for Py_LIMITED_API (e.g., for setuptools + cmake, it’s usually hidden in CMakeLists.txt; setuptools only has a py_limited_api boolean keyword in Extension that controls the filename, nothing else) - another reason for adding a config setting to let the package author select between abi3 and abi3.abi3t wheel tags.
  2. Binding generators (e.g., Cython, nanobind) with support for using the limited API should be updated for PEP 793.
    • Cython is already in progress I see (cython#7276) - nice!
  3. Then port all extension modules in the package to support both ABIs.
    • This will happen one by one. Is there an easy way to detect which ones have been ported? I believe that this is easy by design because -DPy_LIMITED_API=0x030f0000 will error out at compile time for not-yet-ported modules. @encukou the PEP doesn’t say so explicitly I think, can you confirm that this is the case? The “discontinued” in the table in compatibility overview might be saying the opposite - that’s unclear to me.
      • Should wheels tagged with cp315-abi3 be prevented from uploading to PyPI? And what are installers supposed to do with it, treat it as compatible or incompatible with a 3.15t interpreter? EDIT: sorry, the table with checkmarks and crosses does make it clear; let me think some more based on that.

Installers cannot really know this. The responsibility is for build backends I’d think to produce valid wheels, and installers just have to assume that. If the .abi3.so extension was changed, they could validate that at least - but I’m not sure that that’s a useful request either.

PEP 809 says nothing about filename extensions either way. I assume the intent there is to change it to .abi2026.so @steve.dower?

  1. This depends a bit on the answer to my questions further down, it’s not quite clear to me if it’s still possible to build the old-style abi3 with `-DPy_LIMITED_API=0x030f0000. ↩︎

2 Likes