I’d like to share the results of some tests I’ve been doing on a version of CPython that exposes critical sections in the stable ABI.
This is in the vein of my post in the packaging thread on PEP 803 where I did some end-to-end tests on real-world packages. I based my tests on the CPython branch I used there, with additional patches to expose the critical section structs, functions, and macros (but not the mutex variants) in Include/critical_section.h. I also exposed PyMutex as an opaque pointer in pytypedefs.h because there are PyMutex * pointers in the critical section structs. You can look over the changes here:
The main reason I had to make these changes is because PyO3 uses critical sections in its implementation, as Petr described in the post opening this thread.
I also created branches of PyO3 and maturin that make the following changes:
- Add new abi3t and abi3t-py315 PyO3 features to enable abi3t builds.
- Update PyO3’s FFI wrappers to model the
_Py_OPAQUE_PYOBJECTmacro and set it at build time along withPy_LIMITED_APIandPy_GIL_DISABLEDwhen doing builds with the abi3t feature. - Add support in maturin for generating abi3.abi3t wheels.
You can look over the PyO3 changes and maturin changes to get an idea of what I did. These diffs are pretty substantial but I’m hoping I’ll be able upstream part of my changes now, before PEP 803 is decided. I think the refactoring I did to replace boolean flags with enums makes the code clearer than it was before.
I also had to make a small change for cryptography’s CFFI wrappers to account for the adjusted limited API build and enable the abi3t feature crate-wide. Crytography’s CI does it at build time by passing --config-settings, I made the source-level change for convenience.
I also used branches of setuptools and CFFI, as described in my other post.
In the end, I’m able to successfully build cryptography abi3.abi3t wheels using this setup on my Mac. The same wheel installs and passes all tests on both builds. I do need to use a patched version of pip and packaging to actually install the wheels I get. I’d also need to use a patched version of uv to install the wheel on the free-threaded build, because uv doesn’t yet know about the abit3t tag. The current releases of uv and pip will both install abi3.abi3t wheels on the GIL-enabled build, so patching of installers is only needed to support these wheels on the free-threaded build.
All in all, I’m pretty happy with this! So far I haven’t found any issues with testing builds for real-world packages that simultaneously support both builds. I’d also like to re-try this on Linux and Windows to see if there are issues on other platforms.
For what it’s worth, in the context of whatever remaining disagreements there still are in this thread: I don’t think changing the critical section struct or API would make any major difference here. I’d just adapt to it in PyO3 and add conditional compilation to support older free-threaded builds. That said, I personally would still prefer to simply expose the critical section API as-is as I’ve done in my CPython branch, since that requires zero internal churn in CPython.