Yes.
(More of a constraint than intent, but, yes.)
I (and the PEP) treat packaging as part of the installer. I don’t think that’s a problem in practice :)
It’s mainly there because, to use your words, this “can be “built” simply by building a wheel with any old tags, and then renaming it.” :)
Keeping all these tags clear is hard, but I’m also juggling reader viewpoints here (packaging, implementation, developer experience…). Hope you understand if I won’t highlight the answer to this confusion.
With a build tool, this isn’t much of a concern. There’s unfortunately a bunch of effort needed to port any given extension to abi3t. We’ll not see everyone switching instantly, and it’s not really a problem to continue building cp310.abi3 along with cp314.cp314t and cp315.cp315t and cp316.cp316t until both the tool and the project want to consolidate.
The easiest way (settings-wise) is to build with a free-threading interpreter, and then test the result on both FT and GIL-ful. Of course, the tool needs to grow support for this (i.e. apply the abi3.abi3t wheel tag in this configuration).
The common way, I imagine, will be a setting for the build backend in pyproject.toml, CMake file, etc. Ideally, take the way you select abi3 today (which is tool-specific), and do the same with a t added to the option name.
I also have a PR, for a related 3.15 feature. I’m looking for a possible way to get feedback earlier while staying respectful to the maintainers.
Anyway, as above – if the timeline’s measured in years, it’s fine.
(FWIW, my “If there’s anything I can help with, let me know” comment there is genuine.)
Thanks for pointing this out. It should be corrected in my packaging PR now.
This is how I implemented it for setuptools. If you build a limited API wheel using a free-threaded 3.15 interpreter, you get a wheel tagged abi3.abi3t. You select Py_LIMITED_API in the same way as you would for a GIL-enabled interpreter, using the extension module option. See for example the setuptools/c example in the stable-abi-testing repository:
The build settings comes from the tool.setuptools.ext-modules section of the pyproject.toml and and the [bdist_wheel] section of the setup.cfg. The compile-time flags might also have to happen outside of pyproject.toml, e.g. in build configuration, since presumably at first projects will distribute both an older abi3 wheel targeting an older stable ABI version and an abi3.abi3t wheel targeting Python 3.15 and up. Setuptools doesn’t set compile-time macros to enable limited API builds, it’s up to users of setuptools to manage that.
Note that setup.cfg is necessary, since support for setting distutils settings in pyproject.toml only has experimental support in setuptools and the warning about this says the syntax might change in the future.
The Py_TARGET_ABI3T macro should also allow a similar workflow, possibly with a different UX in setuptools based on a hypothetical py-target-abi3t option in tool.setuptools.ext-modules. I decided to make a more minimal change to setuptools to enable PEP 803 testing.
For PyO3, I will make it so that the abi3-py315 feature produces an abi3 extension on the free-threaded build and then maturin will produce an abi3.abi3t wheel. Cryptography already produces two sets of abi3 wheels per architecture, so they’ll just need to add another case that enables the abi3-py315 PyO3 feature at build time on a free-threaded interpreter:
I commented on the SC thread, but I wanted to follow up here as well.
I think that PEP 803 as it stands is somewhat broken?
Most of the PEP treats abi3 and abi3t as distinct entities, which I think is good, it should treat them that way. However, for the filenames of the extension modules themselves it’s treating abi3 and abi3t as equal– Which I think is wrong?
The Stable ABI (abi) promises forwards compatibility, if a target interpreter supports abi3 and it’s >= the version compiled against, then the module is ABI compatible. Treating abi3t as .abi3.so means that documented property is broken. The PEP handles this case for wheels (where it would be most impactful) by treating abi3 and abi3t as distinct things, but it explicitly doesn’t do that for the extensions themselves.
As far as I can tell, the rationale for this is because both GIL and Free Threading can support abi3t, and so they want a shared filename for them both.
I believe the correct answer for handling that case is to have both “types” of Python support .abi3t.so, and only GIL Python support .abi3.so, which would then be reflected in the wheel tags as well (as I believe they already are).
The only downside I can think of to this is that Python 3.13 and 3.14 already exist and look for .abi3.so, but I don’t personally think that’s a compelling argument. All of the PEPs and documentation explicitly call out free threading as not having a stable ABI yet, so I’d consider the fact they’d load a .abi3.so extension a bug.
tl;dr: Yeah, recognizing .abi3t.so in both builds and abi3.so only in non-free-threaded ones is probably a good move to make, but it’s more of a practical hack than conceptually the right thing to do.
The promise still holds for GIL-enabled builds.
For free-threaded builds, this is broken, needs to break now, or will break in a a free-threaded future. (Which of those? That depends on the exact wording of the promise just as much as on what PEP 803 or CPython do.)
Right. The PEP treats abi3 and abi3t as distinct – but, crucially, it allows a single installed extension to be compatible with both. This works for the actual 3.15 ABIs (they have a usable overlap), and for wheel tags where you can have abi3.abi3t.
The place where it doesn’t work is the filename extension. You can’t have both there.
Some possible solutions in this space:
make the FT builds accept *.abi3.so (what the PEP does). This reuses an extension that already exists; tooling that doesn’t care continues to work unchanged.
make the GIL build accept *.abi3t.so (which you’re proposing, AFAIK). Conceptually, this is a lie just as much as the first option.
use just *.so. Look ma, no lying! Also, Windows does this (with .pyd), with no ill effects.
If we’re avoiding lies and broken promises, the last solution seems best. But of course, that’s not what we’re doing – we’re trying to be most helpful to users.
Could we frame the discussion that way? Should be more actionable than complaints of brokenness.
In yet another instance of confusing terminology, 3.13t does have a (lowercase-s) stable ABI, and so does 3.14. An extension compiled for 3.13.0 will work on 3.13.4. It won’t work on 3.14 of course, but since it’s fixed and known, you can theoretically enumerate the differences between 3.13 and 3.14, and add runtime ifs for all those differences. You’d get an extension that works on both 3.13 and 3.14.
The same works for the (capital-S) Stable ABI. You can enumerate the difference between 3.14 abi3 and PEP-803 3.15 abi3t, and build an extension that you can correctly tag cp314-abi3.abi3t. As far as I know, this is not only possible but practical, especially for a project like PyO3 that would already generate wrappers for both ABIs.
The place where it’s not very practical to support this is CPython, so the PEP only talks about this as a possibility rather than a guarantee. It does try to keep this as a possibility, though – in response to Emma’s post from (much) earlier:
Do we make that use case harder, or do we make your use case harder? “Your” use case being Debian’s, as discussed on the SC thread.
Should we even support sharing a site-packages between interpreters? That’s perhaps something the SC should decide – the issue is perhaps bigger than this PEP. For example, we have #122917 which is blocked for performance reasons. Or the recent calls to disallow such “overlapping” installations entirely, even between debug/non-debug builds which have the same ABI. (Please chime in there – e.g. Greg’s post should be quite worrying for you.)
As for the “cryptography” case from the SC thread: note that the Stable ABI is, alas, only about ABI stability. There are other reasons why a cp38-abi3 wheel would be incompatible with Python 3.11 – what you can work around using if in Python, may become #ifdef in C. For a simple example: PyImport_Import has a stable ABI, put the module you import might be deprecated or removed.
There’s a question whether, in cryptography’s case, publishing a cp311-abi3 wheel means they can make their cp38-abi3 wheel incompatible with 3.11+ at runtime.
I haven’t asked either, but I’d bet cryptography maintainers would disagree with Debian here. And I doubt they test cp38-abi3 wheels on 3.14.
If we do support “overlapping” installations, we can do better. For example, would it be worth it to rewrite the import system to allow handle hundreds of possible filename tags efficiently, so you can be extremely specific there?
Anyway, all in all, recognizing .abi3t.so in both builds and abi3.so only in non-free-threaded ones is probably a good move to make, but it’s more of a practical hack than conceptually the right thing to do. I wish I had had more months to think about this problem.
I imagine it will complicate things a bit for build tools – but those need some changes anyway.
On the other hand, why wouldn’t Debian be fine building .abi3.so for GIL builds and individual .cpython-3XXt-*.so for free-threaded?
As for cp314-abi3.abi3t wheels (the ones you’d need specialized tooling to build), they can use “plain” .so filenames, with neither .abi3 nor .abi3t tag. So, that use case happens to still work.
Thanks for the follow-up here, I wasn’t even aware that there was significant discussion happening on the SC thread. Luckily it seems to mostly be around the .so naming question, which I have limited interest in, but I do have the following concern, which I think needs to be addressed somehow.
(This was in answer to my question “what will a developer need to do to build and test abi3 and abi3t compatible wheels”).
I think what matters to me, from a packaging perspective, is to get answers to the following question.
For wheels tagged abi3, abi3t or abi3.abi3t respectively, how should .so files be named in the wheels? I would hope that the answer is .abi3.so for abi3 wheels, and .abi3t.so for abi3t wheels, if only because that’s the most intuitive answer to me. To be clear, I’m happy if other names work, this is purely about what naming tools should use (as in, recommendations for best practices - and “something that isn’t confusing for a non-specialist inspecting the wheel”[1] seems like an important requirement to me for that case). And in case it’s not well known, packaging interoperability standards routinely include statements about what tools SHOULD do in various situations. So yes, I would expect this to be stated in the PEP, even if it’s not common practice for language PEPs.
For the abi3.abi3t wheels, I’m much less clear on what name should be used. Either of .abi3.so or .abi3t.so seems like it would be misleading, if only because we’d have an abi3.abi3t wheel containing the same files as a plain abi3 (or maybe abi3t) wheel.
Your answer suggests that .abi3t.so will be used, and that in practice abi3.abi3t wheels will be identical to abi3t wheels, because the exact same build process is used to produce them. If that’s really the case, I’d much rather that abi3t wheels didn’t even exist, as their presence is misleading.
If all of this complexity is caused by the need to work around the situation with Python 3.14t, I feel like we’re contorting the solution to handle a one-off situation, and we’d be better off simply acknowledging that Python 3.14t can’t be handled cleanly, and starting with a clean slate from 3.15.
Consider an auditor reviewing the wheel for suspicious content, for example ↩︎
I don’t think that it’s super useful to rules lawyer the promise, but the actual documented promise basically guarantees that abi3 will remain ABI compatible across all future Python 3.x versions.
Obviously that’s actually impossible to do if we ever make Free Threading on by default. I personally think it would be reasonable to say that the intention is that it will remain ABI compatible across all future versions of Python that continued to support the abi3 ABI, which I think follows the “spirit” of the promise best.
From a practical point of view, FT attempting to use the .abi3.so extension is going to break people, unless we’re very careful to ensure that we never put FT Python in a position to load a .abi3.so that was compiled from a GIL Python.
Right now most people aren’t exposed to FT Python at all, but if FT gets made the default, I suspect we’re going to have a lot more of these random edge cases come out of the woodworking where some tool didn’t realize that abi3 wasn’t going to continue working correctly when they happened to get FT by default.
My question would be whether the fact that abi3 and abi3t are able to overlap is just an implementation detail of abi3 and abi3t in Python 3.15, or whether that’s actually something we’re promising is going to continue into the future.
If it’s just an implementation detail in Python 3.15, then I don’t think we should put much thought into how to enable someone to share a single .so between them. That’s just something we don’t actually support, but if you’re able to figure out how to do it more power to you.
For example, I suspect it’s probably possible, if you’re careful, to share a single .so with the “unstable” ABI between two Python minor versions. That doesn’t mean that it makes sense for us to have Python 3.15 use .cpython-314-x86_64-linux-gnu.so, because you could do that– it just means that if you want to do that, you’re on your own. There’s even a pretty easy “built in” answer of just use the bare .so (or you can just not share a file and ship both .abi3.so and .abi3t.so).
If the answer is that we are promising that abi3t will overlap with abi3 (or more accurately, that a GIL Python is expected to continue to support abi3t and it’s not just some implementation detail of 3.15), then I think the right answer is to have GIL Python support abi3t (in addition to abi3, for as long as it does support abi3) and I don’t think that’s a practical hack– I think it’s just saying we have a new stable ABI, it’s called abi3t, and it’s supported by both GIL and FT Python.
Well, I’d argue that they continue to “work unchanged” as long as they’ve (or some other part of the toolchain) has been updated to ensure that they do not re-use a *.abi3.so that came from a GIL Python.
IOW, tools like packaging had to be updated not to install a abi3 wheel on FT, otherwise it would have installed a wheel with a .abi3.so file and then the user would have gotten linker errors or segfaults or some other ABI related issue.
Which isn’t really working unchanged It’s just working with different changes.
It’s only a lie if the GIL build doesn’t actually support abi3t.
If it does support it, then it’s just honestly reflecting that it supports it. If it doesn’t support it, then we shouldn’t be trying to tie ourselves into knots to make an implementation defined edge case have a clean/supported implementation.
I think the obvious answer is that we have to give preference to the use case that is a documented as supported, went through a PEP process (multiple times for different aspects of it) and got accepted, and has been relied on for 15 years now.
It’s also not really making the “single abi” use case harder, since they can just make a untagged .so, which isn’t at all a wrong or unsupported thing to do! If they’re relying on implementation details to get something working, then they shouldn’t be using the stable tag.
I don’t actually have a strong opinion on whether we should support that or not. My concern is that we currently do support that, and thus we need to either take that into account with our designs so that we continue supporting it or we need to explicitly drop support for it.
I do strongly believe that we shouldn’t just randomly break it and then shrug our shoulders and go “oh well, you shouldn’t do that”, and we particularly shouldn’t, either implicitly or explicitly, say that people relying on that supported feature are doing the wrong thing (up to, and including having people say that they wish the PSF would take legal action against the downstreams relying on that feature).
I also strongly believe that If we decide that backwards compatibility isn’t important here and we should remove support for these overlapping installs, then we should do it “correctly” and actually remove or discourage the feature somehow.
FT would actually be a good time to do this, since it already can’t correctly load a .abi3.so from a GIL build, so it offers a clean “break” where FT builds could just go back to only supporting .so. Any performance issues that may happen from #122917, surely also affect any additional suffix we support over the first one, so if the performance of these suffixes are truly a concern, then that seems like a benefit in and of itself.
Of course The Stable ABI isn’t that it’s going to work, it’s that you’re not going to run into ABI issues.
I doubt Debian cares about cryptography’s wheels one way or the other, since they’re going to build from source (presumably on the oldest version of Python they support) and it’s on them to test whether it works on all their supported versions of Python or not.
This is one of the problems I’ve been thinking about for a while (hence my alternate proposal), and while I’m pretty sure that we’re unlikely to actually be able to feasibly build an extension that works in both,[1] it seems like the easiest choice here is to recognise something like .abi3+abi3t.so in both, but only the specific ones in each.
Consider that per-object locking would have to work identically in both versions, including against the version of the GIL that isn’t aware of per-object locks or any non-Python native locks from a particular language. I rate this as “low probability of actually working for anything interesting”. ↩︎
FWIW I think the answer to this kind of depends on whether we expect GIL Python to actually support abi3t , or whether the fact you can build something that supports GIL and FT Python is just something you can sort of do, maybe only in 3.15, or maybe only if you’re really careful about it.
If we expect GIL Python to actually support abi3t, then I expect the abi3t wheel tag to include GIL Python 3.15+ and it’ll be rare to have abi3.abi3t wheels (just like it’s rare to have wheels tagging two of the unstable ABIs).
If we expect GIL Python to not actually support (versus “well you can kind of make it work if you try”) abi3st, then I expect the answer to be “well you’re off the beaten path, so you’ll have to figure it out “.
Well I guess if something like abi3+abi3t.so happens, then that’s the obvious answer for a abi3.abi3t wheel, but it’s unclear to me whether that makes sense or not?
That’s not the current PEP, but it’s what would work for Debian and wouldn’t break anything. I’ve now sent a draft PR with wording changes for this.
Not promising, but I don’t see any reasons to break this on the horizon.
And there are reasons to try to keep it this way. If a big benefit of Stable ABI in general is that there are fewer wheels to release, allowing abi3.abi3t instead of separate abi3 & abi3t is obviously helpful.
One way we could deal with this is tell Debian that PEP 803 is not for them, and they should keep building version-specific cp3XXt wheels until they can drop Python 3.14 and below.
But, looks like that won’t be necessary.
I guess the rest of what we disagree on is wording – you see breakage where I see a new feature not supporting a use case yet. Maybe we can agree to disagree on that.
Do you have an example of what won’t/can’t work?
AFAIK, free-threading is rather carefully designed to still work if only one thread state can be attached at a time (which is what the GIL does).
My understanding is that any deadlocks that are possible when the GIL is enabled are also possible when the interpreter needs to do a global synchronization event. Like when the GC runs. That means these sorts of bugs will get caught even if extensions aren’t tested with the GIL enabled like we generally do these days.
I’m also hoping to do an end-to-end test of this on some real-world packages. I think I can straightforwardly test brotlicffi and pywavelets with branches I have for CPython, Cython, NumPy, setuptools, meson-python, and CFFI in forks on Github. We might also be able to test astropy, which I learned today recently switched over to shipping abi3 wheels.
I’m currently trying to make sure all the packaging tools work with this updated approach. Hopefully I’ll have more details to share tomorrow.
Yeah, my position is based solely on my own fear and not on evidence (since Nathan is currently gathering that evidence ). But I think after more than ten years in the industry, I’m allowed to gut feel “this may not work perfectly first time, every time”. Happy to be proven wrong, but this one feels like a high bar to me.
As one data point, there are a lot of Py_GIL_DISABLED checks in our headers, and while only the ones in the limited API matter to user code, all the others imply some difference in behaviour on our side of the agreement. If user code is semantically reliant on any of these, there may be incompatibilities that don’t appear at the API/ABI level, but can only be dealt with by using the preprocessor variable or by runtime detecting which implementation is being used.
Again, it’s possible that we’ve gotten this right, but I don’t think it’s safe to assume that, and I know we haven’t strictly prioritised “functionally identical between GIL/no-GIL” (since the whole point is to be functionally different).
Overall, I’m okay with people being able to experiment with the same build for both runtimes, but I’m very hesitant to endorse or encourage it.
To be clear, I don’t mean promise in like a “you’re offering up your first born if you break this” sort of way
Mostly I think that we should be explicit about what you can and can’t rely on. Right now, we have:
The unstable ABI, you can rely on this not breaking within the same minor version and the recommended filename suffix on POSIX systems will ensure that that Python won’t attempt to load an extension with a different ABI.
The stable ABI. You can rely on this not breaking on any newer version of Python that supports abi3, the recommended filename suffix on POSIX systems won’t protect you from a “too new” abi3 extension, but will make sure that it at least is an abi3 extension.
The… I dunno what to call it, YOLO ABI? Basically an undefined ABI, which probably technically most closely maps to the unstable ABI, but in practice the filename suffix (on POSIX) makes not indication of what ABI it is targeting.
I think it’s important to actually document what you can rely on for abi3t, and for the tooling to not rely on things that we’re not willing to say you can rely on.
So for instance, abi3 is documented as having “>=" support. So tooling all over the place bakes in an assumption that if you have Python 3.2+, then you support abi3. The existence of FT Python at all broke that assumption, and that’s “OK”, the hope is that FT Python provides benefits that are worth that breakage. We didn’t correctly anticipate a need to break abi3, that sucks but life moves on, we just have to try and minimize the pain caused by that :). Having FT Python not load .abi3.so helps to minimize that pain I think, because we don’t know if an .abi3.so was compiled for FT Python or not.
Something that the abi3.abi3t wheels are trying to enable is for projects to not have to produce a different wheel for GIL and for FT Python (and ideally without having to produce one per Python version).
I think that goal is only actually achievable if there is a common, stable ABI between GIL and FT Python that has the same “>=" support. If we’re wishy-washy about whether you can rely on that, then tooling shouldn’t (IMO) rely on that, and things like packaging.tags shouldn’t assume the >= behavior like it does currently.
Typing this out made me realize though that there’s maybe 3 things that need to exist?
The Stable ABI for GIL Python.
The Stable ABI for FT Python.
The Stable ABI for GIL + FT Python.
Currently we really only have the first one of those defined (abi3), and I think that should continue to mean >= for GIL Python.
The PEP currently defines the second of those as abi3t, and relies on wheel’s compressed tags to implement the third of those (abi3.abi3t) for wheels and re-uses the .abi3.so file extension to implement the third (and I guess second) of those at the import layer.
Which re-using that .abi3.so I think is “dangerous” (for lack of a better word), because it relies on some other “layer” making sure that you don’t use something that is actuallyabi3 in a FT Python context, because it doesn’t actually support abi3.
So far I’ve been thinking about this in terms of abi3t covering both the second and third ABIs above, because that made sense to me in the current state of things where abi3t is a subset of abi3 and the actual specifics of ABIs are outside of my wheel house so I’m not sure if/when we’d ever need to make that not the case.
That being said, if we’re worried about telling people that they can rely on abi3t working for GIL Python because we’re unsure of what we might need to add in the future and whether it would work on GIL Python. Maybe it would make sense to explicitly define/name the third ABI from above (let’s just hypothetically call it abi3c, for abi3 common).
In this hypothetical, abi3 is only supported on GIL Python, abi3t is only supported on FT Python, and abi3c is supported on both. abi3t and abi3c are currently the same thing, and are a subset of abi3. Importantly abi3c would always be a subset of abi3t and abi3, but abi3 and abi3t could evolve additional items independently of each other.
Practically speaking, we’ve currently implicitly defined abi3c in this PEP, but called it abi3t and not given any indication of what we can actually rely on for users to know if it’s actuallyabi3c or if it’s actuallyabi3t. Of course we could just decide to only define abi3c (whether it’s called abi3c or abi3t doesn’t actually matter) currently, and punt on defining the “FT only” ABI until we actually need it .
I think that this is important though, because without knowing what they can rely on, different tools are going to bake in different assumptions throughout. I suspect (for instance), that a non zero number of build tools and/or projects are going to bake in assumptions that if they build on FT Python and produce abi3.abi3t wheels that it’s going to be compatible with >= Python versions for both GIL and FT.
Which I think is a reasonable assumption since that’s at least explicitly mentioned in a quote in the motivation for PEP 803 .
When all those tools and projects bake those assumptions in, it means that if we try to go against those assumptions, Hyrum’s law is going to kick in and we’re going to get a number of very annoyed/upset users :).
So regardless of what the actual list is, I hope that PEP 803 will define what you can actually rely on for abi3t, and will explicitly discourage relying on things that are not defined as a thing you can rely on– If that means 803 discourages abi3.abi3t wheels because we don’t want people to rely on there being an overlap, then I think that’s OK.
NOTE: This is a bunch of random thoughts, that aren’t exactly related to PEP 803, but aren’t unrelated to PEP 803, so I’m including them here for whatever benefit it may be
One thing that occurs to me typing all of this out, is that it may make sense to add an API (even if it’s just a sysconfig var) to actually enumerate what ABIs a particular Python supports (and probably whether they are a “Stable” ABI or not). That’s not exactly something that is specifically related to PEP 803, but it is kind of related to FT Python at least.
Tools like packaging made the assumption that you could tell if Python supported abi3 by checking if sys.version was >= 3.2. That assumption broke with FT Python and you’d get broken wheels on FT Python because that hard coded assumption saw Python 3.13+ and said “cool abi3 works here”, and packaging had to be patched to condition that on free threading.
Now there’s a PR to add abi3t support in packaging, and it does the same thing– hard codes an assumption based on Python version and FT. But we’re in this thread hedging our bets about whether abi3t will work on GIL or FT Python both long term or not, so obviously we’re hesitant to make promises– but we’re hard coding assumptions that depend on that question places.
If we had APIs that would tell you what ABIs are actually supported for a given Python (and whether they’re “Stable” or not), then we (hopefully) remove those hard coded assumptions. If packaging could get a list of stable/unstable ABIs from the Python interpreter itself, then Python could (at a technical level) add a new stable API every minor release if it really wanted to (though presumably from a social aspect there would be reasonable pushback against doing that). Likewise, if the mechanism for activating a stable abi compilation was parameterized and gave you the expected suffix somehow, build tools also wouldn’t have to individually know about each ABI, just how to pass the ABI identifier in + get the filename suffix out.
That would mean though (I think anyways), that a new Stable ABI would be a lot less of an effort across the ecosystem, which would make the risk of adding something to be “relied on” to the Stable ABI to be smaller, since if it turned out to be a bad idea, you could roll out a new Stable ABI.
and given the plan to merge the two builds, I’d consider the discrepancies to be bugs, and, like other bugs in free-threading, we’d want to fix them when they’re found.
It’s possible that this would need ABI changes, but given how flexible we … um … want the stable ABI to be wrt. implementation changes/differences, I have hope that we can keep these the same.
Uh, don’t know where to reply. I’ll start where I started – at the other end.
If you build for abi3 version 3.X, your extension will be compatible with GIL-enabled Python version 3.X and above. In return, you can only use a subset of the API in your code. Basically: if you only call the handful of functions which we guarantee to keep “forever”, then your extension will work “forever”.
If you build for abi3t version 3.Y, your extension will be compatible with free-threaded Python version 3.Y and above. In return, you can again only use a subset of the API – possibly a different subset than above (though I’d like to keep it the same in practice).
If you build for bothabi3 3.X andabi3t 3.Y, your extension will be compatible with GIL-enabled Python 3.X+ and with free-threaded Python 3.Y+. Of course, you can only use the intersection of the allowed APIs.
Wheel tags can encode such an “intersection” naturally: abi3.abi3t is defined to be compatible both with interpreters that support abi3, and with ones that support abi3t. (Due to a … historical accident, the version is encoded in a different tag, so the versions 3.X and 3.Y need to be the same, but that’s what you want in practice anyway.)
Such an extension will also, naturally, be compatible with the version-specific cp3X & cp3Yt ABIs. Those “exist” too; there’s not much stopping you from publishing a cp314-abi3.cp314t wheel today.
How do you map that to filename tags? Those aren’t as expressive, and I argued that they don’t need to be. Windows is fine with plain .pyd; we could use plain .so on *nix too, but reusing .abi3.so seemed reasonable (especially given that free-threaded 3.14 already tries to load,abi3.so extensions – an oversight we can’t change at this point). Adding .abi3t.so – a whole new tag – just for symmetry/consistency didn’t seem worth the churn.
Now that I understand Debian’s use case, it looks like it is worth the churn – but as a practical choice to help the “overlapping installations” use case. It is not a way to fully encode the extension’s ABI(s).
What I’d suggest then, personally is PEP 803 should probably take one of two choices:
Extend abi3t to explicitly cover GIL Python, in that we promise the subset of the API you can use will work on both GIL and FT Python and “give up” the flexibility of having the possibility of of abi3t to diverge between GIL and FT. Then abi3t is just the answer for what you use for both GIL and FT. [1]
Keep the flexibility to diverge GIL and FT, and either expect people who produce a abi3.abi3t wheel to use just .so, ship a .abi3.so and a .abi3t.so, or define a different filename tag (.abi3-abi3t.so?) to cover the “intersection” case.
If we do this one, it probably shouldn’t be the default behavior of enabling the Limited API for Free Threading in the various build tools, and you should be somehow opting into the intersection of abi3 and abi3t, ideally with similar support to try and turn using something outside of that intersection into compile errors– but at a minimum people should know they’re opting into something that is possibly stricter than either abi3orabi3t and they’ll have to ensure that they’re compatible with both.
I think those are the only options that make sense to me?
Trying to keep the flexibility to diverge but use either.abi3.so or .abi3t.so to cover both, means that if they ever do diverge, then things are likely going to break for people unless all of the tools are very careful not to let a .abi3.so extension (or a .abi3t.so extension) to get on the sys.path for the “wrong” kind of Python.
For instance, if 3.16 adds a new abi3t only thing that isn’t in the subset supported on GIL Python, the tooling is probably not going to help them here, since AIUI the way to produce a abi3.abi3t wheel is to compile it on FT Python with the limited API… but that only works “ok” because at the moment abi3t is fully supported by GIL Python. Once it’s not, then people have to manually try and guess what Limited API functions are OK to use.
As far as I know the headers you use to enable the Limited API try to protect you from compiling against an unstable ABI, but that won’t help you when you’re attempting to target the intersection of abi3 and abi3t via compiling for abi3t if abi3t has diverged from being a subset of abi3.
On top of that, If I’m reading them correctly, it looks like the existing PRs (they are draft PRs of course, so it’s not entirely fair to use them as an example) are all assuming that if you opt into abi3t you’re producing a abi3.abi3t wheel and don’t make it possible to produce only a abi3t wheel. [2]
It also feels like, if we let abi3t diverge while expecting people to produce shared , we’re sort of re-inventing the problems of PEP 384 (but not nearly as bad), which your PEP 652 fixed because abi3 wasn’t actually defined well, except we’ve shifted it to whether the intersection of abi3 and abi3t is defined well or not?
Where we’re at now, it feels equivalent to a PEP calling out a cp314.cp315 wheel as being an important use case, that we expect that’s how everyone is going to produce wheels for 3.14 and 3.15, but saying it’s on them to make sure that they only use the parts of the ABI that are the same between cp314 and cp315 and we reserve the right to add stuff to cp315 and if we do then the ecosystem has to catch up to not produce cp314.cp315 wheels anymore.
Of course nothing says we can’t release abi4 at some point that does diverge! ↩︎
This is a reasonable thing to do today, but by baking that assumption in it’s going to spread throughout the ecosystem and I suspect make it impossible to actually make abi3t diverge without breaking parts of the ecosystem. ↩︎
Just a note that this choice shouldn’t be taken as determinative or necessary. It was simply the most expedient thing I could do to get something working.
Yea I tried to call out that they were just draft PRs so it’s not really fair. I was mostly just using them as examples of how easy it is for the abi3t = abi3.abi3t hard coded assumptions to get introduced as the “natural” implementation. So it’s probably worth trying to add some friction to making that assumption so people are less likely to actually do it without realizing it
Keep in mind that PEP 803 adds a runtime ABI compatibility check, so “break” means you get a “symbol not found” error (not amazing, but searchable), or a proper informative error message if the Python importer gets to run.
For the purpose of parallel installations, having only .abi3.so and .abi3t.so should be perfectly fine.
If you look at the PEP’s recommendations for build tools, you’ll see the just that. The intersection is first (as it’s more useful if there’s no – or slight – divergence); “abi3t only” is second.
As written, abi3.abi3t is an explicit, well-defined choice. It’s meant to be the default, but there’s a clear way to get abi3t only if that is/becomes useful.
The divergence would need to come in a new feature version of Python. I think it would be reasonable to ask tools to update at that point. That’s what we’re doing here, too :)
I’m still operating under the assumption that GIL & FT are supposed to be converging, not diverging. It’s fine to back door open in case we mess that up, but I don’t think it should translate to mandatory upfront complexity for users of the build tools.
Also to note: if there is a slight divergence – for example, in some API that not everyone uses – the abi3.abi3t intersection should still be useful.
What that section doesn’t cover is how the build tools should name the .so file(s) that are produced. Unless there is another part of the PEP which covers that (I couldn’t find one) that’s something that needs to be added.
Yep, sorry for that. It’s currently a draft PR, as the use case was pointed out after SC submission, and there’s one detail I’m not sure on (.abi3+abi3t.so, which I currently think should go in rejected ideas).
It’s just working with different changes.
The Stable ABI isn’t that it’s going to work, it’s that you’re not going to run into ABI issues.