PEP 803: Stable ABI for Free-Threaded Builds (packaging thread)

Packaging
21

It’s not the original intention – the Stable ABI and wheel tags evolved independently. It’s more of a solution that wheel building tools adopted.
Nevertheless, it’s working fine.

[Pure opinion, stated without sugar-coating for easy disproval:]
Because this would be needless churn. The only reason for the change would be percieved purity.

Also, practically: the version tag already has the correct semantics of “n and up”. Making abi3_15 compatible with abi3_16+ would add new special casing.

Anyway, yes, PEP 809 makes this change. (It starts out saying that “The Stable ABI as abi3 can no longer be preserved, and requires replacement” – a statement I hope will get some elaboration.)

Functions are sometimes (rather rarely) removed from the API. They stay in the ABI, but become no-ops (if they’re redundant), or always raise (if they don’t make sense at all any more), or are horribly slow, etc.
As for field ordering: there’s no common practice for changing structures, since the only relevant structure in the ABI is PyObject (and its subclasses). This PEP is the first – and hopefully last – Stable ABI PEP that needs to change a struct.

22

Practically, functions are just removed from the headers, but the definition remains inside the Python source code. The “old buffer protocol” is an example (look up PyObject_AsCharBuffer as an example)

1 Like
23

As a purely technical point, the standards don’t require that the version tag has those semantics. The tag cp311 means “works on a system that says it supports cp311”. It’s simply convention that Python 3.12 and later systems say that they support cp311 as well as cp312. Making systems that support abi3_16 state that they also support abi3_15 wouldn’t be a problem, it would simply mean adding a similar rule to tools (specifically the packaging library) that implement the current heuristics.

I wish we had made the standards require that cp311 implied cp310 and earlier. We would have had significantly smaller tag lists to check. But that’s a debate that was settled a long time ago, and it’s unlikely anyone will have the energy to revisit it now.

1 Like
24

Functions are sometimes (rather rarely) removed from the API. They
stay in the ABI, but become no-ops (if they’re redundant), or always
raise (if they don’t make sense at all any more), or are horribly
slow, etc.

Doesn’t the original implementation have to remain intact? Otherwise you
are breaking the stable ABI contract.

As for field ordering: there’s no common practice for changing
structures, since the only relevant structure in the ABI is PyObject
(and its subclasses). This PEP is the first – and hopefully last –
Stable ABI PEP that needs to change a struct.

…and therefore (for very good reasons) breaks the stable ABI contract.
In other words the existing ABI has never had an implementation removed.

25

OK but this is just shuffling words around. You are basically removing the public header and trying to hide the symbol and calling it function removed from API.

Essentially what you are doing is encouraging people to not to use the old symbol by making it difficult to be discovered. I don’t think this makes any sense in terms of ABI. You can do the same by removing the documentation and hope nobody uses that function. That’s not ABI stability that’s pretty much API stability. If you in turn, say you pass a struct and you get some padding, because a double and an int swapped where you don’t expect previously and then you have the same API but broke the ABI.

But anyways, if that’s Python’s definition, no problem, it’s good to know. I don’t mean to start a semantics discussion here.

What I am interested is about the name tags and other details.

26

Would it help to standardize the existing practice, which will be used for quite a while even if we change the mechanism?

Plus all tools that reimplement packaging, right? I have feeling that this would be a much bigger problem in practice.
How sure are you that there are “very few” dark corners that assume packaging’s current, non-standardized behaviour?

Looking at the Use section of the standard, I see that CPython 3.3 should look for py31 and py30 – but only for none-any.
(That seems to conflict with “tags are considered separately” you said before?)
Reading the standard, it seems that the exact list is left to the installer tooling – that is, the list is not authoritative, but illustrates that if a tool knows that py32-none-any is compatible with 3.3, then you can install that wheel on 3.3.
Is that a valid reading?

If so, shouldn’t the installer apply the same principle for py32-abi3?
Also, I don’t see an issue with a build tool generating py32-abi3 when it builds on 3.3, but is instructed to stay compatible with 3.2+ (setting `Py_LIMITED_API to 3.2).

I know that’s just one possible reading, but, it does seem to align with current practice.

Alas, the stable ABI protects you from missing symbols and memory corruption, but functionality is covered by the general backwards compatibility policy.

27

Maybe. The existing rules in packaging were created by working out what worked in practice, so they probably include a lot of obscure knowledge that’s not documented anywhere. From what I recall, @brettcannon did most of the work, and it was fairly tricky, so it would probably be worth getting his view.

Generally I’m in favour of standardising whatever we can, but I’m concerned that if we start to try to standardise this, we could end up down a pretty large rabbit hole :roll_eyes:

Yep. There’s likely uv at least.

I’m not at all sure, to be honest. The behaviour implemented by packaging is in effect a de facto standard at this point, and tools could be relying on any part of it. There’s also a number of public “low level” APIs in packaging, documented as:

These functions capture the precise details of which environments support which tags. That information is not defined in the compatibility tag standards but is noted as being up to the implementation to provide.

Tools could quite legitimately depend on any or all of those functions working as documented.

Yes, that’s the sort of arcane knowledge I was alluding to above. I have no idea why that is, and it seems to contradict your idea that abi3 is compatible with earlier versions. Or maybe it’s significant that this is CPython 3.3, when abi3 was introduced? Who knows?

Sorry, yes, that was an oversimplication - how tags are considered is based on what tag sets the system declares it supports, which comes from packaging (or whatever other implementation you’re using) - here. I tend to think of that as implying “each tag must match independently” because that’s the situation in the simpler cases, and it’s hard to explain the real rule :slightly_frowning_face:

It’s not the installer’s job to apply any rules other than “run down the list of supported tags, and pick a wheel that matches the earliest tag set on the list that it can”[1]. It’s the interpreter’s job to say what tag sets it supports - we do that in packaging because getting packaging-specific details like this into the stdlib has generally been too difficult (plus, we’d have no way of getting anything added to older Python versions).

So packaging is essentially a place to store information about the various interpreter implementations, in this context. And tools that can’t (or simply don’t) use packaging need to construct that set of information for themselves somehow.

I also don’t see a problem with that, other than the combinatorial explosion of tag combinations we have to return from packaging.tags.sys_tags, and the need to decide on the correct order of priority (is a py32-abi3 wheel better or worse than a cp33-abi3 wheel, or a cp33-cp33 wheel?)

A big part of the issue here is a lot of the questions that need to be answered are basically theoretical, until someone actually publishes a set of wheels with the particular set of tags that we need to decide between…

  1. Earliest because implementations are supposed to produce tag sets in order of preference ↩︎

28

Sorry, I might have gotten lost in what’s been answered and what hasn’t.
Could you repeat the outstanding question(s) for me?

I see. As an outsider, I see the installer and the libraries it uses as a single box :‍)

Thanks for the clarification!

You’d get the same problem with abi3_15, right? (Especially if you stay with the “tags are considered separately” simplification and allow cp315-abi3_15 to explode.)

(PEP 809 solves the explosion by not making abi2031 backwards compatible with abi2026 at all. But, it requires introducing a new design and immediately freezing it for 10 years…)

29

Possibly, yes. Although if platform and ABI were separated, wheels could be marked as any-abi3_15, so there’s no explosion. I don’t know how practical that is, though, as I really don’t have a good understanding of why people might need to specify cp315 apart from the current need to distinguish “sub-types” of abi3.

These questions are basically what I meant when I talked about a rabbit hole. Apart from the theoretical complexities, there’s also the practical issues of what people do in real life - and once we get beyond the big open source libraries like the scientific stack, there’s an enormous invisible “long tail” of closed source code that could be doing all sorts of things we’re not even aware of.

This is why I’d much prefer a “theoretically sound” model that’s based on compatibility guarantees that we are in control of, even if that pushes complexity back onto library maintainers and runtime checks. It’s one area where I think “practicality beats purity” breaks down… :slightly_frowning_face:

Welcome to the joys of the packaging ecosystem!!!

1 Like
30

Hmm… Well, since PEP 803 makes the runtime check mandatory, py3-abi3.abi3t should be safe. The minimum version could go in wheel metadata, like for common pure-Python py3 wheels.
Should we encourage that?

I don’t think we should avoid introducing cp315-abi3 & cp315-abi3.abi3t in 3.15 though. There are some reasons to build wheels for several Stable ABI versions. See cryptography with cp38-abi3 and cp311-abi3 for an example. (I suspect the reason for that particular case is a bug in CPython… but another project might want to e.g. take advantage of a new faster function while keeping a more backwards-compatible fallback.)

Of course there’d be a py3/py315/cp3/cp315 explosion, and the ordering would probably need a discussion on its own and…

… thanks‽

1 Like
31

Let’s get input from some of the people who would actually be producing wheels before deciding that (and probably also from some build backend maintainers).

But cp3-abi3.abi3t[1] seems wrong to me, as it misrepresents the abi3 policy, which is that you can’t use an abi3 wheel on a Python version older than the one it was built for. I know I said I prefer something “theoretically sound”, but implicit in that was that we use a theoretically sound compatibility policy, and that (to me, at least) means that the ABI name/version defines the ABI precisely, not allowing additions over time.

Now that you say that, I realise that we don’t avoid the combinatorial explosion anyway. Even if cp3-abi3_15 worked and was encouraged, Python 3.15 still supports both cp315 and abi3_15, so you have to include (cp315, abi3_15, arch) in the tag sets you declare support for.

On reflection, I don’t think we can avoid the explosion without redesigning the tag system so that it isn’t based on the idea that you explicitly enumerate everything you support.

At this point, I’m finding it increasingly hard to reason about the possibilities, so I should probably stop making comments for a while, at least until others have had a chance to give their perspective.

However, I do remain uncomfortable about perpetuating the behaviour of having ABI tags that cannot be interpreted on their own, but require supporting information from the python tag.

Is there not a middle ground here, which is to have a versioned ABI, but allow backward compatibility? So a wheel can legitimately claim to be compatible with both abi2026 and abi2031? So adding new symbols is a compatible version bump, but removing symbols is an incompatible version bump. The tag system can certainly handle that (if only because it has to explicitly enumerate everything in any case…)

  1. It should be cp3 by the way, as using the CPython ABI does tie you to the CPython implementation, where py3 says you work on any Python implementation. ↩︎

1 Like
32

AFAIK, most pure-Python packages use py3, even though they have a strict lower version bound (for example, using some new Python API). They keep the bound in wheel metadata.
Is there an issue with that?

[footnote] It should be cp3 by the way, as using the CPython ABI does tie you to the CPython implementation, where py3 says you work on any Python implementation.

Well, that’s kind of the official story for now. But you can run many C API extensions on PyPy or GraalPy, and new additions to abi3 generally seek make that work better.

That’s close to the idea behind abi3.abi3t :‍)

cp315-abi3 (itself, without .abi3t) is what would come normally after …, cp313-abi3, cp314-abi3. Additions only.
abi3t has removals: it is a subset of the corresponding abi3.
It wouldn’t be hard to allow cp315-abi3 (without the abi3t removals). The main reason PEP 803 doesn’t do it is that it would need more complex configuration (an new flag in addition to Py_LIMITED_API).

33

Yes I was basically trying to make a concrete example as the following if that makes sense.

So do I keep 3.15 tag until 3.36 and then I switch to abi336 to be able to use the new stuff? And if I use the new symbol then I’m breaking backwards compatibility. So not sure how this is rendered Stable.

I guess what I am trying to discover is that since 3.15 will be frozen (as I learned above) can I keep abi315 tag if I do not any new additions in version 3.35 (hypothetically speaking)?

I am not able to understand what is the minimum Stable ABI version just by looking at the filenames (or metadata somewhere). It seems like this will create a very complicated situation for the resolvers to handle next to the version constraints.

I would appreciate if these points are demonstrated on concrete examples rather than high-level descriptions.

1 Like
34

In the absence of an actual problem statement / real-world problem, I’d really prefer to make no change. Not only because it’s extra work and there is potential for backwards compatibility impact, but also because removing the lowest-supported CPython version info from the wheel filename seems strictly worse than the current situation. It will be much harder to catch mistakes, and from looking at wheel filenames on PyPI you would no longer be able to see what the supported range of versions is.

1 Like
35

It’s hard to unpack this thread with the back-and-forth and a bunch of misunderstandings, but this stood out to me:

You can make the exact opposite statement here: the any interpreter tag can now no longer be interpreted on its own, but requires supporting information from the ABI tag.

If you wanted the interpreter tag to be standalone too, it’d be something like cp>=311. Given that that’s not valid, using cp311 seems to be the next-best choice, and certainly an understandable decision when that was made.

2 Likes
36

I did create packaging.tags, although the semantics were compiled from the various tools at the time that generated wheel tags.

packaging.tags goes from most strict to less strict, so cp33 > py33, cp33 > abi3. In general, the priority is the tag order itself: interpreter, ABI, platform (with a twist and another).

The docstrings cover what gets emitted (using sys_tags() as the entry point into the code):

37

Yes. To use a function that’s new in 3.36, you require your users to use at least 3.36. There’s no versioning/tagging scheme that can allow using a function that doesn’t exist.
You do have some options though:

  • You could use a backport of PyStruct_Reset that, for example, calls PyStruct_Remove in a loop. The pythoncapi-compat project provides many such backports. If you go this way, you’d typically build separate wheels for 3.35 and 3.36+ (from the same source).
  • PEP 809 (the competing one) would allow querying whether the function exists at runtime, so your extension could stay ABI-compatible with 3.35 to use PyStruct_Reset when it’s available. It still couldn’t use PyStruct_Reset on 3.35 though – some fallback would still be needed.

The 3.35 ABI is stable, and the 3.36 ABI is stable.
When you update from 3.35 to 3.36, your package is no longer compatible with 3.35. It’s your package that changed, though.

Yes.

Let’s look at some cryptography wheels. The selection below is 32-bit Windows (win32 is the shortest platform tag):

  • cryptography-46.0.3-cp38-abi3-win32.whl: Stable ABI 3.8. Compatible with default builds of CPython 3.8 and above.
  • cryptography-46.0.3-cp311-abi3-win32.whl: Stable ABI 3.11. Compatible with default builds of CPython 3.11 and above. Presumably has some advantage (like a bugfix, feature, performance, or such) over the previous cp38 wheel, but nominally you can always use the cp38 instead of this cp311.
  • cryptography-46.0.3-cp314-cp314t-win32.whl: CPython (non-stable) ABI 3.14. Compatible only with the free-threaded build of CPython 3.14; not with 3.15.

If PEP 803 is accepted I’d expect cryptography to release:

  • cryptography-46.0.4-cp315-abi3.abi3t-win32.whl: Stable ABI 3.15. Compatible with default and free-threaded builds of Python 3.15 and above.

The abi3.abi3t is a compressed tag set, roughly meaning the wheel is good for systems that would be good with any of:

  • cryptography-46.0.4-cp315-abi3-win32.whl: Stable ABI 3.15. Compatible with default builds of CPython 3.15+.
  • cryptography-46.0.4-cp315-abi3t-win32.whl: abi3t 3.15. Compatible with free-threaded builds of CPython 3.15+.

(But since you can’t be compatible with PEP 803’s abi3t without also being compatible with abi3, you shouldn’t see abi3t alone in practice – at least while non-free-threaded CPython is still available.)

For a concrete existing example, a compressed tag set appears in cryptography. Let’s unpack some Linux :

  • cryptography-46.0.3-cp38-abi3-manylinux_2_28_x86_64.whl: Stable ABI 3.8 (cp38-abi3) for most Linux distros (manylinux) that have glibc 2.28+ (2_28) on 64-bit Intel/AMD CPU (x86_64). Here the entire manylinux_2_28_x86_64 is a platform tag.
  • cryptography-46.0.3-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl: Stable ABI 3.11 (cp311-abi3) for most Linux distros (manylinux) that have glibc 2.17+ (2_17 or 2014) on an 64-bit ARM CPU (aarch64). Here the manylinux2014_aarch64 and manylinux_2_17_aarch64 are platform tags. (In this case 2014 and 2_17 are synonyms, likely used to support installers that don’t know about a newer versioning scheme.)

Thanks. That’s the thinking behind PEP 803: make the minimal change.
(Also: I believe that accepting PEP 803 would not prevent PEP 809 from articulating a problem statement and making a big change – possibly in 3.15 to avoid churn!)

Some changes are necessary, but, IMO, minimal:

  • packaging needs to output abi3t for free-threaded builds wherever it currently outputs abi3 for (orherwise equal) non-free-threaded builds.
  • build tools need to generate abi3.abi3t instead of abi3 when the Python tag is cp315 and above (or equivalently: when Py_LIMITED_API is 3.15 (0x030f0000) or above)

(I’m surprised that I can put it in such simple terms! Thanks for the discussion that allowed this!)
On the CPython side things are note complex of course.
And on the standardization side, we need to word things in a way that handles “impossible”[2] tags such as cp315-abi3t or cp14-abi3t.abi3. I think that should follow the above changes:

  • cp315-abi3t: compatible with free-threaded builds only for some reason; CPython 3.15 and above
  • cp14-abi3t.abi3: compatible with builds of CPython 3.14 and above, both GIL and FT

  1. I corrected your quote here, hope this is what you meant ↩︎

  2. “impossible” as in: CPython doesn’t provide API to build these. I don’t quite see a reason to artificially prevent Cython or Maturin from doing such builds, if they know what they’re doing. ↩︎

1 Like
38

Presumably as well as still outputting abi3, because abi3 wheels can be used on free-threaded builds?

OK, I think I now understand the proposal better. I still find the naming of the stable ABIs frustratingly confusing. You refer to things like the “3.35 ABI”, but the actual name of the ABI is abi3 (or abi3t). So we have some people talking about abi3, and others talking about the “3.15 free threading ABI”, and unless you’re closely familiar with the guarantees provided by the ABI spec, they feel like different things. I can just about accept the idea that changing that terminology (and the tags that are based on it) is a bigger change than simply splitting abi3 into abi3 and abi3t, but otherwise leaving things as they are - but I don’t see when we’re ever going to get a better chance to fix the terminology.

Maybe PEP 809 goes too far in the other direction (I haven’t read it yet, so I can’t be sure), but one thing I do think it gets right is in taking the position that because we have to change things in order to support free threading, we should look at the wider picture and take the time to set things up for the longer term, rather than simply fixing the immediate issue and dumping the rest of the problem on “future us”. I’m not sure how much weight my opinion matters on that point, though. It’s the people building wheels who are most affected, and if they are happy, and the people like packaging who have to implement the new rules have no objections, my views are secondary[1].

  1. There’s a process question hidden in that statement, though - is this a packaging PEP or a core PEP? Because if it’s a packaging PEP, my views do matter as default PEP delegate. It’s currently marked as “standards track”, although it has significant packaging implications. Also, I assume PEPs 803 and 809 need to be considered as competing alternatives, which adds yet another dimension, given that “do nothing” isn’t really an option here… ↩︎

1 Like
39

Yes. I can’t think of a rename that would make things clearer (in the medium-term).

abi3 is the Stable ABI. It’s the machine-readable tag, synonymous, like cp is for CPython.
But, yes, “the” Stable ABI actually a “family” of ABIs, distinguished by Python version but also the underlying platform (details like the size of a pointer or register allocation in calling conventions).
If you use unqualified “3.35 ABI”, I’d read that as the non-stable (CPython version-specific) ABI.

The complexity added by PEP 803:

  • abi3 is the Stable ABI for “GIL-ful” CPython builds (just like before, but the distinction is now necessary)
  • abi3t is the Stable ABI for free-threaded CPython builds

I sympathize. I thought the free-threading break would be a chance to make several things better. But, alas, we’re not working with volunteer-scale timelines here.

However, I think there will be more chances.

One might be PEP 809. This thread leaves me more and more convinced that PEP 809 would work nicely as a follow-up. A lot in the current PEPs is common; the things unique in PEP 809 need (IMO) more discussion, and the stuff unique to PEP 803 can work as minimal stop-gap measures.
As a follow-up, PEP 806 would work better in some future CPython version, but if it does get in 3.15, perhaps PEP 803’s abi3t is such a small change that we could undo it before beta. Meanwhile, PEP 803 is, hopefully, implementable now. Getting to test the non-packaging parts in 3.15 alpha 3 would be very helpful[1].

Another possible chance would be integrating HPy-style handles.

If it didn’t matter, we wouln’t be having this discussion.

Heh, it’s a weird PEP.
I see PEP 803 as a collection of minimal changes in several areas[2]; packaging stands out because it uses a different PEP process.
IMO, as packaging delegate, you definitely can veto the abi3t change. I don’t think the SC should accept this without your approval.
Whether that change needs its own PEP that would be tagged as “packaging” – you tell me; I’ll do it[3].

However weird this situation is, wouldn’t you say it’s good that a core change takes packaging into consideration? We should do this more often :‍)

  1. no, I definitely don’t have no steering council breathing on my neck ↩︎

  2. The Reference Implementation section lists the pieces. Interestingly, most pieces are already done: The big technical change got its own PEP, 793; the rest either make sense in isolation, or help with testing and would be easy to undo. ↩︎

  3. For better or worse, I’m literally paid to design a stable ABI for free-threaded builds. ↩︎

1 Like
40

Oh, I forgot:

No. You can’t use cp314-abi3 on a free-threaded build.
packaging needs to generate abi3 or abi3t, but not both. (Either that, or we break the “tags are considered in isolation” rule of thumb and only generate both for cp315 and above. A bad idea, as this topic convinced me.)