Except if you build a binary wheel based on 3.14.0a4 and 3.14.0-rc.1 ships with the problematic change. In this case CPython’s version checking at the moment is insignificant here.
Couldn’t you use Py_VERSION_HEX?
Not if one shipped said wheel before the change happened and did not expect for a function to be removed from a built-in capsule pointer.
Well, that’s why you don’t depend on stability in alpha releases 
– this is off the original topic now, though.
1 Like
That’s the version the extension is compiled against, not the version it’s loaded on. If one is using the stable ABI, then this is the wrong information.
1 Like
Nothing, but that isn’t the point.
When you export a capsule, versioning is something you want to have from the start, and we know from experience – both in stdlib and outside – that it’s not something people think about when first exposing a C API.
It is possible to version capsules now, but making it effortless to start will cause people to actually version them. And having a common scheme for metadata should make introspection more fun.
The “out-of-band” size field allows new additions to current capsules. Currently, to do that you either need to a new capsule (with a new name), or export the size (or version) as an additional module attribute (and make users do getattr & PyLong_AsSsize_t… the new name sounds easier).
Treating the “major” version as a number, rather than just having a _vN suffix in the capsule name, is there because of my dislike of “stringly typed” API. The suffix would work just as well, if you don’t share that dislike.
Hopefully major version updates are rare enough that you’ll only need the old method once.
But yes, this is designing for the next decade.
That hurts, don’t do that.
Consider this scheme to remove a field from a capsule:
- version 0 raises a deprecation warning when imported
- version 1 has the same memory layout, but renames the problematic field to
_reservedand always sets it toNULL
Now users can switch to the new version within a proper deprecation period.
(Yes, as said above, you can do the same thing by appending _v1 to the attribute name.)
2 Likes
I agree. My preference is to not break your ABI often enough for this to matter 
That puts the burden on the publisher, yes, but that’s the burden we adopt by publishing code.
In any case, if I provided an ABI that required this, I’d be hiding the tree of calls behind statically linked helper functions that the consumer would know about and would work with whatever build of the module is actually present.
Fields don’t have names in this model, though. If you get v1 and “cast” it to a v0 table, the original field is still there, it’s just set to NULL. The information you need to “don’t touch that” is independent - if you have that information available to you, then you won’t touch the field and the underlying table is the same in both versions, and hence no version is needed.
If the field is that important, then you also need a new field/function to replace it, which means you’re making a layout-incompatible change to the table.
I suspect if you trained people how to design a long-term stable ABI, they’d happily redesign whatever they’re currently providing and put it under a new name. Letting people manage their own versioning and staying out of their way is almost always going to be better for them.
I’m often 100% behind guiding people into doing the right thing by designing the APIs they’ll use well. In this case, I think we’re overcomplicating things more than guiding. And even a soft deprecation on the old API is sending a complicated mixed message to users. I’ll propose an alternative in my next post.
As I say, I’ve worked with both kinds of API before (COM uses “stringly typed”[1] APIs; .NET uses version numbers until people give up on them and start putting "2"s at the end of names). All the version number does is make separate the name into two parts, which is fine if your ABI can transparently adapt to any version (e.g. Python), and annoying if it can’t (e.g. virtually everything else).
This is such a great name!
↩︎
Let’s say we simplify the proposal down to this (int is up for bikeshedding; the in/out-ness is my proposal):
PyObject *PyCapsule_NewWithVersion(
void *pointer,
const char *name,
PyCapsule_Destructor destructor,
/*[in]*/ int version);
void *PyCapsule_GetPointerWithVersion(
PyObject *capsule,
const char *name,
/*[out]*/ int *version);
int PyCapsule_SetPointerWithVersion(
PyObject *capsule,
void *pointer,
/*[in]*/ int version);
void *PyCapsule_ImportWithVersion(
const char *name,
int no_block,
/*[out]*/ int *version);
This is unambiguously going to lead developers towards adding a version number as side-channel information with their capsule. It gives us a place to document why including a version number is a good idea, and even to point out how including it in the structure itself is an alternative. These are all goals of and are achieved by the original proposal, so nothing new here in mine.
The difference from the original proposal is that the additional information is not provided with the “get” request (for validation), but retrieved with the response (for interpretation).
So when you import and get the pointer in a capsule, you’ll also get its version. This is fundamentally identical to putting the version number in a field pointed to by the pointer, apart from it’s available even to capsules that don’t currently have one. A regular PyCapsule_New just sets the version to zero, and a regular GetPointer just doesn’t retrieve it.[1]
Consumers are still going to have to select how to interpret the data retrieved, but this way they can choose whether they are able to support whatever version is present or if they have to raise an error. And we don’t have to define any semantics for the version number (we can recommend semantics in the PEP, but there’s nothing for us to enforce).
(The ctypes parts of the original proposal are best developed independently at first. I’d narrow the whole thing down to the required core changes so it has the best chance of making progress.)
(Side note - we really ought to make any new GetPointer* function return an error status rather than the value. So int PyCapsule_GetPointerWithVersion(PyObject *capsule, const char *name, void **pointer, int *version). I’ve bitten myself hard in the past with a capsule that was supposed to contain NULL )
If you really want to be extra paternalistic, you could make
GetPointerfail if the version is non-zero, but I think that hurts more than it helps. Consumers can’t control what the provider does, so if the provider adds a version just “because it’s the right thing to do”, it ought not break consumers who haven’t updated their code yet. ↩︎
Is this “version number” supposed to refer to the major version, or the minor version? If it’s the major version – then you forgot about the minor version! My proposal used a size for that (which I still think is better than a hardcoded minor version number, see why in the rationale of my draft).
If it’s supposed to be a minor version number, then why not just use a size? Almost all capsules have to (or at least, it’s a very good idea to) be append-only anyway – a size mixed with offsetof is probably better.
For example, I think it’s a bit more clear to do:
if (PyCapsule_GetSize(capsule) > offsetof(FooABI, my_field))
{
foo_abi->my_field(/* ... */);
}
else
{
foo_abi->older_field(/* ... */);
}
Rather than:
if (version == 42) // ???
{
foo_abi->my_field(/* ... */);
}
else
{
foo_abi->older_field(/* ... */);
}
Having used a lot of APIs that require you to pass the size, I think it’s a terrible idea 
It’s fine for validation (on the API implementation side), but don’t rely on the size for anything to do with versioning.
And I deliberately and explicitly left version semantics unspecified. If the user needs to know the version has changed, change the number. If not, don’t change the number. We don’t have to require anything more than that, though we can suggest some ways to design your use of the version.
I think you’re delving just a bit too far into designing an ABI compatibility scheme, rather than providing the tools to let developers design their own. That’s fine, but you need to focus on the ABI scheme as a whole and not the capsule changes - it’s going to be a much bigger proposal. And I hate to dishearten you, but every past attempt at it (and there are thousands) has proven to be insufficient.
They do; you need a header (or for FFI, some description of the API).
The rename means your code won’t compile if you’re touching that field, so it’s safe to bump the version.
Only if they do manage their own versioning.
But for the phase where they’re not yet thinking about this, getting them started with something that’ll gets them far enough is better for them.
Providing values for interpretation sounds good (which is why the draft has PyCapsule_GetMajorVersion and PyCapsule_GetSize). Getting all the info at once sounds good too.
But, I still think also providing validation is better – it avoids common boilerplate that people would be tempted to omit in the first version. And it’s easy enough to skip once you don’t need it.
And we also need to use those suggestions for stdlib capsules.
I don’t disagree, though I also don’t see why the stdlib ones are going to be any different from the Python version number, which is easily accessible. Presumably they follow the same stability rules as the rest of the runtime?
Yes, but this isn’t part of the capsule. It’s part of the headers that the consumer is compiling with. You would have an absolutely identical ABI provided by the publisher, and it would still be usable by any consumer.
The best you get from this scheme is completely breaking anyone who only compiled against v0, regardless of whether they use the changed field or not. They couldn’t predict the future and know how to handle a v1, so when they see a different version, they have to just give up. As the publisher, you might as well change the name of the capsule and you’ll get the same result - the only people who know the new name are those who compiled against v1 and therefore will use it (and the advantage is you can totally remove the field in this case).
A better way overall is to put functions in the ABI rather than fields, and give them all status results so that you can add deprecation warnings or failures later without changing the ABI. (Boy, this sounds awfully like a lot of other discussions we keep having ) The best you can do for data fields or function prototype changes is to replace the entire capsule with one that only those who knew about it at compile time can use. And the advantage of names here is you can provide the old ones as well.
I really think this is a problem that’s better solved with training (and by that I literally mean someone can and should write a book about it and sell it for money, and then get corporate training gigs and spend the money on private jets and such - it’s good money!) than by trying to lead people into it with an API. We can’t possibly provide enough API to get them to a great place, at best we’ll push the problems down a level, which only makes them harder to solve. (We’re also seeing this over in packaging, largely due to similar ABI issues.)
In terms of this proposal, they shouldn’t have to think about a v1 if they’re using v0 – they would just ask for the v0 version, and CPython would ensure that’s what they actually got.
1 Like
I don’t see the part in the original proposal where the provider creates a single capsule with multiple pointers in it, one for each version. Could you point that out for me?
It’s not part of the PyCapsule object itself – it’s done by the Py_mod_capsule slot.
1 Like
I see. Again, I think this proposal is going just deeper enough than the capsule API that it’s better off being presented as its own new thing rather than a change to an existing object type.
“Dynamic ABI versioning for native interfaces” would be a great title, and building it around the idea of a new slot that returns a function table and some helpers on the consumer’s side to invoke functions from that table would all fit nicely and flow well. Perhaps capsules end up with a role to play in that, but more likely they’d enter into the motivation as “here’s how people hack around the lack of this feature today”.
So I’d drop the capsule changes entirely, invent new APIs for getting a module’s versioned native ABI, and keep the focus around the module slot. (Or drop the module slot and minimise the changes to the capsule API, but as previously discussed I don’t think that’s a worthwhile change on its own, so probably go big rather than small ) Your main point to prove will be that it must come from a slot on a native module and not through some other mechanism, because otherwise it’s too easy to say “build it on PyPI first”, but I can definitely see a framing for this that makes it a powerful innovation.
You might also want to consider a per-object slot. Have a look at Python array API standard — Python array API standard 2023.12 documentation for an existing spec that is a slightly different angle from what’s being proposed here, but covers a lot of the same area and concerns. I would fully expect that they’d love to make use of a PyObject_GetNativeInterface(o, name, version) API (you might have to dig into the numpy implementation to see exactly how this lines up though, I don’t think the spec covers this kind of detail).
(And again, I’ll point out that I’m very supportive of this general pattern. I’m only pushing back on the capsule changes because I don’t think they’ll achieve it!)
That’s an interesting idea (and one that will probably end up being quite a bit more work than modifying capsules) – but it doesn’t address the existing problems with capsules. What do you think, @encukou?
OK, I read the proposal now fully, and I have to admit I am starting to share some of the doubts voided. That doesn’t mean I don’t like the ideas/additions at all, though!
However, around the edges, there are some more or less concrete comments:
- Maybe in part due to non-familiarity: I do not like prescribing the
sizeas the “minor version”. In API tables. The minor version will typically be the same, but I don’t really think slots != API. E.g. support for new flags or entries in other structs is reasonable. Maybe the size should just be a suggestion? - The non-module use-case is (IMO) also relatively important and it is too bad that there would be a lot of API to make one easier (which also means that they look different).
- Maybe it would be good to maybe just link to how the usage pattern looks like in the end (even just as a user of the PyDatetime API). I do agree that would clarify things a bit more.
This PEP solves multiple things:
- It solves the problem of evolving capsules (modules or not) in a less awkward way by adding a major version. (I like this, having had to fight with that.)
- It tries to solve the problem of how to manage subinterpreter ready C-API, which will require the module (or some form of context if we think of HPy, but let’s stick with the module).
- It tries to simplify exporting multiple API versions and doing version checks for them.
As much as I like the first point, I am wondering if one should actually ignore it and circle back/reconsider the whole proposal. Maybe that is a no-go, but what if we didn’t even use capsules at all for fetching the API table?!
#define PyArray_MODULE_SLOT numpy_api
Py_c_extension_spec = {
PyObject *module;
PyObject *api_table;
int abi_version;
int api_version; // just because NumPy uses it can also be minor_version of course
}
MyModuleState {
Py_c_extension_spec numpy_api;
}
Init_Func() {
MyModul
PyArray_ImportToModule(self);
}
1 Like
For now – here’s what I’m thinking:
It seems that the use of size is generally disliked. Personally, I think it’s quite convenient, but I understand the concerns. It shouldn’t be much of a change to this proposal to change size to a minor version number. However, going with Steve’s idea, maybe it’s better to leave checking the minor version up to the user (that would allow using it as a size as well).
For example, users would now just:
void whatever() {
// Ignoring error checks for simplicity
PyObject *capsule = PyCapsule_ImportVersioned("foo.foo_abi", 1); // V1
FooABI *foo_abi = PyCapsule_GetPointer(capsule, "foo.foo_abi");
if (PyCapsule_GetMinorVersion(capsule) > offsetof(FooABI, some_field)) {
// We're still using the minor version as size here, but
// now, it's up to the ABI provider to make that decision
foo_abi->some_field(42);
}
}
(This makes ctypes.PyABI a bad idea, though, since the size is no longer standard)
I think that trying to design a new API specifically for ABIs is an interesting idea – but I’m not too sure what benefit it would have over just modifying capsules. Personally, I would rather just modify capsules, since trying to move existing ABIs to some new way to do it sounds like a bit of a hassle – by modifying capsules themselves you at least don’t have to worry much about migration.
If modifying capsules to support versions isn’t feasible, or too unlikely to work in practice, then I would be happy to reimagine the proposal for new APIs – as long as there’s a clear migration path for existing ABIs. (I would like to hear Petr’s thoughts on redesigning this, though – I’m not totally convinced it’s necessary.)
If we do go this route, PyObject_GetNativeInterface sounds promising – I imagine capsules would just return their underlying pointer?