I was thinking more along the lines of a DateTime object (or ndarray) would return the underlying pointer - no capsules involved, just a new API to provide a dedicated native API in a stable/versioned manner for any type that implements the slot. (The capsule is just for smuggling the pointer. If you can get to the pointer directly, you don’t need to smuggle anything.)
Ultimately, the clear migration path is “deprecate the old and add the new”.
The current use of capsules doesn’t directly migrate to the module slot proposal either - for a clean migration, you need to preserve the capsule being assigned to an attribute of a module and accessed through that interface (potentially after the object has been reassigned/passed around by Python code). My simpler capsule proposal would preserve this, but I don’t think any of the other proposals offer a better option than allowing a module to use both the old approach and the new approach simultaneously.
I mentioned this in the “backwards compatibility” section — capsules are supposed to be stored on attributes until the ABI provider drops support for <3.14, then the module slot can be fully depended on. At least with that approach, the migration feels, more or less, seamless. With a totally new API that isn’t backportable, I could see libraries being hesitant to make any use of it before 3.13 is EOL. With a capsule change, it’s more just some macro magic — ABIs could support it on newer versions immediately.
Wait, no, I’ve wrapped my head around the idea: the new slot would be added to PyTypeObject (so every type object would get larger, but non-type instances wouldn’t need to get any bigger). PyObject_GetNativeInterface would check if the object’s type implements the slot, raising an error if not and calling the slot method if so, passing it the instance. The slot method would be responsible for returning a void* to the user, which may point to something statically allocated, or may point to some per-instance data, at that type’s discretion. And presumably that pointer would be invalidated if the object were to be destroyed.
That sounds about right. I’m not too sure if it’s any better than a capsule change, though. At the very least, a new API for this should come with a way to set the pointer on the type from a capsule object.
For example:
PyCapsule_SetNativeInterface(capsule, module);
PyObject_GetNativeInterface(module, "foo.foo_abi", 1) // Equivalent to PyCapsule_GetPointer(capsule, "foo.foo_abi")
And since capsules are technically vtables (with the exception of the type object pointers that I consider to be part of said vtable) I think a new vtable ABI and soft deprecate the capsules ABI could be done as well as explained in the commends on top of the POC header file posted above.
/*
* vtable_abi.h - Python 3 VTable based module abi include file.
*/
/*
* This is a POC (Proof of Concept ABI) For Python 3.14+ that introduces
* an alternative to the Python capsule ABI by avoiding the
* limitations of it. With an Entirely new ABI like this the capsule pitfalls
* can be "fixed" by simply soft deprecating all the capsule ABI's and
* suggesting modules migrate to a VTable based ABI that extension developers
* could Request from a new Import function:
* - PyImport_ImportVTable(const char *name, int32_t major_version)
*
* And a call to a new GetPointer function:
* - PyVTable_GetPointer(PyVTableEntry *entry)
*
* Which allows one to import a specific version of a module's ABI that is returned
* as a "void *" pointer.
*/
But this ABI for 3.14 could change to remove PyVTable_New then to a new slot inside of the module definition that can hold an array of PyVTableABIDef structures instead which I feel would be more memory safe and support sub-interpreters as well (And to rename PyVTableEntry → PyVTableABIDef). And then have the new module def slot be .m_vtables.
Why a new module slot?
Because I do not feel comfortable with something like this as part of the implementation:
/*
* vtable_abi.c - Python 3 VTable based module abi implementation file.
*/
// assume python main source code branch is used as the include path here.
#include <Python.h>
#define VT_ABI_DLL
#include "vtable_abi.h"
typedef struct _PyVTableModule PyVTableModule;
/*
* Internal structures.
*/
struct _PyVTableEntry {
const char *m_name;
PyObject *m_module;
int32_t m_major_version;
Py_ssize_t vtable_size;
void *m_vtable;
};
struct _PyVTableModule {
Py_ssize_t length;
PyVTableEntry *m_entries[];
};
PyVTableModule *_module;
// snip all the code here is simple.
// this function right here makes me feel uncomfortable in terms of "memory usage" that a special module slot for it would resolve entirely.
PyObject *PyVTable_New(void *vtable, const char *name, PyObject *mod, int32_t major_version, Py_ssize_t size) {
if (_module == NULL) {
_module = PyMem_Malloc(sizeof(PyVTableModule));
}
Py_XINCREF(mod);
PyVTableEntry *entry = PyMem_Malloc(sizeof(PyVTableEntry));
entry->m_vtable = vtable;
entry->m_name = name;
entry->m_module = mod;
entry->m_major_version = major_version;
entry->vtable_size = size;
_module->m_entries[_module->length] = entry;
_module->length++;
Py_RETURN_NONE;
}
// with an ".m_vtables" slot this could be simplified to searching up each item in the array of "PyVTableABIDef"'s to then return the version requested (if found) in a slightly more optimized way (it would optimize out the "strcmp" call to compare the 2 strings).
PyVTableEntry *PyImport_ImportVTable(const char *name, int32_t major_version) {
// so that way "PyInit_[name]" would run and call "PyVTable_New" which then
// would add the module that would then be used below this to get the vtable
// abi pointer.
PyObject *mod = PyImport_ImportModule(name);
if (!mod) {
// most likely an ImportError or ModuleNotFoundError.
return NULL;
}
for (Py_ssize_t i = 0; i < _module->length; i++) {
if (strcmp(name, _module->m_entries[i]->m_name) == 0 && _module->m_entries[i]->m_major_version == major_version) {
return _module->m_entries[i];
break;
}
}
// in cases where import succeeds but the module does not add a vtable abi at all.
PyErr_SetString(PyExc_ModuleNotFoundError, "Module not found in the internal VTable list.");
return NULL;
}
No no no, a capsule is a smuggled void *, nothing more, nothing less. The only semantics around it is a name (to confirm you have the right one) and an optional callback on destruction.
Every semantic being attributed to “capsules” in this thread is brand new. They are proposals, not premises/assumptions, and so you can’t just refer to them, you need to define them and try to seek agreement.
But at the same time, you can’t stop people putting a random pointer-sized value into a capsule and passing it around, because that’s what they are for. So any proposal incompatible with that is not going to make it.
Restarting:
There appears to be a desire to have modules/types/objects provide a native table of native function pointers so that a native caller is able to bypass the Python interfaces in order to inspect or modify the value. Is this a fair description of the desired behaviour? Can we start from here and look at building a set of APIs that provide it?
Because I agree, this is an awesome feature to have. I’ve proposed it myself (as a type slot, but it extends to custom module slots almost transparently). But there’s way too much distraction going on here with regards to retrofitting the capsule API to provide it. That is an option, if we happen to determine that the best way for modules to provide a native vtable is through a single, named void*, but we could do so much better by ignoring the constraints at this stage and designing something that properly fits our desires first.
To be fair, capsules are almost exclusively used to expose function tables. I guess they could be used for something else, but even the docs demonstrates them with this use case. I’ve never seen a capsule used for anything else (but if you could link one or provide an example, that would be helpful.)
And that’s exactly what capsules are used for – storing function tables to let a native caller use them. So, if we go the route of designing a new API, we should at least provide some sort of utility to help existing libraries migrate.
With that being said, a new API should cover at least these cases:
Versioning (both for major and minor changes)
Using outside of the native caller (e.g. through ctypes)
Storing things other than function pointers, such as Python objects or other ABI-specific data.
To start on the design, let’s decide if we want to make this specific to modules (i.e. through a module slot), or supporting per-object native interfaces through a type slot. I’m thinking we could do both – a table of function pointers defined through a module slot (something like Py_mod_abi), and a void * returned by per-object interfaces (through a Py_tp_native or something other).
I can’t link private code, unfortunately, but I’ve used them to link COM objects to Python objects, so the strong COM reference is retained for as long as the Python object keeps a reference to the capsule. I’ve also used them for passing pointers to internal data structures into and then back out of Python object models into my own module.
Except they’re migrating from something that can be used for their scenario, but isn’t designed for it. Which means we can’t predict how they all look, and so we can’t automatically migrate anything.
Once we know how the new API looks, we can document “how this may map to what you might currently be doing”, but ultimately it’s going to be a fresh start for anyone who is changing from one to the other. As I said earlier, the advantage of this is that they can run the new and the old in parallel trivially for as long as they like, whereas something that migrates at a Python version boundary can be far more complicated to handle.
I’ll leave details of the design to others for now. I’m far too aware of being the loudest voice in here right now and want others to participate (but I also want it to be focused in beneficial and practical directions, which is why I’ve been speaking up too much).
That feels like a stretch from my NumPy angle. A large C-module will have one function table, but chances are it has other capsules, NumPy has:
An __array_struct__ capsule (not that we use it much).
A capsule to register a new allocator.
supports DLPack capsule
A little capsule to expose some internal API (experimentally)
Internally in some (hackish) code that wants to store user provided C function pointers into a dict. (effetively deprecated, but there).
As a way to provide a user-provided function call back for customizing our dispatching machinery (because creating an object for it seemed not worthwhile).
For a C struct that needs to be stored inside contextvar (IIRC).
Some of those could or maybe should be objects but capsules are a light-weight thing. Others should not be objects because it requires a Python module that owns the type (rather than just a string identifier), and you don’t want to have to look up types/compile time dependencies like that.
Maybe this is a reason for different opinions. I think capsules are often very light-weight (sometimes even private). Bloating them up too much might not be what we want.
Got it, that’s good enough for a rationale on why to make a new API over modifying capsules – although, I’m personally still not 100% convinced that’s the best option. It seems sufficient to modify capsules since this proposal is backwards compatible, meaning existing code that does not use a function table won’t have to modify anything anyway, and just keep their major version at zero. Technically speaking, any void * could (and maybe should) have a version number, in case the provider changes it in the future, regardless of what it holds.
I’m kind of playing devil’s advocate here though – a new API is probably the best way to go (I’m partially just nitpicking so I can reference the counterarguments in a PEP rationale )
I think this is a really interesting idea that I can see a lot of use for, but I’m not sure that it’s quite the same problem as this thread is talking about. This thread’s idea seems to be that we should make it easier and safer for a Python module to expose a native API or ABI. Taking the capsule example in the docs, there’s a header doing #define PySpam_API_pointers 1 but the capsule doesn’t export the number of pointers in an introspectable way, which means something can be compiled to expect 2 pointers but find only 1 at runtime and read past the end of the exported array.
The case where an object (ndarray, datetime, etc) wants to expose a native interface to its underlying representation and the case where a module wants to expose a native API don’t quite seem the same to me. If you squint just right, I guess you can say that any native API exposed by a module is its “native interface”, but “interface” is doing a lot of heavy lifting there. At the very least the lifetimes can be different - the interface to an ndarray’s or datetime’s implementation would obviously only be valid for as long as that object is alive, but the docs’ spam module’s API is statically allocated (and even safe to use in different interpreters, even concurrently).
A module that wants to export a function table could add it to the module state and export a static inline function that takes a module object and:
verifies PyModule_GetDef(m)->m_name, to guard against other modules (or non-modules) in sys.modules – as with capsules
checks a version in any way it wants to
returns ((my_state*)PyModule_GetState(m))->api
That should work now. But someone should still write that book about all the gotchas in function tables, static inline functions, versioning and managing lifetimes.
That’s what it was originally, but it seems the best way to solve it is by adding a new API instead of capsule modifications. (We already have this thread, no need to make a new one on a similar topic.)
This is an interesting point, and one that’s probably relevant here. A “native interface” (which could, technically, be any void * here, not just a function table) should denote whether it’s compatible with multiple interpreters and concurrency (as should the function that exports them). I could see an extension module having an ABI that’s completely separate from Python (for example, just a wrapper around some system functions), but then object-specific interfaces would likely need some way to say whether they are thread-safe.
So, maybe something like this:
static PyABI *
native_func(PyObject *self, int32_t major_version)
{
// Maybe this should get a minor version number as well?
FooObjectAPI *api = malloc(/* ... */);
return PyABI_New(
api, // ABI Pointer (doesn't have to be a function table!)
1, // Major version
Py_ABI_SUPPORTS_GIL | Py_ABI_SUPPORTS_SUBINTERPRETERS // Flags
);
}
PyTypeObject MyType = {
/* ... */
.tp_native = native_func
}
static PyObject *
some_func(PyObject *self, PyObject *foo_object) // METH_O
{
FooObjectAPI *api = PyObject_GetNativeInterface(foo_object, 1); // V1
return api->foo(42);
}
This new PyABI (I reused the name from the previous draft ) should probably hold at least:
My one concern right now is that we’re kind of reinventing capsules by just using a void *. In theory, it might be better to only support function tables, and leave general void * smuggling to capsules (which they can technically version using some of the backwards compatibility hacks from the draft).
)
) should probably hold at least: