C API idea: Making type slots and other function pointers read-only

Core Development C API
1

Hello!
Here’s a C-API idea that’s not quite fleshed out, but it’s becoming more relevant so I’d like to throw it out sooner. First I’ll write about the end goal, then the why, then a possible way to get there.

When you set a type slot like tp_init/nb_add, using whatever mechanism (static PyTypeObject, PyType_Slot, or some new API like Mark’s proposal), we shouldn’t allow getting the function pointer back. Instead, we should add wrapper API to call the underlying function.
This should extend to all cases where a function pointer is passed to CPython.

Why? This allows evolving the function signature & semantics/guarantees. If objects are no longer guaranteed to have a PyObject * tp_add(PyObject *, PyObject *), but instead you need to call PyNumber_Add, Python can, for example:

  • Pass an extra argument, like thread state or defining class, and allow users to set a PyObject * tp_add_ex(PyThreadState*, PyTypeObject*, PyObject *, PyObject *) or something along those lines.
  • Lock a GIL for legacy extensions even in a Python version that no longer uses the GIL normally. (Lots of handwaving here.)

How to get there?

  • Add API to make it possible for users to do everything without reading function pointers.
  • Deprecate PyType_GetSlot, and discourage reading slot values (unfortunately I don’t think you can warn on reads only). These should be discouraged in new code, with no plans to remove them.
  • When needed, add new API to set slots with updated function signatures. The old slots would be replaced by fallback functions that are slower or make extra assumptions, or in the worst case fail (making some new extensions incompatible with some old ones).

Ideally somewhere along the way we can allocate sub-slot tables on demand for heap types, so it’s cheaper to support multiple slot calling conventions. (Nick Coghlan’s idea from 2015 is still in my head…)

Any thoughts?

4 Likes
2

This requires adding new accessor functions for every slot to facility subclassing, calling PyNumber_Add is not the same calling tp_add when a subclass implements tp_add and needs to call the superclass implementation. There’d have to be a PyObject* Py_invoke_tp_add(PyTypeObject*, PyObject*, PyObject) (or some other name).

1 Like
3

Yes. A lot of functions, all alike. I’m not thrilled about that, but I’m convinced that replacing memory accesses by function calls is the right direction in general.

We could have Py_invoke_binop(int slot, PyTypeObject*, PyObject*, PyObject*), called with slot=Py_nb_add, to cut down the number of extra functions.

Ironically, getting the superclass from within tp_add(PyObject *, PyObject *) is not trivial (unless it’s a static class, but those have their own issues) – which is exactly why the slots could use new signatures.

1 Like
4

There’s tp_base if you don’t mind only supporting single-inheritance. That’s good enough for most extensions I write.

BTW. I appreciate the need for having a way to change the signature of slot functions without adding ever more slots to PyTypeObject (e.g. tp_getattr to tp_getattro).

5

tp_base is there, but the problem is getting the type whose tp_base you want.
Py_TYPE(self) may give you a subclass. So you’re left walking the MRO (or tp_base chain), even though in most cases nb_add’s caller knows where it got the nb_add from.