Concerns about PEP 620 ("Hide implementation details from the C API")

Core Development
18

Dear all,

thank you for this very enlightening discussion.

I agree with Victor that restricting to the limited API/ABI has tremendous benefits from a distribution point of view. I remain concerned about the performance impact but am happy to be proven wrong – in any case, this should be benchmarked, and the slowdown may not matter for many users given the benefits.

I was just going through all of nanobind to see what we would really need to be able to pull this off, and the following things would be needed that are currently not possible.

  • Ability to allocate types via PyType_FromModuleAndSpec() while specifying a custom metaclass. This would be needed to allocate co-located space for the binding tool and intercepting the construction of derived classes within Python.

  • Ability to specify tp_vectorcall_offset via PyType_FromModuleAndSpec when constructing the nanobind method object. (EDIT: I realized that this is actually possible via Py_tp_members)

  • Ability to perform vector calls from the extension module: PyObject_Vectorcall, PyObject_VectorcallMethod, PY_VECTORCALL_ARGUMENTS_OFFSET, PyVectorcall_NARGS

  • Ability to query type fields like tp_basicsize, tp_itemsize, tp_name, tp_doc, tp_setattro, tp_descr_set, tp_alloc, tp_new, tp_free. I understand that these should be accessible through PyType_GetSlot but then noticed quite a bit of discussion about tp_name not being compatible with that interface and requiring special workarounds. So it’s not clear to me if all are available.

  • In a few rare places, we need the ability to construct method objects and query then via PyMethod_New, PyMethod_Check, PyMethod_Function.

  • A public alternative to _PyType_Lookup.

Do you think that it would be possible? Objections? What would the process be to introduce such changes into a future version of Python?

Thanks,
Wenzel

2 Likes