CPython heap type use case creates a dangling pointer

Python Help
1

TLDR; I have a weird use case with Python heap types that creates a dangling pointer.

Here is my use case:

We want to be able to create PyTypeObjects at runtime. This is achieved by creating heap types from PyType_Spec. We can specify the properties of the heap type by setting the fields in the PyType_Spec including name and “slots” including tp_methods, tp_members, etc (Type Objects — Python 3.12.5 documentation). In this use case, the tp_methods slot does not have a static lifetime, so is either stack-allocated or it could be heap-allocated (if we can get the lifecycle right).

This is my understanding of how the creation of heap types works in CPython:

When CPython creates a PyTypeObject from a PyType_Spec (e.g. via PyType_FromModuleAndSpec), it copies slots of the spec to the new heap type object that is created. The slots are “shallow” copied in the sense that the value that is copied is just the value of .pfunc. If .pfunc is a pointer, the value of the pointer is copied but not its memory contents. It seems like tp_members is an exception to this, where the contents of tp_members (which is a PyMemberDef array) are memcpyed into some location in the heap type object. However, there is still the same issue because name and doc fields of the PyMemberDef are const char * pointers that do not have their contents copied. This means that overall, the fields of PyType_Spec must be statically defined or outlive the lifetime of e.g. PyType_FromModuleAndSpec.

Here is our problem:

The above behavior is giving us trouble when working with the tp_methods slot (and potentially other slots, but methods is our current issue). Again, rather than copying the contents of the PyMethodDef array that is given in the tp_methods slot, the PyTypeObject keeps a pointer to the PyMethodDef array. This means that this method array must outlive the call to e.g. PyType_FromModuleAndSpec since it is referenced whenever a call to one of the methods is made.

In most existing code using heap types, this is ok: if we create a static PyMethodDef array globally, the pointer to the method array will always be valid until the program exits. However, as described above in the use case, I want to be able to dynamically create the PyType_Spec, including the methods array.

We observed that stack allocation does not work. Whenever the PyType_Spec goes out of scope, we lose the pointer to the method array.

We then looked at heap allocation but ran into a lifecycle problem of making sure that the method array is freed when the resulting PyTypeObject is cleaned up.

To work around this issue with heap allocation, we wrapped the method array into a PyObject and set this PyObject as a member of the PyTypeObject via the tp_members slot. The PyObject will malloc the method array during its creation and free the array during its deallocation. However, when doing so, the PyObject will get decref’ed during deallocation of the PyTypeObject, which could trigger deallocation of the PyObject and hence, free the PyMethodDef array. Then, in the rest of the PyTypeObject cleanup code, we have a dangling pointer to the PyMethodDef array.

To summarize, my workflow is as follows:

  1. Create a PyObject that allocates memory for the PyMethodDef array
  2. Create a PyTypeObject by defining a PyType_Spec, which sets the PyObject from step 1 as a member of the PyTypeObject. Furthermore, the pointer to the method array, which can be accessed via the PyObject, is used as the .pfunc of the tp_methods slot. This means that the PyTypeObject has a reference to the method array PyObject, but also has a pointer to the method array itself.
  3. During deallocation of the PyTypeObject, its reference to the method-array-holding PyObject is removed (via Py_DECREF or something similar), bringing the ref count to 0, which means the PyObject can be deallocated, hence freeing the method array. However, in the remainder of the PyTypeObject’s deallocation, it still has a pointer to the method array, which is now a dangling pointer.

Are there any workarounds for this? If there’s no way to make this work, could we envision an API that allows the creation of PyTypeObjects without requiring a caller to retain any data once the object has been constructed?

2

It’s not particularly straightforward, but you can create a weakref that points to your type with a callable that does the deallocation. The complications are that the function passed to the weakref needs to be a Python callable, and you do also need to clean up the weakref itself in that function.