In the process in updating our simulation software suite (1M+ lines of Python, 100+k lines of C and C++ code in Python extensions) to use Python 3.11, we run into an issue due to
_PyObject_GC_Malloc being removed in Python 3.11.
We have a custom metaclass and base class implemented in C++ that add additional functionality to our Python objects. One of the most important functionality this brings is having fixed “slots” for attributes on the object, something similar to what
__slots__ does in pure Python. Except our slots also have additional functionality. For example, writing into them emits events on the object that can be observed by other objects to get notified when some attribute is changed. Slots on an instance can also have their values “delegated” to some other object in the data model. Another crucial difference is that slots in our classes can be defined even after the class is created, as long as no instances or classes inheriting from it were ever created.
To ensure the best performance, the data for these “slots” is stored at the end of the Python object, after
tp_basicsize bytes. The size of this extra data corresponds to the total number of slots defined on that class and its base classes. Previously we used
_PyObject_GC_Malloc to allocate the necessary memory for our objects:
tp_basicsize + additional memory for our data. As of Python 3.11 this function was removed and there is no non-hackish way for us to allocate the required memory and still have support for GC.
Unfortunately we cannot use VAR objects because our objects are ordinary Python objects, for example, they can have
__weakref__. But VAR objects do not allow this. Calculating
tp_basicsize at the moment of class creation also didn’t work because
tp_basicsize of base classes already takes into account the slots defined on that base, which means that each subclass uses more memory than the base class, even if no additional slots are defined on the subclass.
Would it be acceptable to add a public function with the same functionality
_PyObject_GC_Malloc previously had? Or at least a version of
PyObject_GC_New that also accepts additional memory to be allocated on top of
tp_basicsize? If so, I can submit a pull request with the new function and all the necessary changes if that would be acceptable.