Changes in Python 3.13 object memory model have made it increasingly difficult to use memory alloc slot. As currently implemented it will be almost impossible to implement the JPype object model. JPype is a language bridge between Java and Python. As part of its implementation it must derive a base class which will be used as a mixin for a number of Python types such as PyLong, PyFloat, and PyException. As these types all have different memory layouts this would normally be prohibited.
To deal with this requirement, JPype was copying the formulas found in Python for similar types. Before Python 3.11 it would simply had a copy of the Python allocator which added 8 bytes to the end of all structures requiring Python types. As the 6 or so mixin types the JPype creates are all final and closed to expanding due to a meta class, there was no way to conflict with the extra memory. Starting in Python 3.11 those functions and structures became harder to access with the allocator disappearing entirely in 3.12. Jpype could still function by manually creating a phony type with the required allocation then type shifting to the required layout after allocation.
However, in Python 3.13, Python has make the memory model entirely inaccessable basically rendering the alloc and finalize slots useless to third party. Nominally, a new API in PEP 697 was supposed to allow third party class specific memory to be allocated, but unfortunately that doesn’t work for the JPype use case because the memory it allocates is within the basesize of the object and thus creates exactly the same conflicts. I proposed an alternative to PEP 697 and demonstrated in a patch in which the Python allocator can place class managed space in front of the allocator, but due to other time commitments I have never had time to write the required PEP for this solution to be considered.
Instead the Python memory model 3.13 is now entirely closed. The only way to create GC objects is to call the PyType_GenericNew or one of the private methods. And this call has no provision for allocating any space other than that of the standard layout. In addition both the memory before the object and after is now filled with private objects. The front contains a managed pointers for the dict and weak pointer (which was once added to basicsize on class creation), the area after the class now contains an unknown sized object for inline dictionaries if the size of the object has a certain base size and an item size of 0. The GC_Link method is inaccessible and the type system caches the incoming type meaning that one can’t even polymorph the object like 3.12 to create an object with a non-standard layout. In addition the job of allocator has expanded beyond simply linking the GC, initializing the object type pointer, and set up the reference count as it now also sets up the inline dictionary. In my opinion an allocator should do as little as possible (ie allocate memory and blank it), but this version is adding even more to that already complex path. As far as I am aware most of these changes were internal to the Python team. To try to document the changes I constructed a schema diagram of the current Python object model.
As the sizes of all these special entries are not accessible and the mechanisms to linking the memory allocation from the GC is hidden, effectively the allocator/finalizer slots are Python internal privates and thus no one can implement the same level of concrete classes that Python itself does. One can argue that 3rd party developers shouldn’t be calling private Python functions, though as there are zero examples of how to properly implement GC managed objects and no published API a 3rd party developer has no choice when trying to overcome a basic limitation of Python such as no true multiclass.
I would recommend that Python 3.13 reform its memory model before being released. One or both of the following solutions would alleviate the situation.
-
There should be some sort of public methods such that a 3rd party library is not locked entirely out of creating GC managed objects. It should stop automatically adding private unknown sized objects after the standard model which has always been accessible to 3rd party libraries like JPype. Doing so makes exceptionally difficult to implement multiclass or language bindings. In addition, there should be enough public accessible methods to allocate a GC managed space with pre and size fields. As knowing how big the standard object needs to be is required there needs to be a public method to get the required pre and size for the object allowing future Python versions to add or change the model as they see fit. (Example API: PyType_Alloc(type, pre, post); PyType_Presize(type), PyType_Size(type, nitems), PyType_Init(obj) all of which leave the GC model behind the scenes) Python internals should use these same methods so that everyone implementing concrete classes is on the same playing field and the API object layout stops shifting in unpredictable ways.
-
The PEP 697 should be expanded beyond its limited use case to move its allocation from within basic size to managed space before the object. There is already a proof of concept that demonstrates the concept.
Both solutions have use cases. The first is more flexible especially if the memory requirements for the object do not fit the simple single item size Python one. JPype which needs immediate O(1) access to its extra memory and has closed classes is more like the first case. The second is more tailored to objects that require fixed slot space and want Python to manage it for them, and is a stepping stone to the grand unified object model as it allows for true multiple inheritance without conflicts. If neither solution is implemented then JPype will need to use the most horrible hacks imaginable of monkey patching the basic size and turning off the inline dict function after the type is allocated.
Thank you for your consideration.