Converting a static extension type to a heap exension type

ngoldbaum · April 27, 2023, 5:07pm

I’ve found the documentation here on how to change a static extension type into a heap type. However, I’m having trouble working out the mechanics of actually how to do the conversion based on those docs. It would be really nice to see a version of the extension types tutorial that uses heap types, but that’s not covered there. Is anyone aware of another resource I could look at to compare the implementation of a static type and heap type in a a C extension module?

Thanks!

barry-scott · April 28, 2023, 6:48am

I have code for heap types in PyCXX. You may find its code helps.
Search for PyType_Spec to find your way into the code.

Download source from PyCXX Python-C++ Connection download | SourceForge.net

erlendaasland · April 28, 2023, 10:13am

Thanks for bringing this up. These docs definitely need to be updated to reflect current recommended practise (using heap types, module state, and multi-phase init). There are multiple issues^[1]^[2] regarding such doc updates, but unfortunately nothing has been done yet.

"Embedding Python in Another Application" documentation examples use deprecated functions · Issue #95709 · python/cpython · GitHub ↩︎
"Embedding Python in Another Application" documentation examples use deprecated functions · Issue #95709 · python/cpython · GitHub ↩︎

erlendaasland · April 28, 2023, 10:50am

There’s an example of how to write an extension module using the Limited API in the CPython source:

github.com

python/cpython/blob/main/Modules/xxlimited.c

/* Use this file as a template to start implementing a module that
   also declares object types. All occurrences of 'Xxo' should be changed
   to something reasonable for your objects. After that, all other
   occurrences of 'xx' should be changed to something reasonable for your
   module. If your module is named foo your source file should be named
   foo.c or foomodule.c.

   You will probably want to delete all references to 'x_attr' and add
   your own types of attributes instead.  Maybe you want to name your
   local variables other than 'self'.  If your object type is needed in
   other files, you'll have to create a file "foobarobject.h"; see
   floatobject.h for an example.

   This module roughly corresponds to::

      class Xxo:
         """A class that explicitly stores attributes in an internal dict"""

          def __init__(self):
              # In the C class, "_x_attr" is not accessible from Python code

This file has been truncated. show original

For the legacy approach, see cpython/Modules/xxmodule.c at main · python/cpython · GitHub.

Tres_Seaver · May 22, 2024, 10:08pm

I’m trying to follow along with the guide linked by @ngoldbaum, as well as the xxlimited example reference by @erlendaasland while trying to port the _zope_interface_c_optimizations extension module to multi-phase initialization and heap types.

One thing I don’t see covered in either the guide or the example module is how to deal with weakrefs in the New World Order. The existing _z_i_c module’s Spec holds an explicit weakreflist and clears it in Spec_dealloc. The static TypeObj sets the tp_weaklistoffset field in its type definition as you would expect:

        /* tp_weaklistoffset */ offsetof(Spec, weakreflist),

The docs for tp_members discourage the use of tp_weaklistoffset in favor of Py_TPFLAGS_MANAGED_WEAKREF, but I cannot find any description of the difference in the semantics. Given that my updated module can produce segfaults where the old version did not, and that the other refcount stuff is identical between them, I suspect that I’m not handling the weakrefs case correcty.

Work to date (started Monday at PyCon US sprints) and more musings are at this draft PR.

erlendaasland · May 22, 2024, 10:21pm

Thanks for pointing that out! We do lack good docs for those new flags. Can you open a docs issue on the tracker?