Exposing PyHeapTypeObject in limited API

I wish this actually allowed to be able to use PyHeapTypeObject as a casting type when calling PyType_FromSpec within C code using the limited API for 3.14.0a3. It would also allow one to be able to also access ht_type on it as well and also allow one to store the PyHeapTypeObject pointers using extern PyHeapTypeObject *TypeObj; to be used within their C code even if it spans multiple files. This is practically the only thing on 1 of my C extensions keeping me from using the limited API targeting 3.14 specifically.

Could you explain this in terms of what you need to achieve, as opposed to how it can technically be currently done?

For example, if there was a single (or set of) API to do it, what would the Py_... functions be called?

We are more than willing to add new APIs to enable people. But breaking the abstraction is literally the definition of “use the non-limited API”, so if that’s what you need, then you should be using the normal API.

Type objects are Python objects, so:

  • to get a PyObject* from a PyTypeObject*, cast.
  • to get a PyTypeObject* from a PyObject*, check that it’s a type, and then cast. (You can skip the check if know it’s a type, for example if you got it fromPyType_FromSpec.)

PyHeapTypeObject works the same way. To get the PyTypeObject*, cast. You don’t need to go through ht_type.

If you have control over, the extern pointers, I’d recommend using PyObject * or PyTypeObject*, rather than PyHeapTypeObject.

1 Like

What is the best practice for checking that the object is actually a heap type? PyType_Check is not enough, and there is no convenient PyHeapType_Check function.

PyType_HasFeature(type, Py_TPFLAGS_HEAPTYPE)

But it should be relatively rare to need that in extensions. (Or anywhere outside CPython’s typeobject.c, for that matter.)

1 Like

Yet this is what @AraHaan is asking for. Is the answer “Yes, you can safely cast to PyHeapTypeObject after calling PyType_FromSpec using the limited API in 3.13.0a3”?

PyHeapTypeObject isn’t in the Limited API so I don’t see how you can cast to it at all.

1 Like

No, you cannot do that. But if I understood correctly, @AraHaan ultimately wants:

  • access to ht_type (cast to PyTypeObject* instead)
  • store in an extern pointer (that pointer should be PyObject* or PyTypeObject* instead)

@AraHaan, is that right?

To clarify why avoiding extern PyHeapTypeObject * is a good idea: The PyHeapTypeObject struct itself is not very interesting, and certainly not a candidate for stable ABI. Aside from ht_type, it has:

  • slot tables (whose contents are, with a few exceptions, accessible in the limited API using PyType_GetSlot; and in the non-limited API they’re all exposed via PyTypeObject.tp_as_* pointers)
  • several members with dedicated accessors (like PyType_GetName for ht_name)
  • some details that aren’t useful in extensions and probably should have been fully private (like ht_cached_keys)

And no API functions take PyHeapTypeObject*, so you’d need to cast to PyObject* or PyTypeObject* for any use other than accessing the members.

1 Like

Dear moderators: IMO, the conversation starting with AraHaan’s post should be moved to a new topic in Ideas, titled e.g. “Exposing PyHeapTypeObject in limited API.”

That is correct. However I do also wish that there could be a new type flag that can be used within a specific PyType_Spec that can opt-out of the part of the code in PyType_FromSpec() that creates a type as a heap type (for those who want to create a traditional type that is a “static-like” type that cannot directly be created in the limited API currently (without a spec) and also for when they want the created type to be immortal (after creation) until when said module is unloaded (I seen cleanup like this done from within the old audioop module). I also see that some parts of the cpython codebase also suffers from being unable to use spec’s and such a change could allow them to migrate to using specs, while also not losing the benefits of not being heap types in certain edge cases.

And looking back now, I probably should ideally be using multi-phase and storing all of this information within a structure for which is used when I get the module’s state so it can support multiple interpreters (and could also support free-threaded builds for free).

1 Like