PEP 820 – PySlot: Unified slot system for the C API

PEPs
1

Hello,
I’ve written PEP 820, slightly simplified from last year’s “pre-PEP” thread, a proposal to unify PyType_Slot and PyModuleDef_Slot, and fix some of their shortcomings.
The abstract:

Replace type and module slots with a new, more type-safe structure that allows adding new slots in a more forward-compatible way.

The existing slot structures and related API is soft-deprecated. (That is: it will continue to work without warnings, and it’ll be fully documented and supported, but we plan to not add any new features to it.)

It’s a C API change that isn’t urgent/necessary, but if we want it, it would be good to get in 3.15 so the new PyModExport (PEP 793) can switch to it, and we don’t need a new module export hook.

Another way to look at it: it’s similar in spirit to @steve.dower’s Interfaces API from PEP 809, but for defining things.

What do you think?

7 Likes
Pre-PEP: Unified slot system for the C API
2

I suspect Cython wouldn’t immediately use it (at least for classes… obviously if it goes into PEP 793 then we would for that). Just because it’s mostly targeted at future improvements rather than an immediate new feature. It looks usable though.

I had one thought while reading it. I’m not yet sure if it’s a useful thought. In Steve’s PEP 809 he tried to make interfaces extensible with “namespaces”. Could that be useful here? I could see something like GraalPy wanting some custom type slots (e.g. an “inherits from this Java class” slot[1]). Similarly, at one point PyPy had an extra tp_pypy_flags in their typeobject struct.

  1. I don’t know anything about Java so this is a totally arbitrary example that might have no link to reality ↩︎

1 Like
3

That’s the intention. As the PEP says, users can reuse code they already have written without rewriting/reformatting, and only use the “new” slots if they need any new features.
If this goes to PyModExport, you can still use an array of “old-style” slots to be compatible with older Python versions. You’d need about 4 extra lines:

PyMODEXPORT_FUNC my_modexport(void) {
    static PySlot wrapped_slotswrapped[] = {
        PySlot_STATIC_DATA(Py_mod_slots, the_old_slots_array),
        PySlot_END,
    };
    return wrapped_slots;
}

Not really.
With this proposal, Python won’t “save” the slots array anywhere; it won’t let users retrieve it from the created object.
This is important for forward compatibility: if some of the input needs to be transformed in some way, Python can keep the result, not the original input. For example: “wrapped” old-style slots arrays don’t need to be stored as-is. The Py_mod_gil/Py_mod_multiple_interpreters flags can be forgotten after they’re checked/applied at creation.
Or – very theoretically, if API stability wasn’t a concern – we could turn any Py_nb_invert into an __invert__ method, and remove PyNumberMethods.nb_invert. (Practically this is for any newly added slots: “C function pointer directly on the class object” would be a possible optimization that could be removed at any time as it wouldn’t affect API.)

So, back to the subject of tp_pypy_flags: if you want to save something to retrieve it later, you’ll need to use a different mechanism. Currently, we have module state for modules, and metaclass & PyObject_GetTypeData for types; Steve’s PEP 809 Interfaces API provides “names”/“namespaces” to easily expose the info stored there.

1 Like
4

This seems like a worthwhile improvement. Count me as a +1

I have a question:

What do you anticipate _sl_reserved being used for, or is it really just padding?
If you do anticipate a use, could we give it a name that might make that usage easier in future?

And a comment:

The struct

typedef struct PySlot {
    uint16_t sl_id;
    uint16_t sl_flags;
    union {
        uint32_t _sl_reserved;  // must be 0
    };
    union {
        void *sl_ptr;
        void (*sl_func)(void);
        Py_ssize_t sl_size;
        int64_t sl_int64;
        uint64_t sl_uint64;
    };
} PySlot;

contains anonymous unions.
I thought we generally avoided anonymous unions in the API, as C++ doesn’t support them.
How about:

union _Py_slot_value {
      void *ptr;
      void (*func)(void);
      Py_ssize_t size;
      int64_t int64;
      uint64_t uint64;
};

typedef struct PySlot {
    uint16_t sl_id;
    uint16_t sl_flags;
    uint32_t _sl_reserved;  // must be 0
    union _Py_slot_value sl_val;
} PySlot;

So, instead of:

   {  
       .sl_id=Py_tp_getattro,
       .sl_flags=PySlot_HAS_FALLBACK,
       .sl_func=myClass_getattro,
   },

you’d have

   {  
       .sl_id=Py_tp_getattro,
       .sl_flags=PySlot_HAS_FALLBACK,
       .sl_val.func=myClass_getattro,
   },
2 Likes
5

Just padding now, but reserved for possible future improvements.
In an earlier iteration of the idea, arrays (of slots, MethodDefs ,etc.) could explicit size instead of zero-terminated. You’d set a flag, and put the size here.
I now don’t think this feature is worth the complexity, and I doubt it’ll ever be, but there might be something like it.

tl;dr: it’s unions that are a bit problematic; we don’t gain anything by avoiding anonymous unions.

For the long story: C++ supports anonymous unions, but:

  • They’re only supported in C since C11. But we already require the feature.[1]
  • Until C++20 which adds C-style designated initializers, C++ initializers only allow setting the first member of a union. But that’s an issue for named unions as well.
    The PEP proposes workaround macros (PySlot_PTR & PySlot_PTR_STATIC) for use in C++11-compatible code.
  • C++ doesn’t have anonymous structs (which is irrelevant here but it’s what always confuses me)

  1. We don’t necessarily require the standard spelling: the _Py_ANONYMOUS macro can request a compiler extension. ↩︎

6

I’ve updated the PEP a bit:

  • Be clearer that the PEP 793 API added in 3.15 alpha (PyModExport, PyModule_FromSlotsAndSpec) will be changed to return the new slots.

  • Deprecation of things that were documented to not work, but worked in practice:

    • setting a slot value to NULL (except if the slot explicitly allows this)
    • repeating a slot ID in a definition (except )

    As with all deprecations, happy to hear use cases for un-deprecation.

  • Added “third-party slot ID allocation” and “avoiding anonymous unions” to Rejected Ideas.

7

Hi,

Thanks for this interesting PEP. I read it multiple times but so I far I failed to make my opinion about it. I’m not convinced yet that it’s worth it. But at the same time, I’m not strongly against it neither.

PEP 820 adds many features: typed union, nested slots, single identifier space, optional slot, “has fallback” slot, etc. In exchange, it requires C extension maintainers to update their code from PyType_FromSpec()/PyModuleDef_Init() APIs to PyType_FromSlots()/PyModule_FromSlotsAndSpec(). Since new functions are not available on Python 3.14 and older, C extensions should continue maintaining existing code (using PyType_FromSpec()/PyModuleDef_Init()). At least, “nested slots” should allow to reuse code. But I’m not convinced yet that this migration is worth it.

“Motivation: Type safety”: casting to void* is “technically undefined or implementation-defined behaviour in C”. Well ok, but in practice, it just works for many years and I’m not aware of any open issue about this cast.

Optional slot (PySlot_OPTIONAL): “If the slot ID is unknown, the interpreter should ignore the slot entirely”. We can modify PyType_FromSpec()/PyModuleDef_Init() to ignore unknown slots which would allow building a single C extension working on old and new Python versions (useful for the stable ABI).

PySlot_HAS_FALLBACK: if tp_getattr is being deprecated/removed, PyType_FromSpec() can ignore silently the Py_tp_getattr slot and require the Py_tp_getattro slot. I’m not convinced that PySlot_HAS_FALLBACK is needed (and it looks a little bit complicated to use).

I also have concerns about the macros used to define a slot, such as PySlot_DATA(). They require C11 but I’m not sure that all C extensions can already use C11. The anonymous union may also cause C compatibility issues.

If I put all my concerns aside, sure, an unified API to define types and modules is appealing. It should help the stable ABI in the future. It can be stricter than the previous API, and avoids corner cases (slots set to NULL, slots defined multiple times).

1 Like
8

Same here.
But, if we want this, we want it in 3.15. This is the iteration I have now.

But, they can continue using only PyType_FromSpec()/PyModuleDef_Init(), unless they want some slots-related feature that’s new in 3.15 (and so they’re modifying code anyway). For other extensions there should be no need to migrate immediately, or to maintain two sets of init data.

Silently dropping features on older Python versions is sometimes the right thing to do, but not always. IMO, users are the best judge of that.

It’s the easiest API I could find for dealing with CPython replacing one slot with another (a better, upgraded version), if you want to use the improvement but also support CPythons before the change.
It’s not needed, but not having it is making it hard to evolve the API.

That’s fine. They can use PySlot_PTR, no shame. They lose some type safety, which is (IMO) good to provide for languages that can initialize unions.

There’s a rather extensive section on that. Do you have any concrete issues that aren’t covered?

Yeah. This is a long-term thing. There’ll never be a time when it’s needed, but over the years there have been so many times when I’ve wanted it.
And the interaction with PEP 793 (PyModExport) means that if we want this, we should put it in 3.15. (If we don’t, we’ll want yet another module export hook to return new slots, and that’s not likely to happen.)

1 Like
9

I’m sorry, I still don’t get the rationale for adding yet another API for types and modules. I don’t see which concrete issues are being addressed by the new PySlot API. To me, PySlot would be a nice API to have, but I don’t think that it’s worth the cost of the migration and having to support a new API.

If existing extensions don’t have to migrate to the new API, which projects/use cases will need the new API? Is it an API for new C extensions which only support recent Python versions (3.15 and newer)?

To me, type safety is nice to have, but it’s not really a problem which is preventing developers to write C extensions. The current API just works in practice.

An unified namespace for all slots is nice to have, but I’m fine with the current two namespaces (module slots and type slots).

As I wrote previously, PySlot_OPTIONAL and PySlot_HAS_FALLBACK can be implemented differently, by modifying PyType_FromSpec() to ignore old/deprecated slots and ignore unknown slots (from a more recent Python version).

One of the biggest compatibility issue related to type definition that I recall is https://bugs.python.org/issue37250 : when printfunc tp_print; has been replaced with Py_ssize_t tp_vectorcall_offset; in PyTypeObject. It broke C code generated by Cython which assigned tp_print to 0. If I understand correctly, this issue can already be avoided using PyType_FromSpec() which can silently ignore old/removed tp_print slot.

makes it cumbersome to add/obsolete/adjust the required info (for example, in PEP 697 I gave meaning to negative values of an existing field; adding a new field would be cleaner in similar situations);

You can already add a new slot ID using existing PyType_FromSpec() API, no?

10

Yes.
We won’t be able to add optional slots functionality in the version where they would be more useful. They need to be supported in the previous version.

Indeed; that’s a nice to have and an implementation detail.

They can not. Ignoring unknown slots means removing the error checking – if a slot is required but the interpreter doesn’t support it, it would be silently ignored.
Optionality needs to be opt-in.

To show some recent examples: NumPy has code like:

#if PY_VERSION_HEX >= 0x030c00f0  // Python 3.12+
    {Py_mod_multiple_interpreters, Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED},
#endif
#if PY_VERSION_HEX >= 0x030d00f0  // Python 3.13+
    // signal that this module supports running without an active GIL
    {Py_mod_gil, Py_MOD_GIL_NOT_USED},
#endif
  • #ifdef is not compatible with the stable ABI
  • this uses version detection rather than feature detection, which is problematic for alternate implementations. For example, PyPy or MicroPython would implement all (or most) of a given Python version but also some features from newer versions.
    Of course, there currently aren’t that many alternate implementations of the C API – but that’s partly because we don’t care about them :‍)

These particular slots are “ignorable”, so yes, this issue could be solved by making Python ignore unknown slots.
But, if we added a slot like Py_tp_metaclass, Python should probably error out if it doesn’t support the feature.

Another example: PyO3 wants to specify Py_mod_name & co. when using PyModExport, but not when using PyModuleDef. There’s a workaround they can use, but if we had PEP 820 nested slots, that workaround wouldn’t be necessary. (Again – you need PEP 820 before it’ll be useful.)

Or: the PyType_From* family of functions grows a new function every time it needs some information that’s not a good fit for immutable static data: PyType_FromSpec only had the spec, then bases was added, then module, then metaclass.
Nested slots array allow specifying such values in a small slots array on the stack, and refer to a big static array for the “regular” static slots.

11

Thanks for additional examples of slots, it helps me to understand the big picture.

So if I understand correctly, PyType_FromModuleAndSpec() and PyType_FromMetaclass() were added instead of adding Py_tp_module and Py_tp_metaclass, because slots are almost always defined at the module level as a static array like static PyType_Slot type_slots[] = {...};. Interesting. I now see better the advantage of nested slots:

static PySlot type_static_slots[] = {
    ...  // values known at build time
};

static int
init_type(PyObject *module)
{
    PySlot type_dynamic_slots[] = {
        ... // values only known at runtime
        PySlot_PTR(Py_tp_slots, type_static_slots),
        PySlot_END,
    };

    state->type = PyType_FromSlots(type_dynamic_slots);
    ...
}

For example, the module isn’t known at build time, but it can only be used in “dynamic slots”.

For PySlot_OPTIONAL, if I understood correctly the use case, the idea is to build an extension with Python 3.16 and new slots which only exists in Python 3.16, but mark these slots as “optional” to be able to use this extension on Python 3.15: so PyType_Slots() know that these unknown slots should be ignored.

If I understand correctly, all slots added after Python 3.15 should be used with PySlot_OPTIONAL. That’s not convenient. I would instead expect PyType_Slots() to ignore all “new” slots by default.

But if we add a new “mandatory” slot in Python 3.16, it should not be silently ignored by Python 3.15 (but fail with an error). In that case, I would prefer to mark such specific slot as PySlot_MANDATORY.

I don’t understand the PySlot_HAS_FALLBACK example with tp_getattr to tp_getattro.

For me, it seems easy to modify PyType_FromSlots() to handle migrations. For example, if tp_getattr and tp_getattro are defined, only use tp_getattro (and ignore tp_getattr). I don’t see how it would be better to implement the migration logic in slots.

I would prefer to remove PySlot_HAS_FALLBACK for now.

12

We could do that, but:

  • The behaviour would be more error-prone: when you take a codebase and compile it for an earlier version of Python, some slots would be silently ignored. I’d much prefer only ignoring things if the user code explicitly says it’s OK.
  • Currently, all slots are “mandatory”. To preserve the behaviour, when you convert from “old” slots to PySlot, all of them would need the flag.

OK. This one can be added later, in the version that needs it.

13

I’ve removed PySlot_HAS_FALLBACK from the PEP. How does it look now?

15

My feeling is that this is pretty unrelated to the this PEP. This PEP is describing changes to how extension types and modules are initialized from C. That probably doesn’t actually lead to any changes to how any Python type is structured once it’s been created.

1 Like
16

I suppose I should clarify: I might be wrong, but I’m 99.9% sure that the restriction disallowing multiple parent classes with __slots__ was originally put in place only because of the way slots entries are (currently) handled in C – child class __slots__ are appended to the (sole) parent’s C struct so that parent methods accessing inherited slots can use the same pointer offset as they would in the parent class.

There seems to be no language-theoretical reason that Python should disallow multiple inheritance with slots, but again, I could be completely wrong.

If this PEP would completely rework the C backend to __slots__ then it may also remove the limitation mentioned above. I’m specifically looking at this part of the PEP:

Nested slot tables

In this proposal, the array of slots can reference another array of slots, which is treated as if it was merged into its “parent”, recursively. This complicates slot handling inside the interpreter, but allows:

  • Mixing dynamically allocated (or stack-allocated) slots with static ones. This solves the issue that lead to the PyType_From* family of functions expanding with values that typically can’t be static. For example, the module argument to PyType_FromModuleAndSpec() should be a heap-allocated module object.

  • Sharing a subset of the slots to implement functionality common to several classes/modules.

  • Easily including some slots conditionally, e.g. based on the Python version.

If the array of slots can reference another array, then wouldn’t that allow two disjoint inherited arrays from different parents to be contained in the same child? If so, then the limitation on multiple inheritance could be safely removed.

17

There are two different meanings of the word “slot” being used in this
discussion: type slots, which live in the type are function pointers to
implementations of the type’s methods, and instance slots, which live in
the instance and hold values of instance variables.

If I understand correctly, this PEP is only concerned with type slots,
not instance slots, so it won’t affect the way the |slots| mechanism
works.

However, it seems to me that it wouldn’t be difficult to remove the
limitation on |slots| inheritance by recomputing the offsets of all
of a class’s instance slots when the class is created, instead of
inheriting the offsets of the base class’s slots.

I’m not sure if it would be advisable to do this, because it might break
extension modules that make assumptions about the layout of an object’s
C struct.

In any case, this ought to be the subject of another PEP, since it’s an
independent issue.

2 Likes
18

tl;dr Ok, I’m now convinced that treating all slots as mandatory by default is a sane behavior for backward and forward compatibility. Only slots explicitly marked with PySlot_OPTIONAL should be ignored.

Hum. Let me took at type slots added over the last years:

  • Py_nb_matrix_multiply (Python 3.5).
  • Py_tp_module (PEP 820).
  • Py_tp_metaclass (PEP 820).
  • Py_tp_token (Python 3.14).
  • Py_tp_vectorcall (Python 3.14).
  • Py_am_send (Python 3.10).
  • Py_tp_finalize (Python 3.5).
  • Python 3.11 added Py_bf_getbuffer and Py_bf_releasebuffer to the limited C API.
  • Special tp_pypy_flags: only available on PyPy.

MANDATORY: IMO PyType_FromSlots() should fail if it gets the following unknown slots: Py_tp_metaclass, Py_am_send.

OPTIONAL: IMO it should be safe to silently ignore the following slots if they are unknown: Py_nb_matrix_multiply, Py_tp_module, Py_tp_token, Py_tp_vectorcall, Py_tp_finalize, tp_pypy_flags. But I’m not sure that an extension would work as expected if some type slots are silently ignored…

What if the extension uses PyType_GetModuleByToken() whereas Py_tp_token is ignored? The function is likely to fail which is not the behavior expected by the extension (written for a Python version which understands Py_tp_token). It can lead to crashes if the extension doesn’t handle PyType_GetModuleByToken() failures.

If Py_tp_finalize is silently ignored, the extension may leak memory since the finalization code is ignored.

If Py_bf_getbuffer and Py_bf_releasebuffer are silently ignored by Python 3.10, the object doesn’t implement the buffer protocol and operations such as memoryview(obj) start failing.

Hum. It seems like it would be safer to treat all of these slots as “mandatory” by default, and only ignore them if they are marked with an explicit PySlot_OPTIONAL.

Ok, I’m convinced that treating all slots as mandatory by default is a sane behavior for backward and forward compatibility.

1 Like
19

I changed my mind and I’m now a supporter of PEP 820 :slight_smile: It took me a while but I now see the PEP 820 advantages: it enhances the backward compatibility and the stable ABI.

PEP 820 avoids the need to add new PyType_FromXXX() functions when new members/slots are added thanks to the ability to define some slots on the heap memory, and some others as static data. It helps with backward compatibility, before new functions were not available on old Python versions preventing to import the extension module.

PEP 820 enhances the backward compatibility by allowing to skip new slots on an old Python version using the PySlot_OPTIONAL flag. It’s useful for the stable ABI: a single binary works on multiple Python versions.

There are some other nice features, not really needed but good to have:

  • Unique namespace for slot identifiers (instead of 2 before: type and module slots).
  • Type safety: replace void* with an union.
1 Like
20

Thank you!

Submitted to the SC: PEP 820 – PySlot: Unified slot system for the C API · Issue #337 · python/steering-council · GitHub

I’ve also added a section with some quotes from this thread.