PEP 820 – PySlot: Unified slot system for the C API

PEPs
1

Hello,
I’ve written PEP 820, slightly simplified from last year’s “pre-PEP” thread, a proposal to unify PyType_Slot and PyModuleDef_Slot, and fix some of their shortcomings.
The abstract:

Replace type and module slots with a new, more type-safe structure that allows adding new slots in a more forward-compatible way.

The existing slot structures and related API is soft-deprecated. (That is: it will continue to work without warnings, and it’ll be fully documented and supported, but we plan to not add any new features to it.)

It’s a C API change that isn’t urgent/necessary, but if we want it, it would be good to get in 3.15 so the new PyModExport (PEP 793) can switch to it, and we don’t need a new module export hook.

Another way to look at it: it’s similar in spirit to @steve.dower’s Interfaces API from PEP 809, but for defining things.

What do you think?

7 Likes
Pre-PEP: Unified slot system for the C API
2

I suspect Cython wouldn’t immediately use it (at least for classes… obviously if it goes into PEP 793 then we would for that). Just because it’s mostly targeted at future improvements rather than an immediate new feature. It looks usable though.

I had one thought while reading it. I’m not yet sure if it’s a useful thought. In Steve’s PEP 809 he tried to make interfaces extensible with “namespaces”. Could that be useful here? I could see something like GraalPy wanting some custom type slots (e.g. an “inherits from this Java class” slot[1]). Similarly, at one point PyPy had an extra tp_pypy_flags in their typeobject struct.

  1. I don’t know anything about Java so this is a totally arbitrary example that might have no link to reality ↩︎

1 Like
3

That’s the intention. As the PEP says, users can reuse code they already have written without rewriting/reformatting, and only use the “new” slots if they need any new features.
If this goes to PyModExport, you can still use an array of “old-style” slots to be compatible with older Python versions. You’d need about 4 extra lines:

PyMODEXPORT_FUNC my_modexport(void) {
    static PySlot wrapped_slotswrapped[] = {
        PySlot_STATIC_DATA(Py_mod_slots, the_old_slots_array),
        PySlot_END,
    };
    return wrapped_slots;
}

Not really.
With this proposal, Python won’t “save” the slots array anywhere; it won’t let users retrieve it from the created object.
This is important for forward compatibility: if some of the input needs to be transformed in some way, Python can keep the result, not the original input. For example: “wrapped” old-style slots arrays don’t need to be stored as-is. The Py_mod_gil/Py_mod_multiple_interpreters flags can be forgotten after they’re checked/applied at creation.
Or – very theoretically, if API stability wasn’t a concern – we could turn any Py_nb_invert into an __invert__ method, and remove PyNumberMethods.nb_invert. (Practically this is for any newly added slots: “C function pointer directly on the class object” would be a possible optimization that could be removed at any time as it wouldn’t affect API.)

So, back to the subject of tp_pypy_flags: if you want to save something to retrieve it later, you’ll need to use a different mechanism. Currently, we have module state for modules, and metaclass & PyObject_GetTypeData for types; Steve’s PEP 809 Interfaces API provides “names”/“namespaces” to easily expose the info stored there.

1 Like
4

This seems like a worthwhile improvement. Count me as a +1

I have a question:

What do you anticipate _sl_reserved being used for, or is it really just padding?
If you do anticipate a use, could we give it a name that might make that usage easier in future?

And a comment:

The struct

typedef struct PySlot {
    uint16_t sl_id;
    uint16_t sl_flags;
    union {
        uint32_t _sl_reserved;  // must be 0
    };
    union {
        void *sl_ptr;
        void (*sl_func)(void);
        Py_ssize_t sl_size;
        int64_t sl_int64;
        uint64_t sl_uint64;
    };
} PySlot;

contains anonymous unions.
I thought we generally avoided anonymous unions in the API, as C++ doesn’t support them.
How about:

union _Py_slot_value {
      void *ptr;
      void (*func)(void);
      Py_ssize_t size;
      int64_t int64;
      uint64_t uint64;
};

typedef struct PySlot {
    uint16_t sl_id;
    uint16_t sl_flags;
    uint32_t _sl_reserved;  // must be 0
    union _Py_slot_value sl_val;
} PySlot;

So, instead of:

   {  
       .sl_id=Py_tp_getattro,
       .sl_flags=PySlot_HAS_FALLBACK,
       .sl_func=myClass_getattro,
   },

you’d have

   {  
       .sl_id=Py_tp_getattro,
       .sl_flags=PySlot_HAS_FALLBACK,
       .sl_val.func=myClass_getattro,
   },
2 Likes
5

Just padding now, but reserved for possible future improvements.
In an earlier iteration of the idea, arrays (of slots, MethodDefs ,etc.) could explicit size instead of zero-terminated. You’d set a flag, and put the size here.
I now don’t think this feature is worth the complexity, and I doubt it’ll ever be, but there might be something like it.

tl;dr: it’s unions that are a bit problematic; we don’t gain anything by avoiding anonymous unions.

For the long story: C++ supports anonymous unions, but:

  • They’re only supported in C since C11. But we already require the feature.[1]
  • Until C++20 which adds C-style designated initializers, C++ initializers only allow setting the first member of a union. But that’s an issue for named unions as well.
    The PEP proposes workaround macros (PySlot_PTR & PySlot_PTR_STATIC) for use in C++11-compatible code.
  • C++ doesn’t have anonymous structs (which is irrelevant here but it’s what always confuses me)

  1. We don’t necessarily require the standard spelling: the _Py_ANONYMOUS macro can request a compiler extension. ↩︎

6

I’ve updated the PEP a bit:

  • Be clearer that the PEP 793 API added in 3.15 alpha (PyModExport, PyModule_FromSlotsAndSpec) will be changed to return the new slots.

  • Deprecation of things that were documented to not work, but worked in practice:

    • setting a slot value to NULL (except if the slot explicitly allows this)
    • repeating a slot ID in a definition (except )

    As with all deprecations, happy to hear use cases for un-deprecation.

  • Added “third-party slot ID allocation” and “avoiding anonymous unions” to Rejected Ideas.

7

Hi,

Thanks for this interesting PEP. I read it multiple times but so I far I failed to make my opinion about it. I’m not convinced yet that it’s worth it. But at the same time, I’m not strongly against it neither.

PEP 820 adds many features: typed union, nested slots, single identifier space, optional slot, “has fallback” slot, etc. In exchange, it requires C extension maintainers to update their code from PyType_FromSpec()/PyModuleDef_Init() APIs to PyType_FromSlots()/PyModule_FromSlotsAndSpec(). Since new functions are not available on Python 3.14 and older, C extensions should continue maintaining existing code (using PyType_FromSpec()/PyModuleDef_Init()). At least, “nested slots” should allow to reuse code. But I’m not convinced yet that this migration is worth it.

“Motivation: Type safety”: casting to void* is “technically undefined or implementation-defined behaviour in C”. Well ok, but in practice, it just works for many years and I’m not aware of any open issue about this cast.

Optional slot (PySlot_OPTIONAL): “If the slot ID is unknown, the interpreter should ignore the slot entirely”. We can modify PyType_FromSpec()/PyModuleDef_Init() to ignore unknown slots which would allow building a single C extension working on old and new Python versions (useful for the stable ABI).

PySlot_HAS_FALLBACK: if tp_getattr is being deprecated/removed, PyType_FromSpec() can ignore silently the Py_tp_getattr slot and require the Py_tp_getattro slot. I’m not convinced that PySlot_HAS_FALLBACK is needed (and it looks a little bit complicated to use).

I also have concerns about the macros used to define a slot, such as PySlot_DATA(). They require C11 but I’m not sure that all C extensions can already use C11. The anonymous union may also cause C compatibility issues.

If I put all my concerns aside, sure, an unified API to define types and modules is appealing. It should help the stable ABI in the future. It can be stricter than the previous API, and avoids corner cases (slots set to NULL, slots defined multiple times).

1 Like
8

Same here.
But, if we want this, we want it in 3.15. This is the iteration I have now.

But, they can continue using only PyType_FromSpec()/PyModuleDef_Init(), unless they want some slots-related feature that’s new in 3.15 (and so they’re modifying code anyway). For other extensions there should be no need to migrate immediately, or to maintain two sets of init data.

Silently dropping features on older Python versions is sometimes the right thing to do, but not always. IMO, users are the best judge of that.

It’s the easiest API I could find for dealing with CPython replacing one slot with another (a better, upgraded version), if you want to use the improvement but also support CPythons before the change.
It’s not needed, but not having it is making it hard to evolve the API.

That’s fine. They can use PySlot_PTR, no shame. They lose some type safety, which is (IMO) good to provide for languages that can initialize unions.

There’s a rather extensive section on that. Do you have any concrete issues that aren’t covered?

Yeah. This is a long-term thing. There’ll never be a time when it’s needed, but over the years there have been so many times when I’ve wanted it.
And the interaction with PEP 793 (PyModExport) means that if we want this, we should put it in 3.15. (If we don’t, we’ll want yet another module export hook to return new slots, and that’s not likely to happen.)

1 Like