PEP 820 – PySlot: Unified slot system for the C API

PEPs
22

(Posting for the SC.)

The Steering Council is reviewing PEP 820 and we are mostly positive, but we have a few concerns. We realize that the PEP came out of the discussions around PEP 793 (PyModExport: A new entry point for C extension modules) and that it would be convenient to make these changes in Python 3.15 to avoid backward compatibility concerns, but we struggle a little with the motivation for this specific design.

Nested slot tables, in particular, seem like a complex feature with no justification other than “it seems useful”, and perhaps as a generalization of PyType_Slot and PyModuleDef_Slot arrays. The arbitrary limit on nesting, for example: what use-cases do you see for nested slots that make the limit of 5 reasonable?

We’re also a little skeptical about the type safety claims the PEP makes about the new API. While the new API makes it harder to pass the wrong data type to a slot-setting macro (like passing a string to PySlot_FUNC), it does nothing to protect from using the wrong slot-setting macro for a particular slot (like PySlot_STATIC(tp_repr, "<myclass repr>")). The new API is marginally better, but the question is if that’s a good enough reason to add yet another type/module initialization API. Is this really the best API we can come up with, long term, or are we expected to see yet another new API to further improve the type safety?

Lastly, we’re concerned about the new deprecation warnings for existing users of slots that may be setting slots to NULL. We don’t want to flood users with warnings for things that are outside of their control. It would be much better if the deprecation warnings were only issued for uses of the new APIs, unless we’re really confident the only cases that would produce the deprecation warning are, in fact, problematic situations. From the description of the state of affairs that doesn’t seem to be the case.

We would also like to invite extension authors to weigh in on the discussion here, if there are any concerns. While we take objections from third-party authors seriously, we wouldn’t necessarily expect active endorsement from them as this PEP probably doesn’t materially make their lives easier. We just want to make sure that we aren’t making it dramatically harder.

We know time is running out for Python 3.15, but we would appreciate your opinion on these concerns before we decide on this PEP.

1 Like
23

Thank you for considering the PEP!

The justification is in the rationale – Nested slot tables & Nested “legacy” slot tables.

The key use cases are mixing static & heap tables, and migrating from the old format. Both are rather important.

The limit is supposed to be unreasonable; you shouldn’t hit it in practice :‍)

You need roughly up to one level of nesting for each feature listed in the rationale (mixing static & heap tables, migrating from the old format, sharing slots, including slots conditionally), but I don’t think you’d need all of them at once.
I expect two levels to be common for migrating from the old format (i.e. PyModInit & PyModExport sharing an old-style slots array); nested array (with the slot ID as a “format header”) is the most natural API I found for that. Limiting to 2 levels wouldn’t simplify the implementation much.

Yeah, type safety is mainly about avoiding undefined behavior: data⭤function pointer casts. (This is not an issue in current compilers, and probably won’t become a problem in new platforms, but as Python/emscripten_trampoline.c shows, even new platforms can be “weird”.)

(And also about having an explicit number of bits for flags in a slots – as we drop 32 platforms, we could easily start relying on 64-bit intptr_t, making the API unportable back to 32 bits.)

This is not the best API, but IMO it’s ready for making incremental improvements rather than adding yet another API. (In particular, I’m thinking about slot-specific macros, i.e. instead of PySlot_FUNC(tp_repr, myClass_repr) you’d do something PySlot_tp_repr(myClass_repr). But to be usable in a static array, that needs C _Generic or C++ constexpr, and of course something else in other languages. I’d still want the PEP 820 API as a fallback.)

The ABI is best I can come up with, long term (though it’s of course missing stuff we don’t need now but can add later).
The API is a basic wrapper plus a few convenience macros.

I couldn’t find much evidence of NULL slots being popular, but I agree this would make for a smoother transition where they are used. Let’s do it.

24

Thanks! With the change to omit the deprecation warnings for NULL slots, the SC has decided to accept the PEP. Congrats :slight_smile:

10 Likes
25

Thank you for reviewing it!

I’ve marked the PEP as accepted, and sent a PR.

It would be much better if the deprecation warnings were only issued for uses of the new APIs, unless we’re really confident the only cases that would produce the deprecation warning are, in fact, problematic situations.

Probably well below SC-level resolution, but: “is this new API?” is ambiguous in case of nested “legacy” slots.
In my PR, the “outer-most” layer (or equivalently, the function you call/export), determine whether warnings are shown. So if you switch to PyModExport but keep the struct old format, you’ll get the warnings.
If this turns out to be a problem (in practice with the beta, or with the SC), I’d rather remove deprecation of repeated slots entirely. (Tracking which struct a slot occurred in isn’t worth it; I don’t want to base the warn-or-not decision only on the second occurrence: slot order mostly doesn’t matter, and I don’t want warnings showing up after you alphabetize a “set” definition.)