We almost ran out of type flag bits.
Let’s add more space in 3.16, and while we’re at it, decouple the public API from the internal storage (kinda what PyType_Spec/PySlot do for PyTypeObject struct itself). That way, we can later rearrange the flags or remove obsolete ones… except for the public ones we have in tp_flags now.
I’d like to have a design ready before we need to add the next flag, so we don’t need to rush at that point.
Proposal:
For storage, let’s start with uint8_t _tp_extra_flags after PyTypeObject.tp_watched, filling up some padding.
When 8 bits run out, we can add another field. (Or reuse bits if some are obsolete.)
For API:
Flag IDs
Currently, type flags are represented directly by single-bit masks, for example:
#define Py_TPFLAGS_DISALLOW_INSTANTIATION (1UL << 7)
#define Py_TPFLAGS_IMMUTABLETYPE (1UL << 8)
#define Py_TPFLAGS_HEAPTYPE (1UL << 9)
#define Py_TPFLAGS_BASETYPE (1UL << 10)
There’s already a function to get a single flags: PyType_HasFeature.
This does (tp->tp_flags & flag != 0), which technically you can get multiple flags at once (with the result ORed into a single bit). Maybe that’s out-of-spec usage, but it’s a function from 1998; let’s try not to break it.
I still think we can identify flags by 32-bit numbers. We can:
- let existing flags stay as they are (single bits);
- for new flags, set the currently unassigned bit (1 << 21), with the other bits spelling a unique ID.
Given the name PyType_HasFeature, I’m thinking about calling the flags type features in docs, to make it obvious when the new API needs to be is used.
Getters
In the current PyType_HasFeature, the int argument changes to uint32_t. (This should be an API-compatible change. Also, the function was half-unusable on platforms with 16-bit int.)
int PyType_HasFeature(PyTypeObject *tp, uint32_t feature_id);
Return 0 or 1 based on the value of the given feature.
Return -1 with exception set on error (TypeErroron non-type objects,
SystemErroron non-existent or private flags).
Outside Stable ABI, this and the setter below will be static inline function that should, if feature_id is a compile-time constant, compile to a simple read & mask.
I don’t think we need a DuringGC variant – the author of tp_traverse function should know what their type does.
There should be an internal variant: _PyType_HasFeature would be like the above, but abort on error, and allow getting private features.
Setters
Let’s add a slot for heap types, and a setter function for making static types:
-
A slot for creating heap types:
PySlot_INT64(Py_tp_feature, Py_TPFLAGS_xxx)will set the given flag to 1.The
PySlot_OPTIONALslot flag will cause entries with unknownPy_TPFLAGS_xxxto be ignored.The
Py_tp_flagsslot will be soft-deprecated. -
int PyType_SetFeature(PyTypeObject *tp, uint32_t feature_id, int value);Set the given flag to
(value != 0), if the flag is settable.
Return 0 on success; -1 with exception set on error.Before tp is ready, the function can be called on a static type
for flags thatPyType_HasFeatureaccepts, except ones that the VM can
reliably set itself (Py_TPFLAGS_*_SUBCLASS,Py_TPFLAGS_READY*,
Py_TPFLAGS_HEAPTYPE), internal/private ones (_Py*,Py_TPFLAGS_HAVE_FINALIZE,_Py_TPFLAGS_STATIC_BUILTIN, bothPy_TPFLAGS_HAVE_STACKLESS_EXTENSIONbits,Py_TPFLAGS_HAVE_VERSION_TAG,Py_TPFLAGS_VALID_VERSION_TAG,_Py_TPFLAGS_MATCH_SELF), andPy_TPFLAGS_PREHEADER.(After tp is ready, the only flag that makes sense for users to set is
Py_TPFLAGS_IMMUTABLETYPE, but we already havePyType_Freezefor that, so this isn’t worth implementation complexity.
However, we might want to make future flags safely (un)settable without a new dedicated function.)
Plus an internal function:
-
int _PyType_SetFeature(PyTypeObject *tp, uint32_t feature_id, int new_value);Like
PyType_SetFeature, but allows setting at any time.
No locking or world-stopping; needs external synchronization.
Sunsetting Py_TPFLAGS_DEFAULT
And let’s soft-deprecate Py_TPFLAGS_DEFAULT and officially let people use 0. It has been zero for a while anyway. (Note that Stackless Python doesn’t put “their” bit there.)
Thoughts?