Getting exceptions & built-in types using `Py_GetConstant`

Core Development C API
1

I propose defining identifiers for use with Py_GetConstant for exceptions
(e.g. Py_CONSTANT_Exc_ValueError to get ValueError) and other built-in types
(e.g. Py_CONSTANT_Bool_Type to get bool).

When compiling for Stable ABI 3.16+, the current API will be defined via calls to Py_GetConstant(). For example:

#define PyExc_ArithmeticError (Py_GetConstantBorrowed(Py_CONSTANT_Exc_ArithmeticError))
#define PyBool_Type (*(PyTypeObject*)Py_GetConstantBorrowed(Py_CONSTANT_Bool_Type))

This means they can’t be used as lvalues (good) or outside C functions (like in static initializers, which didn’t work on Windows across DLL boundaries).

This means 103 new constants – one for every PyObject* data export in the Stable ABI except exceptions below, plus Py_CONSTANT_Method_Type (to expose method type as requested in capi-workgroup/decisions#108).

Expand the full list
  • Py_CONSTANT_Exc_ArithmeticError
  • Py_CONSTANT_Exc_AssertionError
  • Py_CONSTANT_Exc_AttributeError
  • Py_CONSTANT_Exc_BaseException
  • Py_CONSTANT_Exc_BaseExceptionGroup
  • Py_CONSTANT_Exc_BlockingIOError
  • Py_CONSTANT_Exc_BrokenPipeError
  • Py_CONSTANT_Exc_BufferError
  • Py_CONSTANT_Exc_BytesWarning
  • Py_CONSTANT_Exc_ChildProcessError
  • Py_CONSTANT_Exc_ConnectionAbortedError
  • Py_CONSTANT_Exc_ConnectionError
  • Py_CONSTANT_Exc_ConnectionRefusedError
  • Py_CONSTANT_Exc_ConnectionResetError
  • Py_CONSTANT_Exc_DeprecationWarning
  • Py_CONSTANT_Exc_EOFError
  • Py_CONSTANT_Exc_EncodingWarning
  • Py_CONSTANT_Exc_Exception
  • Py_CONSTANT_Exc_FileExistsError
  • Py_CONSTANT_Exc_FileNotFoundError
  • Py_CONSTANT_Exc_FloatingPointError
  • Py_CONSTANT_Exc_FutureWarning
  • Py_CONSTANT_Exc_GeneratorExit
  • Py_CONSTANT_Exc_ImportError
  • Py_CONSTANT_Exc_ImportWarning
  • Py_CONSTANT_Exc_IndentationError
  • Py_CONSTANT_Exc_IndexError
  • Py_CONSTANT_Exc_InterruptedError
  • Py_CONSTANT_Exc_IsADirectoryError
  • Py_CONSTANT_Exc_KeyError
  • Py_CONSTANT_Exc_KeyboardInterrupt
  • Py_CONSTANT_Exc_LookupError
  • Py_CONSTANT_Exc_MemoryError
  • Py_CONSTANT_Exc_ModuleNotFoundError
  • Py_CONSTANT_Exc_NameError
  • Py_CONSTANT_Exc_NotADirectoryError
  • Py_CONSTANT_Exc_NotImplementedError
  • Py_CONSTANT_Exc_OSError
  • Py_CONSTANT_Exc_OverflowError
  • Py_CONSTANT_Exc_PendingDeprecationWarning
  • Py_CONSTANT_Exc_PermissionError
  • Py_CONSTANT_Exc_ProcessLookupError
  • Py_CONSTANT_Exc_RecursionError
  • Py_CONSTANT_Exc_ReferenceError
  • Py_CONSTANT_Exc_ResourceWarning
  • Py_CONSTANT_Exc_RuntimeError
  • Py_CONSTANT_Exc_RuntimeWarning
  • Py_CONSTANT_Exc_StopAsyncIteration
  • Py_CONSTANT_Exc_StopIteration
  • Py_CONSTANT_Exc_SyntaxError
  • Py_CONSTANT_Exc_SyntaxWarning
  • Py_CONSTANT_Exc_SystemError
  • Py_CONSTANT_Exc_SystemExit
  • Py_CONSTANT_Exc_TabError
  • Py_CONSTANT_Exc_TimeoutError
  • Py_CONSTANT_Exc_TypeError
  • Py_CONSTANT_Exc_UnboundLocalError
  • Py_CONSTANT_Exc_UnicodeDecodeError
  • Py_CONSTANT_Exc_UnicodeEncodeError
  • Py_CONSTANT_Exc_UnicodeError
  • Py_CONSTANT_Exc_UnicodeTranslateError
  • Py_CONSTANT_Exc_UnicodeWarning
  • Py_CONSTANT_Exc_UserWarning
  • Py_CONSTANT_Exc_ValueError
  • Py_CONSTANT_Exc_Warning
  • Py_CONSTANT_Exc_ZeroDivisionError
  • Py_CONSTANT_BaseObject_Type
  • Py_CONSTANT_Bool_Type
  • Py_CONSTANT_ByteArray_Type
  • Py_CONSTANT_Bytes_Type
  • Py_CONSTANT_CFunction_Type
  • Py_CONSTANT_Capsule_Type
  • Py_CONSTANT_ClassMethodDescr_Type
  • Py_CONSTANT_Complex_Type
  • Py_CONSTANT_DictProxy_Type
  • Py_CONSTANT_Dict_Type
  • Py_CONSTANT_Ellipsis_Type
  • Py_CONSTANT_Enum_Type
  • Py_CONSTANT_Filter_Type
  • Py_CONSTANT_Float_Type
  • Py_CONSTANT_FrozenSet_Type
  • Py_CONSTANT_GetSetDescr_Type
  • Py_CONSTANT_List_Type
  • Py_CONSTANT_Long_Type
  • Py_CONSTANT_Map_Type
  • Py_CONSTANT_MemberDescr_Type
  • Py_CONSTANT_MemoryView_Type
  • Py_CONSTANT_MethodDescr_Type
  • Py_CONSTANT_Module_Type
  • Py_CONSTANT_Property_Type
  • Py_CONSTANT_Range_Type
  • Py_CONSTANT_Reversed_Type
  • Py_CONSTANT_Set_Type
  • Py_CONSTANT_Slice_Type
  • Py_CONSTANT_Super_Type
  • Py_CONSTANT_TraceBack_Type
  • Py_CONSTANT_Tuple_Type
  • Py_CONSTANT_Type_Type
  • Py_CONSTANT_Unicode_Type
  • Py_CONSTANT_WrapperDescr_Type
  • Py_CONSTANT_Zip_Type
  • Py_CONSTANT_GenericAliasType
  • Py_CONSTANT_Method_Type

Why

Data exports are process-wide globals, which means that any implementation that exposes the current ABI (with PyExc_ValueError, PyBool_Type etc.) must implement these as static types, or share them across interpreters and manage their lifetime.
This constrains CPython (where we can’t really do much about it). In RustPython, it’s reportedly a blocker for supporting the Stable ABI.

Data exports are problematic in general; see capi-workgroup/problems#80.

Exceptions, soft-deprecations and removals

The OSError aliases won’t get an identifier for Py_GetConstant, but will be defined as aliases to OSError, and formally soft-deprecated:
- PyExc_EnvironmentError
- PyExc_IOError
- PyExc_WindowsError

Iteration types (which have major caveats w.r.t. instantiation & type checking, see docs) will be formally soft-deprecated, and no longer defined when compiling for Stable ABI 3.16:
- PyCallIter_Type
- PySeqIter_Type
- PyUnicodeIter_Type
- PyByteArrayIter_Type
- PyBytesIter_Type
- PyListIter_Type
- PyListRevIter_Type
- PySetIter_Type
- PyTupleIter_Type
- PyRangeIter_Type
- PyLongRangeIter_Type
- PyDictIterKey_Type
- PyDictRevIterKey_Type
- PyDictIterValue_Type
- PyDictRevIterValue_Type
- PyDictIterItem_Type
- PyDictRevIterItem_Type

The module def type will be soft-deprecated. It will no longer be defined for abi3t 3.16+, but it’ll remain when compiling for abi3:

  • PyModuleDef_Type

When compiling for Stable ABI 3.16+, the following will be functions rather than macros; the private _PyWeakref_*Type objects won’t be defined:

  • PyWeakref_CheckRef
  • PyWeakref_CheckRefExact
  • PyWeakref_CheckProxy
  • PyWeakref_Check

The following private APIs will no longer be defined when compiling for Stable ABI 3.16, as they’re not needed for the relevant macros:

  • _Py_EllipsisObject
  • _Py_FalseStruct
  • _Py_NoneStruct
  • _Py_NotImplementedStruct
  • _Py_TrueStruct
  • _Py_RefTotal (this is a Py_ssize_t, not PyObject*)
  • _Py_SwappedOp (this is an int array, not PyObject*)

What remains

With this, all PyObject data exports in Stable ABI will be “ABI only” (available for backwards compatibility, but not defined in the headers when compiling for 3.16+). The exception is PyModuleDef_Type in abi3.

There are also some non-PyObject ones, out of scope here (except _Py_SwappedOp/_Py_RefTotal above):

  • Py_Version (ulong)
  • PyStructSequence_UnnamedField (C string)
  • PyOS_InputHook (function pointer; niche)
  • Py_FileSystemDefaultEncodeErrors (C string; deprecated)
  • Py_FileSystemDefaultEncoding (C string; deprecated)
  • Py_HasFileSystemDefaultEncoding (C int; deprecated)
  • Py_UTF8Mode (C int; deprecated)
4 Likes
C API: Add `Py_GetBuildInType` and `Py_GetBuildInTypeBorrowed`
2

Looks good to me and should work for Cython.

3

I’m worried about reference counting correctness with the proposed API.

When you want a type, in most cases you want a borrowed reference to it. Also, many functions accepting a type use PyTypeObject* type, rather than PyObject* type. Examples:

  • Py_IS_TYPE(obj, &PyBool_Type)
  • mytype.tp_base = &PyLong_Type; – declare a new type
  • PyObject *encoded = PyLong_Type.tp_repr(obj); – access directly PyTypeObject members (not possible in the limited C API)
  • PyObject_IsSubclass(obj, (PyObject*)&PyBool_Type) – the exception case: it uses a PyObject* type

For these reasons, IMO we should add a Py_GetType() API instead. Example:

#define PyBool_Type (*Py_GetType(Py_TYPE_Bool))

It’s safe to request a borrowed reference to a built-in type since built-in types are valid until Python is finalized: they are destroyed late during Python finalization.

Py_GetType(Py_TYPE_Bool) is shorter to type than (PyTypeObject*)Py_GetConstantBorrowed(Py_CONSTANT_Bool_Type).

In CPython, static types are immortal, so borrowed vs strong references is not a big deal in practice. But other Python implementations may implement them differently, as mortal objects (as CPython heap types).

Note: the C API has PyObject_Type(obj) which gets the type of the object obj. It’s similar to Py_TYPE() but returns a strong reference, instead of a borrowed reference. Py_GetType() name is different than Py_TYPE(obj) and PyObject_Type(obj).

For exception types, we also usually want a borrowed reference. Examples:

  • PyErr_SetString(PyExc_OSError, "Invalid data stream");
  • PyErr_SetFromErrno(PyExc_OSError);

Note: the PyObject* type is used here.

Exception types are also valid until the Python finalization: they are deleted very late in the finalization. So using borrowed references is safe.

So we may add Py_GetException() API to get a borrowed reference. Example:

#define PyExc_OSError Py_GetException(Py_EXC_OSError)

4

If the primary way is accessing these in C is still via PyBool_Type and that gets macro-expanded into Py_GetConstantBorrowed then I’m not sure there’s a huge need for Py_GetType and Py_GetException since few C users will be manually typing them anyway.

That doesn’t apply in Rust (where C macros don’t work directly) but my impression is that PyO3 has to get this right once and then all downstream users will be fine. So I doubt the slightly verbose Py_GetConstantBorrowed is a problem there.

I quite like the idea that there’s one way to get PyObject constants.

3 Likes
5

That’s why we have Py_GetConstantBorrowed.

Yes, but many others need PyObject*. The two main things you can do with a type object in C API are instantiation (with the call API, which needs a PyObject*) and type checking (which tends to need PyTypeObject*). Unless we add two functions, one of the cases will need casts.

(I’ll note that upcast (PyTypeObject*→PyObject*) is the safe direction to cast in. That’s a point for adding a new function for types, but, it didn’t convince me.)

As Da Woods said, I expect the primary API to be &PyBool_Type defines, and any new type that’s “important enough” will probably get functions like PyLong_Check/PyLong_CheckExact that hide the detail.

I wouldn’t be too opposed to naming the defines Py_TYPE_*. But I still think they should work with Py_GetConstantBorrowed.

I did consider proposing specialized function than Py_GetType, just for type-checking:

int PyObject_TypeCheckConstant(PyObject *o, unsigned int constant_id);

but don’t think it’s necessary. This or Py_GetType can be added later.

FWIW,

  • mytype.tp_base = &PyLong_Type; – declare a new type

You can’t do this in the limited API either; you use PyType_FromMetaclass which takes PyObject *bases (either a tuple or a single type), or PySlot_PTR(Py_tp_bases, PyLong_Type).

That’s the case for all constants you get from Py_GetConstantBorrowed. The user-facing docs say they’re borrowed from the interpreter :‍)

1 Like