I would like to know if you agree with me with the general guidelines: hide as many implementation details as possible in the C API header files to make the Python C API usable by the majority of users. I made many changes in last years, but I got backfire on some of them so I had to write PEP 670 (accepted) and PEP 674 (rejected) for example.
I’m working on the C API to hide “implementation details”. Here I would like to talk about changes to avoid using
<string.h> functions like
memset() in the header files, but only used them in the implementation of regular functions. It’s related to PEP 670 which “advices” converting macros to static inline functions, or even regular functions.
Since Python 3.11,
<Python.h> no longer includes the following header files in the limited C API version 3.11 or newer:
The intent is to make the C API to be used as easily as possible by the majority of use cases, to reduce the dependencing on an exact C/C++ compiler or on specific C/C++ compiler flags. And also to be able to use the whole C API in programming languages others than C and C++, like Rust.
In the past, the worst example for me was when atomic variables (“pyatomic.h”) were exposed as part of the C API: it causes many troubles when the C API was used in C++, or with some C compilers. The fix was to remove atomic variables from the C API (move them to the internal C API).
A recent concrete problem is related to my
Py_CLEAR() macro fix to avoid the duplication of side effects: issue #98724. My fix uses
memset() in the header files, whereas the limited C API no longer includes
<string.h> on purpose.
I proposed a fix which implements the macro with a call to a regular function which calls the
memset() function: PR #100121. Thomas Wouters proposed instead to revert the fix to keep the bug (duplication of side effects).
Py_CLEAR() problem is very tricky since my first incorrect fix leaded to a strict aliasing bug which miscompiled Python and so caused crashes! It took me a while to dig into the machine code to fully understand this very tricky compiler issue. The problem is called type punning and is related to strict a aliasing.
It seems like other core devs like Guido van Rossum prefers to fix the bug (don’t duplicate side effects), but I don’t see a very clear consensus here, since some discussions were far from the issue, and not in the public space.