Since the previous version of this PEP, I’ve teamed up with Victor to rewrite it. The result is available as the current version of PEP 743, and quoted below.
In short, we propose a single macro that will:
- Hide “non-recommended API” – API that’s soft-deprecated, and easy to replace with a variant. In effect, we’re playing the role of a rudimentary linter (but the aim is also to provide better, dogfooded data for actual linters, which can do advanced things like ignore individual warnings).
- Hide deprecated API.
- Hide API that lacks the
Pyprefix, so it can clash with other libraries. Note that this is something a linter can’t help with.
The macro is versioned, so users can upgrade (or not) at their own pace. Each version is frozen at 3.x.0b1.
The macro name is kept from the previous version of the PEP; it’s up for bikeshedding.
What do y’all think?
Click for full text (with broken ReST formatting)
Abstract
Add Py_COMPAT_API_VERSION C macro that hides some deprecated and
soft-deprecated symbols, allowing users to opt out of using API with known
issues that other API solves.
The macro is versioned, allowing users to update (or not) on their own pace.
Also, add namespaced alternatives for API without the Py_ prefix,
and soft-deprecate the original names.
Motivation
Some of Python’s C API has flaws that are only obvious in hindsight.
If an API prevents adding features or optimizations, or presents a serious
security risk or maintenance burden, we can deprecate and remove it as
described in :pep:387.
However, this leaves us with some API that has “sharp edges” – it works fine
for its current users, but should be avoided in new code.
For example:
- API that cannot signal an exception, so failures are either ignored or
exit the process with a fatal error. For examplePyObject_HasAttr. - API that is not thread-safe, for example by borrowing references from
mutable objects, or exposing unfinished mutable objects. For example
PyDict_GetItemWithError. - API with names that don’t use the
Py/_Pyprefix, and so can clash
with other code. For example:setter.
It is important to note that despite such flaws, it’s usually possible
to use the API correctly. For example, in a single-threaded environment,
thread safety is not an issue.
We do not want to break working code, even if it uses API that would be wrong
in some – or even most – other contexts.
On the other hand, we want to steer users away from such “undesirable” API
in new code, especially if a safer alternative exists.
Adding the Py prefix
Some names defined in CPython headers is not namespaced: it that lacks the
Py prefix (or a variant: _Py, and alternative capitalizations).
For example, we declare a function type named simply setter.
While such names are not exported in the ABI (as checked by make smelly),
they can clash with user code and, more importantly, with libraries linked
to third-party extensions.
While it would be possible to provide namespaced aliases and (soft-)deprecate
these names, the only way to make them not clash with third-party code is to
not define them in Python headers at all.
Rationale
We want to allow an easy way for users to avoid “undesirable” API if they
choose to do so.
It might be be sufficient to leave this to third-party linters.
For that we’d need a good way to expose a list of (soft-)deprecated
API to such linters.
While adding that, we can – rather easily – do the linter’s job directly
in CPython headers, avoiding the neel for an extra tool.
Unlike Python, C makes it rather easy to limit available API – for a whole
project or for each individual source file – by having users define
an “opt-in” macro.
We already do something similar with Py_LIMITED_API, which limits the
available API to a subset that compiles to stable ABI. (In hindsight, we should
have used a different macro name for that particular kind of limiting, but it’s
too late to change that now.)
To prevent working code from breaking as we identify more “undesirable” API
and add safer alternatives to it, the opt-in macro should be versioned.
Users can choose a version they need based on their compatibility requirements,
and update it at their own pace.
To be clear, this mechanism is not a replacement for deprecation.
Deprecation is for API that prevents new features or optimizations, or
presents a security risk or maintenance burden.
This mechanism, on the other hand, is meant for cases where “we found
a slightly better way of doing things” – perhaps one that’s harder to misuse,
or just has a less misleading name.
(On a lighter note: many people configure a code quality checker to shout at
them about the number of blank lines between functions. Let’s help them
identify more substantial “code smells”!)
The proposed macro does not change any API definitions; it only hides them.
So, if code compiles with the macro, it’ll also compile without it, with
identical behaviour.
This has implications for core devs: to deal with undesirable behaviour,
we’ll need to introduce new, better API, and then discourage the old one.
In turn, this implies that we should look at an individual API and fix all its
known issues at once, rather than do codebase-wide sweeps for a single kind of
issue, so that we avoid multiple renames of the same function.
Adding the Py prefix
An opt-in macro allows us to omit definitions that could clash with
third-party libraries.
Specification
We introduce a Py_COMPAT_API_VERSION macro.
If this macro is defined before #include <Python.h>, some API definitions
– as described below – will be omitted from the Python header files.
The macro only omits complete top-level definitions exposed from <Python.h>.
Other things (the ABI, structure definitions, macro expansions, static inline
function bodies, etc.) are not affected.
The C API working group (:pep:731) has authority over the set of omitted
definitions.
The set of omitted definitions will be tied to a particular feature release
of CPython, and is finalized in each 3.x.0 Beta 1 release.
In rare cases, entries can be removed (i.e. made available for use) at any
time.
The macro should be defined to a version in the format used by
PY_VERSION_HEX, with the “micro”, “release” and “serial” fields
set to zero.
For example, to omit API deemed undesirable in 3.14.0b1, users should define
Py_COMPAT_API_VERSION to 0x030e0000.
Requirements for omitted API
An API that is omitted with Py_COMPAT_API_VERSION must:
- be soft-deprecated (see :pep:
387); - for all known use cases of the API, have a documented alternative
or workaround; - have tests to ensure it keeps working (except for 1:1 renames using
#defineortypedef); - be documented (except if it was never mentioned in previous versions of the
documentation); and - be approved by the C API working group. (The WG may give blanket approvals
for groups of related API; see Initial set below for examples.)
Note that Py_COMPAT_API_VERSION is meant for API that can be trivially
replaced by a better alternative.
API without a replacement should generally be deprecated instead.
Location
All API definitions omitted by Py_COMPAT_API_VERSION will be moved to
a new header, Include/legacy.h.
This is meant to help linter authors compile lists, so they can flag the API
with warnings rather than errors.
Note that for simple renaming of source-only constructs (macros, types), we
expect names to be omitted in the same version – or the same PR – that adds
a replacement.
This means that the original definition will be renamed, and a typedef
or #define for the old name added to Include/legacy.h.
Documentation
Documentation for omitted API should generally:
- appear after the recommended replacement,
- reference the replacement (e.g. “Similar to X, but…”), and
- focus on differences from the replacement and migration advice.
Exceptions are possible if there is a good reason for them.
Initial set
The following API will be omitted with Py_COMPAT_API_VERSION set to
0x030e0000 (3.14) or greater:
-
Omit API returning borrowed references:
==================================== ==============================
Omitted API Replacement
==================================== ==============================
PyDict_GetItem()PyDict_GetItemRef()
PyDict_GetItemString()PyDict_GetItemStringRef()
PyImport_AddModule()PyImport_AddModuleRef()
PyList_GetItem()PyList_GetItemRef()
==================================== ============================== -
Omit deprecated APIs:
==================================== ==============================
Omitted Deprecated API Replacement
==================================== ==============================
PY_FORMAT_SIZE_T"z"
PY_UNICODE_TYPEwchar_t
PyCode_GetFirstFree()PyUnstable_Code_GetFirstFree()
PyCode_New()PyUnstable_Code_New()
PyCode_NewWithPosOnlyArgs()PyUnstable_Code_NewWithPosOnlyArgs()
PyImport_ImportModuleNoBlock()PyImport_ImportModule()
PyMem_DEL()PyMem_Free()
PyMem_Del()PyMem_Free()
PyMem_FREE()PyMem_Free()
PyMem_MALLOC()PyMem_Malloc()
PyMem_NEW()PyMem_New()
PyMem_REALLOC()PyMem_Realloc()
PyMem_RESIZE()PyMem_Resize()
PyModule_GetFilename()PyModule_GetFilenameObject()
PyOS_AfterFork()PyOS_AfterFork_Child()
PyObject_DEL()PyObject_Free()
PyObject_Del()PyObject_Free()
PyObject_FREE()PyObject_Free()
PyObject_MALLOC()PyObject_Malloc()
PyObject_REALLOC()PyObject_Realloc()
PySlice_GetIndicesEx()(two calls; see current docs)
PyThread_ReInitTLS()(no longer needed)
PyThread_create_key()PyThread_tss_alloc()
PyThread_delete_key()PyThread_tss_free()
PyThread_delete_key_value()PyThread_tss_delete()
PyThread_get_key_value()PyThread_tss_get()
PyThread_set_key_value()PyThread_tss_set()
PyUnicode_AsDecodedObject()PyUnicode_Decode()
PyUnicode_AsDecodedUnicode()PyUnicode_Decode()
PyUnicode_AsEncodedObject()PyUnicode_AsEncodedString()
PyUnicode_AsEncodedUnicode()PyUnicode_AsEncodedString()
PyUnicode_IS_READY()(no longer needed)
PyUnicode_READY()(no longer needed)
PyWeakref_GET_OBJECT()PyWeakref_GetRef()
PyWeakref_GetObject()PyWeakref_GetRef()
Py_UNICODEwchar_t
_PyCode_GetExtra()PyUnstable_Code_GetExtra()
_PyCode_SetExtra()PyUnstable_Code_SetExtra()
_PyDict_GetItemStringWithError()PyDict_GetItemStringRef()
_PyEval_RequestCodeExtraIndex()PyUnstable_Eval_RequestCodeExtraIndex()
_PyHASH_BITSPyHASH_BITS
_PyHASH_IMAGPyHASH_IMAG
_PyHASH_INFPyHASH_INF
_PyHASH_MODULUSPyHASH_MODULUS
_PyHASH_MULTIPLIERPyHASH_MULTIPLIER
_PyObject_EXTRA_INIT(no longer needed)
_PyThreadState_UncheckedGet()PyThreadState_GetUnchecked()
_PyUnicode_AsString()PyUnicode_AsUTF8()
_Py_HashPointer()Py_HashPointer()
_Py_T_OBJECTPy_T_OBJECT_EX
_Py_WRITE_RESTRICTED(no longer needed)
==================================== ============================== -
Soft-deprecate and omit APIs:
==================================== ==============================
Omitted Deprecated API Replacement
==================================== ==============================
PyDict_GetItemWithError()PyDict_GetItemRef()
PyDict_SetDefault()PyDict_SetDefaultRef()
PyMapping_HasKey()PyMapping_HasKeyWithError()
PyMapping_HasKeyString()PyMapping_HasKeyStringWithError()
PyObject_HasAttr()PyObject_HasAttrWithError()
PyObject_HasAttrString()PyObject_HasAttrStringWithError()
==================================== ============================== -
Omit
<structmember.h>legacy API:The header file
structmember.h, which is not included from<Python.h>
and must be included separately, will#errorif
Py_COMPAT_API_VERSIONis defined.
This affects the following API:==================================== ==============================
Omitted Deprecated API Replacement
==================================== ==============================
T_SHORTPy_T_SHORT
T_INTPy_T_INT
T_LONGPy_T_LONG
T_FLOATPy_T_FLOAT
T_DOUBLEPy_T_DOUBLE
T_STRINGPy_T_STRING
T_OBJECT(tp_getset; docs to be written)
T_CHARPy_T_CHAR
T_BYTEPy_T_BYTE
T_UBYTEPy_T_UBYTE
T_USHORTPy_T_USHORT
T_UINTPy_T_UINT
T_ULONGPy_T_ULONG
T_STRING_INPLACEPy_T_STRING_INPLACE
T_BOOLPy_T_BOOL
T_OBJECT_EXPy_T_OBJECT_EX
T_LONGLONGPy_T_LONGLONG
T_ULONGLONGPy_T_ULONGLONG
T_PYSSIZETPy_T_PYSSIZET
T_NONE(tp_getset; docs to be written)
READONLYPy_READONLY
PY_AUDIT_READPy_AUDIT_READ
READ_RESTRICTEDPy_AUDIT_READ
PY_WRITE_RESTRICTED(no longer needed)
RESTRICTEDPy_AUDIT_READ
==================================== ============================== -
Omit soft deprecated macros:
====================== =====================================
Omitted Macros Replacement
====================== =====================================
Py_IS_NAN()isnan()(C99+<math.h>)
Py_IS_INFINITY()isinf(X)(C99+<math.h>)
Py_IS_FINITE()isfinite(X)(C99+<math.h>)
Py_MEMCPY()memcpy()(C<string.h>)
====================== ===================================== -
Soft-deprecate and omit typedefs without the
Py/_Pyprefix
(getter,setter,allocfunc, …), in favour of new ones
that add the prefix (Py_getter, etc.) -
Soft-deprecate and omit macros without the
Py/_Pyprefix
(METH_O,CO_COROUTINE,FUTURE_ANNOTATIONS,WAIT_LOCK, …),
favour of new ones that add the prefix (Py_METH_O, etc.). -
Any others approved by the C API workgroup
If any of these proposed replacements, or associated documentation,
are not added in time for 3.14.0b1, they’ll be omitted with later versions
of Py_COMPAT_API_VERSION.
(We expect this for macros generated by configure: HAVE_*, WITH_*,
ALIGNOF_*, SIZEOF_*, and several without a common prefix.)
Implementation
TBD
Open issues
The name Py_COMPAT_API_VERSION was taken from the earlier PEP;
it doesn’t fit this version.
Backwards Compatibility
The macro is backwards compatible.
Developers can introduce and update the macro on their own pace, potentially
for one source file at a time.
Discussions
- C API Evolutions:
Macro to hide deprecated functions <https://github.com/capi-workgroup/api-evolution/issues/24>_
(October 2023) - C API Problems:
Opt-in macro for a new clean API? Subset of functions with no known issues <https://github.com/capi-workgroup/problems/issues/54>_
(June 2023) Finishing the Great Renaming <https://discuss.python.org/t/finishing-the-great-renaming/54082>_
(May 2024)
Prior Art
Py_LIMITED_APImacro of :pep:384“Defining a Stable ABI”.- Rejected :pep:
606“Python Compatibility Version” which has a global
scope.
Copyright
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.