As currently written in PEP 674, the PEP is basically already implemented in Python 3.11. Problem: last February, the SC requested some changes to have a smoother transition towards the new API:
- Document that the old way to set Py_SIZE() with
Py_SIZE(obj) = sizeis deprecated,
- Wait until all supported Python versions provide Python Py_SET_SIZE() (Python 3.9: so wait until Python 3.8 is no longer supported),
- And only then disallow
Py_SIZE(obj) = size(fail with a compiler error).
PEP 674 is mostly about changing two macros, Py_TYPE() and Py_SIZE(). The Py_TYPE() change was approved by the previous SC (last year), and the new SC doesn’t want to change that. The question is about the Py_SIZE() change.
I disagreed with the SC statement and explained that in practice, projects affected by the Py_TYPE() change are already affected by the Py_SIZE() change, so changing both at once is easy. Moreover, all the work (changing Python and updating PyPI top 5000 projects) was already done.
My plan for PEP 674 was to write a new PEP explaining my overall long term vision for the C API: something like PEP 620 “Hide implementation details from the C API” and the HPy project. Such PEP should justify why waiting for 5 years just for changing is a single macro would be a big issue shortly, since Python must change (see Python 3.11 C API changes pulled by the optimization work).
But I got swallowed by Python 3.11 regressions at work. I helped investigating a bunch of Python 3.11 issues, fix Python and help projects to be adapted to Python 3.11. I helped adding many C API functions filling the hole of features removed by Python 3.11 C API changes (structures were made opaque like PyFrameObject, or PyCodeObject changes)
So here we are: PEP 674 is basically implemented since Python 3.11 alpha 1, and the Python 3.11 release candidate 1 got released recently with the changes.
What should be done with PEP 674? Revert the Py_SIZE() change in Python 3.11? Update the PEP? Honestly, I’m now confused, I’m not sure what’s the best.
When PEP 674 issues were discussed, it was proposed to keep the incompatible changes in beta versions and then revert it before the final release. Someone said that a project does something like that (numpy? I failed to find the email).
By the way, I don’t recall any recent bug reports about Py_TYPE() or Py_SIZE() changes, it seems like people just managed to updated their C extensions. Moreover, the pythoncapi-compat project was moved under the GitHub Python organization (python/pythoncapi-compat) and contains a tool to automate the migration (without losing support for old Python versions).