PEP 674 "Disallow using macros as l-values" and Python 3.11

Core Development
1

Hi,

Recently, CAM-Gerlach asked me the status of my PEP 674. Oh. I forgot about that one…

As currently written in PEP 674, the PEP is basically already implemented in Python 3.11. Problem: last February, the SC requested some changes to have a smoother transition towards the new API:

  • Document that the old way to set Py_SIZE() with Py_SIZE(obj) = size is deprecated,
  • Wait until all supported Python versions provide Python Py_SET_SIZE() (Python 3.9: so wait until Python 3.8 is no longer supported),
  • And only then disallow Py_SIZE(obj) = size (fail with a compiler error).

PEP 674 is mostly about changing two macros, Py_TYPE() and Py_SIZE(). The Py_TYPE() change was approved by the previous SC (last year), and the new SC doesn’t want to change that. The question is about the Py_SIZE() change.

I disagreed with the SC statement and explained that in practice, projects affected by the Py_TYPE() change are already affected by the Py_SIZE() change, so changing both at once is easy. Moreover, all the work (changing Python and updating PyPI top 5000 projects) was already done.

My plan for PEP 674 was to write a new PEP explaining my overall long term vision for the C API: something like PEP 620 “Hide implementation details from the C API” and the HPy project. Such PEP should justify why waiting for 5 years just for changing is a single macro would be a big issue shortly, since Python must change (see Python 3.11 C API changes pulled by the optimization work).

But I got swallowed by Python 3.11 regressions at work. I helped investigating a bunch of Python 3.11 issues, fix Python and help projects to be adapted to Python 3.11. I helped adding many C API functions filling the hole of features removed by Python 3.11 C API changes (structures were made opaque like PyFrameObject, or PyCodeObject changes)

So here we are: PEP 674 is basically implemented since Python 3.11 alpha 1, and the Python 3.11 release candidate 1 got released recently with the changes.

What should be done with PEP 674? Revert the Py_SIZE() change in Python 3.11? Update the PEP? Honestly, I’m now confused, I’m not sure what’s the best.

When PEP 674 issues were discussed, it was proposed to keep the incompatible changes in beta versions and then revert it before the final release. Someone said that a project does something like that (numpy? I failed to find the email).

By the way, I don’t recall any recent bug reports about Py_TYPE() or Py_SIZE() changes, it seems like people just managed to updated their C extensions. Moreover, the pythoncapi-compat project was moved under the GitHub Python organization (python/pythoncapi-compat) and contains a tool to automate the migration (without losing support for old Python versions).

2 Likes
2

Sounds like the first thing to do is to ensure that there is a release-blocker issue open about this until it is resolved one way or another.

3 Likes
3

… and to be honest, I think the change should be reverted - having something go in, when the SC explicitly said that it should not, simply because someone implmented it into main before getting approval and then forgot to remove it, is pretty bad :slightly_frowning_face:

Of course, removing it at this late stage would be a huge pain for the RMs, but I think we need to acknowledge that conversely, not doing so undermines the authority of the SC.

@vstinner - why did you merge this without SC approval in the first place?

1 Like
4

That’s something for the RM to decide :wink:

But as someone that was affected by this, I don’t think it is absolutely necessary to roll back this change. Adjusting C extensions for this change is pretty easy and also easy to do cleanly in a backward compatible way. Rolling back this change also has its risks, which includes breaking code that has already adjusted to these changes (depending on how that was done).

4 Likes
5

I’ve added this to the SC’s agenda to figure out what to do about this mistake. There’s at minimum a lesson to be learned here about keeping track of things and using issues as reminder is probably labels to prevent slip-ups from occurring in the future, regardless of whether anything gets reverted.

1 Like
6

For 5 years, I made many changes in the C API. At the end, I started to do change which might affect C extensions (backward compatibility). I changed Py_TYPE() and Py_SIZE() and adapted third-party C extensions to these changes. But some people started to complain about these changes so I wrote a PEP to explain the rationale and communicate on the change.

See PEP 674 – Disallow using macros as l-values | peps.python.org I made these changes in May 2020, but revert them because it broke too many 3rd party extensions. When enough extensions were ready, I made the change again. And then I wrote a PEP.

I know that it’s common that a PEP is already implemented before being written down :slight_smile:

7

Adding abstraction (Py_SET_REFCNT, Py_SET_SIZE, Py_SET_TYPE) layers to no longer access directly PyObject members allows nice enhancements like the nogil project and PEP 683 Immortal objects. For the long term, it should give the ability to move away completely from PyObject and switch to another more efficient structure (or even not have objects at all but tagged pointers, for example).

8

If we decide to revert the change, here is a change for the main branch which should be then be backported to 3.11: gh-89639: Revert Py_SIZE() static inline function by vstinner · Pull Request #96119 · python/cpython · GitHub

9

Ah, I noticed that I forgot something about PEP 674 context.

When I wrote the first draft of PEP 670: Convert macros to functions in the Python C API, I put Py_TYPE() and Py_SIZE() in this PEP. But the problem was how to explain the rationale for Py_TYPE() and Py_SIZE() changes in the same PEP.

It was already tricky to explain PEP 670 rationale (convert macros to functions) and justify that PEP 670 has no significant impact on performance.

PEP 674: Disallow using macros as l-values is built on top of PEP 670. PEP 670 was approved in April 2022, while PEP 670 was being implemented and PEP 674 was still being discussed. Then as I wrote, I got swallowed in other C API issues related to the optimization work.

10

I suggested using the pythoncapi-compat project to projects which wanted to use Py_SET_SIZE() on old Python versions, but many projects went with their own solution. For example, I vaguely recall a project which redefines Py_SIZE macro to get back the Python 3.10 Py_SIZE macro which can be used as a l-value. I didn’t keep track of the diversity of solutions chosen by all projects, but PEP 674 links to many projects (and their fix). It’s not easy to guess if reverting the change now can break a project.

11

I don’t use pythoncapi-compat in my projects, in part because these project are much older that your compatibility headers but also because I don’t want to track another project to see if something changed.

IMHO that would be the wrong solution to deal with changes like this, but at least its compatible with a revert. I’m more worried abut projects that switch to Py_SET_SIZE and define that macro themselves if Py_VERSION_HEX is before 3.11. That would break if the revert entails not just changing the definition of Py_SIZE but also reverts adding Py_SET_SIZE.

As a datapoint: I’m keeping Py_SET_SIZE in my projects, I’ve never really liked how using Py_SIZE as an lvalue looks and like the new macro better.

1 Like
12

My PR only changes Py_SIZE() (convert it back to a macro), it leaves Py_SET_SIZE() unchanged.

13

About pythoncapi-compat, you don’t have to update it frequently. You only have to update it if it becomes incompatible with a newer version of CPython or PyPy, or if you want to get a new compatibility function (use a new Python version on old Python versions). In short, this is no need to update it unless there is a problem.

2 Likes
14

Hi,

I am writing this message on behalf of the Steering Council.

The Steering Council has discussed this in one of the recent meetings. Although we have to say that we are very unhappy that this happened, after discussion with the release management team, the Steering Council have decided not to revert these changes in 3.11 to not affect the stability of the release as at the time this was discussed, the 3.11 branch was already in release candidate 1.

This doesn’t mean that the SC thinks this outcome is ok. We want to emphasise that this outcome is not acceptable, and neither is the way this was done. Changes that require a PEP should never be included in a stable branch before the PEP is done and if they ever land in one of the supported branches without the backing PEP, they should be reverted immediately.

The Steering Council would like to remind everyone that respecting the process is fundamental to a good working environment in the core development team and failing to do so not only makes Python less stable but can also affect the perception of users on the project and its will also make life more difficult to other core devs and colleagues.

Thanks for your understanding.

In the name of the Steering Council,
Pablo Galindo Salgado

4 Likes
15

Sorry about that :frowning: I didn’t come back to PEP 674 until recently for personal reasons. And it seems like nobody else noticed the issue in Python 3.11 before I did.

The situation was special. I wrote the PEP after the changes were made, since I knew that these were incompatible changes, some people had concerns and I didn’t want having the discussion on a bug tracker. I wanted a broader discussion with a PEP which is an efficient tool to take a decision. The history of the changes is documented in the PEP 674: Implementation.

The SC didn’t reject the PEP. Right now it still has the “Draft” status by the way. The SC asked to introduce incompatible changes differently: document well that the old API is deprecated and wait until the new API is available in all supported Python versions.

Next time, I will revert first, before an agreement can be found (when the SC will give his decision). Also, I suggest the SC to request more explicitly an immediate revert next time. Sadly, it’s too easy to lose track of an issue and forget about it. In the meanwhile, nobody reported any issue about the Py_SIZE() change, so I forgot about it. Well, I fixed most projects affected by PEP 674 before writing the PEP, so I’m not surprised :wink:

2 Likes