At the core dev sprint in Brno we decided to create a C API Working Group. The initial WG members have written a draft process PEP: PEP 731. The GitHub file is here.
This is the official discussion thread for this PEP! Please review the draft PEP and discuss it here. We’ll follow the usual (sometimes chaotic) PEP discussion process and once we feel there’s rough consensus we’ll seek approval of the Steering Council.
There are also two new GitHub repos for conversations (although what will happen to these depends on how this PEP fares):
API Evolution (for gradually evolving the existing C API – we expect another PEP to come out of this)
API Revolution (for disruptive proposals, each of which will require a PEP by itself to make any chance)
And of course there’s the earlier problems repo, which will be turned into an informational PEP (regardless of whether PEP 731 is accepted).
This looks great - a promising venue for a more organized and intentional evolution of the C API!
A few thoughts and questions on the PEP:
The working group has at least three members, comprised of prominent Python core developers. The members should consider the needs of the various stakeholders carefully.
Considering “the various stakeholders” include “all users of Python, especially maintainers of code that uses the C API […]” - why limit membership in the working group to core developers?
I do think it’s crucial to include “prominent core devs” (btw what is “prominent” exactly?) in order to represent the interest of the maintainers of the C API. I can’t imagine such a working group without @vstinner for example. Why not include members representing “prominent stakeholders”, like HPy, PyPy, Cython, pybind11, … - even if they are not core developers?
Also, what happens if for whatever reason the working group membership drops below 3 members? Is it disbanded? Is it “frozen” until membership grows back?
The working group’s mandate is to ensure that the Python C API is suitable for all users of and contributors to the API
Is the scope here well-defined (or intentionally fuzzy)?
For example, should the working group govern the “technically public (as in, can be used by invoking an incantation of includes and defines) but not officially public” API? What about the unstable API?
Concretely, will core devs need to seek the approval of the working group when introducing, removing, or changing “private” (_Py...) or unstable APIs?
To me, ‘prominent’ means respected, trusted to consider all stakeholders and balance change versus stability. I not sure what adjective would be better.
I expect stakeholder representatives to be biased and argue in favor of the needs of their particular projects. That is their self-chosen job, so to speak. I expect that they will continue to contribute their viewpoints as further PEPs are proposed. I was not at the sprint, but agree that the proposed group is more likely to produce viable proposals than a mixed group of several stakeholders.
I don’t think that the plan is to hold any PR touching Include/ until all members of the C API Working Group approve it. It’s more about defining guidelines and seeking consensus on C API changes, and taking decisions on technical questions on the C API, than reviewing every single C API change.
For example, I’m interested by no longer allowing to add new private API (Py prefix, but not doc nor tests) in the C API: an API should be either public (Py prefix, exported, tested, documented) or internal (_Py prefix, usually not exported, need Py_BUILD_CORE, special include with pycore prefix). PyUnstable is part of the public API (Py prefix, exported, tested, documented), only the backward compatibility policy changes (PEP 689).
The Steering Council can step in and decide. Like in any unforeseen/ambiguous/harmful situation.
This PEP doesn’t really need to be a bulletproof charter.
IMO: No, but it’d be good to follow the public-API guidelines where they make sense.
IMO, if they are trusted enough, they should be promoted to core developers.
The working group is small, so its members must take other viewpoints than their own into account. We can’t add every stakeholder directly.
It might be useful to define the terms “private”/“internal” in a PEP, devguide or docs. Their meaning seems to subtly change every time I look at them… Edit: I filed this as #33 in the WG repo.
I second the idea of having someone prominent from the C extension community join the WG, but happy for the WG itself to add such members.
I don’t know whether the requirement to be a core developer hinders adding such members or not. I would assume that, e.g., a core numpy developer, already fulfills many of the requirements and simply being willing to put time into the C API WG might make them qualify?
Since the WG would be proposing changes that are only directly binding on the core development team, I’m okay with the core developer requirement.
If the WG doesn’t appear to be soliciting contributions from prominent users of the API and factoring them in, that’s a reason to go to the steering council with a complaint.
Being directly on the WG isn’t a prerequisite to contribute. It’s just a burden of having to take responsibility for the decisions and how they impact other competing interests.
This feels off-topic for the WG charter, but I wholeheartedly agree that these terms can be used confusingly. The WG, once established, should discuss this topic and come up with agreed-upon, useful and future-proof definitions of the terms. But not right now.
I’ve taken this to the Steering Council. I’ve not seen significant pushback on the current text (there was some commentary that Steve responded to), so I think it’s ready for this step.
For me, this working group was designed to solve a problem: the Python community is great to list problems, list solutions, but not to take decisions. The Steering Council called to create such group to delegate since they don’t want to take decisons on every C API question. Maybe on some major ones, we will see
for example. Why not include members representing “prominent stakeholders”, like HPy, PyPy, Cython, pybind11, … - even if they are not core developers?