Hi everyone,
I’m a member of the SciPy team. Lately, I have touched a lot of extension module code due to some internal compiled code changes. Hence I have pondered about this part of the code for a brief period as much as my constraints about FOSS contributions allow for.
From SciPy point of view, this is not much of a design decision we need to make, just as the multi-phase initialization discussion lead to, if you folks say jump, we’ll jump since we typically don’t know the implications (yet) and so far we have been copy pasting what is provided to us.
Besides, our problems are very rarely at the extension module wrapper code as we typically write it once and forget about it. I have not tried yet but if the shim is working then I can go around and copy paste things as we started doing for PEP489 MAINT: Use multi-phase initialisation (PEP 489) · Issue #23067 · scipy/scipy · GitHub though not sure if PEP793 shim also includes the details about multi-phase initialization and the sub-interpreter support.
I would like to see all of them are handled in unison such that they use all the same shim examples or common talking points. Currently it reads as some insider lingo as none of the object types or the function names are documented for outsiders but expects the reader to know already the inner workings of CPython.
Just to give a simple example out in the wild, I have been under the impression that the #ifdef guards are needed to signal for subinterpreters and/or free threading versions; here is something we rewrapped very recently
I already confess that I don’t know what I am doing here but following two, three guidelines from different PEPs and crossing my fingers, hoping for the best. This part of the API is quite difficult for me to decrypt as I can’t quite get to the starting point and many things are referring to each other in the documentation. Porting guide is also referring to prior art so that’s not too much of help either. But these are all “me” problems, not really objectively evaluated.
However the example code provided PEP 793 – PyModExport: A new entry point for C extension modules | peps.python.org is convoluting a lot of things already starting from its top comment section. That needs a very simple example first, having a function C code and the related _mymodmodule.c file or something along those lines. Because that’s the common situation not this global state one (especially when free-threading is coming soon).
If we are to move the Python compiled code wranglers to the new versions, these multiple issues happening at the same time are the details we should have, so that the transitions happen as smooth as possible. However typically each PEP is leaning towards hygiene (“xxx is out of scope of this PEP” is a very common phrase) and a bit underestimating the state of affairs that different PEPs causing some “merge conflicts” as they operate around the same places of the extension module code. But then again, these are issues we can work around. So no actual big issues here.
What is bothering me, not on behalf of SciPy team but personally, is where this is all going. I have not seen a roadmap of all these issues converging to a common point say Python 3.19 or v4 or whatever that all these finally come together and form a stable API (or two) and unicorns and rainbows from there on.
There is free-threading, there is sub-interpreters, there is stable api, and strict api discussions, multi-phase initializers all stemming from different Python versions and targeting different Python versions for graduation (see again the #ifdef guards in the block above).
It seems to me that the folks who are doing one PEP are not really involved in the other (or from outside that is the picture). Or the overseers of these PEPs are not coordinating with end result in mind. I guess I would call this a project management issue. If this is not the case then my apologies in advance as I did not scan every location where discussions being held.
Also the void* APIs are always a scary thing to have and typically are future grenades. Hence I hope there is enough discussion happened around that choice.
TL;DR: If you give me a shim, I’ll paste around and be done with it.
But hopefully you will also consider the number of wheels we are providing Release SciPy 1.16.2 · scipy/scipy · GitHub that it is not that straightforward to add a few more binaries for each t enabled thing, that’s a lot of binaries to provide on PyPI.