@encukou Thank you for your work on PEP 793 and thanks to everyone for your insighful comments.
After reviewing the PEP and the discussion several times, the Steering Council believes that while the conversation has been interesting and technically thorough, it has been primarily dominated by members of the C API Working Group, where there are some disagreements on implementation details and timing. We feel we don’t have sufficient community data to make an informed decision at this point.
Given the potential impact of this PEP on the broader Python ecosystem, we would like to gather input from additional communities that will be affected by this, particularly:
We understand that getting input from busy maintainers can be challenging, and of course, if people don’t respond that won’t be a blocker to moving forward.
@encukou could you help us reach out to these communities and gather their perspectives on PEP 793? Once we have this broader input, please post the feedback either here in this discussion thread or in the SC tracker, and we can continue discussing it.
If you have any concerns, pleas reach out here, in the SC tracker or directly via email and we ar happy to consider alternatives. Thanks again for your continued work on this area!
As maintainer of PyArrow, I would defer to Cython maintainers as that is what we use for the Python-level scaffolding (we do have C++ code directly calling the CPython API, but it does not create any modules). @scoder@da-woods
However, I have minor comments on the PEP:
When CPython accepts a pointer where it doesn’t expect to mutate the pointed data, the pointer should be qualified as const (especially when the PEP recommends it point to a static constant)
The paragraph about the new token mechanism could be clearer. Is the token supposed to be entirely opaque? If so, why is there this weird special case where “when the address of a PyModuleDef is used as a module’s token, the module should behave as if it was created from that PyModuleDef”. Does CPython actually check for that?
Why PyModule_GetToken? Why not require that module authors always choose their token themselves using the dedicated slot? It seems like one should call PyModule_GetToken in the module exec and then store the token statically… somewhere?
It would be nice if there was an example of using a module’s token to fetch the module state.
I would also expect a small subsection about the token mechanism’s performance for accessing module state.
The primary motivation seems to be unifying the ABI of the regular and free-threaded variants of future CPython releases, with the module initialisation getting in the way.
Quick question, was it considered to decouple PyModuleDef_HEAD_INIT from the Python object header? (I didn’t find a mention of it in this discussion page.) I don’t see a reason why it needs to be the same as a future unified object header. It could keep the current stable ABI layout even in a freethreaded build, thus avoiding the need for a completely new module init API.
Changing PyModuleDef_HEAD_INIT was my first intuition, too, and it doesn’t seem unreasonable to me. I don’t think PyModuleDef needs to be an object – it just needs to be distinguishable from one, and we control its header. Instead of adding yet another way of initialising extension modules, we could make PyModuleDefa more flexible struct, possibly configured through its no-longer-ob_type field.
In fact, can’t we just allow returning a third type of object? – EDIT: Ah, right, we can’t, because the object layout is different. That was the original problem.
well, two well defined ones plus a really weird one ↩︎
Adding an interface that doesn’t return a PyObject* seems like it’s necessary to make freethreading work with the Stable ABI. It’s possible that there might be hacky ways to do it with the existing interface, but a new interface seems like a clean way of doing it. I don’t think it’ll be too hard for something like Cython to use. And I think there’s demand for Stable ABI + freethreading, so it’s something I’d try to support relatively quick.
I’m pretty neutral on the “move everything to be slots instead of struct members” change. From a user’s point of view it doesn’t make a huge amount of difference.
Edit: Ignore the following bullet-point. Petr points out that I misunderstood… The token mechanism seems like a welcome addition. It’s broadly just PyState_FindModule but will work on multi-phase modules. Cython’s module-state support is currently still a bit limited, but what I’ve had to implement to get something like PyState_FindModule is fairly nasty and I’ll be glad to ifdef it out on future versions. (I imagine the performance of it will be broadly similar to PyState_FindModule - my impression of that is it wasn’t as bad as you might think for most uses, but I’ve not done rigorous benchmarking).
That’s required so that PyType_GetModuleByToken is equivalent to PyType_GetModuleByDef, which should ease migration from old code.
CPython itself doesn’t check for it.
If you have a module object and want to check if its state has “your” memory layout, you should compare its token to an expected value. You get the module object’s token using PyModule_GetToken. The expected value should usually be the address of the static slots array – which is the PyModExport default.
For dynamic creation the slots array might not be static, so you need to use something else if you want the token functionality.
It needs to be distinguishable from anyPyObject memory layout, past and future.
Past means two flavours – classic and free-threaded – which already have the ob_type field in different spots. For the future, there’s pressure to reorganize the struct and put every bit to good use; compatibility with PyModuleDef puts some constraints on that.
Another issue is that the init function returns a PyObject* – users are currently free to call it and call e.g. Py_TYPE on the result. In fact, that’s how you’d distinguish between single-phase and multi-phase init in a custom loader.
I’m afraid it doesn’t do what you describe. For PyModule_GetToken you need to already have the module object; for PyType_GetModuleByToken you need to have a type from the module (or a subclass, or an instance). In “classic” multi-phase init, you get the same with PyType_GetModuleByDef, if you use a static def.
It’s not a PyState_FindModule, because CPython doesn’t have a one-to-one (interpreter, extension_dll)→(module object) mapping. If you need that mapping you need to build it yourself; I assume that’s what the “fairly nasty” code is.
You can iterate over sys.modules, check the token of each module against a given token, and call PyModule_GetState once you get a match. Is that the operation you want? (That’s a real question; I want to confirm we’re talking about the same thing.)
Alas, the PEP doesn’t make this easier.
Ah fair enough - Cython assumes (and enforces) a one-to-one mapping but of course that isn’t generally true. That doesn’t change my opinion on the rest of it (which can be summarised “this seems like an acceptable amount of disruption to solve a problem that is blocking freethreading+stable ABI”)
It’s one part of the solution, it doesn’t solve it yet. My opposition to doing it now is because it’s a non-zero amount of disruption that doesn’t (quite) do what you summarised it as.
(In other words, it’s an amount of disruption now with the promise of more future disruption, and I’d prefer to merely promise future disruption without causing any now.)
I’m a member of the SciPy team. Lately, I have touched a lot of extension module code due to some internal compiled code changes. Hence I have pondered about this part of the code for a brief period as much as my constraints about FOSS contributions allow for.
From SciPy point of view, this is not much of a design decision we need to make, just as the multi-phase initialization discussion lead to, if you folks say jump, we’ll jump since we typically don’t know the implications (yet) and so far we have been copy pasting what is provided to us.
Besides, our problems are very rarely at the extension module wrapper code as we typically write it once and forget about it. I have not tried yet but if the shim is working then I can go around and copy paste things as we started doing for PEP489 MAINT: Use multi-phase initialisation (PEP 489) · Issue #23067 · scipy/scipy · GitHub though not sure if PEP793 shim also includes the details about multi-phase initialization and the sub-interpreter support.
I would like to see all of them are handled in unison such that they use all the same shim examples or common talking points. Currently it reads as some insider lingo as none of the object types or the function names are documented for outsiders but expects the reader to know already the inner workings of CPython.
Just to give a simple example out in the wild, I have been under the impression that the #ifdef guards are needed to signal for subinterpreters and/or free threading versions; here is something we rewrapped very recently
I already confess that I don’t know what I am doing here but following two, three guidelines from different PEPs and crossing my fingers, hoping for the best. This part of the API is quite difficult for me to decrypt as I can’t quite get to the starting point and many things are referring to each other in the documentation. Porting guide is also referring to prior art so that’s not too much of help either. But these are all “me” problems, not really objectively evaluated.
However the example code provided PEP 793 – PyModExport: A new entry point for C extension modules | peps.python.org is convoluting a lot of things already starting from its top comment section. That needs a very simple example first, having a function C code and the related _mymodmodule.c file or something along those lines. Because that’s the common situation not this global state one (especially when free-threading is coming soon).
If we are to move the Python compiled code wranglers to the new versions, these multiple issues happening at the same time are the details we should have, so that the transitions happen as smooth as possible. However typically each PEP is leaning towards hygiene (“xxx is out of scope of this PEP” is a very common phrase) and a bit underestimating the state of affairs that different PEPs causing some “merge conflicts” as they operate around the same places of the extension module code. But then again, these are issues we can work around. So no actual big issues here.
What is bothering me, not on behalf of SciPy team but personally, is where this is all going. I have not seen a roadmap of all these issues converging to a common point say Python 3.19 or v4 or whatever that all these finally come together and form a stable API (or two) and unicorns and rainbows from there on.
There is free-threading, there is sub-interpreters, there is stable api, and strict api discussions, multi-phase initializers all stemming from different Python versions and targeting different Python versions for graduation (see again the #ifdef guards in the block above).
It seems to me that the folks who are doing one PEP are not really involved in the other (or from outside that is the picture). Or the overseers of these PEPs are not coordinating with end result in mind. I guess I would call this a project management issue. If this is not the case then my apologies in advance as I did not scan every location where discussions being held.
Also the void* APIs are always a scary thing to have and typically are future grenades. Hence I hope there is enough discussion happened around that choice.
TL;DR: If you give me a shim, I’ll paste around and be done with it.
But hopefully you will also consider the number of wheels we are providing Release SciPy 1.16.2 · scipy/scipy · GitHub that it is not that straightforward to add a few more binaries for each t enabled thing, that’s a lot of binaries to provide on PyPI.
The ultimate point of this (one or two steps on) is that a single Stable ABI build can serve freethreading/non-freethreading. So in principle it would allow SciPy to reduce the number of binaries to ship and improve the situation.
However: I believe SciPy is heavily using PyBind11, which doesn’t aim to support the Stable ABI. That probably means you won’t benefit from this PEP, but also that you can safely ignore it for the time being and keep using your existing module initialization code.
Yeah, that’s a problem. By including those Py_mod_multiple_interpreters and Py_mod_gil flags you’re promising that your codebase behaves a certain way; if you just copy & paste them then you’ll get crashes in edge cases. If you don’t add them, you’ll get safe defaults, and clearer error messages in cases that require them.
I hear you. Sadly, the PEP can’t serve user documentation and porting guide. The “interesting” parts of a typical extension module aren’t affected by this PEP.
The goal is to put the info you need in documentation, rather than PEPs – those document individual change documents so they have those drawbacks by design.
You’re perfectly right. Unfortunately, the project manager would probably also need a crystal ball :)
You can have stability now – this API is optional, as are the Py_mod_multiple_interpreters & Py_mod_gil slots, etc. But if you omit them, you’ll not support the latest features.
For the tokens? Those should be fine as they’re only used for pointer comparison.
Or for the slots? That’s an existing design, which I’d like to change but I’m worried that it’d mean even more churn.
I think @encukou and I are most broadly involved across all of these, and as you’ll see (if you find the discussions), we have quite differing opinions about how to approach the management of these changes.
All I’ll say here is thank you for speaking up, I feel you’ve confirmed a number of things that I spend a lot of time arguing for (though Petr probably feels the same, hopefully about different parts of your feedback than me ), and that gives me confidence that I’m not just wasting my time or being obstructionist.[1]
As I push for a bounded, scheduled, “all-at-once” API/ABI modernisation transition rather than the annual trickle of little changes that trick you into modernisation without ever admitting it. ↩︎
I did not mean to walk into a minefield with my comments but let me try to unify you both with my extra comments But before that spicy bit, a quick question;
Do you mean I should remove the guards? In other words am I understanding correctly that you would prefer these features to be turned on by default and lead to errors when built from non-compatible architectures/versions?
Back to spice
These are all fine and I think for every sufficiently sophisticated project, it is bound to happen that folks will start to disagree. You can find similar frictions within SciPy too even for the order of a keyword etc. Hence I don’t necessarily shy away from disagreement per se.
But I should make myself clearer;
It is fine either way, piecemeal or at one-shot, coming to a state where things are X-capable, X being free-threading, subinterpreter, etc. And I acknowledge that every issue is a potential candidate to be the center of attention of the folks who care about it deeply.
What I wanted to mention is not just you folks but it seems like steering council also weighed on this above and mentioned that, kind of tongue-in-cheek, “we want you to get the opinions of the libs involved”. That’s fine.
As I mentioned previously, from C code point-of-view, very little change is needed; arrays and structs vs objects, someone comes and says “use this struct and stop using that function”. To that I’d say “OK cool” because it’s OK either way; this change means very little to me. In other words, I’d say “OK cool“ to anything you folks say because all I am seeing is a shim to be included and it makes no difference to us because typically the extension module wrapper code does not cause any issues.
But why you are offering this; free-threading, or switching to Stable API to have good interoperability or improve Linux libc, musl fragmentation etc. I would not know from these PEPs.
So optional means are they going to stay optional forever as a nice feature offering or is this something we want to move towards eventually but “it is early for enforcement” kind of optional? Our users are typically involve cutting-edge/early-adapting number crunching folk hence our door is knocked very quickly when you folks roll a feature out. Hence we need to know.
I am aware how much work goes into PEPs so I did not want to imply ill-intentions or sloppiness at all. But what is the end goal of the steering council by accepting these PEPs? Are we preparing for a GIL-less state or whatever the case might be? If two PEPs are rigorous enough, even though they rub each other, do they get in? And if they are accepted shouldn’t they be implemented in a coordinated fashion since they are both affecting C-API?
That is to say, are they being accepted based on their individual statements or are they serving a generic roadmap so that they solve a piece of the puzzle albeit partially? I find it hard to believe that these PEPs 793, 489, 652, and many more are just individual attempts and hence require crystal ball at the time of their inception. I would expect C-API WG or SC has at least a rough picture in mind to drive these efforts such that we package maintainers also know what is about to come and help you folks realize that roadmap.
To re-iterate; it’s totally fine from my side, to get the latest and greatest shim and implement it in our modules, but it would be much nicer if different PEP authors work on a single shim and distribute that collectively such that multiple issues resolve together, and we get to the destination quicker.
I’m not sure what you mean “default”, but, yeah, I’d prefer if you don’t copy-paste them without understanding what they’re asserting about your code :)
Not including them might lead to errors, but the error messages will be nicer than thread-safety bugs or subinterpreter cross-talk issues.
Yeah, but that might be because you might not be the target audience of the PEPs. (Unless you want to help steer the evolution – but that’s a different hat to wear.)
What’s New documents, and porting guides, might be better for you as a maintainer. And we should make those more useful – preferably with your feedback.
As you hint, it’s more about users pushing[1]. I’d say that if they knock saying that your module doesn’t work with free-threaded subinterpreters, then it’s time to switch to API that enables it. The concerned user(s) might also help with explaining the use case, and testing the result in their setup.
I think there are several ideas of how the future should look, and many ideas about what the priorities should be. (Free-threading in particular pushed a bunch my “nice-to-have”s to “needed now”, and PEP 793 is definitely one of those.)
Progress happens when and where the rough roadmaps align.
The trouble here is that the perfect shim depends on your use case :(
A binding generator like Cython or nanobind can go wild on indirection, macros and version/feature checks; a smaller library with a single module might instead want to omit the features they don’t need, or run a bit more slowly.
And on the other side, IMO we need a few cohorts of adopters, starting from the early ones. Otherwise we’re driving blind. A single big update makes it hard to correct small mistakes.
Not that CPython can’t push, but I think it’s better to keep even the ugly things as stable as it’s practical. ↩︎
I think I don’t understand this part. Why is it depending on my use case? I was hoping for something like; say a foobar.c function is given and we want to have it wrapped into an extension module. What is needed for package authors is a file that showcases different features at the same place and not in 10 different PEPs that we need to go around and collate. Consider a pseudo example as below
/**
Compile with the following added to C-args for Stable/Limited API, say for Py 3.12
Py_LIMITED_API=0x030C0000
*/
static struct
PyModuleDef_Slot mylib_slots[] = {
{interpreter stuff, Py_MOD_long_macro}, // Remove if no sub-interpreter support
{Py_mod_gil, Py_MOD_GIL...}, // Remove if no free-threading needed
{Py_some_other_thing, Py_MOD_ANOTHER_MACRO}, // This one is for bla
...
{0, NULL}
}
// Other cryptic settings
static struct
PyModuleDef moduledef = {
.m_base = PyModuleDef_HEAD_INIT, // Don't question, just include this
.m_name = "mylib", // lib name
.m_size = 0, // if you are not sure just keep the 0
.m_methods = mylib_methods, // Methods exposed
.m_slots = mylib_slots, // Slots slotted
...
}
// More stuff
With such an example, every passed PEP or new C-API thing is added to the same common extension module example and/or explicitly stated how it is going to play nice with the rest of the new features. If Cython is needed then I’ll go and talk to Cython folks how to go about it. Same with pybind11/nanobind. You don’t need to worry about all that since it’s not your problem to solve all that. But I want the CPython parts lego’d up nicely such that CPython drives coding practices by explicit examples and not with prose.
Otherwise just like I did above, all folks will frankenstein bits and pieces from discussion boards, or PEP drafts. Hence I don’t quite get the “it depends“ situation here. These PEPs need to come together somehow at some point. I don’t quite understand how they are accepted if they are not pre-planned to play nice, to be honest.
I don’t have a dog in that race but I don’t think Python still has this luxury of growing 15 branches then chopping the ones that don’t grow fast enough. Free-threading in particular is in some circles the hot new thing. Hence either it should deliver or CPython folks need to do some expectation management.
But anyways, I think I made the points clear enough and now I’m getting into discussions I did not mean to and I need to shut up. Thank you for letting us know, and I’ll be watching from the bench for the final decision.
Would xxlimited.c work for that – or at least as something we could iterate on to bring it toward that goal? Does it show too much? Too little?
I opened #140214 to add more comments to it.
As a project made primarily by volunteers, Python has little choice than letting branches grow. The main lever the Steering Council has for steering is saying no to things that are too risky – but if they do too much of that, we aren’t getting any exciting features…
(Full disclosure: I’m a PSF employee and the SC can tell me what to do. But there’s not many devs it my situation, and anyway the project isn’t really set up for top-down management.)
I don’t want to be the one who pushes you to add free-threading support against your wishes. A much better reason to add support would be your users wanting it. At that point, there’ll hopefully be enough support docs and and examples – we’re working towards it, but if we’re not there, feel free to ask on this forum.
I did ask you how much of a roadblock PEP 793’s API would be for you, but please don’t take that as pushing you to implement it, if you don’t see the benefits & support.
I’m sorry that you feel you need to shut up. Thank you for the points you made; I hope I can use them to improve the situation :)
), and that gives me confidence that I’m not just wasting my time or being obstructionist.
But before that spicy bit, a quick question;