Concerns About PyUnstable_Module_SetGIL()

Core Development
1

(This is a response to a comment in a separate thread discussing a soft deprecation of support for single-phase init extension modules, which I don’t want to get distracted by free-threading stuff. :smiley:)

PyUnstable_Module_SetGIL() was added unofficially. The only discussion about it was my objection to it [1] a year ago in the PR that added it (with an ensuing temporary compromise). It was added strictly only so folks can try out the free-threading build sooner (and originally named PyModule_ExperimentalSetGIL() to reflect this).

From an official standpoint, PEP 703 only proposes support for multi-phase init modules (via Py_mod_gil). Thus, PyUnstable_Module_SetGIL() (or PyModule_SetGIL()) should not be treated as official until it is added to PEP 703 and gets appropriate attention and discussion. It should not be a recommended tool until then, outside of experimentation. (That was kind of the point of the original PyModule_ExperimentalSetGIL() name.)

My original concerns remain. Furthermore, it feels like they were mostly ignored when I first brought them up and that they are still considered inconvenient.

---------------------

To be clear, my main concern is that Py*Module_SetGIL() is both unnecessary and reinforces use of an outdated, legacy feature (single-phase init) which we would like to eliminate. My comments in the original PR thread cover things fairly well. (See gh-116322: Add Py_mod_gil module slot by swtaarrs · Pull Request #116882 · python/cpython · GitHub)

IMHO, Py*Module_SetGIL() should be treated as strictly temporary and eventually removed. It should not be advertised as part of the long-term solution.

Clearly users are under the impression that it is.

That’s exactly what I expected would happen and why I pushed back on the original addition. At the time it was added, the argument for Py*Module_SetGIL() was that the multi-phase init requirement was getting in the way of people trying out the free-threading build and thus providing feedback.

I definitely understand and relate to such barriers to adoption all too well. Furthermore, I was on board with adding PyModule_ExperimentalSetGIL() as a temporary workaround. However, it’s getting entrenched. [2] Renaming it to PyUnstable_Module_SetGIL() made it worse. Just because we’re worried about barriers to adoption, however minor, it doesn’t mean we have to cut corners. That’s not how we operate. [3] The free-threading project should not be given special treatment. The zen of Python is quite appropriate here.

So what do I suggest is the long-term solution? It’s the already proposed (and implemented) multi-phase init module slot, Py_mod_gil. Extension maintainers can use it right now. I’m not recommending some new invention.

You might say “but then users have to implement multi-phase init!” For one thing, that’s a good thing. For another, that concern represents a misunderstanding (which we’re discussing in another thread), where people have conflated multi-phase init with isolation (which involves switching to module state and heap types). [4]

Just implementing multi-phase init is fairly trivial, as I noted a year ago. I’ll post an elaborating comment in a moment.

With all that in mind, I don’t thing Py*Module_SetGIL() is needed. I do think it should be removed.

  1. https://github.com/python/cpython/pull/116882#discussion_r1575006053 ↩︎

  2. With the status quo, eventually we’ll end up keeping Py*Module_SetGIL(), without a deliberate public decision to do so, because “people are used to it” and we don’t want to inconvenience them. That’s a mistake we can avoid. ↩︎

  3. Otherwise I’d have done a number of things quite differently with my own projects. ↩︎

  4. Implementing extension module isolation is a very good thing, but if a module uses static types or (C) global variables then the effort to isolate becomes non-trivial. That said, switching to multi-phase init remains trivial. ↩︎

7 Likes
2

As I noted, implementing multi-phase init is fairly trivial [1]:

  1. move the content of the module’s init function to a corresponding “module exec” function (dropping the call to create the module object)
  2. set the module def’s m_slots field to an array with:
    • a Py_mod_exec slot, set to the new module exec function
    • a Py_mod_multiple_interpreters slot [2], set to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED [3]
    • to try free-threading, a Py_mod_gil slot [4], set to Py_MOD_GIL_NOT_USED
  3. set def.m_size to 0 (if negative)
  4. update the module init function to only return PyModuleDef_Init(def); for the corresponding module def

(Also see Module Objects — Python 3.13.3 documentation and https://docs.python.org/3/howto/isolating-extensions.html.)

There’s no need to do any of the isolation stuff (heap types, module state, etc.) immediately.

At that point, the only extra step to run on a free-threading build would be to add the Py_mod_gil slot, set to Py_MOD_GIL_USED , etc.

IMHO, the free-threading build should always treat single-phase init modules as though they had Py_MOD_GIL_USED set.

  1. my original outline of the steps: https://github.com/python/cpython/pull/116882#discussion_r1586449047 ↩︎

  2. the default is Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED ↩︎

  3. think of Py_mod_multiple_interpreters as a hypothetical Py_mod_isolated, where Py_MOD_PER_INTERPRETER_GIL_SUPPORTED means “isolated” and Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED means “not isolated” ↩︎

  4. the default is Py_MOD_GIL_USED ↩︎

5 Likes
3

From a Cython point of view we don’t really need this function, so I’m pretty relaxed if it gets removed. We do use it if people declare purposefully make the module single-phase init and declare free-threading compatibility. But I don’t see why anyone would.

However - I’m not really sure I understand the problem. There doesn’t seem anything intrinsic about free-threading which really needs multi-phase init modules. So this seems more like using people’s desire to support free-threading to push them towards multi-phase initialization rather than something that’s really needed.

It’d probably be useful to find out why things like PyO3 and PyBind11 have been slow to adopt multi-phase initialization? (My feeling is that no-one’s really pushed it for PyBind11, and the Rust culture of “everything should be safe” maybe discourages experimental WIP implementations. But this is complete, reading-between-the-lines guesswork from me). But either way, both seem to have prioritised free-threading rather than multi-phase initialization, and that decision seems reasonable to me (as an outsider to both projects).

1 Like
4

I think it’s more that it’s a big change to very complicated procedural macros that are generating unsafe FFI code, and someone needs to carry the change over the finish line.

See RFC: Implement Multi-Phase Module Initialization as per PEP 489 by Aequitosh · Pull Request #4162 · PyO3/pyo3 · GitHub.

2 Likes
5

Exactly this. I am keen to move PyO3 to multi-phase init, just have had to balance this with all other ongoing work. If it’s a priority for CPython to get that done, that helps me focus where I need to be putting efforts.

3 Likes
6

Fair enough - I stand corrected. That does seem to contradict

of course (at least in some cases).

1 Like
7

FWIW, implementing multi-phase init for a module is a relatively small task. If you are talking about full module isolation, then I suggest you break the work down into the following steps:

  1. implement multi-phase init
  2. identify process-global state (variables) in your code
  3. identify process-global state in external dependencies (I’m looking at you, openssl)
    ^^^ this one is often overlooked and I expect it will cause plenty of headaches in the future
  4. consolidate your globals that aren’t static types, if any, into a global struct
  5. store that struct as module state instead of a global variable
  6. convert any static types to heap types (requires module state)

It’s totally worth doing, but you don’t have to do it all at once. Start with multi-phase init, which should be a trivial effort. Steps 2-4 are a good idea anyway. Steps 5 and 6 are where some projects face the most effort. There’s a nice how-to doc with helpful info. We’d definitely love feedback on pain points with any of the steps and especially about anything we can do to make the last two steps easier.

(Ironically, I created this discussion thread so we could keep my other thread focused on single-phase init vs. multi-phase init, so naturally we end up talking about that here. :smiley:)

Good point. That we haven’t encouraged/forced the changed is probably the main reason why so many extension modules still use single-phase init (along with the perceived cost of module isolation.)

FWIW, we work fairly hard to avoid giving extension maintainers extra work. That’s part of why we haven’t pushed multi-phase init and module isolation very hard.

1 Like
8

The problem binding generators have isn’t quite either of those: it’s taking a templating process that currently emits a single module initialisation function and instead making it emit a pair of functions and a struct, preferably with the ability to appropriately configure the contents of that struct.

That’s a harder design problem than porting a single module, even if it isn’t as hard as ensuring a complex module is subinterpreter compatible.

It’s also a problem we don’t have great docs for - binding generator authors have to infer from the docs for porting a single module. While it’s been a while since I read those, I think we do tend to conflate supporting subinterpreters and runtime reinitialization with mechanically migrating to multi-phase init.

2 Likes
9

Thanks for pointing this out. Binding generators are definitely something with which I’m not especially familiar. That clearly applies to the complexities involved here, as my only-moderate experience with implementing code generation doesn’t lead me to expect much difficulty. What is the difference between generating [one PyModuleDef plus the init function] and [the module def plus the init-function-turned-into-exec-function plus the new init function]? Wouldn’t it be no more than emitting an extra (very minimal) function? I’m sure I’m missing something.

Yeah, documentation to support binding generator authors is definitely an area where we could do substantially better (along with supporting embedders). I know @steve.dower has significant interest and knowledge in this area. Regardless, it would be great if we could start with a specific analysis to clearly identify the needs of these users, to inform us of how we could improve the docs and the C-API. That’s especially important as we work toward substantial C-API changes.

1 Like
10

One concrete documentation issue I’d really like to see solved is porting this docs section to use multi-phase init: 1. Extending Python with C or C++ — Python 3.14.0a7 documentation

There’s a note to look at some internal modules in cpython for multi-phase init examples, but IMO it’s not great that the docs themselves don’t have a good example.

I also personally learned how to write C extensions following this tutorial:

It’s a pretty great tutorial! But, it’s out of date and uses single phase init and static types. IMO someone should write a tutorial like this that covers how to write an extension in the new, preferred style. It would also be great to have a similar tutorial for how to port an existing extension module, with worked examples covering common patterns.

5 Likes
11

I expect it would be less about the exact code being generated, and more about having to add a second way of doing things without completely dropping the old way. How hard that is going to be will vary across the different binding generators, but they’ll all at least encounter the problem of it being an additional configuration to test.

2 Likes