neonene
May 26, 2025, 3:52pm
41
I can give an (expanded) interpretation to the section:
“Multi-phase init modules are not expected to support subinterpreters until full isolation is achieved.”
In that case, the multiple-interpreters support sounds like a non-default option at least when introducing. I’d prefer the reconsideration if possible.
ngoldbaum
May 26, 2025, 4:14pm
42
Following up on NumPy, @AA-Turner has been busy and at least has an outline towards converting NumPy’s extension modules to multi-phase init!
opened 07:28PM - 21 May 25 UTC
### Proposed new feature or change:
This is intended to be a tracking issue for… a series of PRs I will shortly submit.
Multi-phase initialisation was introduced in [PEP 489](https://peps.python.org/pep-0489/) for Python 3.5. It replaces the previous 'single-phase' mechanism by splitting the module creation process into creation and execution phases. Quoting from [the documentation](https://docs.python.org/3/c-api/module.html#multi-phase-initialization):
> Extension modules created this way behave more like Python modules: the initialization is split between the creation phase, when the module object is created, and the execution phase, when it is populated. The distinction is similar to the `__new__()` and `__init__()` methods of classes.
This is also the stable mechanism to declare compatibility with both free-threading and subinterpreters (though note that **this PR does not attempt to add support for subinterpreters**).
A common point of confusion/conflation is multi-phase initialisation and module isolation. As noted on the [Python Discourse](https://discuss.python.org/t/89374) forum, "*Implementing extension module isolation is a very good thing, but if a module uses static types or (C) global variables then the effort to isolate becomes non-trivial.*". This issue only seeks to implement multi-phase initialisation, generally following the guide that Eric set out in that thread:
1. move the content of the module’s init function to a corresponding “module exec” function (dropping the call to create the module object)
2. set the module def’s [`m_slots`](https://docs.python.org/3/c-api/module.html#c.PyModuleDef.m_slots) field to an array with:
* a `Py_mod_exec` slot, set to the new module exec function
* a `Py_mod_multiple_interpreters` slot, set to `Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED`
* to try free-threading, a `Py_mod_gil` slot, set to `Py_MOD_GIL_NOT_USED `
3. set `def.m_size` to 0 (if negative)
4. update the module init function to only `return PyModuleDef_Init(def);` for the corresponding module def
cc @ngoldbaum
xref:
* https://github.com/numpy/numpy/issues/26157#issuecomment-2079486111
* https://github.com/numpy/numpy/issues/28271#issuecomment-2634857326
* https://github.com/numpy/numpy/issues/24755
* https://github.com/numpy/numpy/issues/11925#issuecomment-421082106
* https://github.com/numpy/numpy/issues/665
A
PRs:
* #29022
* #29023
* #29024
* #29025
* #29026
* #29027
* #29028
* #29029
* #29030
* #29031
* TODO: simd_dispatch
* TODO: f2py
7 Likes
AA-Turner
May 26, 2025, 4:27pm
43
1 Like
encukou
May 26, 2025, 4:28pm
44
The section starts with:
Extensions using the new initialization scheme are expected to support subinterpreters and multiple Py_Initialize/Py_Finalize cycles correctly, avoiding the issues mentioned in Python documentation.
I’m not sure how you can interpret that as multi-phase modules not being expected to support subinterpreters.
Anyway, the PEP is history; current docs have been saying this since 3.5:
All modules created using multi-phase initialization are expected to support sub-interpreters . Making sure multiple modules are independent is typically enough to achieve this.
1 Like
encukou
May 27, 2025, 6:44am
45
I opened PR #134775 to give more info here.
1 Like
encukou
May 29, 2025, 11:30am
46
For the record: PR #134764 proposes to document single-phase init as legacy .
The only reason to use single-phase init should be backwards compatibility. It can be a very strong reason, of course.
8 Likes
