Proposal: Officially Deprecate Support for Legacy (Single-Phase Init) Extension Modules

Core Development
21

I can’t think of a specific API misuse example off the top of my head where the consequence isn’t already “Your extension module doesn’t work because you got the boilerplate wrong” (for example, creating multiple module objects in the same init function, and the first one isn’t the “main” one for that init function).

There’s definitely janky horribleness in the way single-phase init works (anyone that has experienced the baffling results of attempting to reload single-phase init extension modules will be able to attest to that), but from the perspective of an extension module author, the main benefits of multi-phase init are:

  • less manual boilerplate in the first place (so fewer opportunities for mistakes the compiler won’t catch)
  • you can actually reload the modules (but still not the shared libraries) at runtime
  • access to additional tools for managing global state in a way that allows subinterpreters, module reloading, and runtime reinitialization to all work properly even for complex use cases

It’s that last point that is the most common source of issues with single-phase init, as it’s hard to directly use C/C++ globals for an extension module’s state management without causing problems for the interpreter’s own dynamic state management (and much of the janky horribleness in the implementation arises from attempts to mitigate those inevitable problems).

The binding generators/libraries (SWIG, Cython, pybind11, etc) historically have done a pretty good job of dealing with the boilerplate parts of the problem, but even they can’t make reloading or global state management work consistently without the extra tools that multi-phase init makes available.

So I think the case for soft deprecation (and updating the docs accordingly) is strong, while the case for programmatic deprecation is more “Eh, maybe someday, when the legacy mechanism is causing problems less obscure than people sometimes wondering why importlib.reload doesn’t appear to be doing anything, or why some extension modules can’t be imported in a subinterpreter”.

2 Likes
22

This will be hit more frequently from Python 3.14 as InterpreterPoolExecutor becomes adopted, and/or when the interpreters module is added. Hopefully this may provide some incentive to port to multi-phase for extension modules.

A

3 Likes
23

I have some clarification about PyUnstable_Module_SetGIL(). However, I don’t want discussion of free-threading to take over this thread, which can easily happen. :smiley: I’ve opened Concerns About PyUnstable_Module_SetGIL() instead.

2 Likes
24

FYI, I did the same thing for the CPython runtime (and several of us did for stdlib extension modules). At the beginning, around 2018, there were over 4000 global variables. We moved the ones in extension modules into module state (and switched to heap types). I moved most of the remaining global variables into a new _PyRuntimeState struct (and then moved a bunch of that down into PyInterpreterState). The project took roughly 5 years of intermittent effort. It was worth it.

Most of the work was mechanical, with the exception of sorting out the memory allocators and isolating various builtin objects (e.g. singletons, types). To avoid breaking the C-API, we kept those objects/types in global variables, introduced “immortal” objects (PEP 683), and added an internal-only mechanism to keep static types isolated between interpreters. I’ve considered proposing a similar mechanism in the public C-API for general use. Also, I wanted to be sure we didn’t accidentally add any new global variables later, so I added a tool for that, which we use in CI.

My motivation for this long-term project was my goal of a per-interpreter GIL (which eventually became PEP 684 and was realized with the 3.12 release). However, I knew from the beginning that almost all the work was worth doing anyway, helping the project’s code health and enabling other improvements. Without this effort to consolidate and isolate all that global state, various projects including the free-threading build, would not have been realistically possible.

My point is: thanks for your effort! I understand it well. It’s worth doing.


p.s. I’ll reiterate that, while all that isolation effort is amazing and worth doing, the first step of implementing multi-phase init is essentially trivial.

7 Likes
25

One follow-up question: do you think the effort to consolidate (and isolate) global state will help with support for running under the free-threading build?

Given my own experience, I’d guess that the consolidation will help reduce maintenance burden for extension maintainers.

1 Like
26

(This is essentially a repost of a comment on another thread. I think it’s worth repeating here.)

I want to elaborate on how implementing multi-phase init is fairly trivial [1]:

  1. move the content[2] of the existing module init function to a new “module exec” function
  2. set the module def’s m_slots field to an array with:
    • the Py_mod_exec slot set to the new module exec function
    • the Py_mod_multiple_interpreters slot[3] set to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED[4]
  3. set def.m_size to 0 (if negative)
  4. update the module init function to only return PyModuleDef_Init(moddef);

(Also see Module Objects — Python 3.13.3 documentation and https://docs.python.org/3/howto/isolating-extensions.html.)

There’s no need to do any of the isolation stuff (heap types, module state, etc.) immediately.

FYI, at that point the only extra step to run without the GIL on a free-threading build would be to add the Py_mod_gil slot[^7], set to Py_MOD_GIL_NOT_USED . (The default is Py_MOD_GIL_USED.)


  1. my original outline of the steps: https://github.com/python/cpython/pull/116882#discussion_r1586449047 ↩︎

  2. You’ll drop the call to create the module object. ↩︎

  3. the default is Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED ↩︎

  4. think of Py_mod_multiple_interpreters as a hypothetical Py_mod_isolated, where Py_MOD_PER_INTERPRETER_GIL_SUPPORTED means “isolated” and Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED means “not isolated” ↩︎

4 Likes
27

Yes, it was necessary to finish shipping free-threaded support in NumPy 2.1 last year. Unless you’re asking generically for other libraries that you’re encouraging to do this. In which case, yes, identifying global module state is necessary for a C extension to declare free-threaded support, and if you’re doing that you might as well centralize it, if only to keep things organized. And you might find horrors like storing the state for a context manager in a C global variable, which works about as well as you might expect it in any non-trivial multithreaded code.

See MNT: Reorganize non-constant global statics into structs by ngoldbaum · Pull Request #26607 · numpy/numpy · GitHub if you’re curious about the code details in NumPy and want to dive into a thousand-line C diff :slight_smile:

2 Likes
28

Hi @eric.snow. nanobind maintainer here. A correction to what you said: multi-phase initialization is not a prerequiste for free-threading. Extensions with single phase initialization can opt in via PyUnstable_Module_SetGIL, and this is what e.g. nanobind currently does.

Conceptually, it would not be hard to adapt nanobind to multi-phase initialization. However, in the past I decided against doing this because the benefit seemed unclear. Please correct me if I’m wrong with the following: to my knowledge, the main point of multi-phase initialization is to encapsulate module state in a non-global way, which in turn permits the use of subinterpreters.

However, there are problems with subinterpreters for binding tools like nanobind or pybind11. To enable C++/Python interoperability, they rely on internal data structures that track the state of type and instance objects. Those data structures are global because types could be used by many different modules running in the same interpreter. Different extensions built with nanobind communicate with each other through this shared state.

Currently, nanobind caches a global pointer (static Internals *internals = ...) in all extensions, which points to this data structure following a costly lookup. A significant chunk of code relies on the ability to dereference this pointer. With subinterpreters, that is no longer possible, and it becomes rather complicated to access the module state depending on the context where it is needed. I am concerned that this will add significant costs to the fast path in nanobind. That’s why I never seriously pursued subinterpreter integration, which in turn made it logical to stick with single-phase initialization.

3 Likes
29

Eric’s made the point elsewhere that this is a temporary hack to allow immediate unblocking of some users. It’s not a permanent API, and we’d like to see users move onto the permanent API (multi-phase init).

The main point of multi-phase initialization is to future-proof your modules (or your user’s modules, in this case) against work that we do to improve imports. Without it, we are massively constrained on the runtime side of things, because we have no way to determine whether a module supports new features before it actively runs code.

With multi-phase init, we can safely load your module, discover that it doesn’t support (e.g.) subinterpreters, and wrap it up so that it doesn’t break. Without multi-phase init, we basically have to assume that we can’t trust the module to support anything new since around 2013. We don’t want to add new public API for each feature to activate a module’s support for features - we already have that public API (multi-phase init).

10 Likes
30

Thanks for the explanation. @steve.dower covered some of what I would have said. (For more discussion about the free-threading build and PyUnstable_Module_SetGIL(), check out my other thread: https://discuss.python.org/t/concerns-about-pyunstable-module-setgil/89374).

I do want to reiterate something I pointed out in an earlier comment, that implementing multi-phase init is not the same thing as implementing module isolation. Your explanation implies to me that you understand the two to be the same, which seems to be a fairly common misunderstanding. You can actually implement multi-phase init without the rest of module isolation. (See my earlier post in this thread and my post in the other thread for more info.)

Regarding subinterpreters, what you’ve written is very helpful. Thanks for taking the time to write it up. There isn’t a problem with saying “this module does not support use in multiple interpreters.” Multi-phase init supports that. That said, I’m sure the Python community will ultimately dictate how much users expect extension modules to support use in multiple interpreters.

Also note that we are seriously looking into eventually adjusting the C-API to make the CPython runtime context explicit, instead of relying on a thread-local variable (accessed through PyThreadState_Get()). This would mean nearly all functions would add a context argument (like HPy has), which would effectively be equivalent to an opaque PyThreadState. While this might not change the constraints on nanobind state management much (I’m not sufficiently familiar with nanobind to say), I expect such a change would have some impact on those constraints.

1 Like
31

@eric.snow @steve.dower Thank you for the explanation. This makes sense to me now. I will switch to multi-phase init for the next version of nanobind. I am not involved in pybind11 development anymore, but there is an open PR for subinterpreters+multi-phase initialization that is also likely to be merged soon.

I was wondering about two related points:

  1. AFAIK both free-threading and subinterpreters are solutions to improve parallelism. Are they exclusive, or will there will be python builds with both free-threading and subinterpeters?

  2. The need to distribute gigabytes of binary wheels for many different platforms and Python minor versions has been a pain point in the past, especially for bigger binding efforts. nanobind has a feature to make stable ABI extensions, which is possible since Python 3.12. However, with work on improving parallelization in Python, my impression is that there is no concept of a stable ABI (at least, that is definitely the case for free-threaded Python). What is the status of subinterpreters on this point?

4 Likes
32

I believe that subinterpeters isn’t a separate build of Python. It’s just in every build of Python including the free-threading one. So not exclusive.

Stable ABI is compatible with sub-interpreters I believe. From an extension author’s point of view you “just” have to mark the extension as compatible with sub-interpreters (which you do via moduledef slots) and make sure that all global state is on the module state structure. The latter is the hard bit, but is all in the stable ABI.

1 Like
33

FWIW pybind11 does look to support multi-phase initialization now as of about 3 weeks ago https://github.com/pybind/pybind11/pull/5574

8 Likes
34

As Da Woods said, subinterpreters are already in the main release, so it’s as stable as the Stable ABI.

Free-threaded is an experimental build, because we don’t yet know exactly what shape things will have to be in order to work. Expect some breaking changes, or potentially an entirely new stable ABI (we’re discussing elsewhere whether to move to abi4 for all builds, to add a new parallel abi3t just for free-threaded, or just add a set of build-time options to choose which parts are stable). There’s a proposal that free-threaded is stable enough now that we can commit to its ABI, and so if that’s accepted then the new stable ABI can be determined, but until then we need to keep the flexibility to avoid designing in flaws and being committed to them for a long period of time.

4 Likes
35

SWIG also uses single-phase init when Python extensions are created. A feature request on the github issue tracker has been created Support Python multi-phase init · Issue #3168 · swig/swig · GitHub.

1 Like
36

I’ve just remembered one place where single-phase initialization may be useful for some Cython users.

With single-phase initialization its possible to import a module directly by calling PyInit_module_name which is useful when you want to build all your code into one big binary rather than making each module a separate .so file as Python expects.

We try to encourage people to use PyImport_AppendInittab but that only works if you can set everything up ahead of time before the interpreter is initialized which not everyone can.

This isn’t a use-case that we try to encourage. But I know it’s something which some people do.

(I should also say, I have worked out and documented how to do the same thing with PyModule_FromDefAndSpec and PyModule_ExecDef but it’s pretty verbose and I’m honestly not convinced it’s less hacky)

2 Likes
37

That should work almost just as well with multi-phase init, with some extra steps involved. We do this in the CPython test suite:

For a specific example, see PyInit__test_non_isolated() in Modules/_testmultiphase.c. It is defined in the same file with a bunch of other modules. We load it in a test with the following helper code:

def import_extension_from_file(modname, filename, *, put_in_sys_modules=True):
    loader = importlib.machinery.ExtensionFileLoader(modname, filename)
    spec = importlib.util.spec_from_loader(modname, loader)
    module = importlib.util.module_from_spec(spec)
    loader.exec_module(module)
    if put_in_sys_modules:
        sys.modules[modname] = module
    return module

Would it be worth adding a helper in importlib.util or as a method on ExtensionFileLoader, and/or a helper in the C-API?

This implies to me helpers would be worth adding.

FWIW, the approach of the helper I copied above is definitely the correct way. If that’s how you implemented it using PyModule_FromDefAndSpec() and PyModule_ExecDef() then that wasn’t hacky. The fact that you are uncertain tells me we have a deficiency in the docs and C-API.

1 Like
38

It’s not quite the same. My approach is Issues when embedding — Cython Guidelines documentation

I think the main difference is that I bypass using a loader because the use-case is that the module doesn’t exist as a separate file - it’s just built into the main program.

1 Like
39

The implementation would be more trivial if Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED (NULL) becomes the default.

Now that modern subinterpreters support per-interpreter GIL, the Py_MOD_MULTIPLE_INTERPRETERS_SUPPORTED option seems too special to be implicitly chosen? I suppose no authors are currently omitting the slot. If that is not wrong, the default could be switched.

40

Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED is awkward because it’s the backwards-compatible option. It’s been the default since 2013, when PEP 489 introduced PyModuleDef_Slot itself – long before the Py_mod_multiple_interpreters slot was added. See the Subinterpreters and Interpreter Reloading section in PEP 489.

IMO, the isolation “opt-out” pattern is a bit more general (and more explicit) than the Py_mod_multiple_interpreters slot. But unfortunately, it also blocks more scenarios that can be unsafe:

  • Multiple Py_Initialize/Py_Finalize cycles: if you have a static PyObject* pointer, it can be left dangling in that situation.
  • What NumPy tests call “full reimport” – del sys.modules[...]; import .... In this case, the multi-phase behaviour (real full reimport) is very different from single-phase (reused __dict__ contents).
1 Like