But that doesn’t come with an easy and safe way to handle lifetimes. A big improvement would be some sort of cross-interpreter memoryview object.
I don’t dispute that. Luckily, that memoryview object can be implemented on top of what I described, so we don’t need to block the initial implementation upon having such an object available 
Agreed, but given that it interacts with memory management and potentially interpreter finalization, its implementation probably needs to be provided by core Python.
1 Like
Using multiple interpreters does not (nor has ever) change the behavior of threads in a single interpreter. Each interpreter in the process is isolated from the others. Within a single interpreter, its threads will continue to share objects/data. A per-interpreter GIL does not change that. Basically, there shouldn’t be any change in behavior from Python 3.11, especially if you don’t already use multiple interpreters.
There is no copy-on-write between interpreters. When an additional interpreter is created, it is completely fresh. Anything you might have done in another interpreter must be done all over again in the new one. We would certainly benefit from mechanisms that facilitate explicitly crossing the isolation boundary between interpreters in a limited way, e.g. safely sharing objects, but that’s not where things are at right now. The first step is to get isolation correct, including a per-interpreter GIL.
4 Likes
I would say there is a difference. If you write a C extension that uses a static global to share an object between interpreters, you could get away with this in 3.11 and before. But in 3.12 it will invite race conditions since access to that object (from refcount to “real” mutations) is not protected by a single GIL.
Of course, this is all well understood and the solution is for extension modules not to do this, and to advertise this fact using the two-phase module initialization.
4 Likes
This should only be the case if using per-interpreter GILs, though, which is not the default (probably should never be the default) and isn’t something you accidentally do. If not using per-interpreter GILs, everything should work as before: one GIL for the process, the same allocators for all interpreters.
Fair, but that choice is up to the user or application, not the extension. Basically such behavior used to be harmless and now it may be a hazard. And the problem is that it’s hard to debug — an app may usually work in this configuration but rarely do something wrong.
1 Like
Thanks for explaining and making it clear that interpreters and threads are different and no change in behavior is expected (rereading the PEP now, I realize we somehow jumped to wrong conlusions).
If anyone is interested in trying/helping this avenue - I am working on a module that does that.
RAM sharing using this process is working - The plan is tol build some high level stuff on top of that now. “Naive” subinterpreter single function calling passing pickle-able objects using this RAM is also working fine.
(The project itself, of course, requires Eric’s cPython fork where subinterpreters are being advanced)
4 Likes
To be clear, I got the “cross interpreter memoryview” thing that is being commented rolling in tihs project.
There is a small C file with only two methods: 1 to extract the buffer pointer, another to build
a memoryview from it, respectively for use on the parent interpreter which allocates the actual buffer, and in the sub-interpreter
The high level interface currently is super nice to use, even on repl, but will only pass one object at a time back and forth (through pickle). I am working on proper support for multiple running tasks “everywhere”
(n interpreters X m threads all able to communicate through specialized queues)
2 Likes
That reminds me of the concept called isolate in Dart. It’s basically a thread, but it doesn’t share memory with other isolates.
1 Like
I hope this is the right place to ask this –
Is there a per-process lock to serialize calls to PyInit_<module> functions in extension modules? In general, code can and should use locks to serialize access to per-process global state, but those locks need to be allocated somewhere and I’d think the PyInit_<module> function would be the place to do it.
I’m trying to use the C API in the release candidate and I find that the documentation says that PyInterpreterConfig_OWN_GIL requires use_main_obmalloc==0 which requires check_multi_interp_extensions==1. Does this mean that subinterpreters that have their own GIL can’t import modules that use single-phase init? Is this a hard technical limitation?
That’s correct. Single-phase init modules are expected to be incompatible with a per-interpreter GIL (and subinterpreters generally). This is strictly a consequence of global state (including objects). If an extension doesn’t have any global state then switching it to multi-phase init should be a small effort.
2 Likes
I would like to allow isolated subinterpreters in the jep project and I am trying to determine the correct way to initialize a PyInterpreterConfig. The PEP mentions new macros for PyInterpreterConfig_INIT and PyInterpreterConfig_LEGACY_INIT, these would fit my needs perfectly but I cannot find those in Python 3.12. In pylifecycle.h I found _PyInterpreterConfig_INIT and _PyInterpreterConfig_LEGACY_INIT but it is my understanding the _Py prefix is intended for internal use so I would like to know if those macros should be used outside of cpython itself or if there is a different way that is recommended for initializing a PyInterpreterConfig.
1 Like
Those two macros should be fine to use directly, even though they have a leading underscore. IIRC, I left them with an underscore because they felt more like a power-user tool, where power-users are more comfortable using “private” API like that.
If that’s a problem, I’m not opposed to making the macros properly public. Feel free to open an issue. Note, though, that the “public” name wouldn’t be available in 3.12. In the meantime, I wouldn’t expect copying the initializer to be much of a pain since it isn’t complex, nor long.
1 Like
I have no problem using the initializers as is. I appreciate you confirming that the “private” macros are an acceptable way to initialize the struct in 3.12. The main reason I hesitate to use “private” API is because I want to maximize source compatibility with future versions of Python and public would give me slightly more confidence the API is stable. But this is a brand new feature I don’t need or expect perfect stability, I just want to make sure I am doing the best I can.
2 Likes
Thanks for asking.
FYI, the PyInterpreterConfig struct is public and stable, so that aspect of those two initializers is stable too.
1 Like