Playing around sub-interpreters in 3.14

Python Help
1

Hi -

I thought that with the first class support for subinterpreters finally in the stdlib, my extrainterpreters pet project would be rendered all but obsolete.
Nonetheless, I made a new release this week, changing just (and just) the importing names so that it could function with Python 3.14.

To my surprise, it still provides by default a lot of niceties the stdlib concurrent.interpreter.Interpreter wrapper won’t achieve.

If for nothing else, I welcome you to play around with it (and the unfinished data structures within the code) so that we can improve it, and ultimately get new ideas/patterns that could be useful to a shared-data concurrent execution model (there is a nice proposal now as the pre-PEP 805 )

This is mostly pure-Python, with a simple extension to create a cross-interpreter lock - (and a way to cast a memory address (pointer) to a new Python object reference - at the time one could not import ctypes, which allows this in the sub-interpreter. ctypes works now)

For an example, this is what is possible (from Python 3.12 - 3.14):

>>> import extrainterpreters as ET
>>> bb = ET.Interpreter()
>>>
>>> bb.start()
Sub-Interpreter <#2>
>>> import math
>>> def bla():
...    print(math.cos(0))
>>> bb.run(bla)
1.0

This will automatically check the loaded modules in sys.modules, and import the same modules in the sub-interpreter - and serialize ‘bla’ (once it figures out it is not in a file to be imported) using inspect.getsource so that it exists in the subinterpreter as well.

You are welcome to try and test.

1 Like
2

This is really dangerous, especially in a library. Any concurrent modifications to the object’s reference count will race, because it’s supposed to be protected by a single interpreter’s GIL. Similarly, some important state on an object relies on the owning interpreter. For example, some_list.append(...) may crash, because the list will call PyMem_Realloc in the wrong interpreter.

3

Yes - I take care of that around there -

I don’t do that for user objects - just for carefully managed buffer and shared objects the library itself uses, with locks for what is in use and what not - there is even a guard mechanism to raise an exception if code from outside the library modules try to use this mechanism .

And I don’t do lists or other complex types - there are objects with simple scalar attributes I modify using the locking mechanism.

Thank you for paying attention.