PEP 734: Multiple Interpreters in the Stdlib

PEPs
9

I think this will be a nice addition to Python. I would definitely like to try it out and see how it works in practice once it’s available.

Some comments on the PEP:

Minor comments

ExecFailure

I think this name doesn’t match Python stdlib convention. Maybe ExecError?

This exception is a subclass of RuntimeError.

Why RuntimeError?

create_queue(maxsize=0) -> Queue

Why is it a standalone function rather than Queue(maxsize=0) as in queue.Queue / asyncio.Queue /multiprocessing.Queue?

Here’s the initial list of supported objects:
…

  • Queue

I suggest writing this as interpreters.Queue for clarity.

Shared buffer safety?

However, memoryview does not have any native accommodations. The user is responsible for managing thread-safety, whether passing a token back and forth through a queue to indicate safety (see Synchronization), or by assigning sub-range exclusivity to individual interpreters.

What happens if the user does not properly synchronize? That is, multiple interpreters running in parallel writing and reading to the same address at the same time.

If the answer is “data race”, then this ties back to the no-GIL discussions about needing a memory model. Without one, it will not be possible to provide proper semantics for Python programs in the face of data races. Javascript (which has SharedArrayBuffer and Atomics) takes this approach. Here is the JS memory model: ECMAScript® 2024 Language Specification

If the answer is “undefined behavior”, i.e. it’s not allowed and it’s the programmer’s responsibility to avoid it and no saying what happens if it does happen, that would be the first case of undefined behavior in Python AFAIK, and should be stated explicitly.

Transferable Objects

Speaking of JS (whose prior work is very relevant for this IMO), it has the notion of Transferable Objects, which are a different notion than the Sharable Objects defined in the PEP.

While Sharable Objects remain available on both sides, and therefore must be immutable or synchronized, Transferable Objects are transferred to the receiving side (without copying) and “hollowed out” on the sending side, i.e. transferring ownership. This is very useful point between “copying” and “shared memory+synchronization”.

The reason I’m bringing up Transferable Objects is not that it should be a part of the PEP, but that I think it would be good to either make sure the design does not preclude it as a future enhancement, or that it explicitly does preclude it in case it’s not relevant for Python.

3 Likes