PEP 734: Multiple Interpreters in the Stdlib

[eric.snow]

@ajoino

I’ve been following this work with excitement since I started writing Python in 2017, it seems like a very nice approach to parallelism!

Thanks for saying so! That’s definitely the goal.

@ajoino

my initial question is regarding object “sharing”. The Queue seems a bit too restricted atm, but I think the motivation makes sense.

Yeah, I definitely want it to be more flexible.

@ajoino

Is the current idea that if I want to share more complex objects that I pickle them, send the strings and unpickle them on the other side? I think that’s what the text hints at but I didn’t see that stated clearly anywhere. Maybe it could be stated more clearly if that is the intended workflow?

Yep, that’s pretty accurate. I’ll make sure the PEP gets updated to make that more clear.

@pitrou

instead of hardcoding the list of types that support sharing, would it be possible to make this a protocol? Something like __reduce__, but for in-process sharing.

I’m currently working on a separate PEP for exactly that.

@pitrou

Also, I think interpreters.Queue deviating from the threading and multiprocessing queue semantics by only allowing shareable objects will be annoying users. Perhaps you could have two queue classes, for example a LowLevelQueue for shareable objects only, and a regular Queue that would use pickle for most objects?

That’s a good point. Honestly, I was already considering falling back to pickle in interpreters.Queue. Here’s my main hesitation: I want a consistent explanation for the relationship between the send object and the received one. My preference for that relationship is “they may not be the exact same object, but they might as well be”. At the least, it’s an illusion I’d like to preserve. For immutable objects, that’s easy. For mutable ones, not so much.

Sharing mutable objects would either require that the underlying data is truly shared between the interpreters (a hard-ish problem) or that each interpreter have its own copy. The former would preserve my desired relationship between the objects. The latter would invalidate it unrecoverably. Pickling would mean the latter.

That said, having two distinct classes does seem like a way to preserve the constraint I want on a selective basis. I’ll think it over. It might also be doable as an optional flag on create_queue().

@pitrou

(note that with pickle 5 out-of-band buffers, you could perhaps simply share those buffers while still using pickle for most datatypes?)

That’s a good observation. I’ll keep it in mind.

@bluetech

create_queue(maxsize=0) -> Queue

Why is it a standalone function rather than Queue(maxsize=0) as in queue.Queue / asyncio.Queue /multiprocessing.Queue?

That’s a good point.

Partly it’s because queues live in the space between interpreters. create_queue() seemed to communicate that well. In contrast, Queue() creates an interpreter-specific object and might communicate the wrong thing.

That said, I had the idea for create_queue() (rather, its equivalent) over 6(?) years ago and haven’t thought about it much since. I might think of it differently if I consider it now. There also isn’t much of a technical reason for the separation at the point, especially since I recently changed Queue instances to be singletons (in each interpreter) and require that they match existing queues. I’ll have to think this over.

(FWIW, the same goes for interpreters.create() and `Interpreter’.)

@bluetech

Here’s the initial list of supported objects:
…

• Queue

I suggest writing this as interpreters.Queue for clarity.

@bluetech

Shared buffer safety?

However, memoryview does not have any native accommodations. The user is responsible for managing thread-safety, whether passing a token back and forth through a queue to indicate safety (see Synchronization), or by assigning sub-range exclusivity to individual interpreters.

What happens if the user does not properly synchronize? That is, multiple interpreters running in parallel writing and reading to the same address at the same time.

As @pitrou noted, that’s something users already have to manage, whether from Python code or in extension modules.

@bluetech

Transferable Objects

Speaking of JS (whose prior work is very relevant for this IMO), it has the notion of Transferable Objects, which are a different notion than the Sharable Objects defined in the PEP.

While Sharable Objects remain available on both sides, and therefore must be immutable or synchronized, Transferable Objects are transferred to the receiving side (without copying) and “hollowed out” on the sending side, i.e. transferring ownership. This is very useful point between “copying” and “shared memory+synchronization”.

The reason I’m bringing up Transferable Objects is not that it should be a part of the PEP, but that I think it would be good to either make sure the design does not preclude it as a future enhancement, or that it explicitly does preclude it in case it’s not relevant for Python.

Yeah, that’s a cool idea. We’ve discussed it before. As you noted, it’s something to be addressed after this PEP. You’re right about awareness affecting design though. Thanks for bringing this up.