I was talking with @mitsuhiko at EuroRust this morning about what per-interpreter GIL support in Rust can look like (and specifically in PyO3). This is just a couple of thoughts we explored which might be interesting to share wider.
In my eyes, the challenge of subinterpreter support for a framework like PyO3 is the need to have object isolation between subinterpreters. User-facing APIs need to be constrained to account for this.
Verifying object provenance - at the fundamental level, it seems we need a way at runtime to verify that objects belong to the current subinterpreter. We already can identify subinterpreters by interpreter ID, maybe a solution is to add interpreter ID to
PyHeapTypeObject, so instances of that type can have their subinterpreter known. There may need to be some exploration how this interacts with immortal / static types.
Thread affinity - currently subinterpreter API as implemented creates a Python thread state which can then be activated with
PyEval_RestoreThread. If I understand this correctly it means C code can change the subinterpreter on the current thread by swapping the thread state to one from a different thread. Is there a way we could prevent that, so that there is a guarantee that per host thread there is only ever one subinterpreter which can run on it? Otherwise I think the implication is that after any call into unknown C code you might have been swapped onto a different subinterpreter.
Message passing - I see
Py_AddPendingCallto send calls back to the main interpreter. Are there other APIs to use to pass messages between subinterpreters? I didn’t see any; I guess we can always build our own thread-safe datastructures but it might be nice to have something in the C API for this.
Shared objects - To avoid the need to serialize and message pass, maybe there are subsets of objects we can share safely? If I recall correctly, nogil is introducing per-object locks. Maybe these can be explored as a way to lock objects to enable sharing them in a synchronized way across subinterpreters?
Overall we are optimistic Rust can help bring some cool use cases to the table in this space!