PEP 684: A Per-Interpreter GIL

[encukou (Petr Viktorin)]

@eric.snow

@encukou

My vote goes to no: make 3.12 safe, then remove the limitations.

Agreed. The PEP shouldn’t need more than that.

That said, a thread-safety restriction on the allocators is the simplest way forward for a safe 3.12 (under a per-interpreter GIL). Or were you talking only about the constraint on extension modules?

I was talking about both :‍)

@eric.snow

Do you mean if someone sets a custom mem/object allocator then subinterpreters with their own GIL should not be allowed? That is reasonable, […]

Yes, that seems like the easiest safe way forward.

@eric.snow

@encukou

and new PyMem_SetGlobalAllocator could be added.

What would this do?

Same as PyMem_SetAllocator, but allow subinterpreters with their own GILs – i.e. that allocator would be assumed to be thread-safe.
(Yes, it needs a better name.)

@eric.snow

[PyMem_GetAllocator/PyMem_SetAllocator is] a race we’d have to resolve. However, rather than disallowing it, I’d expect a solution with a granular global lock, like we have for the interpreters list.

Yes. It’s out of scope for this PEP, but :

We probably should expose API for user-defined granular global locks. AFAIK we don’t have a good way to “allocate lock if not already allocated” that would work with multiple GILs.
Such a lock would be useful one-per-process modules (the isolation opt-out), as well as for Marc-André’s use case. IMO, this should be addressed relatively quickly, so people don’t start writing extensions that are only usable in the main interpreter. (I see relying on a single main interpreter as technical debt. Eventually I’d like to allow a library to call PyInitialize without caring whether there’s already an interpreter around. The concept of a main interpreter complicates that, but if it’s contained in the core, it should be manageable.)