PEP 684: A Per-Interpreter GIL

[eric.snow]

@encukou

@eric.snow

Are we okay to require that the “mem” and “object” allocators be thread-safe, whereas currently we say they can rely on the GIL?
Can we avoid making extensions opt in to supporting per-interpreter GIL (if they already implement multi-phase init)?

My vote goes to no: make 3.12 safe, then remove the limitations.

Agreed. The PEP shouldn’t need more than that.

That said, a thread-safety restriction on the allocators is the simplest way forward for a safe 3.12 (under a per-interpreter GIL). Or were you talking only about the constraint on extension modules?

@encukou

For example, PyMem_SetAllocator with PYMEM_DOMAIN_MEM or PYMEM_DOMAIN_OBJ could block creating independent GILs,

Do you mean if someone sets a custom mem/object allocator then subinterpreters with their own GIL should not be allowed? That is reasonable, if we don’t have enough information to conclude that existing custom allocators (used with PyMem_SetAllocator()) are thread-safe.

@encukou

and new PyMem_SetGlobalAllocator could be added.

What would this do?

@encukou

And, I guess setting memory allocators should be blocked if multiple GILs exist? Apparently, after Python is initialized, PyMem_SetAllocator should be only used only for hooks that wrap the current allocator, but creating such a hook using PyMem_GetAllocator gets you a race condition. IMO the best thing the initial implementation can do is to fall, and leave a better solution for later.

Yeah, that’s a race we’d have to resolve. However, rather than disallowing it, I’d expect a solution with a granular global lock, like we have for the interpreters list.

@encukou

A wrinkle is that PyMem_SetAllocator has no way to signal failure – it silently ignores errors. Guess it predates PyStatus?

Right. We’d have to do something like leave the current allocator in place and return. Then you’d have to call PyMem_GetAllocator() afterward to see if your allocator is set. A function that returned a result could be helpful.

Regardless, it would make more sense to me if we had a separate API for wrapping the existing allocator after init (e.g. PyMem_WrapAllocator()). Then PyMem_SetAllocator() would apply only to the actual allocator and only be allowed before runtime init. However, that is definitely not part of this PEP (nor necessary for it).

@encukou

@malemburg

External C libraries often use global structures and corresponding initialization procedures which can only be run once per process.

IMO, the solution is to not opt in for now. If synchronization/introspection API is missing, let’s add it after the PEP is in place. (IMO there are many issues in this area – that’s why I’m trying to convince Eric to make the initial implementation safe but limited.)

Agreed.