PEP 684: A Per-Interpreter GIL

[encukou (Petr Viktorin)]

@eric.snow

Are we okay to require that the “mem” and “object” allocators be thread-safe, whereas currently we say they can rely on the GIL?
Can we avoid making extensions opt in to supporting per-interpreter GIL (if they already implement multi-phase init)?

My vote goes to no: make 3.12 safe, then remove the limitations.
For example, PyMem_SetAllocator with PYMEM_DOMAIN_MEM or PYMEM_DOMAIN_OBJ could block creating independent GILs, and new PyMem_SetGlobalAllocator could be added.

And, I guess setting memory allocators should be blocked if multiple GILs exist? Apparently, after Python is initialized, PyMem_SetAllocator should be only used only for hooks that wrap the current allocator (is that right @vstinner?), but creating such a hook using PyMem_GetAllocator gets you a race condition. IMO the best thing the initial implementation can do is to fall, and leave a better solution for later.

A wrinkle is that PyMem_SetAllocator has no way to signal failure – it silently ignores errors. Guess it predates PyStatus?

@malemburg

External C libraries often use global structures and corresponding initialization procedures which can only be run once per process.

IMO, the solution is to not opt in for now. If synchronization/introspection API is missing, let’s add it after the PEP is in place. (IMO there are many issues in this area – that’s why I’m trying to convince Eric to make the initial implementation safe but limited.)