PEP 684: A Per-Interpreter GIL

PEPs
19

I’ve updated PEP 684 after the last set of feedback. You can see the changes in https://github.com/python/peps/pull/2807.

The PEP text is still at https://peps.python.org/pep-0684/.

Significant changes:

  • settled on keeping the allocators global but requiring that they all be thread-safe
  • the state of the existing “small block” will be moved to PyInterpreterState
  • dropped references to mimalloc
  • simplified the C-API changes
  • clarified the situation with incompatible extension modules
  • proposed that extensions always opt in to per-interpreter GIL support with a new PyModuleDef slot (at least until we have enough evidence that multi-phase init is sufficient)
  • expanded “How to Teach This”

For me the most critical things to settle are:

  • Are we okay to require that the “mem” and “object” allocators be thread-safe, whereas currently we say they can rely on the GIL?
  • Can we avoid making extensions opt in to supporting per-interpreter GIL (if they already implement multi-phase init)?

Open questions (from the PEP):

  • Are we okay to require “mem” and “object” allcoators to be thread-safe?
  • How would a per-interpreter tracemalloc module relate to global allocators?
  • Would the faulthandler module be limited to the main interpreter (like the signal module) or would we leak that global state between interpreters (protected by a granular lock)?
  • How likely is it that a module works under multiple interpreters (isolation) but doesn’t work under a per-interpreter GIL?
  • If it is likely enough, what can we do to help extension maintainers mitigate the problem and enjoy use under a per-intepreter GIL?
  • What would be a better (scarier-sounding) name for importlib.util.allow_all_extensions?
2 Likes