PEP 703: Making the Global Interpreter Lock Optional

I suspect the biggest annoyance in the ecosystem will be C extension API-level compatibility. I think two of the backwards-compat issues in PEP 703 can be worked around:

  • The GIL may lock access to non-Python data structures.
  • PyDict_GetItem (and other container) refcounting is unsafe for concurrent access.

We may be able to do the following to make the transition easier:

  • Give legacy C extension modules some kind of pseudo-GIL so their internal locking assumptions don’t break until they have a chance to update their code. I’m unsure how the deadlock cases also mentioned in the PEP would interact with such a pseudo-GIL.
  • Make PyDict_GetItem return a temporary ref that is automatically dropped when you release the GIL (similar to objc’s autorelease).

I don’t have as good of an intuition about the implications of the memory allocator API incompatibility, or whether that could be worked around - but many extensions likely won’t be affected (if there’s no clear workaround, someone could crawl PyPI to estimate how many extensions may be broken by the allocator changes).

1 Like