PEP 703: Making the Global Interpreter Lock Optional (3.12 updates)

[davidfstr (David Foster)]

Thanks @colesbury for working on this PEP! It’s clearly had a lot of thought put in to it and is tackling a really tough problem head on: the GIL.

Feedback:

The global interpreter lock will remain the default for CPython builds and python.org downloads. A new build configuration flag, --disable-gil will be added to the configure script that will build CPython with support for running without the global interpreter lock.

• Does the PEP (want to) recommend that C extension authors add support for --disable-gil?
  • The Python Build Modes section seems to lay out a suggested roadmap for extension authors that implies that support for --disable-gil will be encouraged.
• Adding support for --disable-gil in an extension would seem to imply that APIs like PyList_GetItem, PyDict_GetItem, and PyWeakref_GetObject be considered deprecated for regular usage in favor of their “new reference” versions that play nicely with --disable-gil. However the Why Not Deprecate PyDict_GetItem in Favor of PyDict_FetchItem? section explicitly says APIs like PyDict_GetItem are not deprecated even while acknowledging that using them can be unsafe. I’m a bit confused.

Additionally, the implementation proposed PEP supports […]

• Nit: Remove either the word “PEP” or the phrase “implementation proposed”

• Container Thread-Safety
• Borrowed References
• Python Critical Sections
• Optimistically Avoiding Locking
• Mimalloc Changes for Optimistic list and dict Access
• Mimalloc Page Reuse
• Optimistic dict and list Access Summary

• The content of the above adjacent sections from the table of contents is sufficiently related that it seems they would be more logically organized if they were under a single supersection. For example:

  • Container Thread-Safety
    • Optimistic dict and list Access Summary (reordered from end to beginning)
    • Borrowed References
    • Python Critical Sections
    • Optimistically Avoiding Locking
    • Mimalloc Changes for Optimistic list and dict Access
    • Mimalloc Page Reuse

  Grouping under a single supersection would also be consistent with the high-level outline given in Overview of CPython Changes.

• I’m a bit concerned at the level of complexity required to obtain safe and performant access to lists & dicts in a GIL-less context. It seems like it would be very easy for a contributor to introduce new data races, which would be difficult to detect/debug.

  • Can this be mitigated somehow via additional static analysis of the CPython source? (I have never maintained lockless algorithms before, so I am unfamiliar with common tools for ensuring that such implementations resist introduction of data race bugs over time.)

the interpreter pauses all threads and enables the GIL before continuing.

• Wait. So it’s possible for a --disable-gil build of CPython to start using a GIL at runtime? I feel like it might be worth mentioning this possiblity earlier in the Build Configuration Changes section. I was previously assuming that --disable-gil would compile away GIL-related code using #ifdefs or similar, but it sounds like code will have to regularly check whether a GIL is enabled or not since it can be enabled at runtime and not just at compile time.
  • Indeed,

  • The author believes a worthwhile goal is to […] have the global interpreter lock controlled at runtime, possibly disabled by default.

At least for some time, there will be two versions of Python requiring separately compiled C-API extensions. It may take some time for C-API extension authors to build --disable-gil compatible packages and upload them to PyPI.

• Does (or will) PyPI support uploading multiple binary wheels, one for each build version of Python?
• How is it proposed that CPython automated builds (“buildbots”?) be updated? Will there now be double the number of builds, to add --disable-gil builds?