PEP 703: Making the Global Interpreter Lock Optional (3.12 updates)

[colesbury (Sam Gross)]

Hi @davidfstr, thanks for your detailed feedback and suggestions. I’ve made some edits to the PEP and I’ve included responses are below.

@davidfstr

Does the PEP (want to) recommend that C extension authors add support for --disable-gil?

Working C-API extensions are important, but I don’t think it’s necessary to make an “official” recommendation in the PEP.

@davidfstr

Adding support for --disable-gil in an extension would seem to imply that APIs like PyList_GetItem, PyDict_GetItem, and PyWeakref_GetObject be considered deprecated for regular usage…

I’ll expand on the rationale given in the PEP:

• PyDict_GetItem and other functions that return borrowed references can already be “unsafe” when there’s code between the call and the use of the “borrowed” value if that code releases the GIL, modifies the dict, triggers a GC, etc. That can happen more often than it sounds because macros like Py_DECREF can call arbitrary code via destructors and weak refs. So, the PEP expands the hazards from “overlapping operations” to “concurrent operations”, but doesn’t categorically change a “safe” API into an “unsafe” one. (The safety is context dependent)
• Most calls to these APIs are still safe. The reasons are varied and context dependent, but includes things like the dict being private to the calling thread (e.g., kwargs).
• I expect that wholesale replacing these calls will introduce more bugs that it would fix. This was my experience when I tried this approach.
• Given that the “correct” decision is uncertain, I think the not deprecating those APIs initially is less risky, even if that turns out to be the wrong decision in the long run. We can still later decide to deprecate those APIs. (The opposite scenario, in which we initially deprecate the APIs, and then decide that was a mistake would cause much more churn.)

@davidfstr

I’m a bit concerned at the level of complexity required to obtain safe and performant access to lists & dicts in a GIL-less context… Can this be mitigated somehow via additional static analysis of the CPython source?

Tools like thread sanitizer (TSAN) help. We already have a GitHub actions runner for address sanitizer (ASAN). I’d expect to add one for TSAN as well.

@davidfstr

Wait. So it’s possible for a --disable-gil build of CPython to start using a GIL at runtime? I feel like it might be worth mentioning this possiblity earlier in the Build Configuration Changes section. I was previously assuming that --disable-gil would compile away GIL-related code using #ifdef s or similar, but it sounds like code will have to regularly check whether a GIL is enabled or not since it can be enabled at runtime and not just at compile time.

This is already mentioned in the Build Configuration Changes section. The code path that acquires the GIL (e.g., PyEval_SaveThread) checks if the GIL is enabled. Runtime vs. compile time support doesn’t affect performance or code complexity, but does improve some debugging and compatibility scenarios.

@davidfstr

• Does (or will) PyPI support uploading multiple binary wheels, one for each build version of Python?

It already works with PyPI because the wheels have different ABI tags.

@davidfstr

How is it proposed that CPython automated builds (“buildbots”?) be updated? Will there now be double the number of builds, to add --disable-gil builds?

There will be additional stable build bots for --disable-gil builds, but I don’t expect the total number of bots to double.