PEP 703: Making the Global Interpreter Lock Optional (3.12 updates)

At OpenAI, we have extension modules that would not have been written if we had no GIL. It would also make maintenance of the bindings to native libraries we would still have easier. In just the past two weeks, we’ve run into two problems in our C extensions that wouldn’t have been a problem in a no GIL world (performance issue caused by not dropping the GIL somewhere, segfaults caused by dropping GIL somewhere where an existing GIL drop should have been more narrowly scoped). With the kinds of distributed ML systems that OpenAI runs, it’s pretty common for us to spend time fighting the GIL. In addition to the performance / latency / availability issues that others have mentioned, the GIL has also contributed to multiple deadlocks we’ve had.

28 Likes