PEP 703: Making the Global Interpreter Lock Optional

PEPs
108

How will you take out the GIL when you call into the extension?

Its not just the loading of the extension that will have potential race conditions.

Each method call of the extension would need a GIL equilivant I would think.

109

To be clear, I am suggesting that the GIL is enabled from that point forward and remains enabled after the extension loading process is completed.

4 Likes
110

This solution would work very well for PyO3 - we would be able to default to requiring GIL and then extension authors can opt out of the GIL when they are comfortable.

I fear that the downside of this approach is that nogil would take a long time to build traction, because it becomes a problem of extensions being built for it plus extensions being upgraded for it. Nevertheless it’s probably worth it for the safety advantages - a slow and stable rollout would be good from perception of users.

3 Likes
111

I’ve been thinking the transition to a GIL-free world would look a lot like the 2-to-3 transition, dragging out interminably. The details would obviously be much different, but giving extension authors an escape hatch would likely delay the widespread adoption of nogil. Would an application writer know that their presumed GIL-free application had been hamstrung by some extension they might well have no control over? Would they be able to pinpoint the source?

112

Yes, explained above.

1 Like
113

This is assuming a GIL-free world is the end goal, which, as long as the GIL-ful version is considerably faster in many common Python applications, is a notion I disagree with. Being able to choose between GIL-ful and GIL-free at run-time feels like a better end goal to me but I’m sure there are many reason why that is bad/hard/impractical.

114

Thanks. The thread has gotten kinda long. I’m only about halfway through the PEP itself and didn’t recall seeing it there.

115

In preventive case nogil doesn’t make it for 3.12, wouldn’t a convention be already nice on how wheels for this alternate reality shall be named ? … replacing the “c” of “cp” per a “n” for nogil cpython ?

  • cramjam-2.6.2-cp312-none-win_amd64.whl = with gil
  • cramjam-2.6.2-np312-none-win_amd64.whl = nogil
  • cramjam-2.6.2-cp312.np312-none-win_amd64.whl = with or without you nogil

It would allow to use existing pypi infrastructure.

1 Like
116

There’s no chance of it making it in for 3.12. Not only has this PEP not been sent to the SC yet to even consider accepting it, there isn’t a working 3.12 patch to even begin reviewing (GitHub - colesbury/nogil: Multithreaded Python without the GIL is 8824 commits behind main as it is).

No because I don’t even necessarily agree it’s an interpreter-level tag difference compared to an ABI tag difference. And I personally don’t want to expound the energy on that debate unless it’s going to be useful (and my active COVID recovery says it wouldn’t be at this time).

1 Like
117

New repo for nogil 3.12 GitHub - colesbury/nogil-3.12: Multithreaded Python without the GIL (experimental rebase on 3.12)

6 Likes
118

it looks about 80 commits still to apply in the todo of nogil-3.12 , plus about 420 commits from cpython-3.12a4 to now, so nogil-3.12 is roughly 500 commits behind.

Given the benefit it brings to some workloads in the cloud, I would expect that unofficial nogil-3.12 will get a bigger community than pypy, and offering them a pypi way-of-blooming, aside of any SC approval for “mainline cpython”, would be great.

3 Likes
119

I really like this nogil implementation.
Hope it will be in 3.12 as long as progress on GitHub - colesbury/nogil-3.12: Multithreaded Python without the GIL (experimental rebase on 3.12) continuous

This PR open to Python a new world of “fast” execution and it wouldn’t be needed to place slow code in C++ library

4 Likes
120

Hi all,
a new discussion has been made, and it seems relevant to notify people who are in this discussion too: PEP 703: Making the Global Interpreter Lock Optional (3.12 updates)

3 Likes
121

Hi all,

(I’m not sure where PEP 703 discussion is happening now that several of the major threads here have been locked by the moderators, so apologies if I should post this somewhere else.)

We at Anaconda have been watching the nogil work with great interest for some time. After some internal discussion, I wanted to announce that we are able to commit Anaconda engineer time toward the packaging challenges that will be associated with adopting PEP 703, including any work on pip, cibuildwheel, and conda-forge that will be needed to get nogil-compatible packages into the hands of the Python community. We can coordinate with the PyPA folks to see where we can be most useful if the PEP moves forward.

67 Likes
122

FYI: The steering committee decided that “We intend to accept PEP 703, although we’re still working on the acceptance details.”

A Steering Council notice about PEP 703 (Making the Global Interpreter Lock Optional in CPython)

4 Likes
123

The steering council has now formally accepted PEP 703:

12 Likes
124

FYI: The tracking issue for PEP 703 implementation in Python: PEP 703 -- Making the Global Interpreter Lock Optional in CPython · Issue #108219 · python/cpython · GitHub

4 Likes
125

Has the no-gil code been fully merged to main at this point or is there more to do? I’ve tried configuring with and without --disable-gil and was mildly surprised not to see a change in the number of test cases run. I would have expected to see some new test cases.

126

It’s work in progress.

Follow this issue for the current state:

2 Likes
127

Thanks. I now recall that, but must have lost track of it with all the other chit-chat (a couple threads with ~250 posts).