PEP 703: Making the Global Interpreter Lock Optional (3.12 updates)

[h-vetinari]

@pf_moore

Pip, for example, will try to build a wheel if one doesn’t exist - and it does that using default build settings, so there’s a built in assumption that “build a wheel” will by default build one that’s compatible with the running python.

This point is crucial – is pip and related infrastructure ready to serve a dual-ABI world? That’s also breaking an ancient assumption. I brought it up in the other thread, but didn’t get much response.

Given the situation in the packaging space (maintainers far above capacity, huge amount of complexity / tooling diversity, many long-standing & severe painpoints, lots of legacy cruft, etc.), I doubt that trying to push such a huge change through this bottleneck will be helpful^[1], i.e. this should factor into the decision of whether to aim for parallel ABIs or version-separated ones (i.e. Python 3.x=with gil / Python 4.x=nogil).

@jamestwebber

As H Vetinari pointed out, it’s totally possible that there are some deep assumptions somewhere in CPython that make this untrue, but it doesn’t sound like they’ve been uncovered by the testing so far.

It doesn’t need “deep assumptions somewhere in CPython” for pure python code to be possibly affected.^[2]

@jamestwebber

For more complicated dependency graphs, it seems like there’s a transitive property. If I need A, and A depends on B, … and Z is a C extension that hasn’t been released for nogil, then I can’t install A because pip won’t resolve the dependencies.

Yes, and that’s entirely reasonable. If you depend on Z, and it’s not nogil-safe, you cannot use it in a nogil-only context without stuff potentially blowing up. The situation does explain the desire towards the runtime switch though (or more generally, running gil & nogil code in the same interpreter). If that encapsulation could somehow be made fool-proof, then you could just^[3] run Z with the gil, but keep all the other dependencies running without it.

To be clear – I want nogil to succeed – but I’m trying to point out how large the surface for potential problems is (which explains a large part of the hesitation in this thread). There’s also a severe mismatch here (well, even more than usual) in how much the natural desire of users for an obvious improvement will translate into a huge chunk of work for volunteer maintainers all over. It’s great that there’s so much excitement about it, but more often than not, that excitement will turn into impatience rather than someone assisting the maintainers in climbing down some really deep debugging rabbit holes.

---

1. but then, as someone who’s very involved in packaging, I acknowledge my biases on this ↩︎

2. In a footnote because I’m repeating myself: it’s enough to rely on non-trivial third-party dependencies, or do some things (like IO) where threading might change observable behaviour (especially losing some implicit ordering in handling external resources like files, opening the possibility for data races, concurrent access problems, etc.); aside from the fact that even pure python libraries might not be ready for being called in parallel from other, now suddenly nogil, python code. ↩︎

3. a very load-bearing “just” ↩︎