Posting for the whole Steering Council, on the subject of @colesbury’s PEP 703 (Making the Global Interpreter Lock Optional in CPython).
Thank you, everyone, for responding to the poll on the no-GIL proposal. It’s clear that the overall sentiment is positive, both for the general idea and for PEP 703 specifically. The Steering Council is also largely positive on both. We intend to accept PEP 703, although we’re still working on the acceptance details.
As we’ve done a few times in the past, we want to communicate our intent to accept the PEP along with where our current thinking is on the details around acceptance.
Our base assumptions are:
- Long-term (probably 5+ years), the no-GIL build should be the only build. We do not want to create a permanent split between with-GIL and no-GIL builds (and extension modules).
- We want to be very careful with backward compatibility. We do not want another Python 3 situation, so any changes in third-party code needed to accommodate no-GIL builds should just work in with-GIL builds (although backward compatibility with older Python versions will still need to be addressed). This is not Python 4. We are still considering the requirements we want to place on ABI compatibility and other details for the two builds and the effect on backward compatibility.
- Before we commit to switching entirely to the no-GIL build, we need to see community support for it. We can’t just flip the default and expect the community to figure out what work they need to do to support it. We, the core devs, need to gain experience with the new build mode and all it entails. We will probably need to figure out new C APIs and Python APIs as we sort out thread safety in existing code. We also need to bring along the rest of the Python community as we gain those insights and make sure the changes we want to make, and the changes we want them to make, are palatable.
- We want to be able to change our mind if it turns out, any time before we make no-GIL the default, that it’s just going to be too disruptive for too little gain. Such a decision could mean rolling back all of the work, so until we’re certain we want to make no-GIL the default, code specific to no-GIL should be somewhat identifiable.
As such, what we currently see as the way forward is three stages:
- Short term, we add the no-GIL build as an experimental build mode, presumably in 3.13 (if it slips to 3.14, that is not a problem). We want the build mode to be experimental to make it clear that while the core devs support that build mode, we can’t expect the community to support it outright. We need time to figure out what we need to do, at the very least in terms of API design and packaging and distribution, to enable the community to support it. We also want to discourage distributors from shipping the experimental no-GIL build as a default interpreter.
- Mid-term, after we have confidence that there is enough community support to make production use of no-GIL viable, we make the no-GIL build supported but not the default (yet), and set a target date/Python version for making it the default. The timing is going to depend a lot on, for example, how backward compatible the API changes end up being (e.g., what to do about the stable ABI), and how much work the community thinks they still need to do. We expect this to take at least a year or two, possibly more. Once we declare it supported we expect some distributors may start shipping no-GIL by default, although it will probably vary greatly by how many other Python packages support no-GIL at that point.
- Long-term, we want no-GIL to be the default, and to remove any vestiges of the GIL (without unnecessarily breaking backward compatibility). We don’t want to wait too long with this, because having two common build modes may be a heavy burden on the community (as, for example, it can double test resources and debugging scenarios), but we can’t rush it either. We think it may take as much as five years to get to this stage.
Throughout the process we (the core devs, not just the SC) will need to re-evaluate the progress and the suggested timelines. We don’t want this to turn into another ten year backward compatibility struggle, and we want to be able to call off PEP 703 and find another solution if it looks to become problematic, and so we need to regularly check that the continued work is worth it.
We hope that this gives some clarity into the future of the PEP while we work on the exact acceptance details. The SC will work to finalise the acceptance over the coming weeks.