Thank you, everyone, for responding to the poll on the no-GIL proposal. It’s clear that the overall sentiment is positive, both for the general idea and for PEP 703 specifically. The Steering Council is also largely positive on both. We intend to accept PEP 703, although we’re still working on the acceptance details.
As we’ve done a few times in the past, we want to communicate our intent to accept the PEP along with where our current thinking is on the details around acceptance.
Our base assumptions are:
Long-term (probably 5+ years), the no-GIL build should be the only build. We do not want to create a permanent split between with-GIL and no-GIL builds (and extension modules).
We want to be very careful with backward compatibility. We do not want another Python 3 situation, so any changes in third-party code needed to accommodate no-GIL builds should just work in with-GIL builds (although backward compatibility with older Python versions will still need to be addressed). This is not Python 4. We are still considering the requirements we want to place on ABI compatibility and other details for the two builds and the effect on backward compatibility.
Before we commit to switching entirely to the no-GIL build, we need to see community support for it. We can’t just flip the default and expect the community to figure out what work they need to do to support it. We, the core devs, need to gain experience with the new build mode and all it entails. We will probably need to figure out new C APIs and Python APIs as we sort out thread safety in existing code. We also need to bring along the rest of the Python community as we gain those insights and make sure the changes we want to make, and the changes we want them to make, are palatable.
We want to be able to change our mind if it turns out, any time before we make no-GIL the default, that it’s just going to be too disruptive for too little gain. Such a decision could mean rolling back all of the work, so until we’re certain we want to make no-GIL the default, code specific to no-GIL should be somewhat identifiable.
As such, what we currently see as the way forward is three stages:
Short term, we add the no-GIL build as an experimental build mode, presumably in 3.13 (if it slips to 3.14, that is not a problem). We want the build mode to be experimental to make it clear that while the core devs support that build mode, we can’t expect the community to support it outright. We need time to figure out what we need to do, at the very least in terms of API design and packaging and distribution, to enable the community to support it. We also want to discourage distributors from shipping the experimental no-GIL build as a default interpreter.
Mid-term, after we have confidence that there is enough community support to make production use of no-GIL viable, we make the no-GIL build supported but not the default (yet), and set a target date/Python version for making it the default. The timing is going to depend a lot on, for example, how backward compatible the API changes end up being (e.g., what to do about the stable ABI), and how much work the community thinks they still need to do. We expect this to take at least a year or two, possibly more. Once we declare it supported we expect some distributors may start shipping no-GIL by default, although it will probably vary greatly by how many other Python packages support no-GIL at that point.
Long-term, we want no-GIL to be the default, and to remove any vestiges of the GIL (without unnecessarily breaking backward compatibility). We don’t want to wait too long with this, because having two common build modes may be a heavy burden on the community (as, for example, it can double test resources and debugging scenarios), but we can’t rush it either. We think it may take as much as five years to get to this stage.
Throughout the process we (the core devs, not just the SC) will need to re-evaluate the progress and the suggested timelines. We don’t want this to turn into another ten year backward compatibility struggle, and we want to be able to call off PEP 703 and find another solution if it looks to become problematic, and so we need to regularly check that the continued work is worth it.
We hope that this gives some clarity into the future of the PEP while we work on the exact acceptance details. The SC will work to finalise the acceptance over the coming weeks.
I wish I had the words to express my gratitude for the work that you folks put in on things like this. I don’t, so, just: thank you, so much. Another reason why I am really glad I decided to start teaching myself Python.
I respect the decision however unfortunately I am not in favor of it. Nevertheless I hope for and wish the core devs all the success and luck for a good positive outcome.
Thanks for the carefulness you are applying to the whole process whilst being positive in getting the changes. The whole CPython community will really appreciate not to have again a python2/3 situation.
Thank you for sharing this notice in anticipation of the official acceptance - this is super exciting!
@colesbury is currently out on vacation, but should be back in a couple of weeks.
We at Meta are excited about the intention to accept PEP-703, and are looking forward to getting to work on a smooth landing of the implementation!
I’m happy to be an early adopter, and I know that a handful of packages have already been modified to work with nogil. Are those versions available somewhere?
Maybe this question is jumping the gun and there will be a more “official” way to do this in the future?
Excited to follow this! As soon as cibuildwheel supports it, I’ll add the additional wheels for MarkupSafe. Or add some experimental builds somewhere before that.
I’m beyond stoked for this. If Python can run truly concurrent code without sacrificing the current single core execution speed of 3.12, that’d be a huge win for the community and people who are heavily invested in this ecosystem.
I’ll start testing it on my code the moment the nogil flag becomes publicly available. Also curious to see how it’ll break single threaded code with tons of global mutable state.
It is very much NOT official, but you should be able to play with the prototype implementation @colesbury published together with the PEP (iirc there’s a 3.9-based implementation and a 3.12-alpha (or beta?) based implementation)
Hope for backward compatibility and easy migration.
Data races caused by removing GIL are sometimes hard to catch and reproduce. Migrating single-threaded code is much more error-prone than writing everything in a multi-threaded fashion in the first place.
This is one of the areas that will meaningfully affect the entire packaging and distribution tooling ecosystem for Python (as discussed to varying in the various PEP 703 threads), largely based on how this would interact with platform compatibility tags which are a foundational part of how wheels “work”.
This will affect installers (i.e. pip), build-backends (like setuptools, scikit-build-core, meson-python etc) and build orchestration tooling (i.e. cibuildwheel, conda-build). It, indirectly, also affects downstream redistributors (like Conda & Linux distros), who use pip+build-backends under the hood to actually build the packages and repackage them in their own formats.
Just to help set expectations, the SC is still talking about how best to land the changes, what will be necessary to expose it to users even as an experiment, etc. As such, don’t be shocked if this doesn’t go public in an official release until Python 3.14 (October 2025). Obviously those who test against main will have much earlier, experimental access as things land. And it’s obviously possible things get in fast enough to make Python 3.13 (October 2024).
