C API docs for free-threading?

Documentation
, ,
1

I noticed today that this page in the C API docs which describes the GIL and the macros for releasing and acquiring it has no mention of the free-threaded build. I think that is a little confusing, since the semantics are different on the free-threaded build, but you still need to save and restore a thread state.

I’m not opening a docs issue about this because I’m not sure about the wider policy of how much there is a desire to include free-threading specific copy in the documentation.

I personally found it useful to think about this specific topic in terms of “attaching” or “detaching” from the interpreter runtime. See the section in the PyO3 docs I wrote on this topic. This came up in PyO3 because lots of the PyO3 API uses gil in names or the docs talk about GIL lifetimes, and the meaning is now subtly different on the free-threaded build.

Maybe we could add some text to the C API docs that notes the existence of the free-threaded build and talks about what the meaning of e.g. Py_BEGIN_ALLOW_THREADS, Py_END_ALLOW_THREADS is and the need to use them, particularly if you need to re-attach to the runtime in a spawned thread to call into the C API, lest you run into a deadlock triggered by a global synchronization event, even in the free-threaded build.

Also see this PyO3 docs PR, which is trying to clear up some user confusion about how to use the rust bindings for the C API around thread states.

3 Likes
2

To add a little more context, there’s some discussion in PEP 703 about this: PEP 703 – Making the Global Interpreter Lock Optional in CPython | peps.python.org.

3

Agreed, the presence (and contents) of docs is going to be an important factor in whether it gets merged[1] or not.

Let’s just give it very clear markings for now. Possibly we need a new block type in the docs to make it stand out (which will also make it easy to remove the specific docs later if that’s what the decision is).

I also don’t see any harm in a dedicated (temporary) section specific to free-threaded in the main docs. It can link out to the specific areas. This would also be a helpful porting guide, as well as an overview of what’s actually changed.

  1. Referring to “making nogil the default”, not each individual merge into main. ↩︎

1 Like
4

Thanks to @ZeroIntensity for opening an issue about the specific problem I pointed about about thread states: Refer to the GIL as a thread state in the C API · Issue #127989 · python/cpython · GitHub

I’m taking the lack of pushback on this idea and the positive response from @steve.dower that this is something the project wants to do. I’ll take future specific asks to GitHub directly.

For a free-threading specific page in the CPython docs, there are some HOWTO guides that cover some of the material (e.g. this one and this one), and expanding those would be good. It might even make sense to move the first one out of the HOWTO section and into the broader docs somehow.

There’s some overlap with the material at https://py-free-threading.github.io, but we should try to make a distinction between information that belongs in the CPython docs and information that belongs in community docs. I don’t think the information about how to declare free-threading support in a cython extension makes sense in the CPython docs, for example.

1 Like