A new API for ensuring/releasing thread states

Core Development C API
21

I’m not entirely convinced by the use cases and I’m concerned about a number of behavioral details. I think it’s important that any new API interacts correctly with existing APIs like PyGILState_Ensure()/Release(). Real use cases often involve multiple C API extensions so any given thread might use a mix of existing APIs like PyGILState_Ensure and new APIs like PyThreadState_Ensure.

I don’t think the linked PR is in a good enough state to evaluate, and I think having an actual implementation is important here to understand the behavior.

  • This API and PyGILState_Ensure() center around thread-local variables. For PyGILState_Ensure, it’s autoTSSkey, which is the potentially inactive PyThreadState per-OS thread. Here I think you need something that’s per (OS thread x interpreter). How is that going to interact with PyGILState_Ensure()? The PR currently uses current_fast_get() (_Py_tss_tstate) – that’s definitely not correct.
  • What happens if PyThreadState_Ensure(interp2, ...) is called while a thread state from a different interpreter is active? What does PyThreadState_Release() do in that case? If it reactivates the original thread state, then it might fail. Can PyThreadState_Release() fail?
  • This API isn’t enough to reliably avoid hanging C++ initialized threads [1] during Python shutdown outside of trivial cases. There’s a good deal of code that calls Py_BEGIN/END_ALLOW_THREADS internally and Py_END_ALLOW_THREADS [2] will hang during Python shutdown in 3.14+.

  1. Thread states created by PyGILState_Ensure() or PyThreadState_New() outside of the threading module. ↩︎

  2. or PyEval_RestoreThread() or PyEval_AcquireThread() ↩︎