Safely using the C API when Python might shut down

A question for @wjakob, @pitrou, @davidhewitt and others in similar situations: what will you do if PyGILState_EnsureOrSafelyFail() returns an error code indicating failure? How are you going to handle that?

For PyTorch, I think the only safe thing we can do is to hang the thread, like PyGILState_Ensure() now does in 3.14. That case is going to happen sometimes even with a PyGILState_EnsureOrSafelyFail(): if you have non-trivial code between PyGILState_EnsureOrSafelyFail() and PyGILState_Release(), there’s a chance you may incidentally call code that internally releases and attempts to re-acquire the GIL, which will hang the thread. For example, any Python code that calls into the interpreter may trigger a GC, and collecting things like file objects whose finalizers temporarily release the GIL.

Furthermore, we can’t reliably pass an error up because we’re sometimes in situations where C++ calls into Python, which calls back into C++, which calls into Python:

C++ -> Python -> C++ -> Python
                    [^^ PyGILState_EnsureOrSafelyFail() fails here]

I don’t think there’s any way to safely pass an error or throw a C++ exception through the earlier Python context while the Python interpreter is shutting down.

1 Like