How can I do a `Py_IsFinalizing` check "atomically"?

Core Development C API
1

I am implementing a daemon periodic thread using C++. These threads need to call into Python periodically. Because any instances of this thread might be alive when the interpreter is finalising, I seem to be getting terminate called without an active exception as a result of trying to acquire the GIL through PyEval_RestoreThread (looking at the core dumps, I can see this is due to pthread_exit being called in the process). The linked documentation suggests using Py_IsFinalizing, which I am doing, but it doesn’t seem to help. My hunch is that, by the time PyEval_RestoreThread is called after the Py_IsFinalizing check, the interpreter might be shutting down. So I was wondering if there is a lock that could/should be acquired before calling Py_IsFinalizing, e.g.

{
    <acquire lock>
 
    if (!Py_IsFinalizing()) {
        PyEval_RestoreThread(...);
    }

    <release lock>
}
1 Like
2

Indeed, it seems the recommandation given in the docs isn’t thread-safe, ironically :slight_smile:

We would need variants of PyEval_RestoreThread and PyGILState_Ensure that return an error instead of aborting the thread.

1 Like
3

I don’t think any potential changes would be backported as far back as 3.7 though? I might have to rely on atexit for a workaround, as it seems any registered functions are called just before the finalising state is set.

4

This would be useful for PyO3 too; users have hit the same type of issue of destructors calling these APIs and triggering a crash.

5

@eric.snow is actively thinking about/working on this, and it’s ultimately much broader than it seems at first glance. Depending on how much pushback we get, hopefully 3.14 will have some more sensible handling of state when it comes to shutdown.

Ultimately, CPython has spent a long time assuming that its only real use was in its own “python.exe”, and so “shutdown” meant the entire process was about to end. Unwinding these assumptions - without breaking everyone who’s currently made things work! - is not a trivial problem.

1 Like