Is there a safe way to use `PyThreadState_SetAsyncExc` without causing deadlocks?

Python Help
1

Hi.

I’m the author of a library that makes common use of threading.Lock to ensure thread-safety. However, a user has reported a deadlock case. Upon further investigation by this user, it appears that the deadlock was likely related to the use of PyThreadState_SetAsyncExc in another third-party library (see source code here).

Since I always use the lock as a context manager, I was confused because I expected the lock to always be released properly. However, it turns out this pattern does not seems robust against PyThreadState_SetAsyncExc. I was able to reproduce a deadlock on Python 3.14 with the following minimal example (you must run it multiple times, e.g. by using a Bash while loop, and eventually a deadlock will occur):

import ctypes
import threading
import time

lock = threading.Lock()


def lock_thread():
    while True:
        with lock:
            pass


t = threading.Thread(target=lock_thread)
t.start()

time.sleep(0.01)

res = ctypes.pythonapi.PyThreadState_SetAsyncExc(
    ctypes.c_long(t.ident),
    ctypes.py_object(RuntimeError),
)

assert res == 1

t.join()

print("Locked: ", lock.locked())

with lock:
    print("Deadlock?")

I guess it makes sense, because PyThreadState_SetAsyncExc can stop the thread at any arbitrary point, thus causing unexpected behavior since the Lock API isn’t atomic. Yet, I was a bit surprised, since this kind of pitfall is not much emphasized in the documentation.

First, can you please confirm this is not a bug in CPython, and simply a consequence of using the C API this way?

Secondly, is there anything I can do, as a library developer, to prevent such deadlock to occur in my code due to externally raised exceptions? I don’t feel I can exhaustively guard my code against PyThreadState_SetAsyncExc.

2

First, can you please confirm this is not a bug in CPython…?

I think it’s a bug or at least a limitation of CPython. This sounds a lot like Deadlock in logging · Issue #88105 · python/cpython · GitHub.

What’s happening is basically that with lock gets executed as:

lock.__enter__()
try:
   ...
finally:
   lock.__exit__()

The problem is that the asynchronous exception may be thrown between the lock.__enter__() and the try or between the finally and the lock.__exit__.

We might be able to fix this for with statements, but there are still going to be other parts of the standard library that aren’t robust to exceptions being thrown at random places. For example, parts of threading.Condition are implemented in Python and if you throw exceptions in random places, it will mess up the internal state (leading to deadlock).

No, I don’t think so, other than tell people not to use PyThreadState_SetAsyncExc.

3

While ctypes is “you’re on your own” territory, the thing that I would consider regrettable in CPython is if this implied a KeyboardInterrupt with a plain threading.Lock could cause a deadlock.

Related:

4

Thank you both for your answers. I totally understand the technical challenges associated with this API.

I guess a brief warning in the documentation would be helpful, though, to serve as an authoritative reference regarding the risk of deadlocks. It wasn’t obvious at first, though it makes sense to me now.