"breaking in" to Python from a wild native thread (Python 3.8)

I’m writing a C extension for Python that interfaces with some native APIs - for example, a Windows timer API which triggers callbacks from a foreign thread I have no direct control over (until stopping the timer entirely).

What is the right way to allow this ‘intruding’ external thread to call back into Python, on its own schedule?

From reading the documentation, it would seem that wrapping my callback code with PyGILState_Ensure and PyGILState_Release would be enough - however that is not working for my little timer callback.

My main Python code is starting the C timer, sleeping for 10 seconds, and then stopping the timer. The C callback code should be invoking a python callback provided elsewhere (and working fine in a single-threaded / synchronous context), but here seems to be hanging/crashing/etc - basically just not working, right after the timer-start call.

What is the minimum amount of ceremony required from a non-Python-controlled external thread (that even I didn’t create!), to safely and correctly invoke a python callback?

The PyGILState APIs should work for this. However, you must make sure that the main thread gives up the GIL (either by continuing to run Python bytecode, or by giving up the GIL in the C extension with “starting the C timer and sleeping for 10 seconds” is done in a C extension).

1 Like