I realized that the logic I tried to implement might be currently impossible in python. I’ve enumerated all possible approaches to this problem.
This is what I meant. Interestingly I ended up with this conclusion with a wrong assumption that all contextmanagers are immune to async exceptions (interrupts). This was then further explored and it happens to hold true for (and only for) python’s own lock implementations.
In other words, using raw locks with context managers is currently the ONLY way to implement robust multithreaded code that can recover from interrupts. Any additional abstraction to raw lock primitives, including condition variables, will cause potential deadlock.
I am wondering if I should open a separate post just for this problem.
Edit: Note that 3.10+ is much better behaved in this regard than previous versions due to Only check evalbreaker after calls and on backwards egdes. Makes sure… · python/cpython@4958f5d · GitHub (hence the robust results with the native C-implemented locks), but any Python code which directly or indirectly triggers a function call return or a local backwards jump in the opcode evaluation sequence can still encounter an external interrupt while already running some form of exception handling.
Thank you @ncoghlan for the information! I did not know that this problem has been brought into your attention so long ago…
Is there any ongoing attempt/proposal to have “atomic” code blocks which can be free of async exceptions just like C-bindings?
Something like this:
And this:
class Wrapper:
atomic def __exit__(self, *_):
# KeyboardInterrupt here will be deferred
# until execution exits the function
self.cleanup()
Since deferring async exceptions is a breaking change that changes when interrupts are raised, I guess this has to be explicitly requested (hence a new keyword).
There haven’t been any recent suggestions in this area that I’m aware of.
For applications where the status quo is unacceptable, the workaround of separating the main application thread from the external interrupt signal handling thread gives more control over the behaviour than selectively enabling & disabling external interrupts within a thread would do.
With async support in the standard library, the ergonomics of that are also already pretty reasonable (run an async event loop in the main interpreter thread, launch the main application in a child thread).
The one inherently tricky part is that if done poorly, the event loop thread may not be able to reliably shut down the app execution thread when an external interrupt is received.
Which leads to the status quo persisting: for most applications, the risk of occasionally not cleaning up as gracefully as they should is more acceptable than the risk of sometimes not terminating gracefully when requested and the process needing to be externally killed anyway.
That seems redundant – if you’re marking the block as atomic with a keyword, there’s no need for the explicit calls to disable async exceptions. BTW I’m not keen on using the word ‘atomic’ for this, as it would be very misleading. It suggests something much stronger than it actually is, such as preventing thread switches within the block. Something more specific is needed if we have to use keyword – although I hope we don’t, see below.
That wouldn’t help with a related issue I have in mind, which is the
current inability to terminate a thread from outside. If it were
possible to write code that handles async exceptions safely, a way could
be provided to throw an exception into another thread.
[quote=“Greg Ewing, post:26, topic:68838, username:gcewing”]
That wouldn’t help with a related issue I have in mind, which is the
current inability to terminate a thread from outside.
[/quote]
The problem you describe is the one I was alluding to with the comment about “the event loop thread may not be able to reliably shut down the app execution thread”.
The safe non-interrupting way to do it is to use signal.set_wakeup_fd or PySignal_SetWakeupFd to wake up the app execution thread and ask it to terminate when the external signal is received. This only works if the app execution thread is consistently checking for termination requests - it does nothing if the app execution thread is genuinely stuck somewhere (or just running a long operation).
The dangerous way to do it is to use PyThreadState_SetAsyncExc, which means the app execution thread becomes vulnerable to the same reliable resource cleanup risks that arise from the signal checking in the main thread, defeating the purpose of adopting the split thread architecture in the first place.