Alright, after a first round of discussion, I wanted to give my opinionated opinions about the main points of discussion that have been floating around, especially w.r.t. my earlier idea about protected scopes.
I wanted to be thorough in addressing the various concerns raised here, so please bear with this very long wall of text.
Automatically interrupt all running threads on SIGINT by default
While this may be a useful change in a number of scenarios, it is also backwards incompatible.
Suppose for the sake of argument that version 3.15 implements this change, then all Python programs that were using threads in Python <= 3.14 are required to consider that KeyboardInterrupt can now show up in all threads, not just the main one. So a refactoring needs to be performed before upgrading to 3.15. Whereas, having a Thread.interrupt() method that isn’t used implicitly by Python by default, allows individual codebases to designate their own paths towards using interrupts in threads, should they desire to do so.
Don’t have uninterruptible scopes as a language feature, instead have a defer_interrupts context manager
Something along these lines:
with threading.defer_interrupts():
dont_interrupt_me()
maybe_interrupt_here()
First off, note that this context manager must be implemented in C, without the additional language support proposed above. That is, to implement it, it must be that the code in the __enter__ and __exit__ methods themselves must not be interrupted, otherwise it would fail to provide the feature.
Secondly, however it’s implemented, the VM must be informed about the beginning and end of those deferred interruptions scopes, so the VM needs a way to keep track of them. This, I think would end up recreating the previously described stack of protected scopes, or something else which still needs to be stored in PyThreadState. Therefore, providing a language feature for this use-case would be no more costly (implementation-wise) than providing a special decorator.
Furthermore, a language-level feature would allow to apply the protection to all existing context managers already in use. I think that eventually, assuming the change is carried out, the inverse problem would apply: “Did you remember to wrap that context manager with threading.defer_interrupts?”
Instead, it should be fairly straight-forward to implement this context manager in Python with the additional language support, so that the protection can be extended outside the __enter__, __exit__, and finally scopes:
@contextlib.contextmanager
def defer_interrupts():
try:
pass
finally:
yield
Add a Thread.interrupt method, but don’t add interrupt-protected scopes
I think it would be hard for this change to get accepted, because:
- this behavior in other languages has been already painfully deprecated;
- the “naive misuse” alluded to in the docs of
PyThreadState_SetAsyncExcis exactly the case in which necessary cleanup code never gets executed because an exception (interrupt) gets raised out-of-thin-air at just the wrong time.
In essence, this would likely become a footgun-code printing machine, that programmers must be warned against misusing. The addition of protected scopes does not magically make all programs safe, but it provides a language-level mechanism to guarantee the execution of necessary clean up code, should there be any.
Instead of adding Thread.interrupt, add an interruptible thread subclass
I agree with what @gcewing has already said.
While this can be very useful to distinguish which code was written with interruptions in mind or not, I think it would also become problematic in a future where libraries always have to consider whether they’re being passed a Thread or a special InterruptibleThread, and have to avoid using interrupts in the first case.
Furthermore, libraries may get passed callables directly, in which case it’s up to the library to decide whether to dispatch the callable to a Thread, or an InterruptibleThread. And the library would essentially be forced to explicitly state in its documentation which one will be used. That would be no different than having an interrupt method in Thread and having the library explicitly state whether or not it will be used.
OTOH, if a codebase manages threads directly without third-party libraries, then it is up to the codebase’s own chosen standards whether to allow calling .interrupt() or not. Possibly, an existing codebase would never allow using that method if the cost of refactoring was too high. Regardless, the problem of upholding codebase-wide standards wouldn’t change if the codebase had to both use InterruptibleThread and then call .interrupt(), or just call .interrupt() on a regular Thread.
A keyword argument passed to the Thread constructor would be no different than having an InterruptibleThread class, in this sense.
User-defined Thread subclasses may already have an interrupt method
Then, those subclasses need not be changed. The existing interrupt method would get called instead of Thread.interrupt.
If the existing subclasses wanted to additionally use the new Thread.interrupt method they would only need to add a call to super, using the existing idioms.
This holds regardless of the name of the added method, although Thread.interrupt has come up repeatedly by multiple people, so I’ll take that as a sign that it is a good name.
Only have interruptions as a package on PyPI
Without language-level protections implemented with cooperation from the VM, this would be no better than what already outlined above in Add a Thread.interrupt method, but don’t add interrupt-protected scopes.
Don’t defer interruptions at all, prioritize stopping the thread
Interruptions need not necessarily correlate with program shutdown. They may also be used as a way to handle timeouts, or other cancellations, possibly in the context of structured concurrency. In such context, it may be important to a program that a thread stops for the sake of efficiency, when e.g. an operation is no longer necessary.
It may very well make sense for a thread to get interrupted, and decide to carry on doing something else. Thus, it is important that the thread maintains a consistent state, possibly by making use of the newly defined protected scopes.
ExitStack
This may already be the case if ExitStack was being used by the main thread and a KeyboardInterrupt was raised at just the wrong time inside enter_context(). With the proposed defer_interrupts() context manager it would be possible to fix the existing issue:
def enter_context(self, cm):
"""Enters the supplied context manager.
If successful, also pushes its __exit__ method as a callback and
returns the result of the __enter__ method.
"""
# We look up the special methods on the type to match the with
# statement.
cls = type(cm)
try:
_enter = cls.__enter__
_exit = cls.__exit__
except AttributeError:
raise TypeError(f"'{cls.__module__}.{cls.__qualname__}' object does "
f"not support the context manager protocol") from None
with defer_interrupts():
result = _enter(cm)
self._push_cm_exit(cm, _exit)
return result
As a general rule, I’d keep these stdlib usages outside the scope of a possible draft PEP, instead keeping the PEP about the language change itself and leaving further uses of the change to specific issues on the cpython repo.
Have the context manager constructors also be part of the protected scope
This can be very useful to allow open to also enjoy this protection. In this snippet, the file is opened before the __enter__() call:
with open("spam.txt") as f:
f.read()
I’d have to start writing a reference implementation to understand how hard it is to do, but I’m in favor.
Thread.interrupt should interrupt pending I/O, in addition to raising an exception
I also agree with this one. I think it should also be the current behavior, I’ll check this. If it is, then there’s little to be done in way of change here. Otherwise, I’ll weigh in the implementation cost.
with and finally aren’t special enough to deserve special treatment
That may be so.
Let’s suppose, though, that we did want to implement thread cancellation, and also had the problem of cleanup handlers at heart. What do others do to implement this feature?
Let’s take POSIX which has a pthread_cancel API. It is noted that:
When a cancelation request is acted on, the following steps occur for thread (in this order):
- Cancelation clean-up handlers are popped (in the reverse of the order in which they were pushed) and called. (See pthread_cleanup_push(3).)
- Thread-specific data destructors are called, in an unspecified order. (See pthread_key_create(3).)
- The thread is terminated. (See pthread_exit(3).)
So to implement something similar we need a stack of clean-up handlers, to be pushed by the thread as it executes its code, and possibly popped by the OS if the thread gets cancelled. Or, popped by the thread if it doesn’t get cancelled.
This strongly resembles the with statement.
Maybe it’s not special enough to deserve special treatment, but it is no more and no less than what is needed to cleanup a thread that’s being cancelled.
Regardless of the fact that with is exactly what’s needed to implement safe thread interruptions, it already is de facto the recommended best practice to handle resource cleanup. So providing extra protections to this statement may very likely improve the safety of already existing code, at the cost of slightly delayed KeyboardInterrupt exceptions.
What precisely are the semantics being proposed for thread cancellation?
The concern was raised here.
I would propose thread interruption, rather than cancellation, to be the ability granted to any thread to call a new Thread.interrupt method, which acts in a similar way to the current semantics for KeyboardInterrupt with additional guardrails for protected scopes.
Specifically, a call to Thread.interrupt would change the thread’s state to track that an exception (subclass of BaseInterrupt) is to be raised at some point. (Keeping it intentionally vague as it is now for KeyboardInterrupt.) Additionally, all subclasses of BaseInterrupt (including KeyboardInterrupt) may not be raised while a protected scope (__enter__, __exit__, and finally) is active. Thus, the interrupt sent to a thread will be deferred until the thread leaves all protected scopes.
Unsurprisingly, this doesn’t magically guarantee that the program is bug-free or infinite-loop-free. It is the job of the programmer to make sure that the interruption is received without bugs and as promptly as desired.