This thread is a more detailed comparison of asyncio with the paper I mentioned in a prior thread. I’m not going to insist to add something right now to asyncio based on this paper, but just want to share how we are doing well and potential points to improve in the future.
Although the paper does not mention Python asyncio at all but comes with case analysis of cancellation-related bugs in other languages (C#, Java, Go) and applications (Cassandra, Hadoop, etc.), the meaning of the paper is that it provides a structured base to discuss what kinds of issues should be considered when we are going to improve asyncio.
TLDR: Thanks to the core contributors and the community, Python asyncio has constructed a fairly good baseline for writing complex concurrent applications. I think it would be insightful to retrospect asyncio’s achievements with the arguments of the paper as it serves as a reference point.
Reference in the paper | Short desc. | Status of asyncio |
---|---|---|
§5.1.2, §5.1.3 | Broken trigger checks, Excess cancel (unexpected overlapping of tasks without cancelling previously launched ones which may happen with async timers) |
No intrinsic utility for timers or periodic tasks[1] – it may be controversial to include timer APIs in asyncio, though. / Guaranteeing scoped cancellation in complex task trees is partly covered with TaskGroup . |
§5.2.1 | Untimely delivery - cancel race |
asyncio.shield() could be used to circumvent duplicate cancel requests and protect ongoing cancellations[2], but still the event loop still unexpectedly shuts down all ongoing tasks regardless whether they are shielded or not. |
§5.2.1 | Untimely delivery - late polling | asyncio is based on inherently “cooperative” coroutines and it should be fine as long as all dependencies are async-aware (i.e., no hidden blocking calls) / Programmer’s caution is still the main mean of preventing this issue and this is the reason of difficulty when writing asyncio apps. |
§5.3.1, §5.3.2 | Cacnel not checked/carried out |
The situation has improved a lot after promoting asyncio.CancelledError as BaseException (since Python 3.8) / A structured way of ensuring clean-up of long-running tasks are still missing (particularly TaskGroup )[3]
|
§5.4 | Cancel mechanisms - cancellable task interface |
asyncio.Task and asyncio.Future
|
§5.4 | Cancel mechanisms - uninterruptible interface |
asyncio.shield() partly corresponds to this, in that it blocks the cancellation signal. |
§5.4, §5.3.3 | Cancel mechanisms - task dependency tracking |
asyncio.TaskGroup
|
-
aiotools
’ timer module hasTimerDelayPolicy
to express “cancel prior task if it’s not complete until the next wait interval finishes”. ↩︎ -
I have an experience that I had to shield all database transactions in web request handlers because cancelling in the middle of
__aexit__()
ofaiopg
’s transaction blocks had caused unreleased database connections and exhaustion of the connection pool. ↩︎ -
This is why I’ve suggested
PersistentTaskGroup
, to cover the use case ofasyncio.gather(..., return_exceptions=True)
. ↩︎