Asyncio, tasks, and exception handling - recommended idioms?

In trying to debug asyncio code, I’ve run into some various issues, and ended up settling on this idiom for task spawning. I’m curious how other people do things, and if there should be a recommended way in the docs? Conceptually, I want to spawn a task the same way you’d spin off a thread or start an asynchronous subprocess, where I don’t ever want to await that, so any exceptions should be logged to the console.

import asyncio
import traceback

all_tasks = [] # kinda like threading.all_threads()
def task_done(task):
	all_tasks.remove(task)
	exc = task.exception() # Also marks that the exception has been handled
	if exc: traceback.print_exception(exc)
def spawn(awaitable):
	"""Spawn an awaitable as a stand-alone task"""
	task = asyncio.create_task(awaitable)
	all_tasks.append(task)
	task.add_done_callback(task_done)
	return task

Usage would be something like this:

async def task1():
	print("Starting task 1")
	await asyncio.sleep(1)
	print("Ending task 1")

async def task2():
	print("Starting task 2")
	await asyncio.slep(2) # oopsie typo

async def main():
	...
	print("blah blah stuff goes here")
	...
	spawn(task1())
	...
	spawn(task2())
	...
	await asyncio.sleep(3)
	print("Shutting down") # No hard feelings. I don't blame you. Critical error.

if __name__ == "__main__":
	loop = asyncio.new_event_loop()
	asyncio.set_event_loop(loop)
	loop.run_until_complete(main())
	print("Unfinished tasks:", all_tasks)

Critically, every spawned task is retained (see docs note which wasn’t there in Python 3.7 or 3.8), and when the task is done, any exception is immediately reported.

I’d very much like for this to have been more discoverable. Can something be added to the docs, or maybe even have an asyncio.spawn() function that behaves like this?

Hi Chris,

Clearly you’re trying to start a discussion, but I’m not sure about what.

Are you hoping to get a discussion started about the thing that hit Hacker News over the weekend (create_task() not keeping tasks alive)?

Is this your first foray into asyncio, or have you been debugging asyncio code for a long time and are you proposing this approach based on your cumulative experience? I can’t tell from your posting here.

I presume you’re not proposing that we literally hardcode a call to traceback.print_exception() in the asyncio library (it uses logging everywhere).

I personally feel that creating “background tasks” using create_task() is probably an anti-pattern, much more than creating “background threads”. Ideally almost all tasks should be waited for, e.g. using TaskGroup (new in 3.11). I certainly wouldn’t want to have a “spawn” function in the stdlib.

Instead of doing exception logging in a done-callback, why not wrap the task in a try/except?

1 Like

Not the first, because I don’t read Hacker News; consider that to be just a coincidence.

This is an ongoing project of my first really serious foray into asyncio, so what you see above is the result of a number of iterations; it’s not something I have a huge amount of experience with, so it’s entirely possible that I’ve missed something blatantly obvious to someone who spends more time with asyncio.

And no, I’m not thinking that hardcoded print_exception is a good idea, but we have sys.excepthook and threading.excepthook so maybe asyncio.excepthook could be the way to move forward? Again, if this exists, my apologies (it doesn’t exist under that exact name but I may have missed it elsewhere).

Interesting that create_task is an anti-pattern. I’m curious how you would go about doing these sorts of things - would you have a TaskGroup that you dynamically add more tasks to over time, or would there be many such groups? This is most commonly used when the app needs to, for instance, push some data out to an SSH connection (subprocess, write to its stdin), but doesn’t need to wait for it to finish.

(Also, since TaskGroup is so new, it’s going to be a while before it can be used; this project needs to be able to run on an RPi, and I believe it’s currently using Python 3.8. That’s not a fundamental problem as we could just build CPython from source if necessary, but I’d prefer to stick to the system Python if possible.)

Wrapping the task in a try/except is definitely an option. I suspect the reason I had trouble with this is due to not retaining references to spawned tasks, which I didn’t know was a problem when this project started. So it’s possible that simply wrapping everything in a try/except would have been sufficient, although we’d then also needed to have the task retention, so it wouldn’t end up materially simpler than the current way of doing things - it’d just be spelled differently (probably an async wrapper function that just goes “try: thing() except: print_traceback”).

So mainly, I’m hoping to hear from people who’ve been using asyncio actively for years, and how they (you) go about handing the unexpected. I’ve used asynchronous I/O in a wide variety of languages, and part of the art of setting up a usable system is having a robust error trapping framework so you KNOW what you’ve done wrong :slight_smile: And I’d much rather use a standard or well-known idiom than try to invent my own.

In asyncio you can set an exception handler with loop.set_exception_handler Event Loop — Python 3.11.2 documentation.

To add some clarity here: The app uses GTK as its UI, which means that any user-triggered actions come in via callback functions (connect signal to function). These are not async functions, so they have to do their work and then finish. Writing to a subprocess’s stdin is usually fast, but is technically asynchronous, and could potentially take a bit of time (if the other end has stalled or something), so we can’t just wait for it before returning. Thus the need to spin off an asynchronous task and return.

TaskGroup looks really nice for the situations where there’s a well-defined group (“do all these things and don’t return till they’re all done”), but I’m not sure how I would use it here.