Pain point in asyncio: Potentially-failing tasks

Async-SIG
21

I’m honestly not too familiar with asyncio, but for me (on python 3.13.5) the original main_async_bad version does end up printing a traceback (even without awaiting the task):

Starting async!
Ending.
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<adelayed_boom() done, defined at .../foo.py:10> exception=AttributeError("'NoneType' object has no attribute 'sense'")>
Traceback (most recent call last):
  File ".../foo.py", line 12, in adelayed_boom
    None.sense
AttributeError: 'NoneType' object has no attribute 'sense'

Am I missing something?

And you can even get the exception to print “in a timely manner”, if you del task (or don’t save task into a temporary variable in the first place). So, this

async def main_async_bad():
	print("Starting async!")
	asyncio.create_task(adelayed_boom())
	# ^ create the task and immediately drop it
	await asyncio.sleep(3) # it's too fiddly to wait for input
	print("Ending.")

prints the exception traceback before the Ending.:

Starting async!
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<adelayed_boom() done, defined at .../foo.py:10> exception=AttributeError("'NoneType' object has no attribute 'sense'")>
Traceback (most recent call last):
  File ".../foo.py", line 12, in adelayed_boom
    None.sense
AttributeError: 'NoneType' object has no attribute 'sense'
Ending.

My understanding is that the Task exception was never retrieved thing is triggered when a task with a pending exception is deleted before being awaited. It obviously can’t / shouldn’t print an exception before that point, because otherwise you could end up printing the exception twice.

In fact your spawn implementation has just such an issue:

async def main_async_good_but_oops():
	print("Starting async!")
	task = spawn(adelayed_boom())
	await asyncio.sleep(3)
	# At this point the task has already printed its traceback,
	# but awaiting it will re-raise the exception, printing it again.
	await task
	print("Ending.") # Won't happen

prints

Starting async!
Traceback (most recent call last):
  File ".../foo.py", line 12, in adelayed_boom
    None.sense
AttributeError: 'NoneType' object has no attribute 'sense'
Traceback (most recent call last):
  File ".../foo.py", line 80, in <module>
    asyncio.run(main_async_good_but_oops())
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../asyncio/runners.py", line 195, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File ".../asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File ".../asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File ".../foo.py", line 72, in main_async_good_but_oops
    await task
  File ".../foo.py", line 12, in adelayed_boom
    None.sense
AttributeError: 'NoneType' object has no attribute 'sense'

It seems to me that the only real issue with asyncio.create_task is that any tasks that outlive the asyncio.run call end up being cancelled.

But that’s not a case of “Errors passing silently”, but rather “Code being interrupted before managing to reach the error”.

Also, I think that your spawn implementation is still susceptible to this issue (via the except asyncio.exceptions.CancelledError: in handle_errors).

But afaik, this is a fundamental limitation of async loops. Unlike threads and processes, an async task can’t escape past the lifetime of it’s parent async loop, so you are forced to either wait for all tasks to complete (like with asyncio.TaskGroup.create_task) or to cancel any tasks that are still in progress (like with asyncio.create_task).

22

The biggest part you’re missing here is the “timely” aspect - that traceback only shows up when the program terminates. This isn’t a huge deal with a three second sleep, but it makes a lot of difference when you’re running a GUI program and trying to figure out why nothing’s happening, and then eventually you get some tracebacks, completely disconnected from the user inputs that triggered them.

In this example, yes. It happens that dropping the task on the floor does, in this simple example, result in the traceback getting printed. That is not guaranteed though, and in other Python versions or in more complicated programs, it may very well not. (That’s why the docs clearly warn about them being leaked.)

That’s definitely possible. But the task might be deleted BEFORE the exception occurs.

There’s a fundamental difference here between async tasks and every other form of asyncio (threads, processes, and some others that aren’t in the stdlib). Every other thing retains a reference elsewhere - if you spawn a thread and don’t keep a reference to it, the thread keeps itself alive. There is no notion of abandonment. That only happens with asyncio.

Ah, very true. I think the correct solution here is to make spawn no longer return the task; the intent with spawning is that you DON’T then await it - it’s been “spun off” like a started thread. Keep in mind, this whole system of task spawning was developed progressively over several Python versions and at the cost of much hair, trying to figure out what was going wrong; I make no promises that it’s now perfect. This is one of the reasons that I want the stdlib to make this functionality available - there are a lot of odd quirks to it, and as you’ve seen, my code still hasn’t gotten it all right.

Not sure what you mean here, whether you mean just calling create_task and dropping them, or having to await them all at some point (WHAT point though?), or something else. Assume that tasks generally will not outlive asyncio.run(). Assume that the main task may run for hours and is heavily interactive, but the tasks being spawned may only take a fraction of a second, or maybe a few seconds at most.

That isn’t a problem. The problem is what happens with short-lived tasks that aren’t part of a “wait for all of these to finish” group, but should be able to outlive the individual tasks that spawned them (one task should be able to spawn another task, same as threads can).

23

To give a concrete example of something like this that occurred in production systems I used to work on, consider the following:

A user creates a record in the system using a web service. The database transaction happens when the user hits “Submit”, and the webpage displays a notification once the database commit has successfully completed. At the same time, an task to send a confirmation email to the user is submitted in the background. The email is not essential, so the email task must not block the web notification, nor must any failure in the email submission cause the web response to fail. The email should happen “in a timely manner” but there’s no need for it to happen to any particular schedule. If the email fails, that’s fine. Just ignore the error, the user has seen the confirmation on the webapp.

This is precisely where I’d want to use a “fire and forget” mechanism for the email. Larger systems quite probably have some sort of email sender service, fed from a message queue or similar, so there’s infrastructure in place for this type of task. But for small-to-medium sized systems, a simple “fire and forget” background task is ideal, and exactly matches the intent.

I don’t think this mechanism is essential, in the sense that it’s certainly possible to implement it for yourself if the stdlib doesn’t provide it. But it’s convenient to have it as a primitive, because “implement it yourself” could require quite extensive changes to the architecture of your system (and “please can we send a confirmation email” is the sort of feature request that often comes after the basic application architecture is already locked down :slightly_frowning_face:).

3 Likes
24

Well, no. It shows up when we are certain[1] that nobody is going to try awaiting the task - after the task is deleted. It’s true that you are technically not guaranteed immediate deletion, but that’s not the same as “until the program terminates”. Although, I suppose it could take a long time[2], if the user creates a cycle that references the task. But that still requires that the user actually store the task somewhere.

Do you have an example, where a task can be kept alive for an arbitrarily long period of time, despite the user immediately dropping the return value of asyncio.create_task?

Are you sure, that this is possible? I was under the impression that the event loop itself holds references to all scheduled and currently running tasks. So even when you “drop” the created task immediately, it only gets actually deleted after it “completes” (whether by finishing successfully, raising an unhandled exception or being cancelled).

Ah, yes. If by “the main task” you mean the top level async function that was started by asyncio.run, then there is no issue.

I meant that something like this

import asyncio, time

async def adelayed_boom():
    await asyncio.sleep(1.0)

    # This line will never be reached, because asyncio.run(this_is_bad())
    # will exit before the above 1.0 second sleep has the chance to finish
    # and this task will be cancelled.
    None.sense

async def this_is_bad():
    print("Starting async!")
    asyncio.create_task(adelayed_boom())

    # Oops! We only waited for 0.5 seconds here, which is less than 1.0 above.
    await asyncio.sleep(0.5)
    print("Ending.")

def the_main_task():
    asyncio.run(this_is_bad())
    print("At this point, we already cancelled the adelayed_boom.")
    time.sleep(10) # <- the "long running" part

if __name__ == "__main__":
    the_main_task()

But it’s somewhat arguable if this should even be considered an issue (it shouldn’t be a problem under the assumptions that you’ve outlined).

So, if I am understanding your point correctly, your main issue is with the lack of strict guarantees about task deletion timing? If you knew for a fact that Task.__del__ will be called “in a timely manner” after the task raises an Exception, then your problem would be solved?

  1. barring any object resurrection shenanigans ↩︎

  2. until the next gc sweep? ↩︎

25

“Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks.” create_task docs

It doesn’t happen in this simple example, but it absolutely CAN happen, according to the docs. This is the difference I mentioned - threads and subprocesses don’t have this quirk.

Yeah, that shouldn’t be an issue. I’ve only ever used asyncio with something where the master task (a GUI, a socket server, etc) is always running, That task can spawn tasks (which might fail) which might spawn tasks (which might fail), and shouldn’t have to retain responsibility for them.

Based on my reading of the docs, no; task deletion is not a safe way to do things. It’s the same problem as with open files - relying on file object deletion to close the file is not recommended. Task exception handling should be independent of the ownership of any controlling object.

1 Like
26

This is a perfect use-case for a process-scoped TaskGroup. Presumably, once you’ve dealt with the final request when shutting down, you’d want to give the background tasks a grace period to finish (because the final request may have enqueued something)? TaskGroups just solve this.

27

You mean like

main_tg = None

async def main():
    global main_tg
    with TaskGroup as main_tg:
        await real_main()

async def real_main():
    assert main_tg is not None
    ...
    # Start a "fire and forget" task
    main_tg.create_task(...)

I guess that works, although I can’t say I like having to manage a global like this.

In the cases I’m thinking about, the process runs indefinitely. The only shutdown is a controlled shutdown of the service, or a crash. In the case of a controlled shutdown, yes, maybe there would be a “stop servicing requests, wait 5 minutes, then shut down” process. But as I say, I’m thinking of small to medium scale systems[1], where just killing the server is possibly more typical way of shutting down :slightly_smiling_face:.

  1. i.e., hobby projects and the like ↩︎

1 Like
28

Whether you use a global or not depends on what architecture you’re using, not on the technique itself. In a FastAPI app I would initialize it in a lifespan and have it injectable into handlers as a dependency, etc. But yeah, that’s the general idea.

29

What does the TaskGroup buy us that a simple set of unfinished tasks doesn’t? I’m not quite following. I thought the purpose of a TaskGroup was “if any one of these fails, cancel all the rest of the tasks”, which is the exact opposite of the desired behaviour.

1 Like
30

In terms of background tasks, things that actually work like people want are possible.

Abstracting it to the point of submitting tasks to a loop running in another thread (useful for GUI/TUI main thread situations)

Context manager that’s leaves error handling of failed tasks to the user

Executor-like API

I wouldn’t use TaskGroups for the cases people have mentioned here.

31

Using TaskGroups ensures your tasks take the shape of a tree (acyclic DAG or whatever) - no child tasks outlive the parent. This is structured concurrency in a nutshell.

By default, an error in a child will interrupt the parent at certain points, but like I showed you with my snippet upthread, you can change this. Also by default, the parent will wait until all the children are done, but for example you can use something like this to change that too.

32

Then, yes, that is specifically NOT what I’m looking for. Great tool for when it’s wanted, but I am not looking for that. I want tasks to be able to spawn other tasks (think like a button click task that causes a longer-term action - there’s no point keeping the button click task around just because there’s another task). Having a single global TaskGroup kinda says “hey we have no child task outliving the parent, but we only have one parent, and if a child task spawns a task, it’s a sibling”, which isn’t going to give any kind of structured concurrency.

3 Likes
33

So I have to wrap my code in an “ignore exceptions” helper, is that right?

So you’re suggesting that asyncio should get an equivalent of the quattro background_task API? Or am I misunderstanding you? Are you suggesting I should write such a thing?

We seem to keep talking past each other a little. You’re saying that the various things I’m suggesting can be done. I don’t dispute that. What I’m saying is that the idea of a “fire and forget” task is fairly intuitive, and as such it’s worth having in the stdlib, so that people don’t have to reinvent it[1].

Nice! Thanks for the pointer, although the disclaimers suggest that maybe it’s not something I should rely on just yet :slightly_smiling_face:

OK. I have no opinion on whether I should or not, although I would like it if there was more consensus from the experts, so that casual users like me didn’t have to navigate all this complexity :confused:

Edit: To put this another way, the prevalence of low-level functions, and the state of flux and/or unclear recommendations around higher level constructs, feels like what I’d expect from a relatively new framework or paradigm. But I wouldn’t class asyncio as new, so how come these things haven’t been thrashed out yet? That’s a genuine question, not a troll - where’s the disconnect between someone like me thinking “fire and forget tasks make sense for my use case” and the stdlib having no sign that anyone has even considered them as a good idea?

  1. And to be clear, “fairly intuitive” doesn’t mean “easy” - quite the opposite, as you’ve shown there are a number of tricky cases to handle. All the more reason to have a robust stdlib implementation. ↩︎

2 Likes
34

In terms of agreement:

I think most people can agree that exceptions shouldn’t go unhandled without an explicit intent to. However, having an API that returns a future which the user is responsble for handling isn’t a new thing, it’s how the concurrent futures apis work, and a user choosing that “just letting the error hit the default exception handler” or “discard the error entirely”, are valid user choices that the standard library shouldn’t be opinionated about, just provide the right tools and the documentation.

It shouldn’t be controversial to have a one-shot function for a background task that’s “fire and forget” that just adds a done callback to toss any exception to the loop’s default exception handler. This would also ensure the error is handled when it happens rather than at garbage collection of the task.

I’ve been collecting the things I’ve had to write more than once for asyncio in the repo I linked. I’ll probably be promoting it to considering it stable in the near future, and I’d have no issue with contributing any of that to cpython if there’s actual wider demand.

I can’t speak for “in general”, but It’s easy enough to write the abstractions I need that I don’t need them to be in the standard library, and I’ve been more annoyed by some of the higher-level abstractions doing things in a way that gets messy and opinionated around cancellation, so I tend not to use those.

35

Fully agreed. And that’s probably all we should do (and promote the heck out of it, of course).

1 Like
36

Did you ever write that up? Would be helpful to know what we can do better.

37

Not in depth. I’ve brought them up in passing, but the issues are in APIs I don’t need to build what I want, so it’s been generally easy to just not use them as “not for me”.

asyncio.{timeout,Timout,timout_at}, asyncio.TaskGroup, asyncio.shield

TaskGroup can be recreated with semantics I’d use (See above), timeout has a few issues with leaking scope (see pep 789), and has different semantics on it’s actual behavior than I want. It can be replicated without those issues by not designing it as a context manager and instead using asyncio.wait

shield is banned because it’s effectively broken, and the better guidance of needing to be aware of what is safe to cancel before choosing to avoids needing it.

38

I’m saying a true “fire and forget” task (that just gets dropped on the floor when the event loop/root task finishes) is a code smell and we shouldn’t be encouraging it. That’s the gist of my argument.

Then, I’m trying to make the case for how the functionality you’re after can be emulated by systems which don’t just drop it on the floor but handle it somehow.

1 Like
39

Might want slightly more if it becomes part of the standard library. I think essentially people want a “use anywhere” task creation that doesn’t require passing some top level task group around and doesn’t cancel unrelated tasks scheduled in the same way.

Ideally, we’re essentially treating the event loop as the top structured level for those use cases. It may be worth attaching tasks scheduled this way to the event loop, and adding a .join method on the event loop to wait on all tasks scheduled this way to finish, avoiding the other issue below:

2 Likes
Asyncio: How to merge two streams