Following over from the other thread you linked this to as I think the feedback is more appropriate here, than in that thread.
I don’t particularly think the task-group machinery is a good abstraction for the problems people are trying to solve with it, and I don’t think making task creation in general have more overhead to support an even more complex case of it is a good idea.
Recently (it’s part of 3.14a) simply changing the structure tasks were stored in resulted in a roughly 10% performance uplift for asyncio applications. Task creation overhead is extremely sensitive to performance for every asyncio application, and I don’t think the additional complexity here needs to be owned by event loops, it should be applications choosing which tasks to block on, and properly ordering a mix of cancellation and signaled graceful shutdown as appropriate.
I do think we can make low-cost abstractions that can help both library and application code here, and I have some that are public and some that I cannot share yet[1], but I don’t think it should increase the cost for creating tasks that aren’t using that abstraction, and that’s part of the issue I have with task groups as currently designed, and as proposed to extend further.
The other problem I have with task groups is I think they get cancellation wrong in two ways, one of them is “obvious”[2] with known issues, the other is a philosophical design difference I’ll be happy to discuss in more depth after the holidays if you’re interested in an alternative, less-intrusive way of structuring this for applications, but the basics of this is that it is enough for applications to have a ordered set of sets of tasks to wait on with the order corresponding to the lifecycle and not necessarily a level of nesting for where the task is launched.
I have written a bit about the issues people run into with cancellation vs cooperative shutdown before here, and it’s partially relevant in how I would suggest designing a good-general purpose abstraction here that doesn’t require hooking into tasks in general, but I’ll only be able to show what some people might view as a toy example for now[1:1]
Some production code I can’t currently share is in holiday-limbo right now, but I’m in the process of waiting on official permission to share the relevant bits of how asyncio is used at my day job, and why we avoid most APIs that implicitly cancel ↩︎ ↩︎
At least for those who have been following various issues, especially those spelled out in the rationale for PEP 533 ↩︎