I finally remember why asyncio goes to such great lengths to prevent you from calling loop.run_until_complete() on an event loop that’s already running.
The issue is the implementation of _run_once(). This is the internal loop method that acts as scheduler. It works roughly as follows. After waiting for I/O to become ready or a timeout, it makes a list of all runnable callbacks and runs them.
The way this code is currently written it would be disastrous if one of the callbacks ended up (presumably via run_until_complete()) calling _run_once() recursively. The recursive call would consume callbacks from the self._ready queue that the outer call was already planning to call, and the outer call, once resumed, would end up crashing when trying to call self._ready.popleft() when the queue is empty.
I could imagine a number of hacks to fix this, but before we go there let’s look at another prohibition: you also can’t create a new event loop when one is already running! This is done out of an abundance of caution – in general it’s a bad idea to start a new event loop when one is already running. A simple workaround is to run the new event loop in a separate thread.
However, using a second loop in the same thread as an existing loop does not cause problems with _run_once() – it just blocks a callback, and asyncio callbacks shouldn’t block. (It’s like doing a socket.recv() call in a coroutine – it works, but it prevents the concurrency that asyncio was meant to provide.)
So we should actually be able to allow this when the user confirms that it is what they want, for example by passing a new flag to asyncio.run(), e.g. asyncio.run(coro(), allow_recursive_loop=True). This would have to be passed along through a few layers (there are several different places where this check is made) but that should be simple enough: the flag should simply be passed to all calls to _check_running().
If someone wants to submit a PR to fix this please link to the PR here and I will review it.