How to invoke `async` function from `cmd` (Python standard library, sync-async bridge, "coroutine ... was never awaited")?

Python Help
1

Hi,

I am trying to write a terminal application using cmd (Python standard library), which is able to invoke async couroutines from the action callback function.

Minimal example

Code:

# main.py
import cmd
import asyncio


class MyShell(cmd.Cmd):
    async def do_myaction(self, arg):
        # this async task is not executed anymore, program immediately
        # exited, as do_myaction is not blocking
        await asyncio.sleep(5)
        print("Some app results")


async def main():
    await asyncio.gather(shell(), other_task())
    print("Exiting...")


async def shell():
    def run():
        MyShell().cmdloop()

    # Run in separate thread to make interactive shell I/IO non-blocking
    await asyncio.to_thread(run)


async def other_task():
    # represents some concurrent initialization task
    await asyncio.sleep(1)
    print("Concurrent init task")


asyncio.run(main())

Invocation:

[user@sys test]$ python main.py
(Cmd) Concurrent init task
myaction
/tmp/test/main.py:20: RuntimeWarning: coroutine 'MyShell.do_myaction' 
was never awaited
  MyShell().cmdloop()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Exiting...

Problem

Obviously cmd is not able to block on awaited code. Program exits, before coroutine do_myaction(self, arg) can be processed.

Here special case is that a synchronous callback function do_myaction() is invocated from the framework, as soon as myaction is typed in interactive prompt. I tried to annotate the callback as async def do_myaction(), but parent handler doesn’t await this coroutine.

In addition, event loop is missing in spawned thread from asyncio.to_thread() 1. So it is needed to create an additional loop inside. I tried to set it via asyncio.set_event_loop(asyncio.new_event_loop()) in custom MyShell constructor, but then did not know how to actually run said loop in this context.

1: Tested that by writing asyncio.get_running_loop() inside constructor - which errors as expected.

Which lead me to following questions:

Questions

[1] Is it possible to use cmd with async at all, not using any external libraries?

[2] If no: Is there a native alternative to cmd in async contexts? I found an example using code, but I guess the same problem arises with sync callback runsource():

import code 
class Repl(code.InteractiveConsole):
    def runsource(self, source, filename="<input>", symbol="single"):
        print("source:", source)
    
repl = Repl()
repl.interact(banner="", exitmsg="")

[3] In other words, or formulated as more general question: How can I invoke async logic from callback functions of synchronous frameworks? I am missing some bridge between sync and async here.

(4) Optional question

Keeping number of questions small, I’ll collapse this one as possible unrelated. Still appreciating an answer:
I was thinking of crafting a manual Future around code that is incompatible to async. Having some experience with JavaScript await, what would be the Python equivalent to a Promise constructor with manual resolve, like:

const promise1 = new Promise((resolve, reject) => {
  setTimeout(() => {
    resolve('foo');
  }, 300);
});

I am still new to async/await and would be glad to receive some feedback and possible solutions. Thanks!

Related:

2

Following seems to work (without having tested extensively):

  • Keep all action callbacks of cmd.Cmd sync
  • Wrap every action callback by asyncio.run(run()), where run() is the async API function
  • asyncio.run() will be synchronously blocking, till async function completed.

A bit repetitive, as you need to do this for each action callback. That’s at least as good as I got it. Full example code:

import cmd
import asyncio

class MyShell(cmd.Cmd):
    def do_myaction(self, arg):
        async def run():
            await asyncio.sleep(3)
            print("Some app results")

        asyncio.run(run())


async def main():
    await asyncio.gather(shell(), other_task())
    print("Exiting...")


async def shell():
    def run():
        MyShell().cmdloop()

    # Run in separate thread to make interactive shell I/IO non-blocking
    await asyncio.to_thread(run)


async def other_task():
    # represents some concurrent initialization task
    await asyncio.sleep(1)
    print("Concurrent init task")


asyncio.run(main())
3

One addition for future readers (pun intended :-)): If you get error

Future attached to a different loop

, it is because futures received from main event loop and the thread loop get mixed. In this case you need to use asyncio.run_coroutine_threadsafe():

    # this should be a reference to outer/main loop
    loop = asyncio.get_running_loop()    

    def do_myaction(self, arg):
        async def run():
            await asyncio.sleep(3)
            print("Some app results")

        asyncio.run_coroutine_threadsafe(run(),loop).result()

All in all my experience is that async/await quickly “creeps” up the entire application stack, which is not always desirable.

For my case I just needed to reuse common async functions from a server component. The client can be run synchronously in a blocking fashion. The easiest way here is to partially use asyncio.run(coro) in all code locations, which require bridging async to sync.

All functions should be completely aware, if run in context of an existing event loop or not. Everything else might be considered a code smell.