Calling coroutines from sync code (2)

Last month, I asked about calling coroutines from synchronous library code in Python Help Calling coroutines from sync code. To quote myself:

@pf_moore had a similar question two years back: Wrapping async functions for use in sync code.

This seems to be an important use case to me, but it’s currently quite difficult to find information on how to best achieve this, and the solution is non-obvious (at least to me), and potentially non-trivial. Would you consider a contribution of a utility function along the lines from my original thread for the standard library?

def call(coro):
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        return asyncio.run(coro)
    else:
        return loop.run_until_complete(coro)

The documentation could clearly state the difference to asyncio.run(), but also make it clear that the coroutine is run synchronously and will block other async code.

4 Likes

I am unenthusiastic about providing this functionality in the stdlib. This is really not something that a well-behaved async application should do, and having the API would legitimize an idiom that is likely to come back and stab you in the back when you least need it. You would be better off refactoring your application so you don’t need this ugly hack. (Of course, I understand that realistically that’s not always possible, but if your specific app’s transition to an async world requires this hack, well, it’s only 7 lines of code.)

3 Likes

This is not only about transitioning applications, but also required for library code. A library with a synchronous API can’t call async code without facing the same problem. Same for a library that provides both a synchronous and an async interface: If you don’t want to duplicate all code paths, you need to use such a pattern as well.

Another problem is that users might not be aware that there actually is a problem. The coroutines documentation only mentions asyncio.run() to run a coroutine outside of other coroutines. (Although the run() documentation mentions that it can’t be called if there already is a running event loop.) If a user starts to use asyncio down in their stack, their code will break unexpectedly, once they start using asyncio higher up.

A (well documented) convenience function in the standard library will be a lot more visible solution. It’s better suited to handle any potential corner cases that users are unaware of. The same reason why asyncio.run() was introduced in the first place.

But I at least wish there was some guidance for this problem in the asyncio documentation.

3 Likes

This is also relevant to recent unittest/asyncio issues (especially #101018, tangentially #101486) that I’ve opened as well, with the additional wrinkle of ensuring that the correct contextvars.Context is used however a method is run.

I’m not sure the unittest case is enough to justify adding a public helper function, but a suggested solution for the general problem would ideally apply there.

But using your proposed helper doesn’t solve the problem! By the time your helper decides to call loop.run_until_complete() you are already in trouble. async code is written under the assumption that the sync code it calls never blocks for I/O, and definitely never enters the event loop recursively. Only await may enter the event loop. This is used in many places to avoid explicit locks: async code can rely on the fact that no other code runs concurrently unless explicitly allowed by await. (This is important when updating data structiures like queus.)

I see this as a somewhat different use cases. But it can suffer from the same problem.

That breakage is intentional – it tells them that the architecture they’re attempting to create is not safe to use.

No, asyncio.run() does not violate the async model. It is a helper that takes care of a bunch of bookkeeping around creating and finalizing event loops and it captures the best practice. Your proposed helper would be the opposite of best practice.

Maybe we should devote a section to explaining why this is a bad idea then.

7 Likes

That much, I definitely agree with. It’s clearly obvious to experts that this is a bad idea, and there are known ways to deal with it (even if they are “duplicate your code” or “re-architect your app”). But none of that is obvious to newcomers, so explaining it before they get in a mess would be very beneficial.

7 Likes

I have encountered cases where I needed to call async functions from sync code. I did refactor my apps to fix that, but I’m still wandering: @pf_moore would you help me understand these known ways a bit better? Maybe you have a few articles or a bunch or personal examples that you could provide to help me build a firmer understanding in my mind?

I don’t know any - I’m not one of the experts I was referring to in my comment.

My 2c on this. I’d say that a library providing both synchronous and asynchronous interfaces for the same purpose, would not benefit from such helpers.

If you’re wrapping all the coroutine functions with it to provide their “sync version”, you’re spinning up (and then shutting down) an event loop for each one of them, with all that this entails.
Looking at the source code of asyncio.runners:Runner may clarify why such an helper should be (IMO) considered an anti-pattern.

Of course hacks are unavoidable occasionaly, but IMO that’s not a valid reason to include them in stdlib.

1 Like

While, technically speaking, there is a way to make this happen: allowing asyncio.run() to work in a nested fashion (a trick also used by Java’s AWT), I’m with Guido: we should not do this at all.

One of the reasons I use async is that concurrency is easier to understand than with the threaded model. I can be assured that no other task will touch my variables between yield points, so no locking is necessary. If we start allowing synchronous functions to start calling async code, those guarantees go right out the window since this would mean that the loop would run multiple tasks during a synchronous call, which my code is not expecting at all. And if we allow this, we likely can’t ever go back, or code written since would break.

1 Like

There seems to be a misconception. call() should still block coroutines that are not created inside the called code. Everything else would break the contract of calling synchronous functions from async code. Of course that isn’t ideal for performance, but that is a (potentially temporary) trade-off that the developer is consciously making when calling synchronous functions.

That said, this whole discussion leaves me flabbergasted. That asyncio breaks encapsulation quite severely should be a big red box first thing in the asyncio documentation. I am disappointed that I can’t implement functions like def get_all(*urls) using asynchronous libraries. But the inability to refactor existing projects towards using asyncio (at least for non-toy projects) is a complete show stopper. Why isn’t it possible to start using an asynchronous interface for accessing the database without the fear that everything will break when switching from WSGI to ASGI?

1 Like

Asynchronicity is a fundamentally difficult concept, and I don’t think you’ll ever get away from the fact that the entire application has to be built with it in mind. Are you using process-based parallelization? Then you need to think about serializing and unserializing things. Thread-based? Forking is right out. Event loop of some form? The exact choice governs what features you have, and everything has to be aware of what can go through that event loop.

What I would say, though, is that writing thread-style code and then converting it to asyncio is your safest bet. Start by assuming that a context switch can happen literally anywhere, and then you’ll usually be fine even if there’s an odd call into the event loop that you didn’t expect. It’s still important to think about asyncio across the entire application, but at least following a thread-style discipline will make that easier to handle.

(Also: For the most part, “work with your own locals” is sufficient isolation, regardless of the concurrency model used. You only need to think about your concurrency model when mutating global state. Which cuts down the number of things to think about drastically.)

1 Like

For example. You run two asynchronous tasks, A and B. Task A calls a third-party asynchronous function foo() which uses an asynchronous lock to guard access to some complex data. Task B calls a synchronous function bar() which calls the same asynchronous function foo() from a synchronous code. You, as a user, have no idea what locking is used in foo(). If the the lock is acquired in task A, and then the execution is switched to task B which calls bar() which calls foo() which tries to acquire the lock, we have a deadlock, because you cannot switch from task B to task A while executing a synchronous code bar(). What is worse, it can happen random and very rarely, so you will miss this in your tests and it can be hard to reproduce the problem. But if it happen, it will just deadlock your program, without any reporting.

For example. You run two asynchronous tasks, A and B. Task A calls a third-party asynchronous function foo() which uses an asynchronous lock to guard access to some complex data. Task B calls a synchronous function bar() which calls the same asynchronous function foo() from a synchronous code. You, as a user, have no idea what locking is used in foo(). If the the lock is acquired in task A, and then the execution is switched to task B which calls bar() which calls foo() which tries to acquire the lock, we have a deadlock, because you cannot switch from task B to task A while executing a synchronous code bar(). What is worse, it can happen random and very rarely, so you will miss this in your tests and it can be hard to reproduce the problem. But if it happen, it will just deadlock your program, without any reporting.

But that is not a problem specific to calling async code from sync code. That problem exist anytime a lock is used when using concurrent code.

Sebastian has a point, and if asyncio had been written as a library then it would likely work the way he’s suggesting.

However, it’s not. Asyncio is an application framework, which means when you’re using it, it assumes that it controls the very base of the stack and the very tip of the stack. It sounds like this could be made clearer in the documentation.

(I think it would also be possible for someone to build an async library on top of the async/await syntax and coroutines, probably quite easily. Unfortunately, libraries that use asyncio almost always rely on the application framework itself, and not just the syntax, so you’d also need a new set of libraries to use on top of the new one.)

Could you show an example? If it is true, concurrent programming would be impossible.

1 Like

If the lock is acquired in task A and then execution is switched, that means there’s a yield point in a critical section. Logically, anything else that wants to enter the same critical section must block. (Though, in general, a yield point inside a critical section is asking for trouble.)

Then task B, which must presumably be an asynchronous task, calls a synchronous function that, by its nature, has to have a concept of blocking. It doesn’t matter whether it’s mixing asyncio and synchronous functions; the problem is that bar() is capable of acquiring a lock in a blocking manner, which means that bar must not be called from an async task. In general, any blocking function called from an asynchronous task is asking for trouble.

So, yes, this will cause problems, but it’s because you’re mixing locking with async yield points without any way to bypass that.

Can you give an example of any other sort of event loop that wouldn’t have this problem? I don’t think it’s asyncio-specific; it’s not that asyncio has to be an application framework, it’s that blocking lock acquisition and yield points are incompatible.

What is a “yield point”? If the lock is acquired in task A and then execution is switched, that means there’s an await (either explicit or implicit via async for or async with) in a critical section. If there is no await, you do not need a lock. Presumably there are more than one await, the state is not valid between two awaits and you want to prevent other code to looking into it when it is not valid.

A synchronous function called in task B has nothing to do with asynchronous locking, you cannot do anything with asynchronous locks from a sync code, and you knows nothing about this particular lock which is an implementation detail of third-party code.

There is no problem with using foo() in an asynchronous code, because if it need to acquire a lock which is currently blocked, the execution is just switched to the task which holds the lock, and finally it will be released. The problem with calling an asynchronous code from synchronous code is that it is not possible to switch to that task and resolve blocking.

The only way to ensure that would be to make call run the event loop in a thread. Although I suspect that you didn’t intend to write what your words seem to imply to me – I’d love me a few good examples of what you’re after here.

Certainly a problem with re-invoking get_running_loop().run_until_complete(coro()) from a synchronous function that’s being called from an async function is that it breaks the cancellation model. By design asyncio’s cancellation only interrupts await expressions. But if I have something like this:

async def compare_urls(urls: list[str]):
    contents = sync_get_urls(urls)
    # Do something else with the collective contents
    ...

def sync_get_urls(urls: list[str]) -> list[bytes]:
    return asyncio.get_running_loop().run_until_complete(get_urls(urls))

async def get_urls(urls: list[str]) -> list[bytes]:
    with asyncio.TaskGroup() as tg:
        # Spawn tasks
        futs = [tg.create_task(get_url(url)) for url in urls]
        # Wait for tasks
        results = [await fut for fut in futs]
    return results

async def get_url(url: str) -> bytes:
    # Async code to fetch data corresponding to one URL
    ...

async def main():
    await compare_urls(sys.argv[1:])

asyncio.run(main())

then cancelling the outer compare_urls() coroutine is unable to cancel the inner async_get_urls() coroutines using the standard asyncio cancellation mechanism.

This is a very emotionally triggering paragraph. “Flabbergasted” doesn’t just mean surprised, it conveys utter indignation about the surprise. In addition you use"breaks encapsulation quite severely", “big red box”, “disappointed”, “complete show stopper”, and “fear”, all strong negative concepts. Did you really have to respond so negatively when the technical reasons for the restriction you are experiencing were explained to you?

If you were just a Discourse newbie, I would just have muted the thread at this point, but I know you as a thoughtful, intelligent and experienced contributor in the typing world, so I hope that further dialog is still possible, and we can help you understand asyncio a little better. (I hope that my example above is enlightening.)

Correct; in asyncio, a yield point is an await or equivalent. (With other forms of asynchronous I/O, a yield point may be spelled differently, but the concept is the same.)

And yes, any time you yield inside a critical section, you are implying that (a) other tasks can run, but (b) no other task can run this same critical section. And mixing that with blocking calls that require the same critical section, without threading, is asking for trouble.

Is it that asyncio (the module) requires that the entire application be built that way, or is it that use of asynchronous locks is incompatible with blocking use of those same locks?