Two sync_apis (playwright and procrastinate): cannot use AsyncToSync in the same thread as an async event loop

Async-SIG
1

Hey everyone,

I’ve been struggling with this situation for a while and I have no idea how to approach it. I’ve tried all the localized fixes I could think of and I have no idea what the right way would be.

The situation

I am writing a web application with Django. It’s pretty much exclusively sync, but some dependencies use async under the hood, specifically playwright (which we use for testing) and procrastinate (a postgres-backed job queuing system).
Given that we’re not really doing async, I reached for the projects’ respective sync apis, which leads to a call graph kinda like this:

pytest
-> fixture setup
    -> playwright is started

-> test case
  -> production code
    -> procrastinate, e.g. to queue an email to send later
-> ...more test cases

-> fixture teardown
  -> playwright shuts down

So far so good, but it turns out that both sync apis start an event loop internally, causing RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop - just await the async function directly.

I could use pytest-asyncio to get an event loop at the test case level (in the above figure), which would cover the whole execution scope, but my sync production code is between the event loop and the calls into procrastinate.
Async → sync → async callchains don’t seem to be a thing and nest_asyncio, which is the only approach I found people going for, seems like it will be a bad choice to bet on.

Any guidance on how to tackle this problem would be appreciated. Do we really have to asyncify the whole call chain?

2

Without knowing more about your use case, it’s hard to give great answers.

if procrastinate is just to fire off things in the background (it does sound like you are using it that way) you can start an event loop in a background thread and schedule to it without waiting on the result until the application teardown.

Some boilerplate for that is abstracted here It’s abstracted as a context manager that starts an asyncio event loop in a thread, and has methods for scheduling coroutines to it. there’s a method to wait on all scheduled coroutines that is optional to call. context exit shuts down the event loop and joins the thread.

I’m not sure if the same solution would work for your use of playwright, but it sounds like it might.

This isn’t quite async → sync → async but structuring use of async and event loops to work in concert with a main thread that has sync work.

There’s also a dual-color queue in the same repository that is suitable for communicating between the paradigms in things other than returns.

1 Like