`await asyncio.gather(task)` hangs when `await task` works

Hi! I’m using python 3.11.4 in a django project with websockets.

I have a function that works fine when calling it with await, but when trying to run multiple functions at once with asyncio.gather it hangs. Even if I only call gather with one function:

My code looks like this (hangs):

        evaluations = []
        async for source_pitch in self.source_pitches.all():
            evaluations.append(Evaluation.create(source_pitch, batch=self))
        await asyncio.gather(*evaluations, return_exceptions=True)

While this runs with no problem:

        async for source_pitch in self.source_pitches.all():
            await Evaluation.create(source_pitch, batch=self)

When setting a trace with ipdb I found that the line that gets stuck is:

> /usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/selectors.py:561
kev_list = self._selector.control(None, max_ev, timeout)

And when running with python -m ipdb and then ctrl c it gets stuck in this line:

File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1132, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):

I have no idea how to debug it further given that without gather it works.

More info: the code hangs inside a sync_to_async django call when trying to save to the db.

You don’t share what Evaluation is, but I suspect that when there are multiple concurrent calls to Evaluation.create, a deadlock gets exposed. You’ll have to look into the code for Evaluation.create to figure out the exact cause.

I didn’t share it cause it makes a bunch of things with lot’s of awaits. I hangs in the first lines though:

    @classmethod
    async def create(cls, source_pitch, batch=None):
        """
        Creates and runs an evaluation for a given pitch file
        """
        pitch = Pitch(
            name=f"TEST: {source_pitch.name}",
        )
        await pitch.asave() # <-- hangs here
        [ ... ]

This is a Django model. asave has a sync_to_async inside. The thing is that it hangs with only one coroutine to gather, so it’s strange that it would produce a deadlock. Directly calling await Evaluation.create(..) works with no deadlock.

I’m not sure how to continue debugging this. Any help much appreciated.

have you tried a TaskGroup ?

Yes, exactly the same results with a TaskGroup. It hangs in the same line even if it’s one task.

Can you replicate this without Django being involved? I’d like to try this on different OSes and Python versions and see if there’s any difference.

I’ll try, the thing is it hangs in a django function…

I’m running OS X but I’ve also tried it inside a docker container. I’m now trying to remove as much code as possible keeping the hanging behaviour

Yeah. Fortunately, you have the option of delving into Django’s source to try to pin down what’s happening.

Thanks!

Ok, this is a django issue. I cannot replicate it outside their views engine.

Following the issue here: #34747 (Django hangs on async views with asycio.gather and an async ORM call) – Django

Thanks everyone for the help!

Cool cool. It’s still interesting, though, and definitely possible (even likely!) that there are multiple factors involved.