`await asyncio.gather(task)` hangs when `await task` works

Python Help
1

Hi! I’m using python 3.11.4 in a django project with websockets.

I have a function that works fine when calling it with await, but when trying to run multiple functions at once with asyncio.gather it hangs. Even if I only call gather with one function:

My code looks like this (hangs):

        evaluations = []
        async for source_pitch in self.source_pitches.all():
            evaluations.append(Evaluation.create(source_pitch, batch=self))
        await asyncio.gather(*evaluations, return_exceptions=True)

While this runs with no problem:

        async for source_pitch in self.source_pitches.all():
            await Evaluation.create(source_pitch, batch=self)

When setting a trace with ipdb I found that the line that gets stuck is:

> /usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/selectors.py:561
kev_list = self._selector.control(None, max_ev, timeout)

And when running with python -m ipdb and then ctrl c it gets stuck in this line:

File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1132, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):

I have no idea how to debug it further given that without gather it works.

More info: the code hangs inside a sync_to_async django call when trying to save to the db.

2

You don’t share what Evaluation is, but I suspect that when there are multiple concurrent calls to Evaluation.create, a deadlock gets exposed. You’ll have to look into the code for Evaluation.create to figure out the exact cause.

3

I didn’t share it cause it makes a bunch of things with lot’s of awaits. I hangs in the first lines though:

    @classmethod
    async def create(cls, source_pitch, batch=None):
        """
        Creates and runs an evaluation for a given pitch file
        """
        pitch = Pitch(
            name=f"TEST: {source_pitch.name}",
        )
        await pitch.asave() # <-- hangs here
        [ ... ]

This is a Django model. asave has a sync_to_async inside. The thing is that it hangs with only one coroutine to gather, so it’s strange that it would produce a deadlock. Directly calling await Evaluation.create(..) works with no deadlock.

I’m not sure how to continue debugging this. Any help much appreciated.

4

have you tried a TaskGroup ?

5

Yes, exactly the same results with a TaskGroup. It hangs in the same line even if it’s one task.

6

Can you replicate this without Django being involved? I’d like to try this on different OSes and Python versions and see if there’s any difference.

7

I’ll try, the thing is it hangs in a django function…

I’m running OS X but I’ve also tried it inside a docker container. I’m now trying to remove as much code as possible keeping the hanging behaviour

8

Yeah. Fortunately, you have the option of delving into Django’s source to try to pin down what’s happening.

Thanks!

9

Ok, this is a django issue. I cannot replicate it outside their views engine.

Following the issue here: #34747 (Django hangs on async views with asycio.gather and an async ORM call) – Django

Thanks everyone for the help!

10

Cool cool. It’s still interesting, though, and definitely possible (even likely!) that there are multiple factors involved.