Hi! I’m using python 3.11.4 in a django project with websockets.
I have a function that works fine when calling it with await, but when trying to run multiple functions at once with asyncio.gather it hangs. Even if I only call gather with one function:
My code looks like this (hangs):
evaluations = []
async for source_pitch in self.source_pitches.all():
evaluations.append(Evaluation.create(source_pitch, batch=self))
await asyncio.gather(*evaluations, return_exceptions=True)
While this runs with no problem:
async for source_pitch in self.source_pitches.all():
await Evaluation.create(source_pitch, batch=self)
When setting a trace with ipdb I found that the line that gets stuck is:
And when running with python -m ipdb and then ctrl c it gets stuck in this line:
File "/usr/local/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1132, in _wait_for_tstate_lock
if lock.acquire(block, timeout):
I have no idea how to debug it further given that without gather it works.
More info: the code hangs inside a sync_to_async django call when trying to save to the db.
You don’t share what Evaluation is, but I suspect that when there are multiple concurrent calls to Evaluation.create, a deadlock gets exposed. You’ll have to look into the code for Evaluation.create to figure out the exact cause.
I didn’t share it cause it makes a bunch of things with lot’s of awaits. I hangs in the first lines though:
@classmethod
async def create(cls, source_pitch, batch=None):
"""
Creates and runs an evaluation for a given pitch file
"""
pitch = Pitch(
name=f"TEST: {source_pitch.name}",
)
await pitch.asave() # <-- hangs here
[ ... ]
This is a Django model. asave has a sync_to_async inside. The thing is that it hangs with only one coroutine to gather, so it’s strange that it would produce a deadlock. Directly calling await Evaluation.create(..) works with no deadlock.
I’m not sure how to continue debugging this. Any help much appreciated.