Note: copied from async-SIG
Note: I posted this to async-SIG and I’m afraid that was a wrong category. I’m posting an edited version here. If it is not OK, please let me know, I’m quite new here.
The documentation of asyncio.subprocess.Process.wait warns about a deadlock:
Note
This method can deadlock when using
stdout=PIPE
orstderr=PIPE
and the child process generates so much output that it blocks waiting for the OS pipe buffer to accept more data. Use thecommunicate()
method when using pipes to avoid this condition.
My understanding was: When the executed process is blocked by a full pipe, it cannot finish its work and then to exit. The fact the process does not terminate makes the process.wait()
block.
However, the process.wait
will block even when the process in such state receives a signal and terminates. A full buffer in the process.stdout
stream buffer blocks the process.wait
. Draining the buffer unblocks the wait
. I think that in this case wait()
does not behave correctly. It should have returned when the process had exited - regardless of buffer full condition.
Below is a demonstration Linux program:
Preparation: create a 512kB file ./datafile, e.g.
dd if=/dev/zero of=datafile bs=64k count=8
or simply use any existing file not smaller than this.
Code:
import asyncio
async def killproc(proc):
await asyncio.sleep(1)
print(f"{proc.returncode=}")
proc.terminate()
print("signal sent")
await asyncio.sleep(1)
print(f"{proc.returncode=}")
await asyncio.sleep(2)
print("reading pipe buffer")
await proc.stdout.read()
async def main():
proc = await asyncio.create_subprocess_exec(
"/usr/bin/cat", "./datafile",
stdout=asyncio.subprocess.PIPE)
print("process started")
asyncio.create_task(killproc(proc))
print("wait() start")
await proc.wait()
print("wait() stop")
asyncio.run(main())
The output + blank lines where there are delays + my comments:
----- process started wait() start proc.returncode=None # process 'cat' exists signal sent proc.returncode=-15 # process 'cat' terminated by signal 15 # (and the Python is aware of that) reading pipe buffer # <-- without this the 'wait' blocks wait() stop -----