Hello!
I have been using Python to launch child processes, to read from their output and to write to their input. I have been almost successfully doing that for a while, until I encountered some problems with graceful shutdown of the parent process. I decided then to use select in order to avoid blocking reads, but I’m having difficulties with that as well.
My problem is that select returns negative after a call to read on stdout, even though I know for a fact that there is still data available to read.
Here is an example:
child.py
#! /usr/bin/env python3
if __name__ == "__main__":
print("READY", flush=True)
input()
parent.py
#! /usr/bin/env python3
import subprocess
import select
def main():
process = subprocess.Popen(["./child.py"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
# First char read
result, *_ = select.select([process.stdout.fileno()], [], [], 1.0)
print(bool(result))
data = process.stdout.read(1)
print(data)
# Second char read (select returns false)
result, *_ = select.select([process.stdout.fileno()], [], [], 1.0)
print(bool(result))
data = process.stdout.read(1)
print(data)
# Third char read (select returns false)
result, *_ = select.select([process.stdout.fileno()], [], [], 1.0)
print(bool(result))
data = process.stdout.read(1)
print(data)
process.stdin.write("QUIT\n")
process.stdin.flush()
process.wait()
if __name__ == "__main__":
main()
You can see that I erroneously ignore the result of select and read from the pipe anyway. The reading operation doesn’t block, as there is data. But select tells me that the read should block, that there is no data.
This was just an example. Actually I need to read whole terminated lines (with readline) and write whole terminated lines. The child program may sometimes output multiple lines per command from parent (one line of input). This issue makes it impossible to read more than one line of output without blocking, because after reading the first line, or even just one character, select returns false. select correctly returns true after I read the single line of output from the child.
child2.py
#! /usr/bin/env python3
if __name__ == "__main__":
print("READY", flush=True)
input()
print("THING1", flush=True)
input()
print("THING2", flush=True)
input()
print("THING3", flush=True)
# These two lines don't exist for select
print("OH_NO1", flush=True)
print("OH_NO2", flush=True)
input()
parent2.py
#! /usr/bin/env python3
import subprocess
import select
def main():
process = subprocess.Popen(["./child2.py"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
# Read READY
result, *_ = select.select([process.stdout.fileno()], [], [], 1.0)
print(bool(result))
data = process.stdout.readline()
print(data)
process.stdin.write("NEXT\n")
process.stdin.flush()
# Read first THING
result, *_ = select.select([process.stdout.fileno()], [], [], 1.0)
print(bool(result))
data = process.stdout.readline()
print(data)
process.stdin.write("NEXT\n")
process.stdin.flush()
# Read second THING
result, *_ = select.select([process.stdout.fileno()], [], [], 1.0)
print(bool(result))
data = process.stdout.readline()
print(data)
process.stdin.write("NEXT\n")
process.stdin.flush()
# Read third THING
result, *_ = select.select([process.stdout.fileno()], [], [], 1.0)
print(bool(result))
data = process.stdout.readline()
print(data)
# Read first OH_NO (select returns false)
result, *_ = select.select([process.stdout.fileno()], [], [], 1.0)
print(bool(result))
data = process.stdout.readline()
print(data)
# Read second OH_NO (select returns false)
result, *_ = select.select([process.stdout.fileno()], [], [], 1.0)
print(bool(result))
data = process.stdout.readline()
print(data)
process.stdin.write("QUIT\n")
process.stdin.flush()
process.wait()
if __name__ == "__main__":
main()
In my situation I don’t know in advance how many lines of output will the child send after a certain input. In the end, select becomes useless.
My question is if this is the normal behavior for select in Python, because it seems very odd to me. I have written subprocessing code in C and C++ as well in the past and the select system call wasn’t behaving like that.
I’m using Python 3.12.7, Linux 6.11.5.
Maybe you can tell by now that I’m very tired.
