Interact with python using subprocess

Python Help
1

I try to interact to python interpreter using subprocess module like this :

import subprocess


def start(executable_file):
    return subprocess.Popen(
        executable_file,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE)


def read(process):
    return process.stdout.readline().decode("utf-8").strip()


def write(process, message):
    process.stdin.write(f"{message.strip()}\n".encode("utf-8"))
    process.stdin.flush()


def terminate(process):
    process.stdin.close()
    process.terminate()
    process.wait(timeout=0.2)


process = start("python")
while True:
    write(process, input())
    print(read(process))
terminate(process)

But it seems it’s locked a deadlock.

If someone knows how to interact with python with python code and recover stdout, stderr with stream mode.

2

You can’t block on reads. You should perform non-blocking reads in a separate thread, and only have the write in the main thread. Also, you should send keyboard interrupts through to the subprocess

1 Like
3

Start the Python subprocess with the -i flag. When it starts, it’ll print the banner to stderr.

You won’t get anything coming back on stdout until the Python subprocess prints to stdout.

process = start(["python", "-i"])
write(process, "message = 'Hello world!'")
# Nothing printed to stdout yet.
write(process, "print(message)")
# Now there's something you can read.
read(process)
1 Like
4

i want to use same python interpreter (subprocess) to interacte with him without kill him
with a loop

5

You might want to look at using a pty then.

Here’s an example:
https://stromberg.dnsalias.org/~strombrg/pypty/
https://stromberg.dnsalias.org/svn/pypty/trunk/pypty

And here’s a smaller example
https://docs.python.org/3/library/pty.html

6

i try to use thread for reading stdout but it’s seems same problem with deadlock :frowning:

proc = Popen(['python'], stdout=PIPE, stdin=PIPE, stderr=PIPE)

def thread_function(proc):
    print(proc.stdout.read())

while True:
    proc.stdin.write(f"{input()}\n".encode("utf-8"))
    th = Thread(target=thread_function, args=(proc,))
    th.start()
    th.join()
7

As mentioned above by Matthew, you’ll need to run Python in interactive mode. If you don’t force interactive mode, the interpreter reads from stdin until the pipe closes, and only then does it compile and execute the input script. Also, without interactive mode the interpreter buffers standard output to a pipe. If buffering is enabled, it won’t actually flush writes to the pipe until the buffer is full.

Be aware that reading from separate stdout and stderr pipes may deadlock. For example, the parent is blocked reading from an empty stdout pipe, while the child is blocked writing to a full stderr pipe. The simplest approach to avoid deadlock that works cross-platform is to use separate reader threads for stdout and stderr, with separate queue.Queue() FIFOs.

Also, run the child interpreter with UTF-8 mode enabled (e.g. -X utf8 command-line option), or at least with PYTHONIOENCODING set to “UTF-8” for sys.std*. Otherwise the interpreter may use a legacy encoding for standard I/O. For example, on Windows Python uses the ANSI code page of the process for non-console I/O, which is usually a legacy encoding such as code page 1252.

1 Like