Suprocess not connecting to one another

Python Help
1

I have two

p1 = subprocess.Popen(command1, stdout=out1, stderr=err1, text=True)
p2 = subprocess.Popen(command2, stdout=out2, stderr=err2, text=True)

The commands in command1 and command2, when run manually (in Ubuntu inside WSL, in my case), are applications that connect to each other through, e.g. 127.0.0.1 and some port, exchange some data and do some computations.

When I run the commands manually, in two terminals, they run, connect and compute. Same commands, I am printing them with " ".join(command1) and " ".join(command2) and copying them from there.

However, from Python, they run, but the connection doesn’t stablish; they time out and terminate.

Likely this is not enough information to tell, but does someone know, from experience, if there is anything specific to subprocess.Popen that could be a cause for the two processes not being able to connect?

Added.

  1. out1, out2, err1, err2 are all files. I see in them the two applications reporting that they run, that they attempt to connect, the port and host, when they time out and give up, and when they terminate.
  2. The port is chosen during each run to hopefully be a free one, with
def getPort():
	with closing(
		socket.socket(socket.AF_INET, socket.SOCK_STREAM)
	) as s:
		s.bind(('localhost', 0))
		s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
		return s.getsockname()[1]
2

Hello,

is this what you had in mind via this example.

  1. Create two modules: reader.py and writer.py and save them to your Desktop for simplicity.
# C:\Desktop\reader.py

try:
    print('Communication is about to start: "%s"' % input())
    import sys
    data = sys.stdin.readline()[:-1]
    print('The meaning of life is', data, int(data) * 2)

except KeyboardInterrupt:
    pass

and

# C:\Desktop\writer.py

print("Hello! I am the information that is being read from the read.py module.")
print(100)

The third script:

# C:\Desktop\processing.py

import os
from subprocess import Popen, PIPE

os.chdir('C:/Desktop')
print(os.getcwd())

try:
    p1 = Popen('python writer.py')
    p2 = Popen('python reader.py' , stdin=p1.stdout)

    output = p2.communicate()[0]

    print(output)

except KeyboardInterrupt:
    pass

Run the processing.py script. Note that in order for them to communicate, the input to p2 is the output from p1. From the script that you provided, this is what is missing.

1 Like
3

The commands are third party applications. They exchange some information via a network connection. Only how Python launches them is what I get to choose.

Laughing them by hand and using subprocess.Popen both seem to work, with the exception that in the latter case they don’t manage to connect.

4

Long shot, but without more info it’s tough to say. Try setting shell=True on the popen calls. Maybe something about the args are different via the shlexing.

5

I’m guessing here because you don’t give enough info.

You don’t say what stdout and stderr are set to. For debugging purposes, so you can see the results of print(…), you should probably omit these. Or if you’re writing them to files, have you looked at the files?

If you aren’t waiting the processes, e.g., p1.wait(), they may not terminate. If they don’t terminate the socket may remain in use. You may have had one successful run that didn’t terminate and the subsequent runs are failing for that reason.

6

I suspect that the subprocesses are not running concurrently. They run one at a time, and tries to talk to the other, which of course fails.

Try concurrent.futures — Launching parallel tasks — Python 3.13.2 documentation

1 Like
7

In the files out1 and out2 they print a lot of stuff (10s K lines). I keep them open, and see them being written at the same time. I see when both files stay there for a while after printing that they are trying to connect, and when they give up and print the last reporting lines (a summary). Could this be an illusion of the OS flushing the buffers, but one of the applications already finished?

My understanding (which can easily be wrong) is that subprocess.Popen makes the process run in the background, while subprocess.run does wait for termination.

I have tried both with Popen, but most of the time I actually launch the second with subprocess.run. With both using subprocess.run I do see command2 not starting until command1 has finished completely.

p1 = subprocess.Popen(command1, stdout=out1, stderr=err1, text=True)
p2 = subprocess.run(command2, stdout=out2, stderr=err2, text=True)
8

Can you try as @csm10495 stated. I read that the shell=True might be required for Linux like platforms which Ubuntu is.

9

Yes, same thing. I had to change the command1 and command2 from being list[str] to single strings, otherwise it looks like each entry of the list is read as an individual command, but apart from that the same happens. They run fine, get to the connection part, don’t connect, and finish.

Anyway, it might not be related to Popen. I asked just in case it is due to something like the subprocess not inheriting something about the environment of the parent, or some permission.

It looks like it is more specific to the applications and/or my environment.

10

It wasn’t plugged to the electricity …
Essentially I was running command2 twice.

1 Like