part two of tonight’s conundrum is that I have a print statement that inexplicably executes in the wrong order; please take a look at the code below.
### run payload-dumper-go as a subprocess and dump the required firmware images from payload.bin ###
print('Dumping firmware images from payload.bin: ', end='')
try:
completed_process = run(['payload-dumper-go', '-output', fw_dir, '-partitions', 'abl,aop,bluetooth,cmnlib,cmnlib64,devcfg,dsp,fw_4j1ed,fw_4u1ea,hyp,keymaster,LOGO,modem,oem_stanvbk,qupfw,storsec,tz,vendor,xbl,xbl_config', normpath('./opflash/payload.bin')], stdout=PIPE, text=True, check=True)
print('DONE')
except CalledProcessError as called_process_error:
print('FAILED\n\nWARNING: payload-dumper-go failed to extract the firmware images;\nplease ensure the update file is a valid OxygenOS OT
exit(1)
both print() statements in this snippet are executed (according to my human senses) simultaneously. The CPU fan spools up while the subprocess runs and then the entire content of both print() statements is printed at the same time. There are similar patterns elsewhere in the code and they execute just fine
My wild guess is that it prints “Dumping firmware images from payload.bin” and then immediately prints Done. Which is not exactly out of order.
If something else is happening, you will have to tell us what.
If my guess is right, I believe the explanation is that by default stdout buffers written output until either the buffer is full or you print a newline. Since the first print suppresses the newline, stdout stores the text in a buffer and doesn’t display it until (eventually) you print a newline.
The solution to this is to add a flush=True to the first print call:
print('Dumping firmware images from payload.bin: ', flush=True, end='')
Sorry, you are right. My question relates to the first two print() statements. I guess I was so focused on the problem that I had my blinkers on with regards to the third!
Your solution solved it! I can’t say I understand the low level reasons for it to work though. In this case, would it be the python interpreter that buffers it’s own stdout until the buffer is full / a newline is printed, or is it the terminal that receives the interpreters stdout in a buffer? Or perhaps it is a responsibility of the OS?
I’ve a rudimentary understanding of standard file streams but I’ve never had to deal with them outside of IO redirection and using the subprocess module in python.
Mainly Python and the OS, but Python can request that data get pushed all the way out (that’s what flush=True does). The Python interpreter itself will see that and say “oh hey, I should actually write this out now”; the OS can be told “hey, please push that all the way out” (that’s done with a system call). I don’t think terminals ever buffer data, since that’s usually counterproductive, but depending on where you’re sending it, you might need to disable buffering there too. (For example, if you’re piping into grep, you might need to tell that to avoid buffering its output.)
Here’s a quick demonstration that Python’s print has some of its own buffering, prior to getting to the OS’s output:
Without flush=True, we see both “Wait…” and “Done” after the delay, but “Waiting…” comes up earlier. Anything the OS is doing would apply to both print and os.write, but the sys.stdout object has buffering built in. Your results may differ slightly, though, so try it out and have a play
I don’t believe there is anything preventing n processes from all trying to write output in the same moment.
You may want to see multiprocessing.Lock(). Create one in your parent process before spawning your subprocesses, and then have each process wrap its print’s in it.
You could also designate one of your processes as “the one that does prints”, have it look for values in a Queue and print them, and have all your other processes send things they need printed to that Queue.
(CAVEAT: I’m speaking from a Unix perspective, specifically Linux, and with some background knowledge from other platforms. This is NOT guaranteed to be accurate on all platforms.)
The protection from n processess is that they all have independent file descriptors. It may be that there are multiple processes writing to the same console, or even the same pipe, but in that case, you’ll usually see interleaving. This is one of the reasons that line buffering is a Good Thing; if you interleave text line by line, it’s usually less of a problem than interleaving characters within a line.
But the problem here isn’t interleaving; it’s just a simple matter of buffering.
