Python program does not finish completely after running multiple concurrent threads that each launch a subprocess.Popen command

Python Help
1

I came here from the python.org webpage “Dealing with Bugs.” Python is such a mature language that it seems unlikely I could find a bug. It sure seems like I have but it is more likely that I am doing something wrong.

Can I raise this type of question here? If not, where should I do it?

I have a sample program I can share to demonstrate the problem.

I am running on Linux Mint (21.3 and 22.1) with Python 3.10 and 3.12.

The Python program starts multiple threads and each of those threads use subprocess.Popen to run a program. If two of those threads run concurrently, the GUI will not shutdown completely. I removed the Popen calls and substituted time.sleep() to confirm that concurrent threads not using subprocess.Popen do not cause any problem. I have tried both the threading module and the concurrent.futures module. Same result.

When I run the program in a terminal window, the program seems to shutdown because I have a print statement at the very end of the program and the message from that print statement appears in the window. The command line prompt also appears. Everything looks ok but the window is now useless. It will not respond to any keypress except ctrl+C, which only causes another prompt to appear. It is no longer possible to type any command. The window can be closed only by clicking on the X in the corner of the window.

I should add that all threads have finished before I try to shutdown the program.

Again, if this is not the right place to raise this issue, please let me know where I can.

2

Maybe share your code so people can help have a look

3

This example runs ffmpeg but any program that runs for at least a few seconds can be used. To use ffmpeg, put two or more fully-qualified audio file name in mediaList in the main part of the program.

I forgot to mention the TEST variable immediately above the requestGain() routine. If you set TEST to True, ffmpeg will return very quickly and avoid concurrent thread execution. The program will shutdown completely with TEST set to True.

#!/usr/bin/env python3
#

import sys, subprocess, queue, threading
from PyQt5 import QtWidgets, QtCore

class AudioPlayerDialog(QtWidgets.QDialog):
    def __init__(self, mediaList):
        super().__init__()
        vbox = QtWidgets.QVBoxLayout()
        pb = QtWidgets.QPushButton('CLOSE')
        pb.clicked.connect(self.close)
        vbox.addWidget(pb)
        self.setLayout(vbox)
        
        self.calcGainTimer = None
        self.gdCalcThreads = None
        self.finished.connect(self.closeEvent)

        self.mediaList = mediaList
        self.mediaListLen = len(mediaList)
        self.normGainList = [-1. for x in mediaList]
        self.buildNormGainList()

    def closeEvent(self, event):
        print('**** close event')
        if self.calcGainTimer is not None:
            print('**** stopping calcGainTimer')
            self.calcGainTimer.stop()
        if self.gdCalcThreads is not None:
            print('**** checking gain calc threads')
            for idx, t in self.gdCalcThreads.copy().items():
                if t.is_alive():
                    print('**** waiting for thread {} to finish'.format(idx))
                    t.join()
        print('**** closing ...')

    def buildNormGainList(self):
        self.calcGainQueue = queue.LifoQueue(maxsize=self.mediaListLen + 10)
        self.calcResultsQueue = queue.Queue(maxsize=self.mediaListLen)
        for i in range(self.mediaListLen-1, -1, -1):
            self.calcGainQueue.put_nowait(i)
        # setup gain/dur calculation thread dictionary
        self.gdCalcThreads = dict()
        # For each expiration of the following timer, a thread will be
        # started to calculate the replay gain for a particular file
        # and a check will determine if all files have been processed.
        print('@@@@ starting updt timer')
        self.calcGainTimer = QtCore.QTimer()
        self.calcGainTimer.timeout.connect(self.stopGainDurCalcWhenDone)
        self.calcGainTimer.setInterval(500) # interval in msec.
        self.calcGainTimer.start()

    def stopGainDurCalcWhenDone(self):
        # this is the target of the updt timer
        try:
            # start a thread if the queue is not empty
            idx = self.calcGainQueue.get_nowait()
            if idx not in self.gdCalcThreads:
                print('@@@@ creating thread for', idx)
                thread = threading.Thread(None, self.requestGain, args=(idx,))
                self.gdCalcThreads[idx] = thread
                thread.start()
        except queue.Empty:
            pass
        except Exception as err:
            print('**** ERROR: get request from calcGainQueue failed.\n\t'\
                                                        '{}'.format(err))
        # check if any thread finished but updates not yet finished
        try:
            idx, gain = self.calcResultsQueue.get_nowait()
            self.normGainList[idx] = gain
            print('@@@@ update finished for', idx)
        except queue.Empty:
            pass
        except Exception as err:
            print('**** ERROR: get request from calcResultsQueue failed.\n\t'\
                                                        '{}'.format(err))
            
        # check if all files have been processed; stop updt timer if so
        done = True     # assume all calc done
        for i in range(self.mediaListLen):
            if self.normGainList[i] < 0.:
                done = False
                break
        if done:
            print('@@@@ all calc are done -----------------------')
            # all files have been processed
            print('@@@@ stopping updt timer')
            self.calcGainTimer.stop()
            self.calcGainTimer = None
            self.gdCalcThreads = None

    TEST = False  # set True to pass dummy filename to ffmpeg

    def requestGain(self, idx):
        print('@@@@ start subprocess for', idx)
        if self.TEST:
            fileName = 'XXXX'
        else:
            fileName = self.mediaList[idx]
        cmd = ['ffmpeg', '-i', '{}'.format(fileName), '-hide_banner',
               '-af', 'replaygain', '-f', 'null', '-']
        proc = subprocess.Popen(cmd, stderr=subprocess.PIPE)
        try:
            _,err = proc.communicate(timeout=60)
            text = err.decode('utf8')
        except subprocess.TimeoutExpired:
            print('**** ERROR: Wait for report from subprocess timed out')
            proc.kill()
            text = ''
        except Exception as err:
            print('**** ERROR: Unable to get report from subprocess: {}'.format(err))
            proc.kill()
            text = ''
        # actual program parses gain from text
        gain = 1.
        self.calcResultsQueue.put_nowait((idx,gain))


if __name__ == '__main__':

    app = QtWidgets.QApplication(sys.argv)

    # put names of media files in list
    mediaList = [
        ]

    apd = AudioPlayerDialog(mediaList)
    apd.exec()

    print('\nFINI')
    sys.exit()
4

Hello,

have you tried setting your processes to daemon as a test?

thread = threading.Thread(target=function_name_here, args=(misc))
thread.daemon = True
5

Yes I did try daemons. The same problem occurs.

6

What you are stating is that the subprocess.Open running with your GUI is the root cause of the issue since as you stated when you replaced the Popen calls with time.sleep(), the problems were no longer observed.

Can you try creating a non-GUI sample script to see if it is the GUI that is perhaps the issue - the layout maybe? Can you try the same script then using tkinter? Seems like a bit of work, I know. Maybe simplify the processes (start small … then build up and see where it breaks).

7

Hi Paul,

Thanks for the comment.

I provided a simpler example in this post without a GUI but it occurred to me later that I could simplify it even more. That simpler example is in the next post.

I am deleting what I had posted here.

8

Paul,

Your suggestion caused me to rethink what is a minimal example I can give to show the problem. I show a simpler example below.

I am wondering if the way I expressed the problem has caused some confusion. I never intended to say the GUI did not close or shutdown. The problem has been that the process that runs the GUI program does not shutdown completely.

The GUI itself has never been part of the problem, which the previous simplified example and this even simpler example show.

Once again, set TEST to True to see the problem goes away because no two subprocess.Popen instances ever run concurrently.

Here it is:

#!/usr/bin/env python3
#

import sys, subprocess, threading, time

TEST = False  # set True to pass dummy filename to ffmpeg

def requestGain(fileName):
    print('starting subprocess for', fileName)
    if TEST:
        fileName = 'XXXX'

    cmd = ['ffmpeg', '-i', '{}'.format(fileName), '-hide_banner',
           '-af', 'replaygain', '-f', 'null', '-']

    proc = subprocess.Popen(cmd, stderr=subprocess.PIPE)

    try:
        _,err = proc.communicate(timeout=60)
    except Exception as err:
        print('**** ERROR: for {} from subprocess: {}'.format(fileName, err))
        proc.kill()


if __name__ == '__main__':

    # put names of media files in list
    mediaList = [
        ]

    threadDict = dict()

    for media in mediaList:
        print('starting thread for', media)
        thread = threading.Thread(None, requestGain, args=(media,))
        threadDict[media] = thread
        thread.start()
        time.sleep(.25)

    for m,t in threadDict.items():
        if t.is_alive():
            print('waiting on thread for', m)
            t.join()

    print('\nFINI')
    sys.exit()

I would appreciate knowing if others do or do not see the same problem when they run this script and what OS are they running.

It may be useful to re-read my first post where I explain what I mean by saying the process does not shutdown completely. What appears in the terminal window looks like the program ends ok but it does not because the terminal window can no longer be used to run another command. It ignores all keys except a ctrl+C.

9

Do the processes exit?

When the script is stuck you can run ps afx in a terminal to see if the ffmpeg processes have exited.

You can also try usingstrace to see why the script is stuck.

10

Does it work if you use subprocess.run()?

11

Hello,

one observation. Why do you create the thread as:

vs.

thread = threading.Thread(target = requestGain, args=(media,))

?

group is *None* by default anyway.
thread_docs

Would have helped if your sample script did not require including external media files from the user testing the script.

12

I modified your test script slightly such that I call very simple .py files … very bare bones from which to build up. The three test .py files that I created to run concurrently are all located on the Desktop for simplicity. After running the script from the command, I do not observe this:

After running the test script, I am able to interact with the command prompt and run additional commands with no issues.

# multi.py

import time

def multi(x, y):
    return x * y

x = 5
y = 10

time.sleep(5)

print(f'The factor of {x} and {y} is: {multi(x,y)}')

# addi

import time

def addi(x, y):
    return x + y

x = 30
y = 5

time.sleep(5)

print(f'The addition of {x} and {y} is: {round(addi(x, y), 1)}')


# divi.py

import time

def divi(x, y):
    return x / y

x = 30
y = 5

time.sleep(5)

print(f'The division of {x} by {y} is: {round(divi(x, y), 1)}'

The modified test script is then:

import os
os.chdir(r'C:\Desktop')

import sys, subprocess, threading, time

def requestGain(fileName):

    print('starting subprocess for', fileName)

    file_run = 'python' + ' ' + fileName
    pipe = subprocess.Popen(file_run, stdout=subprocess.PIPE)
    stdout, stderr = pipe.communicate()

    print("Print statement in test module: ", fileName)
    print(stdout.decode())


if __name__ == '__main__':

    # put names of media files in list
    mediaList = ['divi.py', 'multi.py', 'addi.py'
        ]

    threadDict = dict()

    for media in mediaList:

        print('starting thread for', media)

        thread = threading.Thread(target=requestGain, args=(media,))
        threadDict[media] = thread
        thread.start()
        time.sleep(.25)

    for m,t in threadDict.items():

        if t.is_alive():

            print('waiting on thread for', m)
            t.join()

    print('\nFINI')
    sys.exit()

Build from here and see where it is that your program begins to not work.

fyi … I am using OS Windows 11 Pro

Update:

Is the word replaygain a typo? Should it be: replayagain?

13

@BarryScott - I used top in another terminal window and it showed the ffmpeg processes disappearing as my program reported the threads that launched them finished.

I will try strace. This could be very helpful – see comments below.

As for subprocess.run(), it is merely a higher-level interface that uses subprocess.Popen underneath.

@Paul - Before reading your reply I tried a few alternatives today. Rather than running ffmpeg I tried:
(1) a Python script with a time.sleep;
(2) a Python script with a for loop for 10 million iterations;
(3) grep looking for files in the /usr directory;
(4) ffplay playing a short media file.

All of those processes ran long enough to overlap but they did not cause the termination problem. Exactly as you reported. It now appears the problem is with ffmpeg or a combination of factors with ffmpeg and the overlapping processes.

Paul, if you are game to try ffmpeg, you could create a list of media files more easily by typing just one name but duplicating that name one or more times.

I will explore this a bit more and report back if I can find anything worthwhile to add.

Thanks to all for the questions and suggestions.

14

Have you tried running the ffmpeg files without the arguments in the cmd list?

Why are you passing in the filename with:

'{}'.format(fileName)

It’s the same as passing in just the filename:

module_name = 'divi.py'
filename = "{}".format(module_name)
print('filename = ', filename)

>>> filename = divi.py

print(type(filename))

>>> <class 'str'>

This would make sense if you were modifying it in some way as in:

module_name = 'divi.py'
filename = "ffmpeg_{}".format(module_name)
print('filename = ', filename)

>>> filename =  ffmpeg_divi.py

What is the purpose of the other arguments in the list? As stated above, can you test your script without the cmd arguments and just passing in the ffmpeg file names as was done with the bare bones test script?

15

The point is that run() does all the details of using popen correctly.
I wonder if you forgot a detail that means you hang?

16

@Paul - The arguments to ffmpeg are needed to get the program to return the information I need, which is replay gain for a particular audio file. Without those arguments there is no need to call ffmpeg. This is not just a coding exercise. It is an extraction from a musicplayer I wrote that calls ffmpeg to get some desired information.

Why use ‘“{}”’.format(fileName) instead of just the file name? You are correct in this example. I had used the string.format() approach because the filenames had spaces and had to enclosed in quotation marks. Somewhere along the way those quote marks got lost.

@Barry - I tried run(). Same result as expected. There is really nothing complicated here. The Popen call is straight forward - something I have used in code for many years.

I ran the ffmpeg call in strace. Using a compare program, I compared results between calls that overlapped and calls that did not. Unfortunately, it did not reveal anything that differed in any material way; just different process numbers.

17

When the program hangs have the threads exited?
If the threads have not exited then they must be in a system call and strace can show you that system call.

If they have exited then it suggests that the python threading code has lost track of a thread. Having the main thead query the number of threads and which ones are know to python would be interesting.

Just to check did you use the strace options that trace into all threads and across fork/exec? You should be able to see each thread be created and later exit.

18

Wonder if this is related:

Can you try setting shell=True in the subprocess.Popen.

On Unix-like platforms, when shell is False (its default), the program command line is run directly by os.execvp, ..., If this argument is True, the command-line string is run through a shell instead, and you can specify the shell to use with additional arguments.
19

That, other then on Windows, is not a robust way to start a process.
It means that the command line is first parsed into the list of args that the system requires, which is not quoted correctly leads to subtle errors.

What are you expecting to change?

20

Per the OPs experience with the ffmpeg threads not working as expected, was wondering if it was in some way related to the link attached.