Code only handling single exception with pool.starmap

Python Help
1

The following code is only handling first exception while carrying on with rest of the code.

from multiprocessing.pool import ThreadPool

class SQLQueryError(Exception):
def init(self, message):
self.message = message
super().init(f"SQL query error: {message}")

def solve(args):
with ThreadPool(len(args)) as pool:
try:
return pool.starmap(func,args)
except Exception as e:
print(e)

def func(a, b):
if(a==“12” or a==“23” or b == “Work”):
raise SQLQueryError( f"Table does not exist {a} and {b}")

li = [(“12”, “Hello”), (“23”, “World”), (“12”, “Hospital”),(“7”,“Work”)]
solve(li)

Output: -
SQL query error: Table does not exist 12 and Hello

2

Put a try except into func. An SQLQueryError in any sub-process is not handled until it reaches the try/except in the main process.

1 Like
3

Thank you. Was also curious to know why it is even giving a single error ?

4

If you have any documentation regarding the same it would be really helpful.

5

The first element of li raises an error due to if (a=="12"

1 Like
6

That right but why doesn’t the rest raise the same because all the threads are executing and execution is not stopped. Thank you for the reply

7

They’re not threads, they’re subprocesses.

Python doesn’t support multiple unhandled exceptions at once, unless they’re in a chain of causality or another one occurs during exception handling.

When the first exception occurs it percolates upwards to parent processes until it is handled.

The code running in the subprocesses has no exception handling, so the exception is handled in the main process, very similarly to any other exception in the main process.

When any exception occurs, the main process terminates all its other child processes before halting itself.

Perhaps it’s possible to make Python crash harder than when a standard user exception is raised. But the normal multiprocessing code tries not to leave zombie/orphan processes that would require the user to clean them up.

1 Like
8

Thanks for the reply got a great overview of how the error are handled in python.

But still confused as to why are the other sub process in this case not halting but executing with their standard results. Actually I made a few changes to the code and found out that the code was running other sub processes unhindered.

9

You made me doubt myself! So I spent €0.10 for half an hour on a 16 v-core instance to check ( :heart: Hetzner, and Fedora has Python 3.12 out of the box). What I’ve described above is how Pool.map and Pool.starmap work. But Pool provides a lot of other tools, some of which handle errors differently, especially its lazy iterators.

processing.starmap requires all the subprocesses it spawns to complete for it to form a return value, so blocks on an error in a subprocess, and the behaviour is pretty much as I’ve described above:

[root@fedora-32gb-hel1-1 multiprocessing]# python run.py
No!!! 7 is Seven!

I’m not sure how you fixed it exactly, but you could try iterating over processing.imap_unordered (or the starmap version) instead. That returns the results from the subprocesses back in whichever order they arrive, and seems to hold on to exceptions until the end doesn’t necessarily

[root@fedora-32gb-hel1-1 multiprocessing]# python run.py
Got: 2
Got: 1
Got: 5
Got: 3
Got: 4
Got: 9
Got: 8
Got: 0
Got: 6
No!!! 7 is Seven!

import os
import sys
import multiprocessing
import random


data = list(range(10))

# Uncomment if testing Pool.starmap
# data = [[x] for x in data]

class Seven(Exception):
    pass

def error_if_seven(x):
    if x == 7:
        raise Seven(f'No!!! {x} is Seven!')
    return x

def main(args = sys.argv):


    num_processes = int(args[1]) if len(args) >= 2 else os.cpu_count()-1


    N = len(data)

    goals_in_random_order = random.sample(data, k = N)

    try:
        with multiprocessing.Pool(processes = num_processes) as pool:
            # for result in pool.starmap(error_if_seven, goals_in_random_order):
            for result in pool.imap_unordered(error_if_seven, goals_in_random_order):
                print(f'Got: {result}')
    except Exception as e:
        print(e)

        

if __name__ == '__main__':
    main()