Concurrent usage of concurrent.futures.ThreadPoolExecutor

Python Help
1

Is concurrent.futures.ThreadPoolExecutor safe to use concurrently?
I seem to run into a deadlock when I have a concurrent.futures.ThreadPoolExecutor.map call when the function being mapped submits additional tasks to the same pool and calls result() on them.

2

You only have a finite pool of workers which can run tasks. If you have a pool of 10 workers, each running a task which schedules a new task and waits for the new task’s result, then you won’t have any workers to run any of the 10 new tasks, so you will be blocked forever.

A solution is to use a different pool (ie executor) to run the new tasks.

2 Likes
3

Yes, it’s safe, but details are important and there’s no way for us to guess at your details.

For example, this little progam hangs as-is, but works fine (and displays 3) if NTHREADS is changed to an int bigger than 1.

import concurrent.futures as cf

NTHREADS = 1

def worker(i):
    return ex.submit(lambda x: x+1, i).result() + 1

with cf.ThreadPoolExecutor(NTHREADS) as ex:
    print(ex.submit(worker, 1).result())

Why does it hang with only 1 thread? The main program is using the single worker thread to run the main ex.submit(), and that thread is waiting for worker() to return a result. When the ex.submit() call is made inside worker(), there are no other threads in the executor available to run it. The attempt waits forever for a thread to become available, but the only worker thread is waiting forever for worker() to finish.

Create an executor managing more than 1 thread, and that problem goes away.

Note that “safe” emphatically does not mean “deadlock-free in all cases”. There are many ways you can create deadlocks, just by asking for the impossible :wink:.

1 Like