How to break out of a Thread/ProcessPoolExecutor

Python Help
1

How do I exit out of these once I am done?

I am trying to calculate a value, I do not know the value, but the worker will return True once it’s done. But I cannot cancel the remaining jobs, they keep running forever.

Things I have tried:

  • executor.shutdown()
  • multiprocessing.Event()
  • multiprocessing.Queue()
  • executor.map() and executor.submit()

Am I misusing it, and this is not supposed to be possible? I am essentially trying to do this, but with multiple processes. However as noted, I cannot break out of the job.

for i in range(100000):
    if i == 12345:
        break

Ideally I want to use executor.map() with a ProcessPoolExecutor.

2

You said you’ve used multiprocessing.Event(), but not how you’ve used it. Do the tasks check the event’s state often?

3

Something like this:

event = Event()
event.set()
executor.submit(worker, iterable, event)

And with each iteration of the loop:

if not event.is_set():
     return

But it doesn’t work at all, it raises this exception.

RuntimeError: Condition objects should only be shared between processes through inheritance
4

Solved using multiprocess.Manager()

I am still taking advice, if there are any improvements. However this is my solution for future readers.

def worker(args, event):
    if not event.is_set():
        break
    # do stuff

with multiprocessing.Manager() as manager:
    event = manager.Event()
    event.set()

    with ProcessPoolExecutor() as executor:
        futures = [executor.submit(worker, i, event) for i in iterable]

        for future in futures:
            if future.result():
                print(future.result())
                event.clear()
                break
1 Like