As_completed not yielding futures that raised until pool exits?

Python Help
1

Is it expected that as_completed doesn’t yield a future that raises an exception until the pool completes all other work?

Take this example:

from concurrent.futures import ThreadPoolExecutor, as_completed
from time import sleep

def raise_ex():
    raise Exception('hello')

def sleep_a_while():
    sleep(5)

if __name__ == '__main__':
    with ThreadPoolExecutor(max_workers=8) as executor:
        results = []
        results.append(executor.submit(raise_ex))
        results.append(executor.submit(sleep_a_while))
        results.append(executor.submit(sleep_a_while))
        results.append(executor.submit(sleep_a_while))

        for future in as_completed(results):
            print(future.result())
            print(future.exception())

I would have thought this would almost immediately print the exception from raise_ex but instead it always waits for all the other workers to complete (so it prints about 5 seconds later). Same thing happens if using ProcessPoolExecutor instead.

From the docs:

...yields futures as they complete (finished or cancelled futures)....

The future completes immediately after raising (i think), so why does it wait till all others complete to be yielded by as_completed… is that a bug?

2

This took more debug than I care to admit.

Ultimately what was happening was that future.result() raises the exception it got. Though we don’t see that printed by the interpreter till the Executor completes. This is because the Executor was used as a contextmanager, and on __exit__ it waits for all processes to complete.

… so in other words, the exception won’t make it until the executor shuts down.

I recommend that folks use .exception() on the future after being returned by as_completed to see if it had an exception, handle it, then use result() to get the result in the event of no exception. (Or try/except the result() call to handle the exception… either way works)

1 Like