How Can I Catch When A Thread Dies In Threadpoolexecutor()?

I have some very simple python code that runs a bunch of inputs through various processes via ThreadPoolExecutor(). Now, sometimes one or more of the threads dies quietly. It is

Solution 1:

Executor.map does not support gathering more than one exception. However, its code can easily be adapted to return the arguments on which a failure occurred.

defattempt(executor: 'Executor', fn: 'Callable', *iterables):
    """Attempt to ``map(fn, *iterables)`` and return the args that caused a failure"""
    future_args = [(self.submit(fn, *args), args) for args inzip(*iterables)]

    deffailure_iterator():
        future_args.reverse()
        while future_args:
            future, args = future_args.pop()
            try:
                future.result()
            except BaseException:
                del future
                yield args
    return failure_iterator()

This can be used to concurrently "map" arguments to functions, and later retrieve any failures.

import concurrent.futures as cf

with cf.ThreadPoolExecutor() as executor:
     a_failures = attempt(executor, process_a, process_a_inputs)
     b_failures = attempt(executor, process_b, process_b_inputs)
     for args in a_tries:
         print(f'failed to map {args} onto a')
     for args in b_tries:
         print(f'failed to map {args} onto b')