skmap.parallel.utils.ProcessGeneratorLazy#

ProcessGeneratorLazy(worker, args, max_workers=4, chunk=8, fixed_args=())[source]#

Execute a function in parallel using a ProcessPoolExecutor [1].

Parameters:

  • worker (Callable) – Function to execute in parallel.

  • args (Iterator[tuple]) – to separate job of the pool.

  • max_workers (int) – Number of CPU cores to use in the parallelization. By default all cores are used.

  • chunk (int) – Number of chunks to split the parallelization jobs.

  • fixed_args (tuple) – Constant arguments added in args in each execution of the worker function.

Returns:

A generator with the return of all workers

Return type:

Generator

References

[1] Python ProcessPoolExecutor class

Examples

>>> from skmap.parallel import ProcessGeneratorLazy
>>>
>>> def worker(i, msg):
...   print(f'{i}: {msg}')
...   return f'Worker {i} finished'
>>>
>>> args = iter([ (i,) for i in range(0,5)])
>>> fixed_args = ("I'm running in parallel", )
>>>
>>> for result in ProcessGeneratorLazy(worker, args, fixed_args=fixed_args):
...   print(result)