skmap.parallel.utils.job#

job(worker, worker_args, n_jobs=-1, joblib_args={})[source]#

Execute a function in parallel using joblib [1].

Parameters:

worker (Callable) – Function to execute in parallel.
worker_args (Iterator[tuple]) – Argument iterator where each element is send to separate job.
joblib_args (set) – Number of CPU cores to use in the parallelization. By default all cores are used.
joblib_args – Joblib argumets to send to Parallel class [1].

Returns:

A generator with the return of all workers

Return type:

Generator

References

[1] joblib.Parallel class

Examples

>>> from skmap import parallel
>>>
>>> def worker(i, msg):
...   print(f'{i}: {msg}')
...   return f'Worker {i} finished'
>>>
>>> msg = ("I'm running in parallel", )
>>> args = iter([ (i,msg) for i in range(0,5)])
>>>
>>> for result in parallel.job(worker, args, n_jobs=-1, joblib_args={'backend': 'threading'}):
...   print(result)