I have not seen clear examples with use-cases for Pool.apply, Pool.apply_async and Pool.map. I am mainly using Pool.map
; what are the advantages of others?
This question is related to
python
multithreading
concurrency
multiprocessing
Regarding apply
vs map
:
pool.apply(f, args)
: f
is only executed in ONE of the workers of the pool. So ONE of the processes in the pool will run f(args)
.
pool.map(f, iterable)
: This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. So you take advantage of all the processes in the pool.
Here is an overview in a table format in order to show the differences between Pool.apply
, Pool.apply_async
, Pool.map
and Pool.map_async
. When choosing one, you have to take multi-args, concurrency, blocking, and ordering into account:
| Multi-args Concurrence Blocking Ordered-results
---------------------------------------------------------------------
Pool.map | no yes yes yes
Pool.map_async | no yes no yes
Pool.apply | yes no yes no
Pool.apply_async | yes yes no no
Pool.starmap | yes yes yes yes
Pool.starmap_async| yes yes no no
Pool.imap
and Pool.imap_async
– lazier version of map and map_async.
Pool.starmap
method, very much similar to map method besides it acceptance of multiple arguments.
Async
methods submit all the processes at once and retrieve the results once they are finished. Use get method to obtain the results.
Pool.map
(or Pool.apply
)methods are very much similar to Python built-in map(or apply). They block the main process until all the processes complete and return the result.
Is called for a list of jobs in one time
results = pool.map(func, [1, 2, 3])
Can only be called for one job
for x, y in [[1, 1], [2, 2]]:
results.append(pool.apply(func, (x, y)))
def collect_result(result):
results.append(result)
Is called for a list of jobs in one time
pool.map_async(func, jobs, callback=collect_result)
Can only be called for one job and executes a job in the background in parallel
for x, y in [[1, 1], [2, 2]]:
pool.apply_async(worker, (x, y), callback=collect_result)
Is a variant of pool.map
which support multiple arguments
pool.starmap(func, [(1, 1), (2, 1), (3, 1)])
A combination of starmap() and map_async() that iterates over iterable of iterables and calls func with the iterables unpacked. Returns a result object.
pool.starmap_async(calculate_worker, [(1, 1), (2, 1), (3, 1)], callback=collect_result)
Find complete documentation here: https://docs.python.org/3/library/multiprocessing.html
Source: Stackoverflow.com