multiprocessing Pool When to use apply apply async or map

Question

I have not seen clear examples with use-cases for Pool apply  Pool apply async and Pool map  I am mainly using Pool map  what are the advantages of others

User · Answer

Regarding apply vs map   pool apply f  args   f is only executed in ONE of the workers of the pool  So ONE of the processes in the pool will run f args    pool map f  iterable   This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks  So you take advantage of all the processes in the pool

User · Answer

Back in the old days of Python  to call a function with arbitrary arguments  you would use apply   apply f args kwargs    apply still exists in Python2 7 though not in Python3  and is generally not used anymore  Nowadays    f  args   kwargs    is preferred  The multiprocessing Pool modules tries to provide a similar interface   Pool apply is like Python apply  except that the function call is performed in a separate process  Pool apply blocks until the function is completed   Pool apply async is also like Python s built-in apply  except that the call returns immediately instead of waiting for the result  An AsyncResult object is returned  You call its get   method to retrieve the result of the function call  The get   method blocks until the function is completed  Thus  pool apply func  args  kwargs  is equivalent to pool apply async func  args  kwargs  get     In contrast to Pool apply  the Pool apply async method also has a callback which  if supplied  is called when the function is complete  This can be used instead of calling get     For example   import multiprocessing as mp import time  def foo pool x       time sleep 2      return x x  result list      def log result result         This is called whenever foo pool i  returns a result        result list is modified only by the main process  not the pool workers      result list append result   def apply async with callback        pool   mp Pool       for i in range 10           pool apply async foo pool  args    i     callback   log result      pool close       pool join       print result list   if   name         main         apply async with callback     may yield a result such as   1  0  4  9  25  16  49  36  81  64    Notice  unlike pool map  the order of the results may not correspond to the order in which the pool apply async calls were made     So  if you need to run a function in a separate process  but want the current process to block until that function returns  use Pool apply  Like Pool apply  Pool map blocks until the complete result is returned   If you want the Pool of worker processes to perform many function calls asynchronously  use Pool apply async  The order of the results is not guaranteed to be the same as the order of the calls to Pool apply async   Notice also that you could call a number of different functions with Pool apply async  not all calls need to use the same function    In contrast  Pool map applies the same function to many arguments  However  unlike Pool apply async  the results are returned in an order corresponding to the order of the arguments

User · Answer

Here is an overview in a table format in order to show the differences between Pool apply  Pool apply async  Pool map and Pool map async  When choosing one  you have to take multi-args  concurrency  blocking  and ordering into account                       Multi-args   Concurrence    Blocking     Ordered-results --------------------------------------------------------------------- Pool map            no           yes            yes          yes Pool map async      no           yes            no           yes Pool apply          yes          no             yes          no Pool apply async    yes          yes            no           no Pool starmap        yes          yes            yes          yes Pool starmap async  yes          yes            no           no   Notes    Pool imap and Pool imap async     lazier version of map and map async  Pool starmap method  very much similar to map method besides it acceptance of multiple arguments  Async methods submit all the processes at once and retrieve the results once they are finished  Use get method to obtain the results  Pool map or Pool apply methods are very much similar to Python built-in map or apply   They block the main process until all the processes complete and return the result    Examples   map  Is called for a list of jobs in one time  results   pool map func   1  2  3     apply  Can only be called for one job  for x  y in   1  1    2  2        results append pool apply func   x  y     def collect result result       results append result    map async  Is called for a list of jobs in one time  pool map async func  jobs  callback collect result    apply async  Can only be called for one job and executes a job in the background in parallel  for x  y in   1  1    2  2        pool apply async worker   x  y   callback collect result    starmap  Is a variant of pool map which support multiple arguments  pool starmap func    1  1    2  1    3  1      starmap async  A combination of starmap   and map async   that iterates over iterable of iterables and calls func with the iterables unpacked  Returns a result object   pool starmap async calculate worker    1  1    2  1    3  1    callback collect result    Reference   Find complete documentation here  https   docs python org 3 library multiprocessing html

[python] multiprocessing.Pool: When to use apply, apply_async or map?

Examples related to python

Examples related to multithreading

Examples related to concurrency

Examples related to multiprocessing