How to use multiprocessing pool map with multiple arguments

Question

In the Python multiprocessing library  is there a variant of pool map which supports multiple arguments  text    quot test quot  def harvester text  case       X   case 0      text  str X   if   name         main         pool   multiprocessing Pool processes 6      case   RAW DATASET     pool map harvester text case  case  1      pool close       pool join

User · Answer

Using Python 3 3  with pool starmap     from multiprocessing dummy import Pool as ThreadPool   def write i  x       print i   ---   x   a     1   2   3   b     4   5   6     pool   ThreadPool 2  pool starmap write  zip a b    pool close    pool join     Result   1 --- 4 2 --- 5 3 --- 6   You can also zip   more arguments if you like  zip a b c d e   In case you want to have a constant value passed as an argument you have to use import itertools and then zip itertools repeat constant   a  for example

User · Answer

There are many answers here  but none seem to provide Python 2 3 compatible code that will work on any version  If you want your code to just work  this will work for either Python version     For python 2 3 compatibility  define pool context manager   to support the  with  statement in Python 2 if sys version info 0     2      from contextlib import contextmanager      contextmanager     def multiprocessing context  args    kwargs           pool   multiprocessing Pool  args    kwargs          yield pool         pool terminate   else      multiprocessing context   multiprocessing Pool   After that  you can use multiprocessing the regular Python 3 way  however you like  For example   def  function to run for each x          return x lower   with multiprocessing context processes 3  as pool      results   pool map  function to run for each    Bob    Sue    Tim       print results    will work in Python 2 or Python 3

User · Answer

Having learnt about itertools in J F  Sebastian answer I decided to take it a step further and write a parmap package that takes care about parallelization  offering map and starmap functions on python-2 7 and python-3 2  and later also  that can take any number of positional arguments   Installation  pip install parmap   How to parallelize   import parmap   If you want to do  y    myfunction x  argument1  argument2  for x in mylist    In parallel  y   parmap map myfunction  mylist  argument1  argument2     If you want to do  z    myfunction x  y  argument1  argument2  for  x y  in mylist    In parallel  z   parmap starmap myfunction  mylist  argument1  argument2     If you want to do  listx    1  2  3  4  5  6  listy    2  3  4  5  6  7  param   3 14 param2   42 listz      for  x  y  in zip listx  listy           listz append myfunction x  y  param1  param2     In parallel  listz   parmap starmap myfunction  zip listx  listy   param1  param2    I have uploaded parmap to PyPI and to a github repository   As an example  the question can be answered as follows   import parmap  def harvester case  text       X   case 0      text  str X   if   name         main         case   RAW DATASET    assuming this is an iterable     parmap map harvester  case   test   chunksize 1

User · Answer

text    test   def unpack args       return args 0   args 1     def harvester text  case       X   case 0      text  str X   if   name         main         pool   multiprocessing Pool processes 6      case   RAW DATASET       args is a list of tuples        with the function to execute as the first item in each tuple     args     harvester  text  c  for c in case        doing it this way  we can pass any function       and we don t need to define a wrapper for each different function       if we need to use more than one     pool map unpack  args      pool close       pool join

User · Answer

How to take multiple arguments  def f1 args       a  b  c   args 0    args 1    args 2      return a b c  if   name       quot   main   quot       import multiprocessing     pool   multiprocessing Pool 4        result1   pool map f1     1 2 3         print result1

User · Answer

Here is another way to do it that IMHO is more simple and elegant than any of the other answers provided   This program has a function that takes two parameters  prints them out and also prints the sum   import multiprocessing  def main         with multiprocessing Pool 10  as pool          params      2  2    3  3    4  4            pool starmap printSum  params        end with    end function  def printSum num1  num2       mySum   num1   num2     print  num1       str num1       num2       str num2       sum       str mySum     end function  if   name         main         main     output is   num1   2  num2   2  sum   4 num1   3  num2   3  sum   6 num1   4  num2   4  sum   8   See the python docs for more info   https   docs python org 3 library multiprocessing html module-multiprocessing pool  In particular be sure to check out the starmap function   I m using Python 3 6  I m not sure if this will work with older Python versions  Why there is not a very straight-forward example like this in the docs  I m not sure

User · Answer

The answer to this is version- and situation-dependent  The most general answer for recent versions of Python  since 3 3  was first described below by J F  Sebastian 1 It uses the Pool starmap method  which accepts a sequence of argument tuples  It then automatically unpacks the arguments from each tuple and passes them to the given function   import multiprocessing from itertools import product  def merge names a  b       return      amp      format a  b   if   name         main         names     Brown    Wilson    Bartlett    Rivera    Molloy    Opie       with multiprocessing Pool processes 3  as pool          results   pool starmap merge names  product names  repeat 2       print results     Output    Brown  amp  Brown    Brown  amp  Wilson    Brown  amp  Bartlett         For earlier versions of Python  you ll need to write a helper function to unpack the arguments explicitly  If you want to use with  you ll also need to write a wrapper to turn Pool into a context manager   Thanks to muon for pointing this out    import multiprocessing from itertools import product from contextlib import contextmanager  def merge names a  b       return      amp      format a  b   def merge names unpack args       return merge names  args    contextmanager def poolcontext  args    kwargs       pool   multiprocessing Pool  args    kwargs      yield pool     pool terminate    if   name         main         names     Brown    Wilson    Bartlett    Rivera    Molloy    Opie       with poolcontext processes 3  as pool          results   pool map merge names unpack  product names  repeat 2       print results     Output    Brown  amp  Brown    Brown  amp  Wilson    Brown  amp  Bartlett         In simpler cases  with a fixed second argument  you can also use partial  but only in Python 2 7    import multiprocessing from functools import partial from contextlib import contextmanager   contextmanager def poolcontext  args    kwargs       pool   multiprocessing Pool  args    kwargs      yield pool     pool terminate    def merge names a  b       return      amp      format a  b   if   name         main         names     Brown    Wilson    Bartlett    Rivera    Molloy    Opie       with poolcontext processes 3  as pool          results   pool map partial merge names  b  Sons    names      print results     Output    Brown  amp  Sons    Wilson  amp  Sons    Bartlett  amp  Sons         1  Much of this was inspired by his answer  which should probably have been accepted instead  But since this one is stuck at the top  it seemed best to improve it for future readers

User · Answer

In the official documentation states that it supports only one iterable argument  I like to use apply async in such cases  In your case I would do   from multiprocessing import Process  Pool  Manager  text    test  def harvester text  case  q   None    X   case 0   res   text  str X   if q    q put res   return res   def block until q  results queue  until counter 0    i   0  while i  lt  until counter    results queue put q get      i  1  if   name         main      pool   multiprocessing Pool processes 6   case   RAW DATASET  m   Manager    q   m Queue    results queue   m Queue     when it completes results will reside in this queue  blocking process   Process block until   q  results queue  len case     blocking process start    for c in case    try     res   pool apply async harvester   text  case  q   None      res get timeout 0 1    except     pass  blocking process join

User · Answer

There s a fork of multiprocessing called pathos  note  use the version on github  that doesn t need starmap -- the map functions mirror the API for python s map  thus map can take multiple arguments  With pathos  you can also generally do multiprocessing in the interpreter  instead of being stuck in the   main   block  Pathos is due for a release  after some mild updating -- mostly conversion to python 3 x     Python 2 7 5  default  Sep 30 2013  20 15 49      GCC 4 2 1  Apple Inc  build 5566   on darwin   Type  help    copyright    credits  or  license  for more information     gt  gt  gt  def func a b             print a b          gt  gt  gt     gt  gt  gt  from pathos multiprocessing import ProcessingPool        gt  gt  gt  pool   ProcessingPool nodes 4     gt  gt  gt  pool map func   1 2 3    1 1 1     1 1   2 1   3 1    None  None  None     gt  gt  gt     gt  gt  gt    also can pickle stuff like lambdas     gt  gt  gt  result   pool map lambda x  x  2  range 10      gt  gt  gt  result    0  1  4  9  16  25  36  49  64  81     gt  gt  gt     gt  gt  gt    also does asynchronous map    gt  gt  gt  result   pool amap pow   1 2 3    4 5 6      gt  gt  gt  result get      1  32  729     gt  gt  gt     gt  gt  gt    or can return a map iterator    gt  gt  gt  result   pool imap pow   1 2 3    4 5 6      gt  gt  gt  result    lt processing pool IMapIterator object at 0x110c2ffd0 gt     gt  gt  gt  list result     1  32  729    pathos has several ways that that you can get the exact behavior of starmap    gt  gt  gt  def add  x         return sum x        gt  gt  gt  x     1 2 3   4 5 6    gt  gt  gt  import pathos  gt  gt  gt  import numpy as np  gt  gt  gt    use ProcessPool s map and transposing the inputs  gt  gt  gt  pp   pathos pools ProcessPool    gt  gt  gt  pp map add   np array x  T   6  15   gt  gt  gt    use ProcessPool s map and a lambda to apply the star  gt  gt  gt  pp map lambda x  add  x   x   6  15   gt  gt  gt    use a  ProcessPool  which has starmap  gt  gt  gt   pp   pathos pools  ProcessPool    gt  gt  gt   pp starmap add  x   6  15   gt  gt  gt

User · Answer

From python 3 4 4  you can use multiprocessing get context   to obtain a context object to use multiple start methods   import multiprocessing as mp  def foo q  h  w       q put h         w      print h         w   if   name         main         ctx   mp get context  spawn       q   ctx Queue       p   ctx Process target foo  args  q  hello    world        p start       print q get        p join     Or you just simply replace   pool map harvester text case  case  1    by   pool apply async harvester text case  case  1

User · Answer

You can use the following two functions so as to avoid writing a wrapper for each new function   import itertools from multiprocessing import Pool  def universal worker input pair       function  args   input pair     return function  args   def pool args function   args       return zip itertools repeat function   zip  args     Use the function function with the lists of arguments arg 0  arg 1 and arg 2 as follows   pool   Pool n core  list model   pool map universal worker  pool args function  arg 0  arg 1  arg 2  pool close   pool join

User · Answer

is there a variant of pool map which support multiple arguments    Python 3 3 includes pool starmap   method      usr bin env python3 from functools import partial from itertools import repeat from multiprocessing import Pool  freeze support  def func a  b       return a   b  def main        a args    1 2 3      second arg   1     with Pool   as pool          L   pool starmap func    1  1    2  1    3  1            M   pool starmap func  zip a args  repeat second arg            N   pool map partial func  b second arg   a args          assert L    M    N  if   name       main         freeze support       main     For older versions      usr bin env python2 import itertools from multiprocessing import Pool  freeze support  def func a  b       print a  b  def func star a b          Convert  f  1 2    to  f 1 2   call         return func  a b   def main        pool   Pool       a args    1 2 3      second arg   1     pool map func star  itertools izip a args  itertools repeat second arg     if   name       main         freeze support       main     Output  1 1 2 1 3 1   Notice how itertools izip   and itertools repeat   are used here   Due to the bug mentioned by  unutbu you can t use functools partial   or similar capabilities on Python 2 6  so the simple wrapper function func star   should be defined explicitly  See also the workaround suggested by uptimebox

User · Answer

A better way is using decorator instead of writing wrapper function by hand  Especially when you have a lot of functions to map  decorator will save your time by avoiding  writing wrapper for every function  Usually a decorated function is not picklable  however we may use functools to get around it  More disscusions can be found here   Here the example  def unpack args func       from functools import wraps      wraps func      def wrapper args           if isinstance args  dict               return func   args          else              return func  args      return wrapper   unpack args def func x  y       return x   y   Then you may map it with zipped arguments  np  xlist  ylist   2  range 10   range 10  pool   Pool np  res   pool map func  zip xlist  ylist   pool close   pool join     Of course  you may always use Pool starmap in Python 3    3 3  as mentioned in other answers

User · Answer

Another simple alternative is to wrap your function parameters in a tuple and then wrap the parameters that should be passed in tuples as well  This is perhaps not ideal when dealing with large pieces of data  I believe it would make copies for each tuple   from multiprocessing import Pool  def f  a b c d        print a b c d     return a   b   c  d  if   name         main         p   Pool 10      data     i 0 i 1 i 2 i 3  for i in xrange 10       print p map f  data       p close       p join     Gives the output in some random order   0 1 2 3 1 2 3 4 2 3 4 5 3 4 5 6 4 5 6 7 5 6 7 8 7 8 9 10 6 7 8 9 8 9 10 11 9 10 11 12  6  10  14  18  22  26  30  34  38  42

User · Answer

for python2  you can use this trick  def fun a b       return a b  pool   multiprocessing Pool processes 6  b 233 pool map lambda x fun x b  range 1000

User · Answer

This is an example of the routine I use to pass multiple arguments to a one-argument function used in a pool imap fork   from multiprocessing import Pool    Wrapper of the function to map  class makefun      def   init   self  var2           self var2   var2     def fun self  i           var2   self var2         return var1 i    var2    Couple of variables for the example  var1    1  2  3  5  6  7  8  var2    9  10  11  12     Open the pool  pool   Pool processes 2     Wrapper loop for j in range len var2          Obtain the function to map     pool fun   makefun var2 j   fun        Fork loop     for i  value in enumerate pool imap pool fun  range len var1     0           print var1 i        var2 j        value     Close the pool pool close

User · Answer

Another way is to pass a list of lists to a one-argument routine   import os from multiprocessing import Pool  def task args       print  PID     os getpid       arg1     args 0      arg2     args 1   pool   Pool    pool map task             1 2            3 4            5 6            7 8           One can than construct a list lists of arguments with one s favorite method

User · Answer

This might be another option  The trick is in the wrapper function that returns another function which is passed in to pool map  The code below reads an input array and for each  unique  element in it  returns how many times  ie counts  that element appears in the array  For example if the input is np eye 3       1  0  0                  0  1  0                  0  0  1     then zero appears 6 times and one 3 times import numpy as np from multiprocessing dummy import Pool as ThreadPool from multiprocessing import cpu count   def extract counts label array       labels   np unique label array      out   extract counts helper  label array   labels      return out  def extract counts helper args  labels       n   max 1  cpu count   - 1      pool   ThreadPool n      results          pool map wrapper args  results   labels      pool close       pool join       return results  def wrapper argsin  results       def inner fun label           label array   argsin 0          counts   get label counts label array  label          results label    counts     return inner fun  def get label counts label array  label       return sum label array flatten      label   if   name       quot   main   quot       img   np ones  2 2       out   extract counts img      print  input array   n   img      print  label counts     out      print  quot          quot                   img   np eye 3      out   extract counts img      print  input array   n   img      print  label counts     out      print  quot          quot            img   np random randint 5  size  3  3       out   extract counts img      print  input array   n   img      print  label counts     out      print  quot          quot    You should get  input array      1  1     1  1    label counts    1 0  4           input array      1  0  0     0  1  0     0  0  1    label counts    0 0  6  1 0  3           input array      4 4 0    2 4 3    2 3 1   label counts    0  1  1  1  2  2  3  2  4  3

User · Answer

A better solution for python2   from multiprocessing import Pool def func  i   a  b         print i  a  b     return a   b pool   Pool 3  pool map func    0  1 2     1  2 3     2  3  4       2 3 4  1 2 3  0 1 2  out      3  5  7

User · Answer

I think the below will be better  def multi run wrapper args      return add  args  def add x y       return x y if   name         main         from multiprocessing import Pool     pool   Pool 4      results   pool map multi run wrapper   1 2   2 3   3 4        print results   output   3  5  7

[python] How to use multiprocessing pool.map with multiple arguments?

Examples related to python

Examples related to multiprocessing