How can I recover the return value of a function passed to multiprocessing Process

Question

In the example code below  I d like to recover the return value of the function worker   How can I go about doing this   Where is this value stored   Example Code   import multiprocessing  def worker procnum          worker function        print str procnum      represent       return procnum   if   name         main         jobs          for i in range 5           p   multiprocessing Process target worker  args  i            jobs append p          p start        for proc in jobs          proc join       print jobs   Output   0 represent  1 represent  2 represent  3 represent  4 represent    lt Process Process-1  stopped  gt    lt Process Process-2  stopped  gt    lt Process Process-3  stopped  gt    lt Process Process-4  stopped  gt    lt Process Process-5  stopped  gt     I can t seem to find the relevant attribute in the objects stored in jobs

User · Answer

This example shows how to use a list of multiprocessing.Pipe instances to return strings from an arbitrary number of processes:

import multiprocessing

def worker(procnum, send_end):
    '''worker function'''
    result = str(procnum) + ' represent!'
    print result
    send_end.send(result)

def main():
    jobs = []
    pipe_list = []
    for i in range(5):
        recv_end, send_end = multiprocessing.Pipe(False)
        p = multiprocessing.Process(target=worker, args=(i, send_end))
        jobs.append(p)
        pipe_list.append(recv_end)
        p.start()

    for proc in jobs:
        proc.join()
    result_list = [x.recv() for x in pipe_list]
    print result_list

if __name__ == '__main__':
    main()

Output:

0 represent!
1 represent!
2 represent!
3 represent!
4 represent!
['0 represent!', '1 represent!', '2 represent!', '3 represent!', '4 represent!']

This solution uses fewer resources than a multiprocessing.Queue which uses

a Pipe
at least one Lock
a buffer
a thread

or a multiprocessing.SimpleQueue which uses

a Pipe
at least one Lock

It is very instructive to look at the source for each of these types.

User · Answer

For anyone else who is seeking how to get a value from a Process using Queue  import multiprocessing  ret     foo   False   def worker queue       ret   queue get       ret  foo     True     queue put ret   if   name         main         queue   multiprocessing Queue       queue put ret      p   multiprocessing Process target worker  args  queue        p start       p join       print queue get       Prints   quot foo quot   True   Note that in Windows or Jupyter Notebook  with multithreading you have to save this as a file and execute the file  If you do it in a command prompt you will see an error like this   AttributeError  Can t get attribute  worker  on  lt module    main     built-in  gt

User · Answer

The pebble package has a nice abstraction leveraging multiprocessing Pipe which makes this quite straightforward   from pebble import concurrent   concurrent process def function arg  kwarg 0       return arg   kwarg  future   function 1  kwarg 1   print future result      Example from  https   pythonhosted org Pebble  concurrent-decorators

User · Answer

I think the approach suggested by  sega sai is the better one  But it really needs a code example  so here goes   import multiprocessing from os import getpid  def worker procnum       print  I am number  d in process  d     procnum  getpid         return getpid    if   name         main         pool   multiprocessing Pool processes   3      print pool map worker  range 5      Which will print the return values   I am number 0 in process 19139 I am number 1 in process 19138 I am number 2 in process 19140 I am number 3 in process 19139 I am number 4 in process 19140  19139  19138  19140  19139  19140    If you are familiar with map  the Python 2 built-in  this should not be too challenging  Otherwise have a look at sega Sai s link   Note how little code is needed   Also note how processes are re-used

User · Answer

Use shared variable to communicate  For example like this  import multiprocessing   def worker procnum  return dict        quot  quot  quot worker function quot  quot  quot      print str procnum     quot  represent  quot       return dict procnum    procnum   if   name       quot   main   quot       manager   multiprocessing Manager       return dict   manager dict       jobs          for i in range 5           p   multiprocessing Process target worker  args  i  return dict           jobs append p          p start        for proc in jobs          proc join       print return dict values

User · Answer

For some reason  I couldn t find a general example of how to do this with Queue anywhere  even Python s doc examples don t spawn multiple processes   so here s what I got working after like 10 tries   def add helper queue  arg1  arg2     the func called in child processes     ret   arg1   arg2     queue put ret   def multi add      spawns child processes     q   Queue       processes          rets          for   in range 0  100           p   Process target add helper  args  q  1  2           processes append p          p start       for p in processes          ret   q get     will block         rets append ret      for p in processes          p join       return rets   Queue is a blocking  thread-safe queue that you can use to store the return values from the child processes  So you have to pass the queue to each process  Something less obvious here is that you have to get   from the queue before you join the Processes or else the queue fills up and blocks everything   Update for those who are object-oriented  tested in Python 3 4    from multiprocessing import Process  Queue  class Multiprocessor         def   init   self           self processes              self queue   Queue         staticmethod     def  wrapper func  queue  args  kwargs           ret   func  args    kwargs          queue put ret       def run self  func   args    kwargs           args2    func  self queue  args  kwargs          p   Process target self  wrapper  args args2          self processes append p          p start        def wait self           rets              for p in self processes              ret   self queue get               rets append ret          for p in self processes              p join           return rets    tester if   name         main         mp   Multiprocessor       num proc   64     for   in range num proc     queue up multiple tasks running  sum          mp run sum   1  2  3  4  5       ret   mp wait     get all results     print ret      assert len ret     num proc and all r    15 for r in ret

User · Answer

If you are using Python 3  you can use concurrent futures ProcessPoolExecutor as a convenient abstraction   from concurrent futures import ProcessPoolExecutor  def worker procnum          worker function        print str procnum      represent        return procnum   if   name         main         with ProcessPoolExecutor   as executor          print list executor map worker  range 5       Output   0 represent  1 represent  2 represent  3 represent  4 represent   0  1  2  3  4

User · Answer

I modified vartec s answer a bit since I needed to get the error codes from the function    Thanks vertec    its an awesome trick   This can also be done with a manager list but I think is better to have it in a dict and store a list within it   That way  way we keep the function and the results since we can t be sure of the order in which the list will be populated   from multiprocessing import Process import time import datetime import multiprocessing   def func1 fn  m list       print  func1  starting      time sleep 1      m list fn     this is the first function      print  func1  finishing        return  func1     no need for return since Multiprocess doesnt return it     def func2 fn  m list       print  func2  starting      time sleep 3      m list fn     this is function 2      print  func2  finishing        return  func2   def func3 fn  m list       print  func3  starting      time sleep 9        if fail wont join the rest because it never populate the dict       or do a try except to get something in return      raise ValueError  failed here         if we want to get the error in the manager dict we can catch the error     try          raise ValueError  failed here           m list fn     this is third      except          m list fn     this is third and it fail horrible            print  func3  finishing            return  func3    def runInParallel  fns        is to accept any input in list     start time   datetime datetime now       proc          manager   multiprocessing Manager       m list   manager dict       for fn in fns            print fn           print dir fn          p   Process target fn  name fn func name  args  fn  m list           p start           proc append p      for p in proc          p join      5 is the time out      print datetime datetime now   - start time     return m list  proc  if   name         main         manager  proc   runInParallel func1  func2  func3        print dir proc 0         print proc 0   name       print proc 0  name       print proc 0  exitcode        here you can check what did fail     for i in proc          print i name  i exitcode    name was set up in the Process line 53        here will only show the function that worked and where able to populate the        manager dict     for i  j in manager items            print dir i     things you can do to the function         print i  j

User · Answer

Thought I d simplify the simplest examples copied from above  working for me on Py3 6  Simplest is multiprocessing Pool   import multiprocessing import time  def worker x       time sleep 1      return x  pool   multiprocessing Pool   print pool map worker  range 10      You can set the number of processes in the pool with  e g   Pool processes 5   However it defaults to CPU count  so leave it blank for CPU-bound tasks   I O-bound tasks often suit threads anyway  as the threads are mostly waiting so can share a CPU core   Pool also applies chunking optimization    Note that the worker method cannot be nested within a method  I initially defined my worker method inside the method that makes the call to pool map  to keep it all self-contained  but then the processes couldn t import it  and threw  AttributeError  Can t pickle local object outer method  inner method   More here  It can be inside a class     Appreciate the original question specified printing  represent   rather than time sleep    but without it I thought some code was running concurrently when it wasn t      Py3 s ProcessPoolExecutor is also two lines   map returns a generator so you need the list      from concurrent futures import ProcessPoolExecutor with ProcessPoolExecutor   as executor      print list executor map worker  range 10         With plain Processes   import multiprocessing import time  def worker x  queue       time sleep 1      queue put x   queue   multiprocessing SimpleQueue   tasks   range 10   for task in tasks      multiprocessing Process target worker  args  task  queue    start    for   in tasks      print queue get      Use SimpleQueue if all you need is put and get  The first loop starts all the processes  before the second makes the blocking queue get calls  I don t think there s any reason to call p join   too

User · Answer

You can use the exit built-in to set the exit code of a process  It can be obtained from the exitcode attribute of the process   import multiprocessing  def worker procnum       print str procnum      represent       exit procnum   if   name         main         jobs          for i in range 5           p   multiprocessing Process target worker  args  i            jobs append p          p start        result          for proc in jobs          proc join           result append proc exitcode      print result   Output   0 represent  1 represent  2 represent  3 represent  4 represent   0  1  2  3  4

User · Answer

It seems that you should use the multiprocessing Pool class instead and use the methods  apply    apply async    map    http   docs python org library multiprocessing html highlight pool multiprocessing pool AsyncResult

User · Answer

A simple solution   import multiprocessing  output    data   range 0 10   def f x       return x  2  def handler        p   multiprocessing Pool 64      r p map f  data      return r  if   name         main         output append handler     print output 0     Output    0  1  4  9  16  25  36  49  64  81

[python] How can I recover the return value of a function passed to multiprocessing.Process?

Examples related to python

Examples related to python-multiprocessing