The best solution for your problem is to utilize a
Queues and having a separate "queue feeding" functionality is probably overkill.
Here's a slightly rearranged version of your program, this time with only 2 processes coralled in a
Pool. I believe it's the easiest way to go, with minimal changes to original code:
import multiprocessing import time data = ( ['a', '2'], ['b', '4'], ['c', '6'], ['d', '8'], ['e', '1'], ['f', '3'], ['g', '5'], ['h', '7'] ) def mp_worker((inputs, the_time)): print " Processs %s\tWaiting %s seconds" % (inputs, the_time) time.sleep(int(the_time)) print " Process %s\tDONE" % inputs def mp_handler(): p = multiprocessing.Pool(2) p.map(mp_worker, data) if __name__ == '__main__': mp_handler()
mp_worker() function now accepts a single argument (a tuple of the two previous arguments) because the
map() function chunks up your input data into sublists, each sublist given as a single argument to your worker function.
Processs a Waiting 2 seconds Processs b Waiting 4 seconds Process a DONE Processs c Waiting 6 seconds Process b DONE Processs d Waiting 8 seconds Process c DONE Processs e Waiting 1 seconds Process e DONE Processs f Waiting 3 seconds Process d DONE Processs g Waiting 5 seconds Process f DONE Processs h Waiting 7 seconds Process g DONE Process h DONE
Edit as per @Thales comment below:
If you want "a lock for each pool limit" so that your processes run in tandem pairs, ala:
A waiting B waiting | A done , B done | C waiting , D waiting | C done, D done | ...
then change the handler function to launch pools (of 2 processes) for each pair of data:
def mp_handler(): subdata = zip(data[0::2], data[1::2]) for task1, task2 in subdata: p = multiprocessing.Pool(2) p.map(mp_worker, (task1, task2))
Now your output is:
Processs a Waiting 2 seconds Processs b Waiting 4 seconds Process a DONE Process b DONE Processs c Waiting 6 seconds Processs d Waiting 8 seconds Process c DONE Process d DONE Processs e Waiting 1 seconds Processs f Waiting 3 seconds Process e DONE Process f DONE Processs g Waiting 5 seconds Processs h Waiting 7 seconds Process g DONE Process h DONE
~ Answered on 2014-01-02 17:36:18