Multiprocessing a for loop

Question

I have an array  called data inputs  containing the names of hundreds of astronomy images files  These images are then manipulated  My code works and takes a few seconds to process each image  However  it can only do one image at a time because I m running the array through a for loop   for name in data inputs      sci fits open name   fits        image is manipulated   There is no reason why I have to modify an image before any other  so is it possible to utilise all 4 cores on my machine with each core running through the for loop on a different image   I ve read about the multiprocessing module but I m unsure how to implement it in my case  I m keen to get multiprocessing to work because eventually I ll have to run this on 10 000  images

User · Accepted Answer

You can simply use multiprocessing Pool   from multiprocessing import Pool  def process image name       sci fits open     fits  format name        lt process gt   if   name         main         pool   Pool                             Create a multiprocessing Pool     pool map process image  data inputs     process data inputs iterable with pool

User · Answer

Alternatively  with Pool   as pool       pool map fits open   name     fits  for name in datainput

User · Answer

You can use multiprocessing Pool   from multiprocessing import Pool class Engine object       def   init   self  parameters           self parameters   parameters     def   call   self  filename           sci   fits open filename     fits           manipulated   manipulate image sci  self parameters          return manipulated  try      pool   Pool 8    on 8 processors     engine   Engine my parameters      data outputs   pool map engine  data inputs  finally    To make sure processes are closed in the end  even if errors happen     pool close       pool join

[python] Multiprocessing a for loop?

Examples related to python

Examples related to multiprocessing