python multithreading wait till all threads finished

Question

This may have been asked in a similar context but I was unable to find an answer after about 20 minutes of searching  so I will ask   I have written a Python script  lets say  scriptA py  and a script  lets say scriptB py   In scriptB I want to call scriptA multiple times with different arguments  each time takes about an hour to run   its a huge script  does lots of stuff   don t worry about it  and I want to be able to run the scriptA with all the different arguments simultaneously  but I need to wait till ALL of them are done before continuing  my code   import subprocess   setup do setup     run scriptA subprocess call scriptA   argumentsA  subprocess call scriptA   argumentsB  subprocess call scriptA   argumentsC    finish do finish     I want to do run all the subprocess call   at the same time  and then wait till they are all done  how should I do this    I tried to use threading like the example here   from threading import Thread import subprocess  def call script args      subprocess call args    run scriptA    t1   Thread target call script  args  scriptA   argumentsA   t2   Thread target call script  args  scriptA   argumentsB   t3   Thread target call script  args  scriptA   argumentsC   t1 start   t2 start   t3 start     But I do not think this is right    How do I know they have all finished running before going to my do finish

User · Answer

I just came across the same problem where I needed to wait for all the threads which were created using the for loop I just tried out the following piece of code It may not be the perfect solution but I thought it would be a simple solution to test   for t in threading enumerate        try          t join       except RuntimeError as err          if  cannot join current thread  in err              continue         else              raise

User · Answer

Maybe  something like    for t in threading enumerate        if t daemon          t join

User · Answer

From the threading module documentation     There is a    main thread    object  this corresponds to the initial   thread of control in the Python program  It is not a daemon thread       There is the possibility that    dummy thread objects    are created    These are thread objects corresponding to    alien threads     which are   threads of control started outside the threading module  such as   directly from C code  Dummy thread objects have limited functionality    they are always considered alive and daemonic  and cannot be join  ed    They are never deleted  since it is impossible to detect the   termination of alien threads    So  to catch those two cases when you are not interested in keeping a list of the threads you create   import threading as thrd   def alter data data  index       data index     2   data    0  2  6  20   for i  value in enumerate data       thrd Thread target alter data  args  data  i   start    for thread in thrd enumerate        if thread daemon          continue     try          thread join       except RuntimeError as err          if  cannot join current thread  in err args 0                 catchs main thread             continue         else              raise   Whereupon    gt  gt  gt  print data   0  4  12  40

User · Answer

Put the threads in a list and then use the Join method   threads        t   Thread       threads append t       repeat as often as necessary        Start all threads  for x in threads       x start       Wait for all of them to finish  for x in threads       x join

User · Answer

In Python3  since Python 3 2 there is a new approach to reach the same result  that I personally prefer to the traditional thread creation start join  package concurrent futures  https   docs python org 3 library concurrent futures html  Using a ThreadPoolExecutor the code would be   from concurrent futures thread import ThreadPoolExecutor import time  def call script ordinal  arg       print  Thread   ordinal   argument    arg      time sleep 2      print  Thread   ordinal   Finished    args     argumentsA    argumentsB    argumentsC    with ThreadPoolExecutor max workers 2  as executor      ordinal   1     for arg in args          executor submit call script  ordinal  arg          ordinal    1 print  All tasks has been finished     The output of the previous code is something like   Thread 1 argument  argumentsA Thread 2 argument  argumentsB Thread 1 Finished Thread 2 Finished Thread 3 argument  argumentsC Thread 3 Finished All tasks has been finished   One of the advantages is that you can control the throughput setting the max concurrent workers

User · Answer

I prefer using list comprehension based on an input list   inputs    scriptA   argumentsA  scriptA   argumentsB       threads    Thread target call script  args  i   for i in inputs   t start   for t in threads   t join   for t in threads

User · Answer

You need to use join method of Thread object in the end of the script   t1   Thread target call script  args  scriptA   argumentsA   t2   Thread target call script  args  scriptA   argumentsB   t3   Thread target call script  args  scriptA   argumentsC    t1 start   t2 start   t3 start    t1 join   t2 join   t3 join     Thus the main thread will wait till t1  t2 and t3 finish execution

User · Answer

You can have class something like below from which you can add  n  number of functions or console scripts you want to execute in parallel passion and start the execution and wait for all jobs to complete        from multiprocessing import Process  class ProcessParallel object               To Process the  functions parallely                  def   init   self   jobs                                   self jobs   jobs         self processes           def fork processes self                       Creates the process objects for given function deligates                     for job in self jobs              proc    Process target job              self processes append proc       def start all self                       Starts the functions process all together                      for proc in self processes              proc start        def join all self                       Waits untill all the functions executed                      for proc in self processes              proc join     def two sum a 2  b 2       return a   b  def multiply a 2  b 2       return a   b    How to run  if   name         main          note  two sum  multiply can be replace with any python console scripts which      you wanted to run parallel       procs    ProcessParallel two sum  multiply       Add all the process in list     procs fork processes        starts  process execution      procs start all        wait until all the process got executed     procs join all

[python] python multithreading wait till all threads finished

Examples related to python

Examples related to multithreading