Forking Multi-Threaded Processes Bash

Question

I would like to make a section of my code more efficient  I m thinking of making it fork off into multiple processes and have them execute 50 100 times at once  instead of just once    For example  pseudo    for line in file  do  foo  foo2  foo3  done   I would like this for loop to run multiple times  I know this can be done with forking  Would it look something like this   while x  lt   50  parent child pid       fork child     child      do     foo  foo2  foo3      done    return child pid       Or am I thinking about this the wrong way   Thanks

User · Answer

With GNU Parallel you can do   cat file   parallel  foo     foo2     foo3       This will run one job on each cpu core  To run 50 do   cat file   parallel -j 50  foo     foo2     foo3       Watch the intro videos to learn more   http   www youtube com playlist list PL284C9FF2488BC6D1

User · Answer

Let me try example  for x in 1 2 3   do   echo a  x   sleep 1   echo b  x      amp   done   sleep 10   And use jobs to see what s running

User · Answer

I don t like using wait because it gets blocked until the process exits  which is not ideal when there are multiple process to wait on as I can t get a status update until the current process is done  I prefer to use a combination of kill -0 and sleep to this   Given an array of pids to wait on  I use the below waitPids   function to get a continuous feedback on what pids are still pending to finish   declare -a pids waitPids         while      pids     -ne 0    do         echo  Waiting for pids    pids              local range   eval echo  0        pids    -1             local i         for i in  range  do             if   kill -0   pids  i   2 gt   dev null  then                 echo  Done --   pids  i                    unset pids  i              fi         done         pids     pids         Expunge nulls created by unset          sleep 1     done     echo  Done       When I start a process in the background  I add its pid immediately to the pids array by using this below utility function   addPid         local desc  1     local pid  2     echo   desc --  pid      pids    pids      pid      Here is a sample that shows how to use   for i in  2  5   do     sleep  i  amp      addPid  Sleep for  i     done waitPids   And here is how the feedback looks   Sleep for 2 -- 36271 Sleep for 3 -- 36272 Sleep for 4 -- 36273 Sleep for 5 -- 36274 Waiting for pids  36271 36272 36273 36274 Waiting for pids  36271 36272 36273 36274 Waiting for pids  36271 36272 36273 36274 Done -- 36271 Waiting for pids  36272 36273 36274 Done -- 36272 Waiting for pids  36273 36274 Done -- 36273 Waiting for pids  36274 Done -- 36274 Done

User · Answer

Here s my thread control function      bin bash   This function just checks jobs in background  don t do more things    if jobs number is lower than MAX  then return to get more jobs    if jobs number is greater or equal to MAX  then wait  until someone finished     Usage      thread max 8     thread max 0      wait  until all jobs completed  thread max         local CHECK INTERVAL  3s      local CUR THREADS      local MAX          1     amp  amp  MAX  1    return 127        reset MAX value  0 is easy to remember        MAX -eq 0    amp  amp            MAX 1         DEBUG  waiting for all tasks finish             while true  do         CUR THREADS  jobs -p   wc -w             workaround about jobs bug  If don t execute it explicitily            CUR THREADS will stick at 1  even no jobs running anymore          jobs  amp  gt  dev null          DEBUG  current thread amount   CUR THREADS          if    CUR THREADS -ge  MAX    then             sleep  CHECK INTERVAL         else             return 0         fi     done

User · Answer

In bash scripts  non-interactive  by default JOB CONTROL is disabled so you can t do the the commands  job  fg  and bg   Here is what works well for me      bin sh  set -m   Enable Job Control  for i in  seq 30   do   start 30 jobs in parallel   sleep 3  amp  done    Wait for all parallel jobs to finish while   1    do fg 2 gt   dev null          1    amp  amp  break  done   The last line uses  fg  to bring a background job into the foreground  It does this in a loop until fg returns 1        1   which it does when there are no longer any more background jobs

User · Answer

Based on what you all shared I was able to put this together      usr bin env bash  VAR1  192 168 1 20 192 168 1 126 192 168 1 36   for a in  VAR1  do   ssh -t -t  a -l Administrator  sudo softwareupdate -l      amp  done  WAITPIDS   WAITPIDS          wait  WAITPIDS echo  Script has finished   Exit 1   This lists all the updates on the mac on three machines at once  Later on I used it to perform a software update for all machines when i CAT my ipaddress txt

User · Answer

I don t know of any explicit fork call in bash  What you probably want to do is append  amp  to a command that you want to run in the background  You can also use  amp  on functions that you define within a bash script   do something with line       line  1   foo   foo2   foo3    for line in file do   do something with line  line  amp  done   EDIT  to put a limit on the number of simultaneous background processes  you could try something like this   for line in file do   while   jobs   wc -l  -ge 50     do     sleep 5   done   do something with line  line  amp  done

User · Answer

haridsv s approach is great  it gives the flexibility to run a processor slots setup where a number of processes can be kept running with new jobs submitting as jobs complete  keeping the overall load up   Here are my mods to haridsv s code for an n-slot processor for a  grid  of ngrid  jobs    I use it for grids of simulation models    Followed by test output for 8 jobs 3 at a time  with running totals of running  submitted  completed and remaining      bin bash                                                                            see haridsv on forking-multi-threaded-processes-bash   loop over grid  submitting jobs in the background    As jobs complete new ones are set going to keep the number running   up to n as much as possible  until it tapers off at the end      8 jobs ngrid 8   3 at a time n 3   running counts running 0 completed 0   previous values prunning 0 pcompleted 0                                                                              process monitoring functions   declare -a pids   function checkPids     echo     pids     if      pids     -ne 0   then     echo  Checking for pids    pids          local range   eval echo  0        pids    -1         local i     for i in  range  do         if   kill -0   pids  i   2 gt   dev null  then             echo  Done --   pids  i                unset pids  i              completed   expr  completed   1          fi     done     pids     pids         Expunge nulls created by unset      running       pids           echo   PIDS    running fi     function addPid         desc  1     pid  2     echo     desc  -   pid     pids    pids      pid                                                                                 Loop and report when job changes happen    keep going until all are completed    idx 0 while    completed -lt   ngrid    do       if    running -lt  n    amp  amp     idx -lt   ngrid        then                                                                          submit a new process if less than n   are running and we haven t finished        get desc for process           name  job    idx    background execution         sleep 3  amp          addPid  name            idx   expr  idx   1                                                                               fi       checkPids   if something changes        if     running  -gt   prunning                    completed  -gt   pcompleted        then         remain   expr  ngrid -  completed          echo    Running     running   Submitted     idx                   Completed    completed  Remaining    remain     fi   save counts to prev values     prunning   running      pcompleted   completed        sleep 1   done                                                                              Test output    job 0 - 75257 1 Checking for pids  75257  PIDS  1  Running  1 Submitted  1  Completed  0 Remaining  8  job 1 - 75262 2 Checking for pids  75257 75262  PIDS  2  Running  2 Submitted  2  Completed  0 Remaining  8  job 2 - 75267 3 Checking for pids  75257 75262 75267  PIDS  3  Running  3 Submitted  3  Completed  0 Remaining  8 3 Checking for pids  75257 75262 75267 Done -- 75257  PIDS  2  Running  2 Submitted  3  Completed  1 Remaining  7  job 3 - 75277 3 Checking for pids  75262 75267 75277 Done -- 75262  PIDS  2  Running  2 Submitted  4  Completed  2 Remaining  6  job 4 - 75283 3 Checking for pids  75267 75277 75283 Done -- 75267  PIDS  2  Running  2 Submitted  5  Completed  3 Remaining  5  job 5 - 75289 3 Checking for pids  75277 75283 75289  PIDS  3  Running  3 Submitted  6  Completed  3 Remaining  5 3 Checking for pids  75277 75283 75289 Done -- 75277  PIDS  2  Running  2 Submitted  6  Completed  4 Remaining  4  job 6 - 75298 3 Checking for pids  75283 75289 75298 Done -- 75283  PIDS  2  Running  2 Submitted  7  Completed  5 Remaining  3  job 7 - 75304 3 Checking for pids  75289 75298 75304 Done -- 75289  PIDS  2  Running  2 Submitted  8  Completed  6 Remaining  2 2 Checking for pids  75298 75304  PIDS  2 2 Checking for pids  75298 75304 Done -- 75298  PIDS  1  Running  1 Submitted  8  Completed  7 Remaining  1 1 Checking for pids  75304 Done -- 75304  PIDS  0  Running  0 Submitted  8  Completed  8 Remaining  0

[bash] Forking / Multi-Threaded Processes | Bash

Examples related to bash

Examples related to shell

Examples related to fork