Bash script processing limited number of commands in parallel

Question

I have a bash script that looks like this      bin bash wget LINK1  gt  dev null 2 gt  amp 1 wget LINK2  gt  dev null 2 gt  amp 1 wget LINK3  gt  dev null 2 gt  amp 1 wget LINK4  gt  dev null 2 gt  amp 1           wget LINK4000  gt  dev null 2 gt  amp 1   But processing each line until the command is finished then moving to the next one is very time consuming  I want to process for instance 20 lines at once then when they re finished another 20 lines are processed   I thought of wget LINK1  gt  dev null 2 gt  amp 1  amp  to send the command to the background and carry on  but there are 4000 lines here this means I will have performance issues  not to mention being limited in how many processes I should start at the same time so this is not a good idea   One solution that I m thinking of right now is checking whether one of the commands is still running or not  for instance after 20 lines I can add this loop   while      ps -ef   grep KEYWORD   grep -v grep   wc -l  -gt 0    do sleep 1 done   Of course in this case I will need to append  amp  to the end of the line  But I m feeling this is not the right way to do it   So how do I actually group each 20 lines together and wait for them to finish before going to the next 20 lines  this script is dynamically generated so I can do whatever math I want on it while it s being generated  but it DOES NOT have to use wget  it was just an example so any solution that is wget specific is not gonna do me any good

User · Answer

See parallel  Its syntax is similar to xargs  but it runs the commands in parallel

User · Answer

Use the wait built-in   process1  amp  process2  amp  process3  amp  process4  amp  wait process5  amp  process6  amp  process7  amp  process8  amp  wait   For the above example  4 processes process1     process4 would be started in the background  and the shell would wait until those are completed before starting the next set   From the GNU manual    wait  jobspec or pid            Wait until the child process specified by each process ID pid or job specification jobspec exits and return the exit status of the last   command waited for  If a job spec is given  all processes in the job   are waited for  If no arguments are given  all currently active child   processes are waited for  and the return status is zero  If neither   jobspec nor pid specifies an active child process of the shell  the   return status is 127

User · Answer

In fact  xargs can run commands in parallel for you  There is a special -P max procs command-line option for that  See man xargs

User · Answer

You can run 20 processes and use the command   wait   Your script will wait and continue when all your background jobs are finished

[linux] Bash script processing limited number of commands in parallel

Examples related to linux

Examples related to bash

Examples related to shell