Running multiple commands with xargs

Question

cat a txt   xargs -I   echo    In the example above  xargs takes echo   as the command argument  But in some cases  I need multiple commands to process the argument instead of one  For example  cat a txt   xargs -I    command1  command2         But xargs doesn t accept this form  One solution I know is that I can define a function to wrap the commands  but I want to avoid that because it is complex  Is there a better solution

User · Answer

Try this  git config --global alias all   f     find   -d -name  quot  git quot    sed s      git  g   xargs -P10 -I   git --git-dir     git --work-tree     1     f   It runs ten threads in parallel and does what ever git command you want to all repos in the folder structure  No matter if the repo is one or n levels deep  E g  git all pull

User · Answer

I prefer style which allows dry run mode  without   sh     cat a txt   xargs -I   echo  command1  command2          sh   Works with pipes too   cat a txt   xargs -I   echo  echo     cat     sh

User · Answer

cat a txt   xargs -d    n  sh -c  for arg do command1   arg   command2   arg        done         or  without a Useless Use Of cat    lt a txt xargs -d    n  sh -c  for arg do command1   arg   command2   arg        done        To explain some of the finer points    The use of   arg  instead of    and the absence of -I in the xargs command line  is for security reasons  Passing data on sh s command-line argument list instead of substituting it into code prevents content that data might contain  such as   rm -rf     to take a particularly malicious example  from being executed as code  Similarly  the use of -d    n  is a GNU extension which causes xargs to treat each line of the input file as a separate data item  Either this or -0  which expects NULs instead of newlines  is necessary to prevent xargs from trying to apply shell-like  but not quite shell-compatible  parsing to the stream it reads   If you don t have GNU xargs  you can use tr   n    0   lt a txt   xargs -0     to get line-oriented reading without -d   The   is a placeholder for  0  such that other data values added by xargs become  1 and onward  which happens to be the default set of values a for loop iterates over

User · Answer

With GNU Parallel you can do  cat a txt   parallel  command1     command2             Watch the intro videos to learn more  https   www youtube com playlist list PL284C9FF2488BC6D1 For security reasons it is recommended you use your package manager to install  But if you cannot do that then you can use this 10 seconds installation  The 10 seconds installation will try to do a full installation  if that fails  a personal installation  if that fails  a minimal installation     wget -O - pi dk 3    lynx -source pi dk 3    curl pi dk 3          fetch -o - http   pi dk 3    gt  install sh   sha1sum install sh   grep 67bd7bc7dc20aff99eb8f1266574dadb 12345678 67bd7bc7 dc20aff9 9eb8f126 6574dadb   md5sum install sh   grep b7a15cdbb07fb6e11b0338577bc1780f b7a15cdb b07fb6e1 1b033857 7bc1780f   sha512sum install sh   grep 186000b62b66969d7506ca4f885e0c80e02a22444 6f25960b d4b90cf6 ba5b76de c1acdf39 f3d24249 72930394 a4164351 93a7668d 21ff9839 6f920be5 186000b6 2b66969d 7506ca4f 885e0c80 e02a2244 40e8a43f   bash install sh

User · Answer

This is just another approach without xargs nor cat   while read stuff  do   command1   stuff    command2   stuff        done  lt  a txt

User · Answer

This seems to be the safest version   tr    n      0    lt  a txt   xargs -r0  bin bash -c  command1       command2              -0 can be removed and the tr replaced with a redirect  or the file can be replaced with a null separated file instead   It is mainly in there since I mainly use xargs with find with -print0 output   This might also be relevant on xargs versions without the -0 extension   It is safe  since args will pass the parameters to the shell as an array when executing it  The shell  at least bash  would then pass them as an unaltered array to the other processes when all are obtained using        1   If you use      xargs -r0 -I   bash -c  f       command   f        the assignment will fail if the string contains double quotes  This is true for every variant using -i or -I   Due to it being replaced into a string  you can always inject commands by inserting unexpected characters  like quotes  backticks or dollar signs  into the input data   If the commands can only take one parameter at a time   tr    n      0    lt  a txt   xargs -r0 -n1  bin bash -c  command1       command2             Or with somewhat less processes   tr    n      0    lt  a txt   xargs -r0  bin bash -c  for f in       do command1   f   command2   f   done        If you have GNU xargs or another with the -P extension and you want to run 32 processes in parallel  each with not more than 10 parameters for each command   tr    n      0    lt  a txt   xargs -r0 -n10 -P32  bin bash -c  command1       command2             This should be robust against any special characters in the input   If the input is null separated   The tr version will get some invalid input if some of the lines contain newlines  but that is unavoidable with a newline separated file   The blank first parameter for bash -c is due to this   From the bash man page   Thanks  clacke   -c   If the -c option is present  then  commands  are  read  from  the  first  non-option  argument  com-      mand string    If there are arguments after the command string  the first argument is assigned to  0      and any remaining arguments are assigned to the positional parameters   The assignment  to   0  sets      the name of the shell  which is used in warning and error messages

User · Answer

My current BKM for this is        xargs -n1 -I   perl -e  system  echo 1      system  echo 2         It is unfortunate that this uses perl  which is less likely to be installed than bash  but it handles more input that the accepted answer     I welcome a ubiquitous version that does not rely on perl     KeithThompson s suggestion of          xargs -I   sh -c  command1  command2         is great - unless you have the shell comment character   in your input  in which case part of the first command and all of the second command will be truncated   Hashes   can be quite common  if the input is derived from a filesystem listing  such as ls or find  and your editor creates temporary files with   in their name   Example of the problem     bash 1366   gt    bin ls   cat  Makefile   README  Makefile README   Oops  here is the problem     bash 1367   gt   ls   xargs -n1 -I   sh -i -c  echo 1    echo 2    1 1 1 1 Makefile 2 Makefile 1 README 2 README   Ahh  that s better     bash 1368   gt   ls   xargs -n1 -I   perl -e  system  echo 1      system  echo 2       1  Makefile  2  Makefile  1  README  2  README  1 Makefile 2 Makefile 1 README 2 README   bash 1369   gt

User · Answer

You can use  cat file txt   xargs -i  sh -c  command      command2     amp  amp  command3            variable for each line on the text file

User · Answer

Another possible solution that works for me is something like -  cat a txt   xargs bash -c  command1     command2     bash   Note the  bash  at the end - I assume it is passed as argv 0  to bash  Without it in this syntax the first parameter to each command is lost  It may be any word   Example   cat a txt   xargs -n 5 bash -c  echo -n  date   Y m d- H M S     echo   data        echo  data again        bash

User · Answer

One thing I do is to add to  bashrc  profile this function   function each         while read line  do         for f in       do              f  line         done     done     then you can do things like        each command1 command2  command3 has spaces    which is less verbose than xargs or -exec  You could also modify the function to insert the value from the read at an arbitrary location in the commands to each  if you needed that behavior also

[bash] Running multiple commands with xargs

Examples related to bash

Examples related to xargs