[bash] Running multiple commands with xargs

cat a.txt | xargs -I % echo %

In the example above, xargs takes echo % as the command argument. But in some cases, I need multiple commands to process the argument instead of one. For example:

cat a.txt | xargs -I % {command1; command2; ... }

But xargs doesn't accept this form. One solution I know is that I can define a function to wrap the commands, but I want to avoid that because it is complex. Is there a better solution?

This is just another approach without xargs nor cat:

while read stuff; do
  command1 "$stuff"
  command2 "$stuff"
  ...
done < a.txt

Another possible solution that works for me is something like -

cat a.txt | xargs bash -c 'command1 $@; command2 $@' bash

Note the 'bash' at the end - I assume it is passed as argv[0] to bash. Without it in this syntax the first parameter to each command is lost. It may be any word.

Example:

cat a.txt | xargs -n 5 bash -c 'echo -n `date +%Y%m%d-%H%M%S:` ; echo " data: " $@; echo "data again: " $@' bash

You can use

cat file.txt | xargs -i  sh -c 'command {} | command2 {} && command3 {}'

{} = variable for each line on the text file

My current BKM for this is

... | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'

It is unfortunate that this uses perl, which is less likely to be installed than bash; but it handles more input that the accepted answer. (I welcome a ubiquitous version that does not rely on perl.)

@KeithThompson's suggestion of

 ... | xargs -I % sh -c 'command1; command2; ...'

is great - unless you have the shell comment character # in your input, in which case part of the first command and all of the second command will be truncated.

Hashes # can be quite common, if the input is derived from a filesystem listing, such as ls or find, and your editor creates temporary files with # in their name.

Example of the problem:

$ bash 1366 $>  /bin/ls | cat
#Makefile#
#README#
Makefile
README

Oops, here is the problem:

$ bash 1367 $>  ls | xargs -n1 -I % sh -i -c 'echo 1 %; echo 2 %'
1
1
1
1 Makefile
2 Makefile
1 README
2 README

Ahh, that's better:

$ bash 1368 $>  ls | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'
1 #Makefile#
2 #Makefile#
1 #README#
2 #README#
1 Makefile
2 Makefile
1 README
2 README
$ bash 1369 $>  

One thing I do is to add to .bashrc/.profile this function:

function each() {
    while read line; do
        for f in "$@"; do
            $f $line
        done
    done
}

then you can do things like

... | each command1 command2 "command3 has spaces"

which is less verbose than xargs or -exec. You could also modify the function to insert the value from the read at an arbitrary location in the commands to each, if you needed that behavior also.

With GNU Parallel you can do:

cat a.txt | parallel 'command1 {}; command2 {}; ...; '

Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

For security reasons it is recommended you use your package manager to install. But if you cannot do that then you can use this 10 seconds installation.

The 10 seconds installation will try to do a full installation; if that fails, a personal installation; if that fails, a minimal installation.

$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
   fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 67bd7bc7dc20aff99eb8f1266574dadb
12345678 67bd7bc7 dc20aff9 9eb8f126 6574dadb
$ md5sum install.sh | grep b7a15cdbb07fb6e11b0338577bc1780f
b7a15cdb b07fb6e1 1b033857 7bc1780f
$ sha512sum install.sh | grep 186000b62b66969d7506ca4f885e0c80e02a22444
6f25960b d4b90cf6 ba5b76de c1acdf39 f3d24249 72930394 a4164351 93a7668d
21ff9839 6f920be5 186000b6 2b66969d 7506ca4f 885e0c80 e02a2244 40e8a43f
$ bash install.sh

I prefer style which allows dry run mode (without | sh) :

cat a.txt | xargs -I % echo "command1; command2; ... " | sh

Works with pipes too:

cat a.txt | xargs -I % echo "echo % | cat " | sh

This seems to be the safest version.

tr '[\n]' '[\0]' < a.txt | xargs -r0 /bin/bash -c 'command1 "$@"; command2 "$@";' ''

(-0 can be removed and the tr replaced with a redirect (or the file can be replaced with a null separated file instead). It is mainly in there since I mainly use xargs with find with -print0 output) (This might also be relevant on xargs versions without the -0 extension)

It is safe, since args will pass the parameters to the shell as an array when executing it. The shell (at least bash) would then pass them as an unaltered array to the other processes when all are obtained using ["$@"][1]

If you use ...| xargs -r0 -I{} bash -c 'f="{}"; command "$f";' '', the assignment will fail if the string contains double quotes. This is true for every variant using -i or -I. (Due to it being replaced into a string, you can always inject commands by inserting unexpected characters (like quotes, backticks or dollar signs) into the input data)

If the commands can only take one parameter at a time:

tr '[\n]' '[\0]' < a.txt | xargs -r0 -n1 /bin/bash -c 'command1 "$@"; command2 "$@";' ''

Or with somewhat less processes:

tr '[\n]' '[\0]' < a.txt | xargs -r0 /bin/bash -c 'for f in "$@"; do command1 "$f"; command2 "$f"; done;' ''

If you have GNU xargs or another with the -P extension and you want to run 32 processes in parallel, each with not more than 10 parameters for each command:

tr '[\n]' '[\0]' < a.txt | xargs -r0 -n10 -P32 /bin/bash -c 'command1 "$@"; command2 "$@";' ''

This should be robust against any special characters in the input. (If the input is null separated.) The tr version will get some invalid input if some of the lines contain newlines, but that is unavoidable with a newline separated file.

The blank first parameter for bash -c is due to this: (From the bash man page) (Thanks @clacke)

-c   If the -c option is present, then  commands  are  read  from  the  first  non-option  argument  com-
     mand_string.   If there are arguments after the command_string, the first argument is assigned to $0
     and any remaining arguments are assigned to the positional parameters.  The assignment  to  $0  sets
     the name of the shell, which is used in warning and error messages.

Try this:

git config --global alias.all '!f() { find . -d -name ".git" | sed s/\\/\.git//g | xargs -P10 -I{} git --git-dir={}/.git --work-tree={} $1; }; f'

It runs ten threads in parallel and does what ever git command you want to all repos in the folder structure. No matter if the repo is one or n levels deep.

E.g: git all pull