[bash] How can I use inverse or negative wildcards when pattern matching in a unix/linux shell?

Say I want to copy the contents of a directory excluding files and folders whose names contain the word 'Music'.

cp [exclude-matches] *Music* /target_directory

What should go in place of [exclude-matches] to accomplish this?

Not in bash (that I know of), but:

cp `ls | grep -v Music` /target_directory

I know this is not exactly what you were looking for, but it will solve your example.

In bash, an alternative to shopt -s extglob is the GLOBIGNORE variable. It's not really better, but I find it easier to remember.

An example that may be what the original poster wanted:

GLOBIGNORE="*techno*"; cp *Music* /only_good_music/

When done, unset GLOBIGNORE to be able to rm *techno* in the source directory.

You can also use a pretty simple for loop:

for f in `find . -not -name "*Music*"`
do
    cp $f /target/dir
done

The extglob shell option gives you more powerful pattern matching in the command line.

You turn it on with shopt -s extglob, and turn it off with shopt -u extglob.

In your example, you would initially do:

$ shopt -s extglob
$ cp !(*Music*) /target_directory

The full available extended globbing operators are (excerpt from man bash):

If the extglob shell option is enabled using the shopt builtin, several extended pattern matching operators are recognized.A pattern-list is a list of one or more patterns separated by a |. Composite patterns may be formed using one or more of the following sub-patterns:

• ?(pattern-list)
  Matches zero or one occurrence of the given patterns
• *(pattern-list)
  Matches zero or more occurrences of the given patterns
• +(pattern-list)
  Matches one or more occurrences of the given patterns
• @(pattern-list)
  Matches one of the given patterns
• !(pattern-list)
  Matches anything except one of the given patterns

So, for example, if you wanted to list all the files in the current directory that are not .c or .h files, you would do:

$ ls -d !(*@(.c|.h))

Of course, normal shell globing works, so the last example could also be written as:

$ ls -d !(*.[ch])

Not in bash (that I know of), but:

cp `ls | grep -v Music` /target_directory

I know this is not exactly what you were looking for, but it will solve your example.

The extglob shell option gives you more powerful pattern matching in the command line.

You turn it on with shopt -s extglob, and turn it off with shopt -u extglob.

In your example, you would initially do:

$ shopt -s extglob
$ cp !(*Music*) /target_directory

The full available extended globbing operators are (excerpt from man bash):

If the extglob shell option is enabled using the shopt builtin, several extended pattern matching operators are recognized.A pattern-list is a list of one or more patterns separated by a |. Composite patterns may be formed using one or more of the following sub-patterns:

• ?(pattern-list)
  Matches zero or one occurrence of the given patterns
• *(pattern-list)
  Matches zero or more occurrences of the given patterns
• +(pattern-list)
  Matches one or more occurrences of the given patterns
• @(pattern-list)
  Matches one of the given patterns
• !(pattern-list)
  Matches anything except one of the given patterns

So, for example, if you wanted to list all the files in the current directory that are not .c or .h files, you would do:

$ ls -d !(*@(.c|.h))

Of course, normal shell globing works, so the last example could also be written as:

$ ls -d !(*.[ch])

If you want to avoid the mem cost of using the exec command, I believe you can do better with xargs. I think the following is a more efficient alternative to

find foo -type f ! -name '*Music*' -exec cp {} bar \; # new proc for each exec



find . -maxdepth 1 -name '*Music*' -prune -o -print0 | xargs -0 -i cp {} dest/

If you want to avoid the mem cost of using the exec command, I believe you can do better with xargs. I think the following is a more efficient alternative to

find foo -type f ! -name '*Music*' -exec cp {} bar \; # new proc for each exec



find . -maxdepth 1 -name '*Music*' -prune -o -print0 | xargs -0 -i cp {} dest/

You can also use a pretty simple for loop:

for f in `find . -not -name "*Music*"`
do
    cp $f /target/dir
done

The extglob shell option gives you more powerful pattern matching in the command line.

You turn it on with shopt -s extglob, and turn it off with shopt -u extglob.

In your example, you would initially do:

$ shopt -s extglob
$ cp !(*Music*) /target_directory

The full available extended globbing operators are (excerpt from man bash):

If the extglob shell option is enabled using the shopt builtin, several extended pattern matching operators are recognized.A pattern-list is a list of one or more patterns separated by a |. Composite patterns may be formed using one or more of the following sub-patterns:

• ?(pattern-list)
  Matches zero or one occurrence of the given patterns
• *(pattern-list)
  Matches zero or more occurrences of the given patterns
• +(pattern-list)
  Matches one or more occurrences of the given patterns
• @(pattern-list)
  Matches one of the given patterns
• !(pattern-list)
  Matches anything except one of the given patterns

So, for example, if you wanted to list all the files in the current directory that are not .c or .h files, you would do:

$ ls -d !(*@(.c|.h))

Of course, normal shell globing works, so the last example could also be written as:

$ ls -d !(*.[ch])

The following works lists all *.txt files in the current dir, except those that begin with a number.

This works in bash, dash, zsh and all other POSIX compatible shells.

for FILE in /some/dir/*.txt; do    # for each *.txt file
    case "${FILE##*/}" in          #   if file basename...
        [0-9]*) continue ;;        #   starts with digit: skip
    esac
    ## otherwise, do stuff with $FILE here
done

1. In line one the pattern /some/dir/*.txt will cause the for loop to iterate over all files in /some/dir whose name end with .txt.

2. In line two a case statement is used to weed out undesired files. – The ${FILE##*/} expression strips off any leading dir name component from the filename (here /some/dir/) so that patters can match against only the basename of the file. (If you're only weeding out filenames based on suffixes, you can shorten this to $FILE instead.)

3. In line three, all files matching the case pattern [0-9]*) line will be skipped (the continue statement jumps to the next iteration of the for loop). – If you want to you can do something more interesting here, e.g. like skipping all files which do not start with a letter (a–z) using [!a-z]*, or you could use multiple patterns to skip several kinds of filenames e.g. [0-9]*|*.bak to skip files both .bak files, and files which does not start with a number.

this would do it excluding exactly 'Music'

cp -a ^'Music' /target

this and that for excluding things like Music?* or *?Music

cp -a ^\*?'complete' /target
cp -a ^'complete'?\* /target

A trick I haven't seen on here yet that doesn't use extglob, find, or grep is to treat two file lists as sets and "diff" them using comm:

comm -23 <(ls) <(ls *Music*)

comm is preferable over diff because it doesn't have extra cruft.

This returns all elements of set 1, ls, that are not also in set 2, ls *Music*. This requires both sets to be in sorted order to work properly. No problem for ls and glob expansion, but if you're using something like find, be sure to invoke sort.

comm -23 <(find . | sort) <(find . | grep -i '.jpg' | sort)

Potentially useful.

One solution for this can be found with find.

$ mkdir foo bar
$ touch foo/a.txt foo/Music.txt
$ find foo -type f ! -name '*Music*' -exec cp {} bar \;
$ ls bar
a.txt

Find has quite a few options, you can get pretty specific on what you include and exclude.

Edit: Adam in the comments noted that this is recursive. find options mindepth and maxdepth can be useful in controlling this.

Not in bash (that I know of), but:

cp `ls | grep -v Music` /target_directory

I know this is not exactly what you were looking for, but it will solve your example.

In bash, an alternative to shopt -s extglob is the GLOBIGNORE variable. It's not really better, but I find it easier to remember.

An example that may be what the original poster wanted:

GLOBIGNORE="*techno*"; cp *Music* /only_good_music/

When done, unset GLOBIGNORE to be able to rm *techno* in the source directory.

If you want to avoid the mem cost of using the exec command, I believe you can do better with xargs. I think the following is a more efficient alternative to

find foo -type f ! -name '*Music*' -exec cp {} bar \; # new proc for each exec



find . -maxdepth 1 -name '*Music*' -prune -o -print0 | xargs -0 -i cp {} dest/

My personal preference is to use grep and the while command. This allows one to write powerful yet readable scripts ensuring that you end up doing exactly what you want. Plus by using an echo command you can perform a dry run before carrying out the actual operation. For example:

ls | grep -v "Music" | while read filename
do
echo $filename
done

will print out the files that you will end up copying. If the list is correct the next step is to simply replace the echo command with the copy command as follows:

ls | grep -v "Music" | while read filename
do
cp "$filename" /target_directory
done

One solution for this can be found with find.

$ mkdir foo bar
$ touch foo/a.txt foo/Music.txt
$ find foo -type f ! -name '*Music*' -exec cp {} bar \;
$ ls bar
a.txt

Find has quite a few options, you can get pretty specific on what you include and exclude.

Edit: Adam in the comments noted that this is recursive. find options mindepth and maxdepth can be useful in controlling this.

The following works lists all *.txt files in the current dir, except those that begin with a number.

This works in bash, dash, zsh and all other POSIX compatible shells.

for FILE in /some/dir/*.txt; do    # for each *.txt file
    case "${FILE##*/}" in          #   if file basename...
        [0-9]*) continue ;;        #   starts with digit: skip
    esac
    ## otherwise, do stuff with $FILE here
done

1. In line one the pattern /some/dir/*.txt will cause the for loop to iterate over all files in /some/dir whose name end with .txt.

2. In line two a case statement is used to weed out undesired files. – The ${FILE##*/} expression strips off any leading dir name component from the filename (here /some/dir/) so that patters can match against only the basename of the file. (If you're only weeding out filenames based on suffixes, you can shorten this to $FILE instead.)

3. In line three, all files matching the case pattern [0-9]*) line will be skipped (the continue statement jumps to the next iteration of the for loop). – If you want to you can do something more interesting here, e.g. like skipping all files which do not start with a letter (a–z) using [!a-z]*, or you could use multiple patterns to skip several kinds of filenames e.g. [0-9]*|*.bak to skip files both .bak files, and files which does not start with a number.

My personal preference is to use grep and the while command. This allows one to write powerful yet readable scripts ensuring that you end up doing exactly what you want. Plus by using an echo command you can perform a dry run before carrying out the actual operation. For example:

ls | grep -v "Music" | while read filename
do
echo $filename
done

will print out the files that you will end up copying. If the list is correct the next step is to simply replace the echo command with the copy command as follows:

ls | grep -v "Music" | while read filename
do
cp "$filename" /target_directory
done

You can also use a pretty simple for loop:

for f in `find . -not -name "*Music*"`
do
    cp $f /target/dir
done

One solution for this can be found with find.

$ mkdir foo bar
$ touch foo/a.txt foo/Music.txt
$ find foo -type f ! -name '*Music*' -exec cp {} bar \;
$ ls bar
a.txt

Find has quite a few options, you can get pretty specific on what you include and exclude.

Edit: Adam in the comments noted that this is recursive. find options mindepth and maxdepth can be useful in controlling this.

this would do it excluding exactly 'Music'

cp -a ^'Music' /target

this and that for excluding things like Music?* or *?Music

cp -a ^\*?'complete' /target
cp -a ^'complete'?\* /target

A trick I haven't seen on here yet that doesn't use extglob, find, or grep is to treat two file lists as sets and "diff" them using comm:

comm -23 <(ls) <(ls *Music*)

comm is preferable over diff because it doesn't have extra cruft.

This returns all elements of set 1, ls, that are not also in set 2, ls *Music*. This requires both sets to be in sorted order to work properly. No problem for ls and glob expansion, but if you're using something like find, be sure to invoke sort.

comm -23 <(find . | sort) <(find . | grep -i '.jpg' | sort)

Potentially useful.

You can also use a pretty simple for loop:

for f in `find . -not -name "*Music*"`
do
    cp $f /target/dir
done

If you want to avoid the mem cost of using the exec command, I believe you can do better with xargs. I think the following is a more efficient alternative to

find foo -type f ! -name '*Music*' -exec cp {} bar \; # new proc for each exec



find . -maxdepth 1 -name '*Music*' -prune -o -print0 | xargs -0 -i cp {} dest/