[bash] Copy folder recursively, excluding some folders

I am trying to write a simple bash script that will copy the entire contents of a folder including hidden files and folders into another folder, but I want to exclude certain specific folders. How could I achieve this?

Quick Start

Run:

rsync -av --exclude='path1/in/source' --exclude='path2/in/source' [source]/ [destination]

Notes

• -avr will create a new directory named [destination].
• source and source/ create different results:
  • source — copy the contents of source into destination.
  • source/ — copy the folder source into destination.
• To exclude many files:
  • --exclude-from=FILE — FILE is the name of a file containing other files or directories to exclude.
• --exclude may also contain wildcards:
  • e.g. --exclude=*/.svn*

Modified from: https://stackoverflow.com/a/2194500/749232

Example

Starting folder structure:

.
+-- destination
+-- source
    +-- fileToCopy.rtf
    +-- fileToExclude.rtf

Run:

rsync -av --exclude='fileToCopy.rtf' source/ destination

Ending folder structure:

.
+-- destination
¦   +-- fileToExclude.rtf
+-- source
    +-- fileToCopy.rtf
    +-- fileToExclude.rtf

you can use tar, with --exclude option , and then untar it in destination. eg

cd /source_directory
tar cvf test.tar --exclude=dir_to_exclude *
mv test.tar /destination 
cd /destination  
tar xvf test.tar

see the man page of tar for more info

You can use find with the -prune option.

An example from man find:

       cd /source-dir
       find . -name .snapshot -prune -o \( \! -name *~ -print0 \)|
       cpio -pmd0 /dest-dir

       This command copies the contents of /source-dir to /dest-dir, but omits
       files  and directories named .snapshot (and anything in them).  It also
       omits files or directories whose name ends in ~,  but  not  their  con-
       tents.  The construct -prune -o \( ... -print0 \) is quite common.  The
       idea here is that the expression before -prune matches things which are
       to  be  pruned.  However, the -prune action itself returns true, so the
       following -o ensures that the right hand side  is  evaluated  only  for
       those  directories  which didn't get pruned (the contents of the pruned
       directories are not even visited, so their  contents  are  irrelevant).
       The  expression on the right hand side of the -o is in parentheses only
       for clarity.  It emphasises that the -print0 action  takes  place  only
       for  things  that  didn't  have  -prune  applied  to them.  Because the
       default `and' condition between tests binds more tightly than -o,  this
       is  the  default anyway, but the parentheses help to show what is going
       on.

EXCLUDE="foo bar blah jah"                                                                             
DEST=$1

for i in *
do
    for x in $EXCLUDE
    do  
        if [ $x != $i ]; then
            cp -a $i $DEST
        fi  
    done
done

Untested...

Similar to Jeff's idea (untested):

find . -name * -print0 | grep -v "exclude" | xargs -0 -I {} cp -a {} destination/

Use tar along with a pipe.

cd /source_directory
tar cf - --exclude=dir_to_exclude . | (cd /destination && tar xvf - )

You can even use this technique across ssh.

inspired by @SteveLazaridis's answer, which would fail, here is a POSIX shell function - just copy and paste into a file named cpx in yout $PATH and make it executible (chmod a+x cpr). [Source is now maintained in my GitLab.

#!/bin/sh

# usage: cpx [-n|--dry-run] "from_path" "to_path" "newline_separated_exclude_list"
# limitations: only excludes from "from_path", not it's subdirectories

cpx() {
# run in subshell to avoid collisions
  (_CopyWithExclude "$@")
}

_CopyWithExclude() {
  case "$1" in
    -n|--dry-run) { DryRun='echo'; shift; } ;;
  esac

  from="$1"
  to="$2"
  exclude="$3"

  $DryRun mkdir -p "$to"

  if [ -z "$exclude" ]; then
      cp "$from" "$to"
      return
  fi

  ls -A1 "$from" \
    | while IFS= read -r f; do
        unset excluded
        if [ -n "$exclude" ]; then
          for x in $(printf "$exclude"); do
          if [ "$f" = "$x" ]; then
              excluded=1
              break
          fi
          done
        fi
        f="${f#$from/}"
        if [ -z "$excluded" ]; then
          $DryRun cp -R "$f" "$to"
        else
          [ -n "$DryRun" ] && echo "skip '$f'"
        fi
      done
}

# Do not execute if being sourced
[ "${0#*cpx}" != "$0" ] && cpx "$@"

Example usage

EXCLUDE="
.git
my_secret_stuff
"
cpr "$HOME/my_stuff" "/media/usb" "$EXCLUDE"