[bash] recursively use scp but excluding some folders

Assume there are some folders with these structures

/bench1/1cpu/p_0/image/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_0/fl_1/
/bench1/1cpu/p_1/image/
/bench1/1cpu/p_1/fl_1/
/bench1/1cpu/p_1/fl_1/
/bench1/1cpu/p_1/fl_1/
/bench1/1cpu/p_1/fl_1/
/bench1/2cpu/p_0/image/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_0/fl_1/
/bench1/2cpu/p_1/image/
/bench1/2cpu/p_1/fl_1/
/bench1/2cpu/p_1/fl_1/
/bench1/2cpu/p_1/fl_1/
/bench1/2cpu/p_1/fl_1/
....

What I want to do is to scp the following folders

/bench1/1cpu/p_0/image/
/bench1/1cpu/p_1/image/
/bench1/2cpu/p_0/image/
/bench1/2cpu/p_1/image/

As you can see I want to recursively use scp but excluding all folders that name "fl_X". It seems that scp has not such option.

UPDATE scp has not such feature. Instead I use the following command

 rsync -av --exclude 'fl_*' user@server:/my/dir

But it doesn't work. It only transfers the list of folders!! something like ls -R

This question is related to bash scp

The answer is


You can use extended globbing as in the example below:

#Enable extglob
shopt -s extglob

cp -rv !(./excludeme/*.jpg) /var/destination

You can specify GLOBIGNORE and use the pattern *

GLOBIGNORE='ignore1:ignore2' scp -r source/* remoteurl:remoteDir

You may wish to have general rules which you combine or override by using export GLOBIGNORE, but for ad-hoc usage simply the above will do. The : character is used as delimiter for multiple values.


Assuming the simplest option (installing rsync on the remote host) isn't feasible, you can use sshfs to mount the remote locally, and rsync from the mount directory. That way you can use all the options rsync offers, for example --exclude.

Something like this should do:

sshfs user@server: sshfsdir
rsync --recursive --exclude=whatever sshfsdir/path/on/server /where/to/store

Note that the effectiveness of rsync (only transferring changes, not everything) doesn't apply here. This is because for that to work, rsync must read every file's contents to see what has changed. However, as rsync runs only on one host, the whole file must be transferred there (by sshfs). Excluded files should not be transferred, however.


If you use a pem file to authenticate u can use the following command (which will exclude files with something extension):

rsync -Lavz -e "ssh -i <full-path-to-pem> -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --exclude "*.something" --progress <path inside local host> <user>@<host>:<path inside remote host>

The -L means follow links (copy files not links). Use full path to your pem file and not relative.

Using sshfs is not recommended since it works slowly. Also, the combination of find and scp that was presented above is also a bad idea since it will open a ssh session per file which is too expensive.


This one works fine for me as the directories structure is not important for me.

scp -r USER@HOSTNAME:~/bench1/?cpu/p_?/image/ .

Assuming /bench1 is in the home directory of the current user. Also, change USER and HOSTNAME to the real values.