recursively use scp but excluding some folders

Question

Assume there are some folders with these structures   bench1 1cpu p 0 image   bench1 1cpu p 0 fl 1   bench1 1cpu p 0 fl 1   bench1 1cpu p 0 fl 1   bench1 1cpu p 0 fl 1   bench1 1cpu p 1 image   bench1 1cpu p 1 fl 1   bench1 1cpu p 1 fl 1   bench1 1cpu p 1 fl 1   bench1 1cpu p 1 fl 1   bench1 2cpu p 0 image   bench1 2cpu p 0 fl 1   bench1 2cpu p 0 fl 1   bench1 2cpu p 0 fl 1   bench1 2cpu p 0 fl 1   bench1 2cpu p 1 image   bench1 2cpu p 1 fl 1   bench1 2cpu p 1 fl 1   bench1 2cpu p 1 fl 1   bench1 2cpu p 1 fl 1         What I want to do is to scp the following folders   bench1 1cpu p 0 image   bench1 1cpu p 1 image   bench1 2cpu p 0 image   bench1 2cpu p 1 image    As you can see I want to recursively use scp but excluding all folders that name  fl X   It seems that scp has not such option   UPDATE scp has not such feature  Instead I use the following command   rsync -av --exclude  fl    user server  my dir   But it doesn t work  It only transfers the list of folders   something like ls -R

User · Answer

This one works fine for me as the directories structure is not important for me.

scp -r USER@HOSTNAME:~/bench1/?cpu/p_?/image/ .

Assuming /bench1 is in the home directory of the current user. Also, change USER and HOSTNAME to the real values.

User · Answer

Although scp supports recursive directory copying with the -r option  it does not support filtering of the files  There are several ways to accomplish your task  but I would probably rely on find  xargs  tar  and ssh instead of scp   find   -type d -wholename   bench  image      xargs tar cf -     ssh user remote tar xf - -C  my dir   The rsync solution can be made to work  but you are missing some arguments  rsync also needs the r switch to recurse into subdirectories  Also  if you want the same security of scp  you need to do the transfer under ssh  Something like   rsync -avr -e  ssh -l user  --exclude  fl      bench  remote  my dir

User · Answer

You can specify GLOBIGNORE and use the pattern    GLOBIGNORE  ignore1 ignore2  scp -r source   remoteurl remoteDir   You may wish to have general rules which you combine or override by using export GLOBIGNORE  but for ad-hoc usage simply the above will do  The   character is used as delimiter for multiple values

User · Answer

If you use a pem file to authenticate u can use the following command  which will exclude files with something extension    rsync -Lavz -e  ssh -i  lt full-path-to-pem gt  -o StrictHostKeyChecking no -o UserKnownHostsFile  dev null  --exclude    something  --progress  lt path inside local host gt   lt user gt   lt host gt   lt path inside remote host gt    The -L means follow links  copy files not links   Use full path to your pem file and not relative   Using sshfs is not recommended since it works slowly  Also  the combination of find and scp that was presented above is also a bad idea since it will open a ssh session per file which is too expensive

User · Answer

Assuming the simplest option  installing rsync on the remote host  isn t feasible  you can use sshfs to mount the remote locally  and rsync from the mount directory  That way you can use all the options rsync offers  for example --exclude   Something like this should do   sshfs user server  sshfsdir rsync --recursive --exclude whatever sshfsdir path on server  where to store   Note that the effectiveness of rsync  only transferring changes  not everything  doesn t apply here  This is because for that to work  rsync must read every file s contents to see what has changed  However  as rsync runs only on one host  the whole file must be transferred there  by sshfs   Excluded files should not be transferred  however

User · Answer

You can use extended globbing as in the example below    Enable extglob shopt -s extglob  cp -rv     excludeme   jpg   var destination

[bash] recursively use scp but excluding some folders

Examples related to bash

Examples related to scp