How to rsync only a specific list of files

Question

I ve about 50 or so files in various sub-directories that I d like to push to a remote server   I figured rsync would be able to do this for me using the --include-from option  Without the --exclude     option  all the files in the directory are being synced  with the option  no files are   rsync -avP -e ssh --include-from deploy rsync include txt --exclude      root 0 0 0 0  var www  --dry-run   I m running it as dry initially and 0 0 0 0 is obviously replaced by the IP of the remote server  The contents of rsync include txt is a new line separated list of relative paths to the files I want to upload   Is there a better way of doing this that is escaping me on a Monday morning

User · Answer

This answer is not the direct answer for the question  But it should help you figure out which solution fits best for your problem   When analysing the problem you should activate the debug option -vv  Then rsync will output which files are included or excluded by which pattern   building file list       sender  hiding file FILE1 because of pattern FILE1   sender  showing file FILE2 because of pattern

User · Answer

There is a flag --files-from that does exactly what you want  From man rsync    --files-from FILE       Using this option allows you to specify the exact list of files to transfer  as read from the specified FILE or - for standard input    It also tweaks the default behavior of rsync to make transferring just the specified files and directories easier          The --relative  -R  option is implied  which preserves the path information that is specified for each item in the file  use --no-relative or --no-R if you want to turn that off     The --dirs  -d  option is implied  which will create directories specified in the list on the destination rather than noisily skipping them  use --no-dirs or --no-d if you want to turn that off     The --archive  -a  option   s behavior does not imply --recursive  -r   so specify it explicitly  if you want it    These side-effects change the default state of rsync  so the position of the --files-from option on the command-line has no bearing on how other options are parsed  e g  -a works the same before or after --files-from  as does --no-R and all other options           The filenames that are read from the FILE are all relative to the source dir -- any leading slashes are removed and no      references are allowed to go higher than the source dir   For example  take this command   rsync -a --files-from  tmp foo  usr remote  backup       If   tmp foo  contains  the  string   bin   or even   bin    the  usr bin directory will be created as  backup bin on the remote host   If it contains  bin    note the trailing slash   the immediate contents of the directory would also be sent  without needing to be explicitly mentioned in the file -- this began in version 2 6 4    In both   cases  if the -r option was enabled  that dir   s entire hierarchy would also be transferred  keep in mind that -r needs to be specified explicitly with --files-from  since it is not implied by -a    Also note that the effect of the  enabled by default  --relative option is to duplicate only the path info that is read from  the  file  --  it   does not force the duplication of the source-spec path   usr in this case        In addition  the --files-from file can be read from the remote host instead of the local host if you specify a  host   in front of the file  the host must match one end of the transfer    As a short-cut  you can specify just a prefix of     to mean  use the remote end of the transfer    For example   rsync -a --files-from   path file-list src    tmp copy       This would copy all the files specified in the  path file-list file that was located on the remote  src  host       If the --iconv and --protect-args options are specified and the --files-from filenames are being sent from one host to another  the filenames will be translated from the sending host   s charset to the receiving host   s charset       NOTE   sorting the list of files in the --files-from input helps rsync to be more efficient  as it will avoid re-visiting the path elements that are shared between adjacent entries   If the input is not sorted  some path elements  implied directories  may end up being scanned multiple times  and rsync will eventually unduplicate them after   they get turned into file-list elements

User · Answer

--files-from  parameter needs trailing slash if you want to keep the absolute path intact  So your command would become something like below   rsync -av --files-from  path to file    tmp    This could be done like there are a large number of files and you want to copy all files to x path  So you would find the files and throw output to a file like below   find  var   -name   log  gt  file

User · Answer

date   Wed 24 Apr 2019 09 54 53 AM PDT   rsync --version   rsync  version 3 1 3  protocol version 31         Syntax  rsync   lt file   folder list gt    lt source gt    lt target gt   Folder names  here  WITH a trailing    e g  Cancer - Evolution   are in a folder list file  e g   cm folder list test       mnt Vancouver projects ie claws data cm folder list test   test file  2019-04-24 Cancer  Cancer - Evolution  Cancer - Genomic Variants  Cancer - Metastasis  EMT Transition       Cancer Pathways  Networks  Catabolism - Autophagy  Phagosomes  Mitophagy  Catabolism - Lysosomes    If you don t include those trailing    the rsync d target folders are created  but are empty   Those folder names are appended to the rest of their path   home victoria Mail 2 RESEARCH - NEWS   thus providing the complete folder path to rsync  e g    home victoria Mail 2 RESEARCH - NEWS Cancer - Evolution    Note that you also need to use --files-from       NOT --include-from       rsync -aqP --delete --files-from  mnt Vancouver projects ie claws data cm folder list test    home victoria Mail 2 RESEARCH - NEWS    IN    In my BASH script  I defined variable  IN as follows    BASEDIR   mnt Vancouver projects ie claws  IN  BASEDIR data test input   rsync options used    -a      archive  equals -rlptgoD  no -H -A -X      -r      recursive     -l      copy symlinks as symlinks     -p      preserve permissions     -t      preserve modification times      -g      preserve group      -o      preserve owner  super-user only       -D      same as --devices --specials    -q      quiet  https   serverfault com questions 547106 run-totally-silent-rsync     --delete     This  tells  rsync to delete extraneous files from the RECEIVING SIDE  ones     that AREN   T ON THE SENDING SIDE   but only for the directories that are     being synchronized   You must have asked rsync to send the whole directory      e g    dir  or  dir    without using a wildcard for the directory   s contents      e g   dir     since the wildcard is expanded by the shell and rsync thus     gets a request to transfer individual files  not the files    parent directory      Files  that  are  excluded  from  the transfer are also excluded from being     deleted unless you use the --delete-excluded option or mark the rules as     only matching on the sending side  see the include exclude modifiers in the     FILTER RULES section

User · Answer

Edit  atp s answer below is better   Please use that one  You might have an easier time  if you re looking for a specific list of files  putting them directly on the command line instead    rsync -avP -e ssh  cat deploy rsync include txt  root 0 0 0 0  var www   This is assuming  however  that your list isn t so long that the command line length will be a problem and that the rsync include txt file contains just real paths  i e  no comments  and no regexps

User · Answer

For the record  none of the answers above helped except for one  To summarize  you can do the backup operation using --files-from  by using either    rsync -aSvuc  cat rsync-src-files   mnt d rsync test    OR    rsync -aSvuc --recursive --files-from rsync-src-files    mnt d rsync test    The former command is self explanatory  beside the content of the file rsync-src-files which I will elaborate down below  Now  if you want to use the latter version  you need to keep in mind the following four remarks    Notice one needs to specify both --files-from and the source directory One needs to explicitely specify --recursive   The file rsync-src-files is a user created file and it was placed within the src directory for this test The rsyn-src-files contain the files and folders to copy and they are taken relative to the source directory  IMPORTANT  Make sure there is not trailing spaces or blank lines in the file  In the example below  there are only two lines  not three  Figure it out by chance   Content of rsynch-src-files is    folderName1 folderName2

User · Answer

I got similar task  to rsync all files modified after given date  but excluding some directories  It was difficult to build one liner all-in-one style  so I dived problem into smaller pieces   Final solution   find    sourceDIR -type f -newermt  DD MMM YYYY HH MM SS    egrep -v       Downloads FOO   gt  FileList txt rsync -v --files-from FileList txt   sourceDIR  Destination   First I use find -L   sourceDIR -type f -newermt  DD MMM YYYY HH MM SS   I tried to add regex to find line to exclude name patterns  however my flavor of Linux  Mint  seams not to understand negate regex in find  Tried number of regex flavors - non work as desired  So I end up with egrep -v - option that excludes pattern easy way  My rsync is not copying directories like   cache or   config plus some other I explicitly named

User · Answer

None of these answers worked for me  when all I had was a list of directories  Then I stumbled upon the solution  You have to add -r to --files-from because -a will not be recursive in this scenario  who knew     rsync -aruRP --files-from directory list      new location

[rsync] How to rsync only a specific list of files?

Examples related to rsync