[command-line] Use Robocopy to copy only changed files?

I'm trying to find an easy way of deploying only changed files to the webserver for deployment purpose. In times past I've used MSBuild, which could be told to only copy files that were newer than the ones on the target, but I'm in a hurry and don't want to try to figure out the newer version of MSBuild.

Can I use ROBOCOPY for this? There is a list of options for exclusion, which is:

  • /XC :: eXclude Changed files.
  • /XN :: eXclude Newer files.
  • /XO :: eXclude Older files.
  • /XX :: eXclude eXtra files and directories.
  • /XL :: eXclude Lonely files and directories.

What exactly does it mean to exclude? Exclude copying, or exclude overwriting? For example, if I wrote:

ROBOCOPY C:\SourceFolder\ABC.dll D:\DestinationFolder /XO

would this copy only newer files, not files of the same age?

Or is there a better tool to do this?

The answer is

Looks like /e option is what you need, it'll skip same files/directories.

robocopy c:\data c:\backup /e

If you run the command twice, you'll see the second round is much faster since it skips a lot of things.

You can use robocopy to copy files with an archive flag and reset the attribute. Use /M command line, this is my backup script with few extra tricks.

This script needs NirCmd tool to keep mouse moving so that my machine won't fall into sleep. Script is using a lockfile to tell when backup script is completed and mousemove.bat script is closed. You may leave this part out.

Another is 7-Zip tool for splitting virtualbox files smaller than 4GB files, my destination folder is still FAT32 so this is mandatory. I should use NTFS disk but haven't converted backup disks yet.


@REM https://technet.microsoft.com/en-us/library/cc733145.aspx
@REM http://www.skonet.com/articles_archive/robocopy_job_template.aspx

set basedir=%~dp0
del /Q %basedir%backup-robocopy-log.txt

set dt=%date%_%time:~0,8%
echo "%dt% robocopy started" > %basedir%backup-robocopy-lock.txt
start "Keep system awake" /MIN /LOW  cmd.exe /C %basedir%backup-robocopy-movemouse.bat

set dest=E:\backup

call :BACKUP "Program Files\MariaDB 5.5\data"
call :BACKUP "projects"
call :BACKUP "Users\Myname"

@REM Split +4GB file to multiple files to support FAT32 destination disk,
@REM splitted files must be stored outside of the robocopy destination folder.
set srcfile=C:\Users\Myname\VirtualBox VMs\Ubuntu\Ubuntu.vdi
set dstfile=%dest%\Users\Myname\VirtualBox VMs\Ubuntu\Ubuntu.vdi
set dstfile2=%dest%\non-robocopy\Users\Myname\VirtualBox VMs\Ubuntu\Ubuntu.vdi
IF NOT EXIST "%dstfile%" (
  IF NOT EXIST "%dstfile2%.7z.001" attrib +A "%srcfile%"
  dir /b /aa "%srcfile%" && (
    del /Q "%dstfile2%.7z.*"
    c:\apps\commands\7za.exe -mx0 -v4000m u "%dstfile2%.7z"  "%srcfile%"
    attrib -A "%srcfile%"
    @set dt=%date%_%time:~0,8%
    @echo %dt% Splitted %srcfile% >> %basedir%backup-robocopy-log.txt

del /Q %basedir%backup-robocopy-lock.txt

TITLE Backup %~1
robocopy.exe "c:\%~1" "%dest%\%~1" /JOB:%basedir%backup-robocopy-job.rcj

@set dt=%date%_%time:~0,8%
@echo %dt% robocopy completed >> %basedir%backup-robocopy-log.txt
@echo %dt% robocopy completed


:: Robocopy Job Parameters
:: robocopy.exe "c:\projects" "E:\backup\projects" /JOB:backup-robocopy-job.rcj

:: Source Directory (this is given in command line)

:: Destination Directory (this is given in command line)

:: Include files matching these names

/M      :: copy only files with the Archive attribute and reset it.
/XJD    :: eXclude Junction points for Directories.

:: Exclude Directories
    C:\Users\Myname\Saved Games

:: Exclude files matching these names

:: Exclude files with any of the given Attributes set
:: S=System, H=Hidden

:: Copy options
/S          :: copy Subdirectories, but not empty ones.
/E          :: copy subdirectories, including Empty ones.
/COPY:DAT   :: what to COPY for files (default is /COPY:DAT).
/DCOPY:T    :: COPY Directory Timestamps.
/PURGE      :: delete dest files/dirs that no longer exist in source.

:: Retry Options
/R:0        :: number of Retries on failed copies: default 1 million.
/W:1        :: Wait time between retries: default is 30 seconds.

:: Logging Options (LOG+ append)
/NDL        :: No Directory List - don't log directory names.
/NP         :: No Progress - don't display percentage copied.
/TEE        :: output to console window, as well as the log file.
/LOG+:c:\apps\commands\backup-robocopy-log.txt :: append to logfile


@echo off
@REM Move mouse to prevent maching from sleeping 
@rem while running a backup script

echo Keep system awake while robocopy is running,
echo this script moves a mouse once in a while.

set basedir=%~dp0
set IDX=0

IF NOT EXIST "%basedir%backup-robocopy-lock.txt" GOTO :EOF
SET /A IDX=%IDX% + 1
IF "%IDX%"=="240" (
  echo Move mouse to keep system awake
  c:\apps\commands\nircmdc.exe sendmouse move 5 5
  c:\apps\commands\nircmdc.exe sendmouse move -5 -5
c:\apps\commands\nircmdc.exe wait 1000

Examples related to command-line

Git is not working after macOS Update (xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools) Flutter command not found Angular - ng: command not found how to run python files in windows command prompt? How to run .NET Core console app from the command line Copy Paste in Bash on Ubuntu on Windows How to find which version of TensorFlow is installed in my system? How to install JQ on Mac by command-line? Python not working in the command line of git bash Run function in script from command line (Node JS)

Examples related to synchronization

fs.writeFile in a promise, asynchronous-synchronous stuff Use Robocopy to copy only changed files? Wait until flag=true Why is synchronized block better than synchronized method? Printing Even and Odd using two Threads in Java How to use the CancellationToken property? Sharing a variable between multiple different threads How to keep two folders automatically synchronized? Asynchronous Process inside a javascript for loop Java Singleton and Synchronization

Examples related to command-line-arguments

How to pass arguments to Shell Script through docker run How can I pass variable to ansible playbook in the command line? Use Robocopy to copy only changed files? mkdir's "-p" option What is Robocopy's "restartable" option? How do you run a .exe with parameters using vba's shell()? Bash command line and input limit Check number of arguments passed to a Bash script Parsing boolean values with argparse Import SQL file by command line in Windows 7

Examples related to robocopy

How do I force Robocopy to overwrite files? Use Robocopy to copy only changed files? What is Robocopy's "restartable" option? How to exclude subdirectories in the destination while using /mir /xd switch in robocopy How to copy directories with spaces in the name Copy files without overwrite How can I make robocopy silent in the command line except for progress? How can I copy network files using Robocopy? How to copy a directory structure but only include certain files (using windows batch files)