Progress during large file copy Copy-Item Write-Progress

Question

Is there any way to copy a really large file  from one server to another  in PowerShell AND display its progress   There are solutions out there to use Write-Progress in conjunction with looping to copy many files and display progress   However I can t seem to find anything that would show progress of a single file   Any thoughts

User · Answer

This recursive function copies files and directories recursively from source path to destination path If file already exists on destination path  it copies them only with newer files  Function Copy-FilesBitsTransfer           Parameter Mandatory  true   String  sourcePath            Parameter Mandatory  true   String  destinationPath            Parameter Mandatory  false   bool  createRootDirectory    true         item   Get-Item  sourcePath      itemName   Split-Path  sourcePath -leaf     if    item PSIsContainer    Item Is a file           clientFileTime   Get-Item  sourcePath   select LastWriteTime -ExpandProperty LastWriteTime          if    Test-Path -Path  destinationPath  itemName                Start-BitsTransfer -Source  sourcePath -Destination  destinationPath -Description  quot  sourcePath  gt  gt   destinationPath quot  -DisplayName  quot Copy Template file quot  -Confirm  false             if                        return  false                                 else               serverFileTime   Get-Item  destinationPath  itemName   select LastWriteTime -ExpandProperty LastWriteTime              if   serverFileTime -lt  clientFileTime                                Start-BitsTransfer -Source  sourcePath -Destination  destinationPath -Description  quot  sourcePath  gt  gt   destinationPath quot  -DisplayName  quot Copy Template file quot  -Confirm  false                 if                            return  false                                                     else   Item Is a directory         if   createRootDirectory                destinationPath    quot  destinationPath  itemName quot              if    Test-Path -Path  destinationPath -PathType Container                    if  Test-Path -Path  destinationPath -PathType Leaf    In case item is a file  delete it                      Remove-Item -Path  destinationPath                                    New-Item -ItemType Directory  destinationPath   Out-Null                 if                            return  false                                                    Foreach   fileOrDirectory in  Get-Item -Path  quot  sourcePath   quot                           status   Copy-FilesBitsTransfer  fileOrDirectory  destinationPath  true             if    status                   return  false                                    return  true

User · Answer

Trevor Sullivan has a write-up on how to add a command called Copy-ItemWithProgress to PowerShell on Robocopy

User · Answer

Sean Kearney from the Hey  Scripting Guy  Blog has a solution I found works pretty nicely   Function Copy-WithProgress        CmdletBinding        Param                Parameter Mandatory  true              ValueFromPipelineByPropertyName  true              Position 0            Source           Parameter Mandatory  true              ValueFromPipelineByPropertyName  true              Position 0            Destination             Source  Source tolower        Filelist Get-Childitem   Source     Recurse      Total  Filelist count      Position 0      foreach   File in  Filelist                 Filename  File Fullname tolower   replace  Source              DestinationFile   Destination  Filename          Write-Progress -Activity  Copying data from   source  to   Destination   -Status  Copying File  Filename  -PercentComplete    Position  total  100          Copy-Item  File FullName -Destination  DestinationFile          Position             Then to use it   Copy-WithProgress -Source  src -Destination  dest

User · Answer

i found none of the examples above met my needs  i wanted to copy a directory with sub directories  the problem is my source directory had too many files so i quickly hit the BITS file limit  i had   1500 file  also the total directory size was quite large    i found a function using robocopy that was a good starting point at https   keithga wordpress com 2014 06 23 copy-itemwithprogress   however i found it wasn t quite robust enough  it didn t handle trailing slashes  spaces gracefully and did not stop the copy when the script was halted   Here is my refined version   function Copy-ItemWithProgress        lt        SYNOPSIS     RoboCopy with PowerShell progress        DESCRIPTION     Performs file copy with RoboCopy  Output from RoboCopy is captured      parsed  and returned as Powershell native status and progress        PARAMETER Source     Directory to copy files from  this should not contain trailing slashes       PARAMETER Destination     DIrectory to copy files to  this should not contain trailing slahes       PARAMETER FilesToCopy     A wildcard expresion of which files to copy  defaults to           PARAMETER RobocopyArgs     List of arguments passed directly to Robocopy      Must not conflict with defaults   ndl  TEE  Bytes  NC  nfl  Log       PARAMETER ProgressID     When specified   gt  0  will use this identifier for the progress bar       PARAMETER ParentProgressID     When specified   gt   0  will use this identifier as the parent ID for progress bars     so that they appear nested which allows for usage in more complex scripts        OUTPUTS     Returns an object with the status of final copy      REMINDER  Any error level below 8 can be considered a success by RoboCopy        EXAMPLE     C  PS gt    Copy-ItemWithProgress c  Src d  Dest      Copy the contents of the c  Src directory to a directory d  Dest     Without the  e or  mir switch  only files from the root of c  src are copied        EXAMPLE     C  PS gt    Copy-ItemWithProgress   c  Src Files   d  Dest  mir  xf   log -Verbose      Copy the contents of the  c  Name with Space  directory to a directory d  Dest      mir and  XF parameters are passed to robocopy  and script is run verbose       LINK     https   keithga wordpress com 2014 06 23 copy-itemwithprogress       NOTES     By Keith S  Garner  KeithGa KeithGa com  - 6 23 2014     With inspiration by Trevor Sullivan  pcgeek86     Tweaked by Justin Marshall - 02 20 2020        gt        CmdletBinding        param           Parameter Mandatory  true            string  Source           Parameter Mandatory  true            string  Destination           Parameter Mandatory  false            string  FilesToCopy                 Parameter Mandatory    true ValueFromRemainingArguments  true             string     RobocopyArgs           int  ParentProgressID -1           int  ProgressID -1             handle spaces and trailing slashes      SourceDir      0    -f   Source -replace                 TargetDir      0    -f   Destination -replace                   ScanLog     IO Path   GetTempFileName        RoboLog     IO Path   GetTempFileName        ScanArgs      SourceDir  TargetDir  FilesToCopy     RobocopyArgs     ndl  TEE  bytes  Log  ScanLog  nfl  L  Split           RoboArgs      SourceDir  TargetDir  FilesToCopy     RobocopyArgs     ndl  TEE  bytes  Log  RoboLog  NC  Split             Launch Robocopy Processes     write-verbose   Robocopy Scan  n      ScanArgs -join           write-verbose   Robocopy Full  n      RoboArgs -join            ScanRun   start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList  ScanArgs     try                RoboRun   start-process robocopy -PassThru -WindowStyle Hidden -ArgumentList  RoboArgs         try                         Parse Robocopy  Scan  pass              ScanRun WaitForExit                LogData   get-content  ScanLog             if   ScanRun ExitCode -ge 8                                 LogData out-string Write-Error                 throw  Robocopy    ScanRun ExitCode                              FileSize    regex   Match  LogData -4       s   d   s   d     Groups 2  Value             write-verbose   Robocopy Bytes   FileSize  n     LogData -join   n                 determine progress parameters              ProgressParms                 if   ParentProgressID -ge 0                     ProgressParms  ParentID    ParentProgressID                           if   ProgressID -ge 0                     ProgressParms  ID    ProgressID               else                    ProgressParms  ID    RoboRun Id                             Monitor Full RoboCopy             while    RoboRun HasExited                                 LogData   get-content  RoboLog                  Files    LogData -match    s   d   s   S                    if   null -ne  Files                                          copied     Files 0    Files Length-2     ForEach-Object     Split   t   -2     Measure-Object -sum  Sum                     if   LogData -1  -match   100  d  d   d                                                    write-progress Copy -ParentID  ProgressParms  ID   -percentComplete  LogData -1  Trim     t    LogData -1                           Copied     Files -1  Split   t   -2   100     LogData -1  Trim     t                                              else                                               write-progress Copy -ParentID  ProgressParms  ID   -Complete                                           write-progress ROBOCOPY  -PercentComplete   Copied  FileSize 100   Files -1  Split   t   -1   ProgressParms                                           finally               if    RoboRun HasExited   Write-Warning  Terminating copy process with ID    RoboRun Id        RoboRun Kill                    RoboRun WaitForExit                 Parse full RoboCopy pass results  and cleanup              get-content  RoboLog  -11  -2    out-string   Write-Verbose             remove-item  RoboLog             write-output   PSCustomObject    ExitCode    RoboRun ExitCode                     finally           if    ScanRun HasExited   Write-Warning  Terminating scan process with ID    ScanRun Id        ScanRun Kill              ScanRun WaitForExit            remove-item  ScanLog

User · Answer

I haven t heard about progress with Copy-Item  If you don t want to use any external tool  you can experiment with streams  The size of buffer varies  you may try different values  from 2kb to 64kb   function Copy-File       param   string  from   string  to       ffile    io file   OpenRead  from       tofile    io file   OpenWrite  to      Write-Progress -Activity  quot Copying file quot  -status  quot  from - gt   to quot  -PercentComplete 0     try            byte    buff   new-object byte   4096          long  total    int  count   0         do                count    ffile Read  buff  0   buff Length               tofile Write  buff  0   count               total     count             if   total   1mb -eq 0                    Write-Progress -Activity  quot Copying file quot  -status  quot  from - gt   to quot                       -PercentComplete   long   total   100    ffile Length                           while   count -gt 0            finally            ffile Dispose            tofile Dispose           Write-Progress -Activity  quot Copying file quot  -Status  quot Ready quot  -Completed

User · Answer

I amended the code from stej  which was great  just what i needed   to use larger buffer   long  for larger files and used System Diagnostics Stopwatch class to track elapsed time and estimate time remaining   Also added reporting of transfer rate during transfer and outputting overall elapsed time and overall transfer rate   Using 4MB  4096 1024 bytes  buffer to get better than Win7 native throughput copying from NAS to USB stick on laptop over wifi   On To-Do list    add error handling  catch   handle get-childitem file list as input nested progress bars when copying multiple files  file x of y    if total data copied etc  input parameter for buffer size   Feel free to use improve  -   function Copy-File   param   string  from   string  to   ffile    io file   OpenRead  from   tofile    io file   OpenWrite  to  Write-Progress       -Activity  Copying file        -status   from Split      select -last 1        -PercentComplete 0 try        sw    System Diagnostics Stopwatch   StartNew         byte    buff   new-object byte    4096 1024       long  total    long  count   0     do            count    ffile Read  buff  0   buff Length           tofile Write  buff  0   count           total     count          int  pctcomp     int   total  ffile Length  100             int  secselapsed    int   sw elapsedmilliseconds ToString    1000          if    secselapsed -ne 0                  single  xferrate      total  secselapsed  1mb             else                single  xferrate   0 0                   if   total   1mb -eq 0                if  pctcomp -gt 0                     int  secsleft       secselapsed  pctcomp   100 - secselapsed                     else                    int  secsleft   0               Write-Progress                   -Activity   pctcomp ToString        Copying file         0 n2   -f  xferrate     MB s                    -status   from Split      select -last 1                    -PercentComplete  pctcomp                   -SecondsRemaining  secsleft                  while   count -gt 0   sw Stop     sw Reset      finally       write-host    from Split      select -last 1             copied in      secselapsed     seconds at              0 n2   -f  int    ffile length  secselapsed  1mb      MB s           ffile Close          tofile Close

User · Answer

Hate to be the one to bump an old subject  but I found this post extremely useful  After running performance tests on the snippets by stej and it s refinement by Graham Gold  plus the BITS suggestion by Nacht  I have decuded that   I really liked Graham s command with time estimations and speed readings  I also really liked the significant speed increase of using BITS as my transfer method   Faced with the decision between the two    I found that Start-BitsTransfer supported Asynchronous mode  So here is the result of my merging the two  function Copy-File         ref  https   stackoverflow com a 55527732 3626361     param  string  From   string  To       try            job   Start-BitsTransfer -Source  From -Destination  To               -Description  quot Moving   From   gt   To quot                -DisplayName  quot Backup quot  -Asynchronous            Start stopwatch          sw    System Diagnostics Stopwatch   StartNew           Write-Progress -Activity  quot Connecting    quot           while   job JobState ToString   -ne  quot Transferred quot                 switch   job JobState ToString                       quot Connecting quot                        break                                    quot Transferring quot                         pctcomp     job BytesTransferred    job BytesTotal    100                      elapsed     sw elapsedmilliseconds ToString      1000                      if   elapsed -eq 0                             xferrate   0 0                                           else                            xferrate      job BytesTransferred    elapsed    1mb                                              if   job BytesTransferred   1mb -eq 0                            if   pctcomp -gt 0                                 secsleft       elapsed    pctcomp    100  -  elapsed                                                    else                                secsleft   0                                                    Write-Progress -Activity   quot Copying file   quot      From Split  quot   quot     Select-Object -last 1     quot      quot     quot  0 n2  quot  -f  xferrate    quot MB s quot                                 -PercentComplete  pctcomp                               -SecondsRemaining  secsleft                                           break                                    quot Transferred quot                        break                                   Default                       throw  job JobState ToString      quot  unexpected BITS state  quot                                                      sw Stop            sw Reset             finally           Complete-BitsTransfer -BitsJob  job         Write-Progress -Activity  quot Completed quot  -Completed

User · Answer

Not that I m aware of   I wouldn t recommend using copy-item for this anyway   I don t think it has been designed to be robust like robocopy exe to support retry which you would want for extremely large file copies over the network

User · Answer

It seems like a much better solution to just use BitsTransfer  it seems to come OOTB on most Windows machines with PowerShell 2 0 or greater   Import-Module BitsTransfer Start-BitsTransfer -Source  Source -Destination  Destination -Description  Backup  -DisplayName  Backup

User · Answer

Alternativly this option uses the native windows progress bar       FOF CREATEPROGRESSDLG     amp H0 amp     objShell   New-Object -ComObject  Shell Application    objFolder    objShell NameSpace  DestLocation     objFolder CopyHere  srcFile   FOF CREATEPROGRESSDLG

User · Answer

cmd  c copy  z src dest   not pure PowerShell  but executable in PowerShell and it displays progress in percents

[powershell] Progress during large file copy (Copy-Item & Write-Progress?)

This recursive function copies files and directories recursively from source path to destination path

If file already exists on destination path, it copies them only with newer files.

Examples related to powershell

Examples related to progress-bar

Examples related to progress

Examples related to copying

Examples related to copy-item