How to compare 2 files fast using NET

Question

Typical approaches recommend reading the binary via FileStream and comparing it byte-by-byte    Would a checksum comparison such as CRC be faster  Are there any  NET libraries that can generate a checksum for a file

User · Answer

My experiments show that it definitely helps to call Stream.ReadByte() fewer times, but using BitConverter to package bytes does not make much difference against comparing bytes in a byte array.

So it is possible to replace that "Math.Ceiling and iterations" loop in the comment above with the simplest one:

            for (int i = 0; i < count1; i++)
            {
                if (buffer1[i] != buffer2[i])
                    return false;
            }

I guess it has to do with the fact that BitConverter.ToInt64 needs to do a bit of work (check arguments and then perform the bit shifting) before you compare and that ends up being the same amount of work as compare 8 bytes in two arrays.

User · Answer

In addition to Reed Copsey s answer     The worst case is where the two files are identical  In this case it s best to compare the files byte-by-byte  If if the two files are not identical  you can speed things up a bit by detecting sooner that they re not identical    For example  if the two files are of different length then you know they cannot be identical  and you don t even have to compare their actual content

User · Answer

The slowest possible method is to compare two files byte by byte   The fastest I ve been able to come up with is a similar comparison  but instead of one byte at a time  you would use an array of bytes sized to Int64  and then compare the resulting numbers   Here s what I came up with       const int BYTES TO READ   sizeof Int64        static bool FilesAreEqual FileInfo first  FileInfo second                if  first Length    second Length              return false           if  string Equals first FullName  second FullName  StringComparison OrdinalIgnoreCase               return true           int iterations    int Math Ceiling  double first Length   BYTES TO READ            using  FileStream fs1   first OpenRead            using  FileStream fs2   second OpenRead                          byte   one   new byte BYTES TO READ               byte   two   new byte BYTES TO READ                for  int i   0  i  lt  iterations  i                                   fs1 Read one  0  BYTES TO READ                    fs2 Read two  0  BYTES TO READ                    if  BitConverter ToInt64 one 0     BitConverter ToInt64 two 0                       return false                                   return true          In my testing  I was able to see this outperform a straightforward ReadByte   scenario by almost 3 1   Averaged over 1000 runs  I got this method at 1063ms  and the method below  straightforward byte by byte comparison  at 3031ms   Hashing always came back sub-second at around an average of 865ms   This testing was with an  100MB video file   Here s the ReadByte and hashing methods I used  for comparison purposes       static bool FilesAreEqual OneByte FileInfo first  FileInfo second                if  first Length    second Length              return false           if  string Equals first FullName  second FullName  StringComparison OrdinalIgnoreCase               return true           using  FileStream fs1   first OpenRead            using  FileStream fs2   second OpenRead                          for  int i   0  i  lt  first Length  i                                  if  fs1 ReadByte      fs2 ReadByte                        return false                                   return true             static bool FilesAreEqual Hash FileInfo first  FileInfo second                byte   firstHash   MD5 Create   ComputeHash first OpenRead             byte   secondHash   MD5 Create   ComputeHash second OpenRead              for  int i 0  i lt firstHash Length  i                          if  firstHash i     secondHash i                   return false                    return true

User · Answer

If you only need to compare two files  I guess the fastest way would be  in C  I don t know if it s applicable to  NET    open both files f1  f2 get the respective file length l1  l2 if l1    l2 the files are different  stop mmap   both files use memcmp   on the mmap  ed files   OTOH  if you need to find if there are duplicate files in a set of N files  then the fastest way is undoubtedly using a hash to avoid N-way bit-by-bit comparisons

User · Answer

If the files are not too big  you can use   public static byte   ComputeFileHash string fileName        using  var stream   File OpenRead fileName           return System Security Cryptography MD5 Create   ComputeHash stream       It will only be feasible to compare hashes if the hashes are useful to store    Edited the code to something much cleaner

User · Answer

If you d o  decide you truly need a full byte-by-byte comparison  see other answers for discussion of hashing   then the easiest solution is    bull  for  System String  path names  public static bool AreFileContentsEqual String path1  String path2    gt                File ReadAllBytes path1  SequenceEqual File ReadAllBytes path2       bull  for  System IO FileInfo  instances  public static bool AreFileContentsEqual FileInfo fi1  FileInfo fi2    gt      fi1 Length    fi2 Length  amp  amp       fi1 Length    0    File ReadAllBytes fi1 FullName  SequenceEqual                          File ReadAllBytes fi2 FullName       Unlike some other posted answers  this is conclusively correct for any kind of file  binary  text  media  executable  etc   but as a full binary comparison  files that that differ only in  quot unimportant quot  ways  such as BOM  line-ending  character encoding  media metadata  whitespace  padding  source-code comments  etc   will always be considered not-equal  This code loads both files into memory entirely  so it should not be used for comparing truly gigantic files  Beyond that important caveat  full loading isn t really a penalty given the design of the  NET GC  because it s fundamentally optimized to keep small  short-lived allocations extremely cheap   and in fact could even be optimal when file sizes are expected to be less than 85K  because using a minimum of user code  as shown here  implies maximally delegating file performance issues to the CLR  BCL  and JIT to benefit from  e g   the latest design technology  system code  and adaptive runtime optimizations  Furthermore  for such workaday scenarios  concerns about the performance of byte-by-byte comparison via LINQ enumerators  as shown here  are moot  since hitting the disk a t  a l l  for file I O will dwarf  by several orders of magnitude  the benefits of the various memory-comparing alternatives  For example  even though SequenceEqual does in fact give us the  quot optimization quot  of abandoning on first mismatch  this hardly matters after having already fetched the files  contents  each fully necessary to confirm the match

User · Answer

I think there are applications where  hash  is faster than comparing byte by byte  If you need to compare a file with others or have a thumbnail of a photo that can change  It depends on where and how it is using   private bool CompareFilesByte string file1  string file2        using  var fs1   new FileStream file1  FileMode Open       using  var fs2   new FileStream file2  FileMode Open                 if  fs1 Length    fs2 Length  return false          int b1  b2          do                       b1   fs1 ReadByte                b2   fs2 ReadByte                if  b1    b2    b1  lt  0  return false                    while  b1  gt   0             return true     private string HashFile string file        using  var fs   new FileStream file  FileMode Open       using  var reader   new BinaryReader fs                 var hash   new SHA512CryptoServiceProvider            hash ComputeHash reader ReadBytes  int file Length            return Convert ToBase64String hash Hash            private bool CompareFilesWithHash string file1  string file2        var str1   HashFile file1       var str2   HashFile file2       return str1    str2      Here  you can get what is the fastest   var sw   new Stopwatch    sw Start    var compare1   CompareFilesWithHash receiveLogPath  logPath   sw Stop    Debug WriteLine string Format  Compare using Hash  0    sw ElapsedTicks    sw Reset    sw Start    var compare2   CompareFilesByte receiveLogPath  logPath   sw Stop    Debug WriteLine string Format  Compare byte-byte  0    sw ElapsedTicks      Optionally  we can save the hash in a database   Hope this can help

User · Answer

A checksum comparison will most likely be slower than a byte-by-byte comparison   In order to generate a checksum  you ll need to load each byte of the file  and perform processing on it   You ll then have to do this on the second file   The processing will almost definitely be slower than the comparison check   As for generating a checksum  You can do this easily with the cryptography classes   Here s a short example of generating an MD5 checksum with C    However  a checksum may be faster and make more sense if you can pre-compute the checksum of the  test  or  base  case   If you have an existing file  and you re checking to see if a new file is the same as the existing one  pre-computing the checksum on your  existing  file would mean only needing to do the DiskIO one time  on the new file   This would likely be faster than a byte-by-byte comparison

User · Answer

Another improvement on large files with identical length  might be to not read the files sequentially  but rather compare more or less random blocks   You can use multiple threads  starting on different positions in the file and comparing either forward or backwards   This way you can detect changes at the middle end of the file  faster than you would get there using a sequential approach

User · Answer

Edit  This method would not work for comparing binary files   In  NET 4 0  the File class has the following two new methods   public static IEnumerable lt string gt  ReadLines string path  public static IEnumerable lt string gt  ReadLines string path  Encoding encoding    Which means you could use   bool same   File ReadLines path1  SequenceEqual File ReadLines path2

User · Answer

This I have found works well comparing first the length without reading data and then comparing the read byte sequence  private static bool IsFileIdentical string a  string b                   if  new FileInfo a  Length    new FileInfo b  Length  return false     return  File ReadAllBytes a  SequenceEqual File ReadAllBytes b

User · Answer

It s getting even faster if you don t read in small 8 byte chunks but put a loop around  reading a larger chunk  I reduced the average comparison time to 1 4       public static bool FilesContentsAreEqual FileInfo fileInfo1  FileInfo fileInfo2                bool result           if  fileInfo1 Length    fileInfo2 Length                        result   false                    else                       using  var file1   fileInfo1 OpenRead                                  using  var file2   fileInfo2 OpenRead                                          result   StreamsContentsAreEqual file1  file2                                                      return result             private static bool StreamsContentsAreEqual Stream stream1  Stream stream2                const int bufferSize   1024   sizeof Int64           var buffer1   new byte bufferSize           var buffer2   new byte bufferSize            while  true                        int count1   stream1 Read buffer1  0  bufferSize               int count2   stream2 Read buffer2  0  bufferSize                if  count1    count2                                return false                             if  count1    0                                return true                             int iterations    int Math Ceiling  double count1   sizeof Int64                for  int i   0  i  lt  iterations  i                                  if  BitConverter ToInt64 buffer1  i   sizeof Int64      BitConverter ToInt64 buffer2  i   sizeof Int64                                          return false

User · Answer

Here are some utility functions that allow you to determine if two files  or two streams  contain identical data   I have provided a  fast  version that is multi-threaded as it compares byte arrays  each buffer filled from what s been read in each file  in different threads using Tasks   As expected  it s much faster  around 3x faster  but it consumes more CPU  because it s multi threaded  and more memory  because it needs two byte array buffers per comparison thread        public static bool AreFilesIdenticalFast string path1  string path2                return AreFilesIdentical path1  path2  AreStreamsIdenticalFast              public static bool AreFilesIdentical string path1  string path2                return AreFilesIdentical path1  path2  AreStreamsIdentical              public static bool AreFilesIdentical string path1  string path2  Func lt Stream  Stream  bool gt  areStreamsIdentical                if  path1    null              throw new ArgumentNullException nameof path1             if  path2    null              throw new ArgumentNullException nameof path2             if  areStreamsIdentical    null              throw new ArgumentNullException nameof path2             if   File Exists path1      File Exists path2               return false           using  var thisFile   new FileStream path1  FileMode Open  FileAccess Read  FileShare ReadWrite                         using  var valueFile   new FileStream path2  FileMode Open  FileAccess Read  FileShare ReadWrite                                 if  valueFile Length    thisFile Length                      return false                   if   areStreamsIdentical thisFile  valueFile                       return false                                  return true             public static bool AreStreamsIdenticalFast Stream stream1  Stream stream2                if  stream1    null              throw new ArgumentNullException nameof stream1             if  stream2    null              throw new ArgumentNullException nameof stream2             const int bufsize   80000     80000 is below LOH  85000           var tasks   new List lt Task lt bool gt  gt             do                          consumes more memory  two buffers for each tasks              var buffer1   new byte bufsize               var buffer2   new byte bufsize                int read1   stream1 Read buffer1  0  buffer1 Length               if  read1    0                                int read3   stream2 Read buffer2  0  1                   if  read3    0     not eof                     return false                   break                                both stream read could return different counts             int read2   0              do                               int read3   stream2 Read buffer2  read2  read1 - read2                   if  read3    0                      return false                   read2    read3                            while  read2  lt  read1                   consumes more cpu             var task   Task Run      gt                                return IsSame buffer1  buffer2                               tasks Add task                     while  true            Task WaitAll tasks ToArray             return  tasks Any t   gt   t Result              public static bool AreStreamsIdentical Stream stream1  Stream stream2                if  stream1    null              throw new ArgumentNullException nameof stream1             if  stream2    null              throw new ArgumentNullException nameof stream2             const int bufsize   80000     80000 is below LOH  85000          var buffer1   new byte bufsize           var buffer2   new byte bufsize            var tasks   new List lt Task lt bool gt  gt             do                       int read1   stream1 Read buffer1  0  buffer1 Length               if  read1    0                  return stream2 Read buffer2  0  1     0     check not eof                 both stream read could return different counts             int read2   0              do                               int read3   stream2 Read buffer2  read2  read1 - read2                   if  read3    0                      return false                   read2    read3                            while  read2  lt  read1                if   IsSame buffer1  buffer2                   return false                    while  true              public static bool IsSame byte   bytes1  byte   bytes2                if  bytes1    null              throw new ArgumentNullException nameof bytes1             if  bytes2    null              throw new ArgumentNullException nameof bytes2             if  bytes1 Length    bytes2 Length              return false           for  int i   0  i  lt  bytes1 Length  i                          if  bytes1 i     bytes2 i                   return false                    return true

User · Answer

Something  hopefully  reasonably efficient   public class FileCompare       public static bool FilesEqual string fileName1  string fileName2                return FilesEqual new FileInfo fileName1   new FileInfo fileName2                    lt summary gt                    lt  summary gt           lt param name  file1  gt  lt  param gt           lt param name  file2  gt  lt  param gt           lt param name  bufferSize  gt 8kb seemed like a good default lt  param gt           lt returns gt  lt  returns gt      public static bool FilesEqual FileInfo file1  FileInfo file2  int bufferSize   8192                if   file1 Exists     file2 Exists    file1 Length    file2 Length  return false           var buffer1   new byte bufferSize           var buffer2   new byte bufferSize            using  var stream1   file1 Open FileMode Open  FileAccess Read  FileShare Read                         using  var stream2   file2 Open FileMode Open  FileAccess Read  FileShare Read                                  while  true                                        var bytesRead1   stream1 Read buffer1  0  bufferSize                       var bytesRead2   stream2 Read buffer2  0  bufferSize                        if  bytesRead1    bytesRead2  return false                      if  bytesRead1    0  return true                      if   ArraysEqual buffer1  buffer2  bytesRead1   return false                                                            lt summary gt                    lt  summary gt           lt param name  array1  gt  lt  param gt           lt param name  array2  gt  lt  param gt           lt param name  bytesToCompare  gt  0 means compare entire arrays lt  param gt           lt returns gt  lt  returns gt      public static bool ArraysEqual byte   array1  byte   array2  int bytesToCompare   0                if  array1 Length    array2 Length  return false           var length    bytesToCompare    0    array1 Length   bytesToCompare          var tailIdx   length - length   sizeof Int64              check in 8 byte chunks         for  var i   0  i  lt  tailIdx  i    sizeof Int64                         if  BitConverter ToInt64 array1  i     BitConverter ToInt64 array2  i   return false                       check the remainder of the array  always shorter than 8 bytes         for  var i   tailIdx  i  lt  length  i                          if  array1 i     array2 i   return false                     return true

User · Answer

My answer is a derivative of  lars but fixes the bug in the call to Stream Read  I also add some fast path checking that other answers had  and input validation  In short  this should be the answer   using System  using System IO   namespace ConsoleApp4       class Program               static void Main string   args                        var fi1   new FileInfo args 0                var fi2   new FileInfo args 1                Console WriteLine FilesContentsAreEqual fi1  fi2                       public static bool FilesContentsAreEqual FileInfo fileInfo1  FileInfo fileInfo2                        if  fileInfo1    null                                throw new ArgumentNullException nameof fileInfo1                               if  fileInfo2    null                                throw new ArgumentNullException nameof fileInfo2                               if  string Equals fileInfo1 FullName  fileInfo2 FullName  StringComparison OrdinalIgnoreCase                                 return true                             if  fileInfo1 Length    fileInfo2 Length                                return false                            else                               using  var file1   fileInfo1 OpenRead                                          using  var file2   fileInfo2 OpenRead                                                  return StreamsContentsAreEqual file1  file2                                                                            private static int ReadFullBuffer Stream stream  byte   buffer                        int bytesRead   0              while  bytesRead  lt  buffer Length                                int read   stream Read buffer  bytesRead  buffer Length - bytesRead                   if  read    0                                           Reached end of stream                      return bytesRead                                     bytesRead    read                             return bytesRead                     private static bool StreamsContentsAreEqual Stream stream1  Stream stream2                        const int bufferSize   1024   sizeof Int64               var buffer1   new byte bufferSize               var buffer2   new byte bufferSize                while  true                                int count1   ReadFullBuffer stream1  buffer1                   int count2   ReadFullBuffer stream2  buffer2                    if  count1    count2                                        return false                                     if  count1    0                                        return true                                     int iterations    int Math Ceiling  double count1   sizeof Int64                    for  int i   0  i  lt  iterations  i                                          if  BitConverter ToInt64 buffer1  i   sizeof Int64      BitConverter ToInt64 buffer2  i   sizeof Int64                                                  return false                                                                            Or if you want to be super-awesome  you can use the async variant   using System  using System IO  using System Threading Tasks   namespace ConsoleApp4       class Program               static void Main string   args                        var fi1   new FileInfo args 0                var fi2   new FileInfo args 1                Console WriteLine FilesContentsAreEqualAsync fi1  fi2  GetAwaiter   GetResult                        public static async Task lt bool gt  FilesContentsAreEqualAsync FileInfo fileInfo1  FileInfo fileInfo2                        if  fileInfo1    null                                throw new ArgumentNullException nameof fileInfo1                               if  fileInfo2    null                                throw new ArgumentNullException nameof fileInfo2                               if  string Equals fileInfo1 FullName  fileInfo2 FullName  StringComparison OrdinalIgnoreCase                                 return true                             if  fileInfo1 Length    fileInfo2 Length                                return false                            else                               using  var file1   fileInfo1 OpenRead                                          using  var file2   fileInfo2 OpenRead                                                  return await StreamsContentsAreEqualAsync file1  file2  ConfigureAwait false                                                                            private static async Task lt int gt  ReadFullBufferAsync Stream stream  byte   buffer                        int bytesRead   0              while  bytesRead  lt  buffer Length                                int read   await stream ReadAsync buffer  bytesRead  buffer Length - bytesRead  ConfigureAwait false                   if  read    0                                           Reached end of stream                      return bytesRead                                     bytesRead    read                             return bytesRead                     private static async Task lt bool gt  StreamsContentsAreEqualAsync Stream stream1  Stream stream2                        const int bufferSize   1024   sizeof Int64               var buffer1   new byte bufferSize               var buffer2   new byte bufferSize                while  true                                int count1   await ReadFullBufferAsync stream1  buffer1  ConfigureAwait false                   int count2   await ReadFullBufferAsync stream2  buffer2  ConfigureAwait false                    if  count1    count2                                        return false                                     if  count1    0                                        return true                                     int iterations    int Math Ceiling  double count1   sizeof Int64                    for  int i   0  i  lt  iterations  i                                          if  BitConverter ToInt64 buffer1  i   sizeof Int64      BitConverter ToInt64 buffer2  i   sizeof Int64                                                  return false

User · Answer

Yet another answer  derived from  chsh   MD5 with usings and shortcuts for file same  file not exists and differing lengths        lt summary gt      Performs an md5 on the content of both files and returns true if     they match      lt  summary gt       lt param name  file1  gt first file lt  param gt       lt param name  file2  gt second file lt  param gt       lt returns gt true if the contents of the two files is the same  false otherwise lt  returns gt  public static bool IsSameContent string file1  string file2        if  file1    file2          return true       FileInfo file1Info   new FileInfo file1       FileInfo file2Info   new FileInfo file2        if   file1Info Exists  amp  amp   file2Info Exists         return true      if   file1Info Exists  amp  amp  file2Info Exists          return false      if  file1Info Exists  amp  amp   file2Info Exists          return false      if  file1Info Length    file2Info Length          return false       using  FileStream file1Stream   file1Info OpenRead        using  FileStream file2Stream   file2Info OpenRead                   byte   firstHash   MD5 Create   ComputeHash file1Stream           byte   secondHash   MD5 Create   ComputeHash file2Stream           for  int i   0  i  lt  firstHash Length  i                          if  i gt  secondHash Length  firstHash i     secondHash i                   return false                    return true

User · Answer

The only thing that might make a checksum comparison slightly faster than a byte-by-byte comparison is the fact that you are reading one file at a time  somewhat reducing the seek time for the disk head  That slight gain may however very well be eaten up by the added time of calculating the hash   Also  a checksum comparison of course only has any chance of being faster if the files are identical  If they are not  a byte-by-byte comparison would end at the first difference  making it a lot faster   You should also consider that a hash code comparison only tells you that it s very likely that the files are identical  To be 100  certain you need to do a byte-by-byte comparison   If the hash code for example is 32 bits  you are about 99 99999998  certain that the files are identical if the hash codes match  That is close to 100   but if you truly need 100  certainty  that s not it

User · Answer

Honestly  I think you need to prune your search tree down as much as possible   Things to check before going byte-by-byte    Are sizes the same  Is the last byte in file A different than file B   Also  reading large blocks at a time will be more efficient since drives read sequential bytes more quickly  Going byte-by-byte causes not only far more system calls  but it causes the read head of a traditional hard drive to seek back and forth more often if both files are on the same drive   Read chunk A and chunk B into a byte buffer  and compare them  do NOT use Array Equals  see comments   Tune the size of the blocks until you hit what you feel is a good trade off between memory and performance  You could also multi-thread the comparison  but don t multi-thread the disk reads

[c#] How to compare 2 files fast using .NET?

Examples related to c#

Examples related to file

Examples related to compare

Examples related to checksum