Get all files and directories in specific path fast

Question

I am creating a backup application where c  scans a directory  Before I use to have something like this in order to get all the files and subfiles in a directory   DirectoryInfo di   new DirectoryInfo  A       var directories  di GetFiles      SearchOption AllDirectories    foreach  FileInfo d in directories             Add files to a list so that later they can be compared to see if each file           needs to be copid or not     The only problem with that is that sometimes a file could not be accessed and I get several errors  an example of an error that I get is   As a result I created a recursive method that will scan all files in the current directory  If there where directories in that directory then the method will be called again passing that directory  The nice thing about this method is that I could place the files inside a try catch block giving me the option to add those files to a List if there where no errors and adding the directory to another list if I had errors   try       files   di GetFiles searchPattern  SearchOption TopDirectoryOnly                    catch          info of this folder was not able to get      lstFilesErrors Add sDir di         return      So this method works great the only problem is that when I scan a large directory it takes to much times  How could I speed up this process    My actual method is this in case you need it    private void startScan DirectoryInfo di          lstFilesErrors is a list of MyFile objects        I created that class because I wanted to store more specific information        about a file such as its comparePath name and other properties that I need         in order to compare it with another list         lstFiles is a list of MyFile objects that store all the files        that are contained in path that I want to scan      FileInfo   files   null      DirectoryInfo   directories   null      string searchPattern               try               files   di GetFiles searchPattern  SearchOption TopDirectoryOnly                            catch                 info of this folder was not able to get         lstFilesErrors Add sDir di            return                if there are files in the directory then add those files to the list     if  files    null                foreach  FileInfo f in files                        lstFiles Add sFile f                          try               directories   di GetDirectories searchPattern  SearchOption TopDirectoryOnly             catch               lstFilesErrors Add sDir di            return                if that directory has more directories then add them to the list then         execute this function     if  directories    null          foreach  DirectoryInfo d in directories                        FileInfo   subFiles   null              DirectoryInfo   subDir   null               bool isThereAnError   false               try                               subFiles   d GetFiles                    subDir   d GetDirectories                               catch                               isThereAnError   true                                                                             if  isThereAnError                  lstFilesErrors Add sDir d                else                               lstFiles Add sDir d                    startScan d                                  Ant the problem if I try to handle the exception with something like   DirectoryInfo di   new DirectoryInfo  A       FileInfo   directories   null              try                               directories   di GetFiles      SearchOption AllDirectories                              catch  UnauthorizedAccessException e                                Console WriteLine  There was an error with UnauthorizedAccessException                              catch                               Console WriteLine  There was antother error                    Is that if an exception occurs then I get no files

User · Answer

There is a long history of the  NET file enumeration methods being slow   The issue is there is not an instantaneous way of enumerating large directory structures   Even the accepted answer here has its issues with GC allocations   The best I ve been able to do is wrapped up in my library and exposed as the FindFile  source  class in the CSharpTest Net IO namespace   This class can enumerate files and folders without unneeded GC allocations and string marshalling   The usage is simple enough  and the RaiseOnAccessDenied property will skip the directories and files the user does not have access to       private static long SizeOf string directory                var fcounter   new CSharpTest Net IO FindFile directory       true  true  true           fcounter RaiseOnAccessDenied   false           long size   0  total   0          fcounter FileFound                 o  e    gt                                if   e IsDirectory                                        Interlocked Increment ref total                       size    e Length                                            Stopwatch sw   Stopwatch StartNew            fcounter Find            Console WriteLine  Enumerated  0 n0  files totaling  1 n0  bytes in  2 n3  seconds                              total  size  sw Elapsed TotalSeconds           return size          For my local C   drive this outputs the following      Enumerated 810 046 files totaling 307 707 792 662 bytes in 232 876 seconds    Your mileage may vary by drive speed  but this is the fastest method I ve found of enumerating files in managed code  The event parameter is a mutating class of type FindFile FileFoundEventArgs so be sure you do not keep a reference to it as it s values will change for each event raised

User · Answer

In  NET 4 0 there s the Directory EnumerateFiles method which returns an IEnumerable lt string gt  and is not loading all the files in memory  It s only once you start iterating over the returned collection that files will be returned and exceptions could be handled

User · Answer

copied this piece from my other answer in your other question   Show progress when searching all files in a directory  Fast files enumeration  Of course  as you already know  there are a lot of ways of doing the enumeration itself    but none will be instantaneous  You could try using the USN Journal of the file system to do the scan  Take a look at this project in CodePlex  MFT Scanner in VB NET    it found all the files in my IDE SATA  not SSD  drive in less than 15 seconds  and found 311000 files   You will have to filter the files by path  so that only the files inside the path you are looking are returned  But that is the easy part of the job

User · Answer

You can use this to get all directories and sub-directories  Then simply loop through to process the files   string   folders   System IO Directory GetDirectories   C  My Sample Path        System IO SearchOption AllDirectories    foreach string f in folders         call some function to get all files in folder

User · Answer

This method is much faster  You can only tel when placing a lot of files in a directory  My A   external hard drive contains almost 1 terabit so it makes a big difference when dealing with a lot of files     static void Main string   args        DirectoryInfo di   new DirectoryInfo  A           FullDirList di            Console WriteLine  Done        Console Read       static List lt FileInfo gt  files   new List lt FileInfo gt         List that will hold the files and subfiles in path static List lt DirectoryInfo gt  folders   new List lt DirectoryInfo gt        List that hold direcotries that cannot be accessed static void FullDirList DirectoryInfo dir  string searchPattern           Console WriteLine  Directory  0    dir FullName          list the files     try               foreach  FileInfo f in dir GetFiles searchPattern                           Console WriteLine  File  0    f FullName               files Add f                                           catch               Console WriteLine  Directory  0    n could not be accessed       dir FullName                           return      We alredy got an error trying to access dir so dont try to access it again               process each directory        If I have been able to see the files in the directory I should also be able         to look at its directories so I dont think I should place this in a try catch block     foreach  DirectoryInfo d in dir GetDirectories                  folders Add d           FullDirList d  searchPattern                                  By the way I got this thanks to your comment Jim Mischel

User · Answer

Maybe it will be helpfull for you  You could use  DirectoryInfo EnumerateFiles  method and handle UnauthorizedAccessException as you need    using System  using System IO   class Program       static void Main string   args                DirectoryInfo diTop   new DirectoryInfo   d              try                       foreach  var fi in diTop EnumerateFiles                                  try                                          Display each file over 10 MB                       if  fi Length  gt  10000000                                                Console WriteLine   0  t t 1    fi FullName  fi Length ToString  N0                                                             catch  UnauthorizedAccessException UnAuthTop                                        Console WriteLine   0    UnAuthTop Message                                                foreach  var di in diTop EnumerateDirectories                                     try                                       foreach  var fi in di EnumerateFiles      SearchOption AllDirectories                                                 try                                                          Display each file over 10 MB                               if  fi Length  gt  10000000                                                                Console WriteLine   0  t t 1     fi FullName  fi Length ToString  N0                                                                                     catch  UnauthorizedAccessException UnAuthFile                                                        Console WriteLine  UnAuthFile   0    UnAuthFile Message                                                                                     catch  UnauthorizedAccessException UnAuthSubDir                                        Console WriteLine  UnAuthSubDir   0    UnAuthSubDir Message                                                     catch  DirectoryNotFoundException DirNotFound                        Console WriteLine   0    DirNotFound Message                     catch  UnauthorizedAccessException UnAuthDir                        Console WriteLine  UnAuthDir   0    UnAuthDir Message                     catch  PathTooLongException LongPath                        Console WriteLine   0    LongPath Message

User · Answer

I know this is old  but     Another option may be to use the FileSystemWatcher like so   void SomeMethod         System IO FileSystemWatcher m Watcher   new System IO FileSystemWatcher        m Watcher Path   path      m Watcher Filter              m Watcher NotifyFilter   m Watcher NotifyFilter   NotifyFilters LastAccess   NotifyFilters LastWrite   NotifyFilters FileName   NotifyFilters DirectoryName      m Watcher Created    new FileSystemEventHandler OnChanged       m Watcher EnableRaisingEvents   true     private void OnChanged object sender  FileSystemEventArgs e                string path   e FullPath           lock  listLock                        pathsToUpload Add path                     This would allow you to watch the directories for file changes with an extremely lightweight process  that you could then use to store the names of the files that changed so that you could back them up at the appropriate time

[c#] Get all files and directories in specific path fast

Examples related to c#

Examples related to performance

Examples related to fileinfo