[c#] How best to read a File into List<string>

[Edit]

If you are doing this to trim the beginning of a log file, you can avoid loading the entire file by doing something like this:

// count the number of lines in the file
int count = 0;
using (var sr = new StreamReader("file.txt"))
{
    while (sr.ReadLine() != null) 
        count++;
}

// skip first (LOG_MAX - count) lines
count = LOG_MAX - count;
using (var sr = new StreamReader("file.txt"))
using (var sw = new StreamWriter("output.txt"))
{
    // skip several lines
    while (count > 0 && sr.ReadLine() != null) 
        count--;

    // continue copying
    string line = "";
    while ((line = sr.ReadLine()) != null)
        sw.WriteLine(line);
}

First of all, since File.ReadAllLines loads the entire file into a string array (string[]), copying to a list is redundant.

Second, you must understand that a List is implemented using a dynamic array under the hood. This means that CLR will need to allocate and copy several arrays until it can accommodate the entire file. Since the file is already on disk, you might consider trading speed for memory and working on disk data directly, or processing it in smaller chunks.

1. If you need to load it entirely in memory, at least try to leave in an array:

    string[] lines = File.ReadAllLines("file.txt");
   

2. If it really needs to be a List, load lines one by one:

    List<string> lines = new List<string>();
    using (var sr = new StreamReader("file.txt"))
    {
         while (sr.Peek() >= 0)
             lines.Add(sr.ReadLine());
    }
   

   Note: List<T> has a constructor which accepts a capacity parameter. If you know the number of lines in advance, you can prevent multiple allocations by preallocating the array in advance:

    List<string> lines = new List<string>(NUMBER_OF_LINES);
   

3. Even better, avoid storing the entire file in memory and process it "on the fly":

    using (var sr = new StreamReader("file.txt"))
    {
         string line;
         while ((line = sr.ReadLine()) != null) 
         {
             // process the file line by line
         }
    }