[c#] Split a List into smaller lists of N size

I am attempting to split a list into a series of smaller lists.

My Problem: My function to split lists doesn't split them into lists of the correct size. It should split them into lists of size 30 but instead it splits them into lists of size 114?

How can I make my function split a list into X number of Lists of size 30 or less?

public static List<List<float[]>> splitList(List <float[]> locations, int nSize=30) 
{       
    List<List<float[]>> list = new List<List<float[]>>();

    for (int i=(int)(Math.Ceiling((decimal)(locations.Count/nSize))); i>=0; i--) {
        List <float[]> subLocat = new List <float[]>(locations); 

        if (subLocat.Count >= ((i*nSize)+nSize))
            subLocat.RemoveRange(i*nSize, nSize);
        else subLocat.RemoveRange(i*nSize, subLocat.Count-(i*nSize));

        Debug.Log ("Index: "+i.ToString()+", Size: "+subLocat.Count.ToString());
        list.Add (subLocat);
    }

    return list;
}

If I use the function on a list of size 144 then the output is:

Index: 4, Size: 120
Index: 3, Size: 114
Index: 2, Size: 114
Index: 1, Size: 114
Index: 0, Size: 114

Addition after very useful comment of mhand at the end

Original answer

Although most solutions might work, I think they are not very efficiently. Suppose if you only want the first few items of the first few chunks. Then you wouldn't want to iterate over all (zillion) items in your sequence.

The following will at utmost enumerate twice: once for the Take and once for the Skip. It won't enumerate over any more elements than you will use:

public static IEnumerable<IEnumerable<TSource>> ChunkBy<TSource>
    (this IEnumerable<TSource> source, int chunkSize)
{
    while (source.Any())                     // while there are elements left
    {   // still something to chunk:
        yield return source.Take(chunkSize); // return a chunk of chunkSize
        source = source.Skip(chunkSize);     // skip the returned chunk
    }
}

How many times will this Enumerate the sequence?

Suppose you divide your source into chunks of chunkSize. You enumerate only the first N chunks. From every enumerated chunk you'll only enumerate the first M elements.

While(source.Any())
{
     ...
}

the Any will get the Enumerator, do 1 MoveNext() and returns the returned value after Disposing the Enumerator. This will be done N times

yield return source.Take(chunkSize);

According to the reference source this will do something like:

public static IEnumerable<TSource> Take<TSource>(this IEnumerable<TSource> source, int count)
{
    return TakeIterator<TSource>(source, count);
}

static IEnumerable<TSource> TakeIterator<TSource>(IEnumerable<TSource> source, int count)
{
    foreach (TSource element in source)
    {
        yield return element;
        if (--count == 0) break;
    }
}

This doesn't do a lot until you start enumerating over the fetched Chunk. If you fetch several Chunks, but decide not to enumerate over the first Chunk, the foreach is not executed, as your debugger will show you.

If you decide to take the first M elements of the first chunk then the yield return is executed exactly M times. This means:

• get the enumerator
• call MoveNext() and Current M times.
• Dispose the enumerator

After the first chunk has been yield returned, we skip this first Chunk:

source = source.Skip(chunkSize);

Once again: we'll take a look at reference source to find the skipiterator

static IEnumerable<TSource> SkipIterator<TSource>(IEnumerable<TSource> source, int count)
{
    using (IEnumerator<TSource> e = source.GetEnumerator()) 
    {
        while (count > 0 && e.MoveNext()) count--;
        if (count <= 0) 
        {
            while (e.MoveNext()) yield return e.Current;
        }
    }
}

As you see, the SkipIterator calls MoveNext() once for every element in the Chunk. It doesn't call Current.

So per Chunk we see that the following is done:

• Any(): GetEnumerator; 1 MoveNext(); Dispose Enumerator;

• Take():

  • nothing if the content of the chunk is not enumerated.

  • If the content is enumerated: GetEnumerator(), one MoveNext and one Current per enumerated item, Dispose enumerator;

  • Skip(): for every chunk that is enumerated (NOT the contents of the chunk): GetEnumerator(), MoveNext() chunkSize times, no Current! Dispose enumerator

If you look at what happens with the enumerator, you'll see that there are a lot of calls to MoveNext(), and only calls to Current for the TSource items you actually decide to access.

If you take N Chunks of size chunkSize, then calls to MoveNext()

• N times for Any()
• not yet any time for Take, as long as you don't enumerate the Chunks
• N times chunkSize for Skip()

If you decide to enumerate only the first M elements of every fetched chunk, then you need to call MoveNext M times per enumerated Chunk.

The total

MoveNext calls: N + N*M + N*chunkSize
Current calls: N*M; (only the items you really access)

So if you decide to enumerate all elements of all chunks:

MoveNext: numberOfChunks + all elements + all elements = about twice the sequence
Current: every item is accessed exactly once

Whether MoveNext is a lot of work or not, depends on the type of source sequence. For lists and arrays it is a simple index increment, with maybe an out of range check.

But if your IEnumerable is the result of a database query, make sure that the data is really materialized on your computer, otherwise the data will be fetched several times. DbContext and Dapper will properly transfer the data to local process before it can be accessed. If you enumerate the same sequence several times it is not fetched several times. Dapper returns an object that is a List, DbContext remembers that the data is already fetched.

It depends on your Repository whether it is wise to call AsEnumerable() or ToLists() before you start to divide the items in Chunks

I would suggest to use this extension method to chunk the source list to the sub-lists by specified chunk size:

/// <summary>
/// Helper methods for the lists.
/// </summary>
public static class ListExtensions
{
    public static List<List<T>> ChunkBy<T>(this List<T> source, int chunkSize) 
    {
        return source
            .Select((x, i) => new { Index = i, Value = x })
            .GroupBy(x => x.Index / chunkSize)
            .Select(x => x.Select(v => v.Value).ToList())
            .ToList();
    }
}

For example, if you chunk the list of 18 items by 5 items per chunk, it gives you the list of 4 sub-lists with the following items inside: 5-5-5-3.

public static IEnumerable<IEnumerable<T>> SplitIntoSets<T>
    (this IEnumerable<T> source, int itemsPerSet) 
{
    var sourceList = source as List<T> ?? source.ToList();
    for (var index = 0; index < sourceList.Count; index += itemsPerSet)
    {
        yield return sourceList.Skip(index).Take(itemsPerSet);
    }
}

How about this one? The idea was to use only one loop. And, who knows, maybe you're using only IList implementations thorough your code and you don't want to cast to List.

private IEnumerable<IList<T>> SplitList<T>(IList<T> list, int totalChunks)
{
    IList<T> auxList = new List<T>();
    int totalItems = list.Count();

    if (totalChunks <= 0)
    {
        yield return auxList;
    }
    else 
    {
        for (int i = 0; i < totalItems; i++)
        {               
            auxList.Add(list[i]);           

            if ((i + 1) % totalChunks == 0)
            {
                yield return auxList;
                auxList = new List<T>();                
            }

            else if (i == totalItems - 1)
            {
                yield return auxList;
            }
        }
    }   
}

I find accepted answer (Serj-Tm) most robust, but I'd like to suggest a generic version.

public static List<List<T>> splitList<T>(List<T> locations, int nSize = 30)
{
    var list = new List<List<T>>();

    for (int i = 0; i < locations.Count; i += nSize)
    {
        list.Add(locations.GetRange(i, Math.Min(nSize, locations.Count - i)));
    }

    return list;
}

While plenty of the answers above do the job, they all fail horribly on a never ending sequence (or a really long sequence). The following is a completely on-line implementation which guarantees best time and memory complexity possible. We only iterate the source enumerable exactly once and use yield return for lazy evaluation. The consumer could throw away the list on each iteration making the memory footprint equal to that of the list w/ batchSize number of elements.

public static IEnumerable<List<T>> BatchBy<T>(this IEnumerable<T> enumerable, int batchSize)
{
    using (var enumerator = enumerable.GetEnumerator())
    {
        List<T> list = null;
        while (enumerator.MoveNext())
        {
            if (list == null)
            {
                list = new List<T> {enumerator.Current};
            }
            else if (list.Count < batchSize)
            {
                list.Add(enumerator.Current);
            }
            else
            {
                yield return list;
                list = new List<T> {enumerator.Current};
            }
        }

        if (list?.Count > 0)
        {
            yield return list;
        }
    }
}

EDIT: Just now realizing the OP asks about breaking a List<T> into smaller List<T>, so my comments regarding infinite enumerables aren't applicable to the OP, but may help others who end up here. These comments were in response to other posted solutions that do use IEnumerable<T> as an input to their function, yet enumerate the source enumerable multiple times.

One more

public static IList<IList<T>> SplitList<T>(this IList<T> list, int chunkSize)
{
    var chunks = new List<IList<T>>();
    List<T> chunk = null;
    for (var i = 0; i < list.Count; i++)
    {
        if (i % chunkSize == 0)
        {
            chunk = new List<T>(chunkSize);
            chunks.Add(chunk);
        }
        chunk.Add(list[i]);
    }
    return chunks;
}

List<int> orginalList =new List<int>(){1,2,3,4,5,6,7,8,9,10,12};
Dictionary<int,List<int>> dic = new Dictionary <int,List<int>> ();
int batchcount = orginalList.Count/2; //To List into two 2 parts if you 
 want three give three
List<int> lst = new List<int>();
for (int i=0;i<orginalList.Count; i++)
{
lst.Add(orginalList[i]);
if (i % batchCount == 0 && i!=0)
{
Dic.Add(threadId, lst);
lst = new List<int>();**strong text**
threadId++;
}
}
if(lst.Count>0)
Dic.Add(threadId, lst); //in case if any dayleft 
foreach(int BatchId in Dic.Keys)
{
  Console.Writeline("BatchId:"+BatchId);
  Console.Writeline('Batch Count:"+Dic[BatchId].Count);
}

I have a generic method that would take any types include float, and it's been unit-tested, hope it helps:

    /// <summary>
    /// Breaks the list into groups with each group containing no more than the specified group size
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="values">The values.</param>
    /// <param name="groupSize">Size of the group.</param>
    /// <returns></returns>
    public static List<List<T>> SplitList<T>(IEnumerable<T> values, int groupSize, int? maxCount = null)
    {
        List<List<T>> result = new List<List<T>>();
        // Quick and special scenario
        if (values.Count() <= groupSize)
        {
            result.Add(values.ToList());
        }
        else
        {
            List<T> valueList = values.ToList();
            int startIndex = 0;
            int count = valueList.Count;
            int elementCount = 0;

            while (startIndex < count && (!maxCount.HasValue || (maxCount.HasValue && startIndex < maxCount)))
            {
                elementCount = (startIndex + groupSize > count) ? count - startIndex : groupSize;
                result.Add(valueList.GetRange(startIndex, elementCount));
                startIndex += elementCount;
            }
        }


        return result;
    }

public static IEnumerable<IEnumerable<T>> Batch<T>(this IEnumerable<T> items, int maxItems)
{
    return items.Select((item, index) => new { item, index })
                .GroupBy(x => x.index / maxItems)
                .Select(g => g.Select(x => x.item));
}

how about:

while(locations.Any())
{    
    list.Add(locations.Take(nSize).ToList());
    locations= locations.Skip(nSize).ToList();
}

Based on Dimitry Pavlov answere I would remove .ToList(). And also avoid the anonymous class. Instead I like to use a struct which does not require a heap memory allocation. (A ValueTuple would also do job.)

public static IEnumerable<IEnumerable<TSource>> ChunkBy<TSource>(this IEnumerable<TSource> source, int chunkSize)
{
    if (source is null)
    {
        throw new ArgumentNullException(nameof(source));
    }
    if (chunkSize <= 0)
    {
        throw new ArgumentOutOfRangeException(nameof(chunkSize), chunkSize, "The argument must be greater than zero.");
    }

    return source
        .Select((x, i) => new ChunkedValue<TSource>(x, i / chunkSize))
        .GroupBy(cv => cv.ChunkIndex)
        .Select(g => g.Select(cv => cv.Value));
} 

[StructLayout(LayoutKind.Auto)]
[DebuggerDisplay("{" + nameof(ChunkedValue<T>.ChunkIndex) + "}: {" + nameof(ChunkedValue<T>.Value) + "}")]
private struct ChunkedValue<T>
{
    public ChunkedValue(T value, int chunkIndex)
    {
        this.ChunkIndex = chunkIndex;
        this.Value = value;
    }

    public int ChunkIndex { get; }

    public T Value { get; }
}

This can be used like the following which only iterates over the collection once and also does not allocate any significant memory.

int chunkSize = 30;
foreach (var chunk in collection.ChunkBy(chunkSize))
{
    foreach (var item in chunk)
    {
        // your code for item here.
    }
}

If a concrete list is actually needed then I would do it like this:

int chunkSize = 30;
var chunkList = new List<List<T>>();
foreach (var chunk in collection.ChunkBy(chunkSize))
{
    // create a list with the correct capacity to be able to contain one chunk
    // to avoid the resizing (additional memory allocation and memory copy) within the List<T>.
    var list = new List<T>(chunkSize);
    list.AddRange(chunk);
    chunkList.Add(list);
}

I had encountered this same need, and I used a combination of Linq's Skip() and Take() methods. I multiply the number I take by the number of iterations this far, and that gives me the number of items to skip, then I take the next group.

        var categories = Properties.Settings.Default.MovementStatsCategories;
        var items = summariesWithinYear
            .Select(s =>  s.sku).Distinct().ToList();

        //need to run by chunks of 10,000
        var count = items.Count;
        var counter = 0;
        var numToTake = 10000;

        while (count > 0)
        {
            var itemsChunk = items.Skip(numToTake * counter).Take(numToTake).ToList();
            counter += 1;

            MovementHistoryUtilities.RecordMovementHistoryStatsBulk(itemsChunk, categories, nLogger);

            count -= numToTake;
        }

Library MoreLinq have method called Batch

List<int> ids = new List<int>() { 1, 2, 3, 4, 5, 6, 7, 8, 9, 0 }; // 10 elements
int counter = 1;
foreach(var batch in ids.Batch(2))
{
    foreach(var eachId in batch)
    {
        Console.WriteLine("Batch: {0}, Id: {1}", counter, eachId);
    }
    counter++;
}

Result is

Batch: 1, Id: 1
Batch: 1, Id: 2
Batch: 2, Id: 3
Batch: 2, Id: 4
Batch: 3, Id: 5
Batch: 3, Id: 6
Batch: 4, Id: 7
Batch: 4, Id: 8
Batch: 5, Id: 9
Batch: 5, Id: 0

ids are splitted into 5 chunks with 2 elements.

Serj-Tm solution is fine, also this is the generic version as extension method for lists (put it into a static class):

public static List<List<T>> Split<T>(this List<T> items, int sliceSize = 30)
{
    List<List<T>> list = new List<List<T>>();
    for (int i = 0; i < items.Count; i += sliceSize)
        list.Add(items.GetRange(i, Math.Min(sliceSize, items.Count - i)));
    return list;
} 

public static List<List<T>> ChunkBy<T>(this List<T> source, int chunkSize)
    {           
        var result = new List<List<T>>();
        for (int i = 0; i < source.Count; i += chunkSize)
        {
            var rows = new List<T>();
            for (int j = i; j < i + chunkSize; j++)
            {
                if (j >= source.Count) break;
                rows.Add(source[j]);
            }
            result.Add(rows);
        }
        return result;
    }