Split a List into smaller lists of N size

Question

I am attempting to split a list into a series of smaller lists   My Problem  My function to split lists doesn t split them into lists of the correct size  It should split them into lists of size 30 but instead it splits them into lists of size 114   How can I make my function split a list into X number of Lists of size 30 or less   public static List lt List lt float   gt  gt  splitList List  lt float   gt  locations  int nSize 30                List lt List lt float   gt  gt  list   new List lt List lt float   gt  gt          for  int i  int  Math Ceiling  decimal  locations Count nSize     i gt  0  i--            List  lt float   gt  subLocat   new List  lt float   gt  locations             if  subLocat Count  gt     i nSize  nSize               subLocat RemoveRange i nSize  nSize           else subLocat RemoveRange i nSize  subLocat Count- i nSize             Debug Log   Index    i ToString      Size    subLocat Count ToString             list Add  subLocat              return list      If I use the function on a list of size 144 then the output is      Index  4  Size  120   Index  3  Size  114   Index  2  Size  114   Index  1  Size  114   Index  0  Size  114

User · Answer

I find accepted answer  Serj-Tm  most robust  but I d like to suggest a generic version   public static List lt List lt T gt  gt  splitList lt T gt  List lt T gt  locations  int nSize   30        var list   new List lt List lt T gt  gt          for  int i   0  i  lt  locations Count  i    nSize                list Add locations GetRange i  Math Min nSize  locations Count - i                return list

User · Answer

Serj-Tm solution is fine  also this is the generic version as extension method for lists  put it into a static class    public static List lt List lt T gt  gt  Split lt T gt  this List lt T gt  items  int sliceSize   30        List lt List lt T gt  gt  list   new List lt List lt T gt  gt         for  int i   0  i  lt  items Count  i    sliceSize          list Add items GetRange i  Math Min sliceSize  items Count - i         return list

User · Answer

public static List lt List lt T gt  gt  ChunkBy lt T gt  this List lt T gt  source  int chunkSize                           var result   new List lt List lt T gt  gt             for  int i   0  i  lt  source Count  i    chunkSize                        var rows   new List lt T gt                 for  int j   i  j  lt  i   chunkSize  j                                  if  j  gt   source Count  break                  rows Add source j                              result Add rows                     return result

User · Answer

One more  public static IList lt IList lt T gt  gt  SplitList lt T gt  this IList lt T gt  list  int chunkSize        var chunks   new List lt IList lt T gt  gt         List lt T gt  chunk   null      for  var i   0  i  lt  list Count  i                  if  i   chunkSize    0                        chunk   new List lt T gt  chunkSize               chunks Add chunk                     chunk Add list i              return chunks

User · Answer

Based on Dimitry Pavlov answere I would remove  ToList    And also avoid the anonymous class  Instead I like to use a struct which does not require a heap memory allocation   A ValueTuple would also do job    public static IEnumerable lt IEnumerable lt TSource gt  gt  ChunkBy lt TSource gt  this IEnumerable lt TSource gt  source  int chunkSize        if  source is null                throw new ArgumentNullException nameof source              if  chunkSize  lt   0                throw new ArgumentOutOfRangeException nameof chunkSize   chunkSize   The argument must be greater than zero                return source          Select  x  i    gt  new ChunkedValue lt TSource gt  x  i   chunkSize            GroupBy cv   gt  cv ChunkIndex           Select g   gt  g Select cv   gt  cv Value         StructLayout LayoutKind Auto    DebuggerDisplay       nameof ChunkedValue lt T gt  ChunkIndex             nameof ChunkedValue lt T gt  Value          private struct ChunkedValue lt T gt        public ChunkedValue T value  int chunkIndex                this ChunkIndex   chunkIndex          this Value   value             public int ChunkIndex   get         public T Value   get        This can be used like the following which only iterates over the collection once and also does not allocate any significant memory   int chunkSize   30  foreach  var chunk in collection ChunkBy chunkSize         foreach  var item in chunk                   your code for item here            If a concrete list is actually needed then I would do it like this   int chunkSize   30  var chunkList   new List lt List lt T gt  gt     foreach  var chunk in collection ChunkBy chunkSize            create a list with the correct capacity to be able to contain one chunk        to avoid the resizing  additional memory allocation and memory copy  within the List lt T gt       var list   new List lt T gt  chunkSize       list AddRange chunk       chunkList Add list

User · Answer

I had encountered this same need  and I used a combination of Linq s Skip   and Take   methods  I multiply the number I take by the number of iterations this far  and that gives me the number of items to skip  then I take the next group           var categories   Properties Settings Default MovementStatsCategories          var items   summariesWithinYear              Select s   gt   s sku  Distinct   ToList               need to run by chunks of 10 000         var count   items Count          var counter   0          var numToTake   10000           while  count  gt  0                        var itemsChunk   items Skip numToTake   counter  Take numToTake  ToList                counter    1               MovementHistoryUtilities RecordMovementHistoryStatsBulk itemsChunk  categories  nLogger                count -  numToTake

User · Answer

Library MoreLinq have method called Batch List lt int gt  ids   new List lt int gt      1  2  3  4  5  6  7  8  9  0       10 elements int counter   1  foreach var batch in ids Batch 2         foreach var eachId in batch                Console WriteLine  quot Batch   0   Id   1  quot   counter  eachId             counter       Result is Batch  1  Id  1 Batch  1  Id  2 Batch  2  Id  3 Batch  2  Id  4 Batch  3  Id  5 Batch  3  Id  6 Batch  4  Id  7 Batch  4  Id  8 Batch  5  Id  9 Batch  5  Id  0  ids are splitted into 5 chunks with 2 elements

User · Answer

public static IEnumerable lt IEnumerable lt T gt  gt  Batch lt T gt  this IEnumerable lt T gt  items  int maxItems        return items Select  item  index    gt  new   item  index                     GroupBy x   gt  x index   maxItems                   Select g   gt  g Select x   gt  x item

User · Answer

I would suggest to use this extension method to chunk the source list to the sub-lists by specified chunk size        lt summary gt      Helper methods for the lists       lt  summary gt  public static class ListExtensions       public static List lt List lt T gt  gt  ChunkBy lt T gt  this List lt T gt  source  int chunkSize                 return source              Select  x  i    gt  new   Index   i  Value   x                 GroupBy x   gt  x Index   chunkSize               Select x   gt  x Select v   gt  v Value  ToList                 ToList              For example  if you chunk the list of 18 items by 5 items per chunk  it gives you the list of 4 sub-lists with the following items inside  5-5-5-3

User · Answer

How about this one  The idea was to use only one loop  And  who knows  maybe you re using only IList implementations thorough your code and you don t want to cast to List   private IEnumerable lt IList lt T gt  gt  SplitList lt T gt  IList lt T gt  list  int totalChunks        IList lt T gt  auxList   new List lt T gt         int totalItems   list Count         if  totalChunks  lt   0                yield return auxList            else                for  int i   0  i  lt  totalItems  i                                         auxList Add list i                            if   i   1    totalChunks    0                                yield return auxList                  auxList   new List lt T gt                                                else if  i    totalItems - 1                                yield return auxList

User · Answer

List lt int gt  orginalList  new List lt int gt    1 2 3 4 5 6 7 8 9 10 12   Dictionary lt int List lt int gt  gt  dic   new Dictionary  lt int List lt int gt  gt      int batchcount   orginalList Count 2    To List into two 2 parts if you   want three give three List lt int gt  lst   new List lt int gt     for  int i 0 i lt orginalList Count  i      lst Add orginalList i    if  i   batchCount    0  amp  amp  i  0    Dic Add threadId  lst   lst   new List lt int gt      strong text   threadId        if lst Count gt 0  Dic Add threadId  lst     in case if any dayleft  foreach int BatchId in Dic Keys      Console Writeline  quot BatchId  quot  BatchId     Console Writeline  Batch Count  quot  Dic BatchId  Count

User · Answer

I have a generic method that would take any types include float  and it s been unit-tested  hope it helps            lt summary gt          Breaks the list into groups with each group containing no more than the specified group size          lt  summary gt           lt typeparam name  T  gt  lt  typeparam gt           lt param name  values  gt The values  lt  param gt           lt param name  groupSize  gt Size of the group  lt  param gt           lt returns gt  lt  returns gt      public static List lt List lt T gt  gt  SplitList lt T gt  IEnumerable lt T gt  values  int groupSize  int  maxCount   null                List lt List lt T gt  gt  result   new List lt List lt T gt  gt                Quick and special scenario         if  values Count    lt   groupSize                        result Add values ToList                       else                       List lt T gt  valueList   values ToList                int startIndex   0              int count   valueList Count              int elementCount   0               while  startIndex  lt  count  amp  amp    maxCount HasValue     maxCount HasValue  amp  amp  startIndex  lt  maxCount                                  elementCount    startIndex   groupSize  gt  count    count - startIndex   groupSize                  result Add valueList GetRange startIndex  elementCount                    startIndex    elementCount                                    return result

User · Answer

how about   while locations Any              list Add locations Take nSize  ToList         locations  locations Skip nSize  ToList

User · Answer

public static List lt List lt float   gt  gt  SplitList List lt float   gt  locations  int nSize 30                  var list   new List lt List lt float   gt  gt           for  int i   0  i  lt  locations Count  i    nSize                  list Add locations GetRange i  Math Min nSize  locations Count - i                  return list        Generic version   public static IEnumerable lt List lt T gt  gt  SplitList lt T gt  List lt T gt  locations  int nSize 30                  for  int i   0  i  lt  locations Count  i    nSize                  yield return locations GetRange i  Math Min nSize  locations Count - i

User · Answer

public static IEnumerable lt IEnumerable lt T gt  gt  SplitIntoSets lt T gt       this IEnumerable lt T gt  source  int itemsPerSet         var sourceList   source as List lt T gt     source ToList        for  var index   0  index  lt  sourceList Count  index    itemsPerSet                yield return sourceList Skip index  Take itemsPerSet

User · Answer

Addition after very useful comment of mhand at the end  Original answer  Although most solutions might work  I think they are not very efficiently  Suppose if you only want the first few items of the first few chunks  Then you wouldn t want to iterate over all  zillion  items in your sequence   The following will at utmost enumerate twice  once for the Take and once for the Skip  It won t enumerate over any more elements than you will use   public static IEnumerable lt IEnumerable lt TSource gt  gt  ChunkBy lt TSource gt       this IEnumerable lt TSource gt  source  int chunkSize        while  source Any                           while there are elements left            still something to chunk          yield return source Take chunkSize      return a chunk of chunkSize         source   source Skip chunkSize          skip the returned chunk           How many times will this Enumerate the sequence   Suppose you divide your source into chunks of chunkSize  You enumerate only the first N chunks  From every enumerated chunk you ll only enumerate the first M elements   While source Any                   the Any will get the Enumerator  do 1 MoveNext   and returns the returned value after Disposing the Enumerator  This will be done N times  yield return source Take chunkSize     According to the reference source this will do something like   public static IEnumerable lt TSource gt  Take lt TSource gt  this IEnumerable lt TSource gt  source  int count        return TakeIterator lt TSource gt  source  count      static IEnumerable lt TSource gt  TakeIterator lt TSource gt  IEnumerable lt TSource gt  source  int count        foreach  TSource element in source                yield return element          if  --count    0  break            This doesn t do a lot until you start enumerating over the fetched Chunk  If you fetch several Chunks  but decide not to enumerate over the first Chunk  the foreach is not executed  as your debugger will show you   If you decide to take the first M elements of the first chunk then the yield return is executed exactly M times  This means    get the enumerator  call MoveNext   and Current M times   Dispose the enumerator   After the first chunk has been yield returned  we skip this first Chunk   source   source Skip chunkSize     Once again  we ll take a look at reference source to find the skipiterator  static IEnumerable lt TSource gt  SkipIterator lt TSource gt  IEnumerable lt TSource gt  source  int count        using  IEnumerator lt TSource gt  e   source GetEnumerator                   while  count  gt  0  amp  amp  e MoveNext    count--          if  count  lt   0                         while  e MoveNext    yield return e Current                      As you see  the SkipIterator calls MoveNext   once for every element in the Chunk  It doesn t call Current   So per Chunk we see that the following is done    Any    GetEnumerator  1 MoveNext    Dispose Enumerator  Take       nothing if the content of the chunk is not enumerated   If the content is enumerated  GetEnumerator    one MoveNext and one Current per enumerated item  Dispose enumerator  Skip    for every chunk that is enumerated  NOT the contents of the chunk   GetEnumerator    MoveNext   chunkSize times  no Current  Dispose enumerator    If you look at what happens with the enumerator  you ll see that there are a lot of calls to MoveNext    and only calls to Current for the TSource items you actually decide to access   If you take N Chunks of size chunkSize  then calls to MoveNext     N times for Any   not yet any time for Take  as long as you don t enumerate the Chunks N times chunkSize for Skip     If you decide to enumerate only the first M elements of every fetched chunk  then you need to call MoveNext M times per enumerated Chunk   The total  MoveNext calls  N   N M   N chunkSize Current calls  N M   only the items you really access    So if you decide to enumerate all elements of all chunks   MoveNext  numberOfChunks   all elements   all elements   about twice the sequence Current  every item is accessed exactly once   Whether MoveNext is a lot of work or not  depends on the type of source sequence  For lists and arrays it is a simple index increment  with maybe an out of range check   But if your IEnumerable is the result of a database query  make sure that the data is really materialized on your computer  otherwise the data will be fetched several times  DbContext and Dapper will properly transfer the data to local process before it can be accessed  If you enumerate the same sequence several times it is not fetched several times  Dapper returns an object that is a List  DbContext remembers that the data is already fetched   It depends on your Repository whether it is wise to call AsEnumerable   or ToLists   before you start to divide the items in Chunks

User · Answer

While plenty of the answers above do the job  they all fail horribly on a never ending sequence  or a really long sequence   The following is a completely on-line implementation which guarantees best time and memory complexity possible  We only iterate the source enumerable exactly once and use yield return for lazy evaluation  The consumer could throw away the list on each iteration making the memory footprint equal to that of the list w  batchSize number of elements   public static IEnumerable lt List lt T gt  gt  BatchBy lt T gt  this IEnumerable lt T gt  enumerable  int batchSize        using  var enumerator   enumerable GetEnumerator                  List lt T gt  list   null          while  enumerator MoveNext                          if  list    null                                list   new List lt T gt   enumerator Current                             else if  list Count  lt  batchSize                                list Add enumerator Current                             else                               yield return list                  list   new List lt T gt   enumerator Current                                    if  list  Count  gt  0                        yield return list                      EDIT  Just now realizing the OP asks about breaking a List lt T gt  into smaller List lt T gt   so my comments regarding infinite enumerables aren t applicable to the OP  but may help others who end up here  These comments were in response to other posted solutions that do use IEnumerable lt T gt  as an input to their function  yet enumerate the source enumerable multiple times

[c#] Split a List into smaller lists of N size

Examples related to c#

Examples related to list

Examples related to split