How do you split a list into evenly sized chunks

Question

I have a list of arbitrary length  and I need to split it up into equal size chunks and operate on it  There are some obvious ways to do this  like keeping a counter and two lists  and when the second list fills up  add it to the first list and empty the second list for the next round of data  but this is potentially extremely expensive   I was wondering if anyone had a good solution to this for lists of any length  e g  using generators   I was looking for something useful in itertools but I couldn t find anything obviously useful  Might ve missed it  though   Related question  What is the most    pythonic    way to iterate over a list in chunks

User · Answer

I have come up to following solution without creation temorary list object, which should work with any iterable object. Please note that this version for Python 2.x:

def chunked(iterable, size):
    stop = []
    it = iter(iterable)
    def _next_chunk():
        try:
            for _ in xrange(size):
                yield next(it)
        except StopIteration:
            stop.append(True)
            return

    while not stop:
        yield _next_chunk()

for it in chunked(xrange(16), 4):
   print list(it)

Output:

[0, 1, 2, 3]
[4, 5, 6, 7]
[8, 9, 10, 11]
[12, 13, 14, 15] 
[]

As you can see if len(iterable) % size == 0 then we have additional empty iterator object. But I do not think that it is big problem.

User · Answer

This works in v2 v3  is inlineable  generator-based and uses only the standard library   import itertools def split groups iter in  group size       return   x for    x in item  for    item in itertools groupby enumerate iter in   key lambda x  x 0     group size

User · Answer

letting r be the chunk size  and L be the initial list  you can do    chunkL      i for i in L r k r  k 1     for k in range len L  r

User · Answer

AA i i SS  for i in range len AA     SS     Where AA is array  SS is chunk size  For example    gt  gt  gt  AA range 10 21  SS 3  gt  gt  gt   AA i i SS  for i in range len AA     SS     10  11  12    13  14  15    16  17  18    19  20     or  range 10  13   range 13  16   range 16  19   range 19  21   in py3

User · Answer

If you had a chunk size of 3 for example  you could do   zip   iterable i  3  for i in range 3       source  http   code activestate com recipes 303060-group-a-list-into-sequential-n-tuples   I would use this when my chunk size is fixed number I can type  e g   3   and would never change

User · Answer

A generic chunker for any iterable  which gives the user a choice of how to handle a partial chunk at the end   Tested on Python 3   chunker py  from enum import Enum  class PartialChunkOptions Enum       INCLUDE   0     EXCLUDE   1     PAD   2     ERROR   3  class PartialChunkException Exception       pass  def chunker iterable  n  on partial PartialChunkOptions INCLUDE  pad None               A chunker yielding n-element lists from an iterable  with various options     about what to do about a partial chunk at the end       on partial PartialChunkOptions INCLUDE  the default                        include the partial chunk as a short   lt n  element list      on partial PartialChunkOptions EXCLUDE                      do not include the partial chunk      on partial PartialChunkOptions PAD                      pad to an n-element list                        also pass pad  lt pad value gt   default None       on partial PartialChunkOptions ERROR                      raise a RuntimeError if a partial chunk is encountered              on partial   PartialChunkOptions on partial               iterator   iter iterable      while True          vals              for i in range n               try                  vals append next iterator               except StopIteration                  if vals                      if on partial    PartialChunkOptions INCLUDE                          yield vals                     elif on partial    PartialChunkOptions EXCLUDE                          pass                     elif on partial    PartialChunkOptions PAD                          yield vals    pad     n - len vals                       elif on partial    PartialChunkOptions ERROR                          raise PartialChunkException                     return                 return         yield vals   test py  import chunker  chunk size   3  for it in  range 100  107             range 100  109         print   nITERABLE TO CHUNK      format it       print  CHUNK SIZE      format chunk size        for option in chunker PartialChunkOptions   members   values            print   noption    used  format option           try              for chunk in chunker chunker it  chunk size  on partial option                   print chunk          except chunker PartialChunkException              print  PartialChunkException was raised       print       output of test py   ITERABLE TO CHUNK  range 100  107  CHUNK SIZE  3  option PartialChunkOptions INCLUDE used  100  101  102   103  104  105   106   option PartialChunkOptions EXCLUDE used  100  101  102   103  104  105   option PartialChunkOptions PAD used  100  101  102   103  104  105   106  None  None   option PartialChunkOptions ERROR used  100  101  102   103  104  105  PartialChunkException was raised   ITERABLE TO CHUNK  range 100  109  CHUNK SIZE  3  option PartialChunkOptions INCLUDE used  100  101  102   103  104  105   106  107  108   option PartialChunkOptions EXCLUDE used  100  101  102   103  104  105   106  107  108   option PartialChunkOptions PAD used  100  101  102   103  104  105   106  107  108   option PartialChunkOptions ERROR used  100  101  102   103  104  105   106  107  108

User · Answer

I saw the most awesome Python-ish answer in a duplicate of this question   from itertools import zip longest  a   range 1  16  i   iter a  r   list zip longest i  i  i    gt  gt  gt  print r    1  2  3    4  5  6    7  8  9    10  11  12    13  14  15     You can create n-tuple for any n  If a   range 1  15   then the result will be     1  2  3    4  5  6    7  8  9    10  11  12    13  14  None     If the list is divided evenly  then you can replace zip longest with zip  otherwise the triplet  13  14  None  would be lost  Python 3 is used above  For Python 2  use izip longest

User · Answer

Consider using matplotlib cbook pieces  for example   import matplotlib cbook as cbook segments   cbook pieces np arange 20   3  for s in segments       print s

User · Answer

The answer above  by koffein  has a little problem  the list is always split into an equal number of splits  not equal number of items per partition  This is my version  The     chs   1  takes into account that the number of items may not be divideable exactly by the partition size  so the last partition will only be partially filled     Given  l  is your list  chs   12   Your chunksize partitioned     l i chs  i chs  chs  for i in range  len l     chs  1

User · Answer

code   def split list the list  chunk size       result list          while the list          result list append the list  chunk size           the list   the list chunk size       return result list  a list    1  2  3  4  5  6  7  8  9  10   print split list a list  3    result     1  2  3    4  5  6    7  8  9    10

User · Answer

As per this answer  the top-voted answer leaves a  runt  at the end  Here s my solution to really get about as evenly-sized chunks as you can  with no runts  It basically tries to pick exactly the fractional spot where it should split the list  but just rounds it off to the nearest integer   from   future   import division    not needed in Python 3 def n even chunks l  n          Yield n as even chunks as possible from l         last   0     for i in range 1  n 1           cur   int round i    len l    n            yield l last cur          last   cur   Demonstration    gt  gt  gt  pprint pprint list n even chunks list range 100    9      0  1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20  21     22  23  24  25  26  27  28  29  30  31  32     33  34  35  36  37  38  39  40  41  42  43     44  45  46  47  48  49  50  51  52  53  54  55     56  57  58  59  60  61  62  63  64  65  66     67  68  69  70  71  72  73  74  75  76  77     78  79  80  81  82  83  84  85  86  87  88     89  90  91  92  93  94  95  96  97  98  99    gt  gt  gt  pprint pprint list n even chunks list range 100    11      0  1  2  3  4  5  6  7  8     9  10  11  12  13  14  15  16  17     18  19  20  21  22  23  24  25  26     27  28  29  30  31  32  33  34  35     36  37  38  39  40  41  42  43  44     45  46  47  48  49  50  51  52  53  54     55  56  57  58  59  60  61  62  63     64  65  66  67  68  69  70  71  72     73  74  75  76  77  78  79  80  81     82  83  84  85  86  87  88  89  90     91  92  93  94  95  96  97  98  99     Compare to the top-voted chunks answer    gt  gt  gt  pprint pprint list chunks list range 100    100  9      0  1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20  21     22  23  24  25  26  27  28  29  30  31  32     33  34  35  36  37  38  39  40  41  42  43     44  45  46  47  48  49  50  51  52  53  54     55  56  57  58  59  60  61  62  63  64  65     66  67  68  69  70  71  72  73  74  75  76     77  78  79  80  81  82  83  84  85  86  87     88  89  90  91  92  93  94  95  96  97  98     99    gt  gt  gt  pprint pprint list chunks list range 100    100  11      0  1  2  3  4  5  6  7  8     9  10  11  12  13  14  15  16  17     18  19  20  21  22  23  24  25  26     27  28  29  30  31  32  33  34  35     36  37  38  39  40  41  42  43  44     45  46  47  48  49  50  51  52  53     54  55  56  57  58  59  60  61  62     63  64  65  66  67  68  69  70  71     72  73  74  75  76  77  78  79  80     81  82  83  84  85  86  87  88  89     90  91  92  93  94  95  96  97  98     99

User · Answer

You may also use get chunks function of utilspie library as    gt  gt  gt  from utilspie import iterutils  gt  gt  gt  a    1  2  3  4  5  6  7  8  9    gt  gt  gt  list iterutils get chunks a  5     1  2  3  4  5    6  7  8  9     You can install utilspie via pip   sudo pip install utilspie   Disclaimer  I am the creator of utilspie library

User · Answer

def split seq seq  num pieces       start   0     for i in xrange num pieces           stop   start   len seq i  num pieces           yield seq start stop          start   stop   usage   seq    1  2  3  4  5  6  7  8  9  10   for seq in split seq seq  3       print seq

User · Answer

No magic  but simple and correct   def chunks iterable  n          Yield successive n-sized chunks from iterable         values          for i  item in enumerate iterable  1           values append item          if i   n    0              yield values             values          if values          yield values

User · Answer

If you don t care about the order    gt  from itertools import groupby  gt  batch no   3  gt  data    abcdefgh    gt         x 1  for x in x 1        for x in      groupby        sorted           x 0    batch no  x 1            for x in          enumerate data                 key lambda x  x 0                a    d    g      b    e    h      c    f       This solution doesn t generates sets of same size  but distributes values so batches are as big as possible while keeping the number of generated batches

User · Answer

How do you split a list into evenly sized chunks   quot Evenly sized chunks quot   to me  implies that they are all the same length  or barring that option  at minimal variance in length  E g  5 baskets for 21 items could have the following results   gt  gt  gt  import statistics  gt  gt  gt  statistics variance  5 5 5 5 1    3 2  gt  gt  gt  statistics variance  5 4 4 4 4    0 19999999999999998  A practical reason to prefer the latter result  if you were using these functions to distribute work  you ve built-in the prospect of one likely finishing well before the others  so it would sit around doing nothing while the others continued working hard  Critique of other answers here When I originally wrote this answer  none of the other answers were evenly sized chunks - they all leave a runt chunk at the end  so they re not well balanced  and have a higher than necessary variance of lengths  For example  the current top answer ends with   60  61  62  63  64  65  66  67  68  69    70  71  72  73  74    Others  like list grouper 3  range 7     and chunk range 7   3  both return    0  1  2    3  4  5    6  None  None    The None s are just padding  and rather inelegant in my opinion  They are NOT evenly chunking the iterables  Why can t we divide these better  Cycle Solution A high-level balanced solution using itertools cycle  which is the way I might do it today  Here s the setup  from itertools import cycle items   range 10  75  number of baskets   10  Now we need our lists into which to populate the elements  baskets       for   in range number of baskets    Finally  we zip the elements we re going to allocate together with a cycle of the baskets until we run out of elements  which  semantically  it exactly what we want  for element  basket in zip items  cycle baskets        basket append element   Here s the result   gt  gt  gt  from pprint import pprint  gt  gt  gt  pprint baskets    10  20  30  40  50  60  70     11  21  31  41  51  61  71     12  22  32  42  52  62  72     13  23  33  43  53  63  73     14  24  34  44  54  64  74     15  25  35  45  55  65     16  26  36  46  56  66     17  27  37  47  57  67     18  28  38  48  58  68     19  29  39  49  59  69    To productionize this solution  we write a function  and provide the type annotations  from itertools import cycle from typing import List  Any  def cycle baskets items  List Any   maxbaskets  int  - gt  List List Any        baskets       for   in range min maxbaskets  len items         for item  basket in zip items  cycle baskets            basket append item      return baskets  In the above  we take our list of items  and the max number of baskets  We create a list of empty lists  in which to append each element  in a round-robin style  Slices Another elegant solution is to use slices - specifically the less-commonly used step argument to slices  i e   start   0 stop   None step   number of baskets  first basket   items start stop step   This is especially elegant in that slices don t care how long the data are - the result  our first basket  is only as long as it needs to be  We ll only need to increment the starting point for each basket  In fact this could be a one-liner  but we ll go multiline for readability and to avoid an overlong line of code  from typing import List  Any  def slice baskets items  List Any   maxbaskets  int  - gt  List List Any        n baskets   min maxbaskets  len items       return  items i  n baskets  for i in range n baskets    And islice from the itertools module will provide a lazily iterating approach  like that which was originally asked for in the question  I don t expect most use-cases to benefit very much  as the original data is already fully materialized in a list  but for large datasets  it could save nearly half the memory usage  from itertools import islice from typing import List  Any  Generator      def yield islice baskets items  List Any   maxbaskets  int  - gt  Generator List Any   None  None       n baskets   min maxbaskets  len items       for i in range n baskets           yield islice items  i  None  n baskets   View results with  from pprint import pprint  items   list range 10  75   pprint cycle baskets items  10   pprint slice baskets items  10   pprint  list s  for s in yield islice baskets items  10     Updated prior solutions Here s another balanced solution  adapted from a function I ve used in production in the past  that uses the modulo operator  def baskets from items  maxbaskets 25       baskets       for   in range maxbaskets       for i  item in enumerate items           baskets i   maxbaskets  append item      return filter None  baskets    And I created a generator that does the same if you put it into a list  def iter baskets from items  maxbaskets 3          generates evenly balanced baskets from indexable iterable        item count   len items      baskets   min item count  maxbaskets      for x i in range baskets           yield  items y i  for y i in range x i  item count  baskets         And finally  since I see that all of the above functions return elements in a contiguous order  as they were given   def iter baskets contiguous items  maxbaskets 3  item count None               generates balanced baskets from iterable  contiguous contents     provide item count if providing a iterator that doesn t support len               item count   item count or len items      baskets   min item count  maxbaskets      items   iter items      floor   item count    baskets      ceiling   floor   1     stepdown   item count   baskets     for x i in range baskets           length   ceiling if x i  lt  stepdown else floor         yield  items next   for   in range length    Output To test them out  print baskets from range 6   8   print list iter baskets from range 6   8    print list iter baskets contiguous range 6   8    print baskets from range 22   8   print list iter baskets from range 22   8    print list iter baskets contiguous range 22   8    print baskets from  ABCDEFG   3   print list iter baskets from  ABCDEFG   3    print list iter baskets contiguous  ABCDEFG   3    print baskets from range 26   5   print list iter baskets from range 26   5    print list iter baskets contiguous range 26   5     Which prints out    0    1    2    3    4    5     0    1    2    3    4    5     0    1    2    3    4    5     0  8  16    1  9  17    2  10  18    3  11  19    4  12  20    5  13  21    6  14    7  15     0  8  16    1  9  17    2  10  18    3  11  19    4  12  20    5  13  21    6  14    7  15     0  1  2    3  4  5    6  7  8    9  10  11    12  13  14    15  16  17    18  19    20  21      A    D    G      B    E      C    F       A    D    G      B    E      C    F       A    B    C      D    E      F    G      0  5  10  15  20  25    1  6  11  16  21    2  7  12  17  22    3  8  13  18  23    4  9  14  19  24     0  5  10  15  20  25    1  6  11  16  21    2  7  12  17  22    3  8  13  18  23    4  9  14  19  24     0  1  2  3  4  5    6  7  8  9  10    11  12  13  14  15    16  17  18  19  20    21  22  23  24  25    Notice that the contiguous generator provide chunks in the same length patterns as the other two  but the items are all in order  and they are as evenly divided as one may divide a list of discrete elements

User · Answer

Like  AaronHall I got here looking for roughly evenly sized chunks  There are different interpretations of that  In my case  if the desired size is N  I would like each group to be of size  N  Thus  the orphans which are created in most of the above should be redistributed to other groups   This can be done using   def nChunks l  n           Yield n successive chunks from l      Works for lists   pandas dataframes  etc             newn   int 1 0   len l    n   0 5      for i in xrange 0  n-1           yield l i newn i newn newn      yield l n newn-newn      from Splitting a list of into N parts of approximately equal length  by simply calling it as nChunks l l n   or nChunks l floor l n

User · Answer

code   def split list the list  chunk size       result list          while the list          result list append the list  chunk size           the list   the list chunk size       return result list  a list    1  2  3  4  5  6  7  8  9  10   print split list a list  3    result     1  2  3    4  5  6    7  8  9    10

User · Answer

Lazy loading version     import pprint pprint pprint list chunks range 10  75   10     range 10  20    range 20  30    range 30  40    range 40  50    range 50  60    range 60  70    range 70  75          Confer this implementation s result with the example usage result of the accepted answer     Many of the above functions assume that the length of the whole iterable are known up front  or at least are cheap to calculate   For some streamed objects that would mean loading the full data into memory first  e g  to download the whole file  to get the length information   If you however don t know the the full size yet  you can use this code instead   def chunks iterable  size               Yield successive chunks from iterable  being  size  long       https   stackoverflow com a 55776536 3423324      param iterable  The object you want to split into pieces       param size  The size each of the resulting pieces should have              i   0     while True          sliced   iterable i i   size          if len sliced     0                to suppress stuff like  range max  max                break           end if         yield sliced         if len sliced   lt  size                our slice is not the full length  so we must have passed the end of the iterator             break           end if         i    size    so we start the next chunk at the right place        end while   end def   This works because the slice command will return less no elements if you passed the end of an iterable    abc  0 2      ab   abc  2 4      c   abc  4 6          We now use that result of the slice  and calculate the length of that generated chunk  If it is less than what we expect  we know we can end the iteration   That way the iterator will not be executed unless access

User · Answer

I dislike idea of splitting elements by chunk size  e g  script can devide 101 to 3 chunks as  50  50  1   For my needs I needed spliting proportionly  and keeping order same  First I wrote my own script  which works fine  and it s very simple  But I ve seen later this answer  where script is better than mine  I reccomend it  Here s my script   def proportional dividing N  n               N - length of array  bigger number      n - number of chunks  smaller number      output - arr  containing N numbers  diveded roundly to n chunks             arr          if N    0          return arr     elif n    0          arr append N          return arr     r   N    n     for i in range n-1           arr append r      arr append N-r  n-1        last n   arr -1        last number always will be r  lt   last n  lt  2 r       when last n    r it s ok  but when last n  gt  r         if last n  gt  r                and if difference too big  bigger than 1   then         if abs r-last n   gt  1                2  2  2  2  2  2  2  2  2  2  2  7    N 29  n 12               we need to give unnecessary numbers to first elements back             diff   last n - r             for k in range diff                   arr k     1             arr -1    r               and we receive  3  3  3  3  3  2  2  2  2  2  2  2      return arr  def split items items  chunks       arr   proportional dividing len items   chunks      splitted          for chunk size in arr          splitted append items  chunk size           items   items chunk size       print splitted      return splitted  items    1 2 3 4 5 6 7 8 9 10 11  chunks   3 split items items  chunks  split items   a   b   c   d   e   f   g   h   i   g   k   l    m    3  split items   a   b   c   d   e   f   g   h   i   g   k   l    m    n    3  split items range 100   4  split items range 99   4  split items range 101   4    and output     1  2  3  4    5  6  7  8    9  10  11      a    b    c    d      e    f    g    h      i    g    k    l    m       a    b    c    d    e      f    g    h    i    g      k    l    m    n     range 0  25   range 25  50   range 50  75   range 75  100    range 0  25   range 25  50   range 50  75   range 75  99    range 0  25   range 25  50   range 50  75   range 75  101

User · Answer

You may also use get chunks function of utilspie library as    gt  gt  gt  from utilspie import iterutils  gt  gt  gt  a    1  2  3  4  5  6  7  8  9    gt  gt  gt  list iterutils get chunks a  5     1  2  3  4  5    6  7  8  9     You can install utilspie via pip   sudo pip install utilspie   Disclaimer  I am the creator of utilspie library

User · Answer

Consider using matplotlib cbook pieces  for example   import matplotlib cbook as cbook segments   cbook pieces np arange 20   3  for s in segments       print s

User · Answer

The answer above  by koffein  has a little problem  the list is always split into an equal number of splits  not equal number of items per partition  This is my version  The     chs   1  takes into account that the number of items may not be divideable exactly by the partition size  so the last partition will only be partially filled     Given  l  is your list  chs   12   Your chunksize partitioned     l i chs  i chs  chs  for i in range  len l     chs  1

User · Answer

Use list comprehensions   l    1 2 3 4 5 6 7 8 9 10 11 12  k   5  chunk size print  tuple l x y   for  x  y  in   x  x k  for x in range 0  len l   k

User · Answer

I dislike idea of splitting elements by chunk size  e g  script can devide 101 to 3 chunks as  50  50  1   For my needs I needed spliting proportionly  and keeping order same  First I wrote my own script  which works fine  and it s very simple  But I ve seen later this answer  where script is better than mine  I reccomend it  Here s my script   def proportional dividing N  n               N - length of array  bigger number      n - number of chunks  smaller number      output - arr  containing N numbers  diveded roundly to n chunks             arr          if N    0          return arr     elif n    0          arr append N          return arr     r   N    n     for i in range n-1           arr append r      arr append N-r  n-1        last n   arr -1        last number always will be r  lt   last n  lt  2 r       when last n    r it s ok  but when last n  gt  r         if last n  gt  r                and if difference too big  bigger than 1   then         if abs r-last n   gt  1                2  2  2  2  2  2  2  2  2  2  2  7    N 29  n 12               we need to give unnecessary numbers to first elements back             diff   last n - r             for k in range diff                   arr k     1             arr -1    r               and we receive  3  3  3  3  3  2  2  2  2  2  2  2      return arr  def split items items  chunks       arr   proportional dividing len items   chunks      splitted          for chunk size in arr          splitted append items  chunk size           items   items chunk size       print splitted      return splitted  items    1 2 3 4 5 6 7 8 9 10 11  chunks   3 split items items  chunks  split items   a   b   c   d   e   f   g   h   i   g   k   l    m    3  split items   a   b   c   d   e   f   g   h   i   g   k   l    m    n    3  split items range 100   4  split items range 99   4  split items range 101   4    and output     1  2  3  4    5  6  7  8    9  10  11      a    b    c    d      e    f    g    h      i    g    k    l    m       a    b    c    d    e      f    g    h    i    g      k    l    m    n     range 0  25   range 25  50   range 50  75   range 75  100    range 0  25   range 25  50   range 50  75   range 75  99    range 0  25   range 25  50   range 50  75   range 75  101

User · Answer

Here s an idea using itertools groupby   def chunks l  n       c   itertools count       return  it for    it in itertools groupby l  lambda x  next c   n     This returns a generator of generators  If you want a list of lists  just replace the last line with      return  list it  for    it in itertools groupby l  lambda x  next c   n     Example returning list of lists    gt  gt  gt  chunks  abcdefghij   4     a    b    c    d      e    f    g    h      i    j       So yes  this suffers form the  runt problem   which may or may not be a problem in a given situation

User · Answer

I m surprised nobody has thought of using iter s two-argument form   from itertools import islice  def chunk it  size       it   iter it      return iter lambda  tuple islice it  size          Demo    gt  gt  gt  list chunk range 14   3     0  1  2    3  4  5    6  7  8    9  10  11    12  13     This works with any iterable and produces output lazily  It returns tuples rather than iterators  but I think it has a certain elegance nonetheless  It also doesn t pad  if you want padding  a simple variation on the above will suffice   from itertools import islice  chain  repeat  def chunk pad it  size  padval None       it   chain iter it   repeat padval       return iter lambda  tuple islice it  size     padval     size    Demo    gt  gt  gt  list chunk pad range 14   3     0  1  2    3  4  5    6  7  8    9  10  11    12  13  None    gt  gt  gt  list chunk pad range 14   3   a      0  1  2    3  4  5    6  7  8    9  10  11    12  13   a      Like the izip longest-based solutions  the above always pads  As far as I know  there s no one- or two-line itertools recipe for a function that optionally pads  By combining the above two approaches  this one comes pretty close    no padding   object    def chunk it  size  padval  no padding       if padval     no padding          it   iter it          sentinel          else          it   chain iter it   repeat padval           sentinel    padval     size     return iter lambda  tuple islice it  size    sentinel    Demo    gt  gt  gt  list chunk range 14   3     0  1  2    3  4  5    6  7  8    9  10  11    12  13    gt  gt  gt  list chunk range 14   3  None     0  1  2    3  4  5    6  7  8    9  10  11    12  13  None    gt  gt  gt  list chunk range 14   3   a      0  1  2    3  4  5    6  7  8    9  10  11    12  13   a      I believe this is the shortest chunker proposed that offers optional padding   As Tomasz Gandor observed  the two padding chunkers will stop unexpectedly if they encounter a long sequence of pad values  Here s a final variation that works around that problem in a reasonable way    no padding   object   def chunk it  size  padval  no padding       it   iter it      chunker   iter lambda  tuple islice it  size            if padval     no padding          yield from chunker     else          for ch in chunker              yield ch if len ch     size else ch    padval      size - len ch     Demo    gt  gt  gt  list chunk  1  2          5   2     1  2              5     gt  gt  gt  list chunk  1  2  None  None  5   2  None     1  2    None  None    5  None

User · Answer

Since everybody here talking about iterators  boltons has perfect method for that  called iterutils chunked iter   from boltons import iterutils  list iterutils chunked iter list range 50    11     Output     0  1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20  21     22  23  24  25  26  27  28  29  30  31  32     33  34  35  36  37  38  39  40  41  42  43     44  45  46  47  48  49     But if you don t want to be mercy on memory  you can use old-way and store the full list in the first place with iterutils chunked

User · Answer

The toolz library has the partition function for this   from toolz itertoolz core import partition  list partition 2   1  2  3  4      1  2    3  4

User · Answer

heh  one line version  In  48   chunk   lambda ulist  step   map lambda i  ulist i i step    xrange 0  len ulist   step    In  49   chunk range 1 100   10  Out 49      1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20     21  22  23  24  25  26  27  28  29  30     31  32  33  34  35  36  37  38  39  40     41  42  43  44  45  46  47  48  49  50     51  52  53  54  55  56  57  58  59  60     61  62  63  64  65  66  67  68  69  70     71  72  73  74  75  76  77  78  79  80     81  82  83  84  85  86  87  88  89  90     91  92  93  94  95  96  97  98  99

User · Answer

I like the Python doc s version proposed by tzot and J F Sebastian a lot   but it has two shortcomings    it is not very explicit I usually don t want a fill value in the last chunk   I m using this one a lot in my code   from itertools import islice  def chunks n  iterable       iterable   iter iterable      while True          yield tuple islice iterable  n   or iterable next     UPDATE  A lazy chunks version   from itertools import chain  islice  def chunks n  iterable      iterable   iter iterable     while True         yield chain  next iterable    islice iterable  n-1

User · Answer

I ve created these two fancy one-liners which are efficient and lazy  both input and output are iterables  also they doen t depend on any module  First one-liner is totally lazy meaning that it returns iterator producing iterators  i e  each chunk produced is iterator iterating over chunk s elements   this version is good for the case if chunks are very large or elements are produced slowly one by one and should become available immediately as they are produced  Try it online  chunk iters   lambda it  n    e for i  g in enumerate   f    cit   for j  e in zip range  1  n - 1  i    g   for cit in  iter it    for f in cit   Second one-liner returns iterator that produces lists  Each list is produced as soon as elements of whole chunk become available through input iterator or if very last element of last chunk is reached  This version should be used if input elements are produced fast or all available immediately  Other wise first more-lazy one-liner version should be used  Try it online  chunk lists   lambda it  n   l for l in       for i  g in enumerate  it          for e in g for l in  l  len l    n     e   1 - i    if  len l    n    0     i   Also I provide multi-line version of first chunk iters one-liner  which returns iterator producing another iterators  going through each chunk s elements   Try it online  def chunk iters it  n       cit   iter it      def one chunk f           yield f         for i  e in zip range n - 1   cit               yield e     for f in cit          yield one chunk f

User · Answer

Directly from the  old  Python documentation  recipes for itertools    from itertools import izip  chain  repeat  def grouper n  iterable  padvalue None        grouper 3   abcdefg    x   -- gt    a   b   c      d   e   f      g   x   x        return izip   chain iterable  repeat padvalue  n-1    n    The current version  as suggested by J F Sebastian    from itertools import izip longest as zip longest   for Python 2 x from itertools import zip longest   for Python 3 x  from six moves import zip longest   for both  uses the six compat library   def grouper n  iterable  padvalue None        grouper 3   abcdefg    x   -- gt    a   b   c      d   e   f      g   x   x        return zip longest   iter iterable   n  fillvalue padvalue    I guess Guido s time machine works   worked   will work   will have worked   was working again   These solutions work because  iter iterable   n  or the equivalent in the earlier version  creates one iterator  repeated n times in the list  izip longest then effectively performs a round-robin of  each  iterator  because this is the same iterator  it is advanced by each such call  resulting in each such zip-roundrobin generating one tuple of n items

User · Answer

Lazy loading version     import pprint pprint pprint list chunks range 10  75   10     range 10  20    range 20  30    range 30  40    range 40  50    range 50  60    range 60  70    range 70  75          Confer this implementation s result with the example usage result of the accepted answer     Many of the above functions assume that the length of the whole iterable are known up front  or at least are cheap to calculate   For some streamed objects that would mean loading the full data into memory first  e g  to download the whole file  to get the length information   If you however don t know the the full size yet  you can use this code instead   def chunks iterable  size               Yield successive chunks from iterable  being  size  long       https   stackoverflow com a 55776536 3423324      param iterable  The object you want to split into pieces       param size  The size each of the resulting pieces should have              i   0     while True          sliced   iterable i i   size          if len sliced     0                to suppress stuff like  range max  max                break           end if         yield sliced         if len sliced   lt  size                our slice is not the full length  so we must have passed the end of the iterator             break           end if         i    size    so we start the next chunk at the right place        end while   end def   This works because the slice command will return less no elements if you passed the end of an iterable    abc  0 2      ab   abc  2 4      c   abc  4 6          We now use that result of the slice  and calculate the length of that generated chunk  If it is less than what we expect  we know we can end the iteration   That way the iterator will not be executed unless access

User · Answer

heh  one line version  In  48   chunk   lambda ulist  step   map lambda i  ulist i i step    xrange 0  len ulist   step    In  49   chunk range 1 100   10  Out 49      1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20     21  22  23  24  25  26  27  28  29  30     31  32  33  34  35  36  37  38  39  40     41  42  43  44  45  46  47  48  49  50     51  52  53  54  55  56  57  58  59  60     61  62  63  64  65  66  67  68  69  70     71  72  73  74  75  76  77  78  79  80     81  82  83  84  85  86  87  88  89  90     91  92  93  94  95  96  97  98  99

User · Answer

Here s a generator that yields the chunks you want   def chunks lst  n          Yield successive n-sized chunks from lst         for i in range 0  len lst   n           yield lst i i   n      import pprint pprint pprint list chunks range 10  75   10      10  11  12  13  14  15  16  17  18  19     20  21  22  23  24  25  26  27  28  29     30  31  32  33  34  35  36  37  38  39     40  41  42  43  44  45  46  47  48  49     50  51  52  53  54  55  56  57  58  59     60  61  62  63  64  65  66  67  68  69     70  71  72  73  74       If you re using Python 2  you should use xrange   instead of range     def chunks lst  n          Yield successive n-sized chunks from lst         for i in xrange 0  len lst   n           yield lst i i   n      Also you can simply use list comprehension instead of writing a function  though it s a good idea to encapsulate operations like this in named functions so that your code is easier to understand  Python 3    lst i i   n  for i in range 0  len lst   n     Python 2 version    lst i i   n  for i in xrange 0  len lst   n

User · Answer

I was curious about the performance of different approaches and here it is   Tested on Python 3 5 1  import time batch size   7 arr len   298937   ---------slice-------------  print   r nslice   start   time time   arr    i for i in range 0  arr len   while True      if not arr          break      tmp   arr 0 batch size      arr   arr batch size -1  print time time   - start    -----------index-----------  print   r nindex   arr    i for i in range 0  arr len   start   time time   for i in range 0  round len arr    batch size   1        tmp   arr batch size   i   batch size    i   1   print time time   - start    ----------batches 1------------  def batch iterable  n 1       l   len iterable      for ndx in range 0  l  n           yield iterable ndx min ndx   n  l    print   r nbatches 1   arr    i for i in range 0  arr len   start   time time   for x in batch arr  batch size       tmp   x print time time   - start    ----------batches 2------------  from itertools import islice  chain  def batch iterable  size       sourceiter   iter iterable      while True          batchiter   islice sourceiter  size          yield chain  next batchiter    batchiter    print   r nbatches 2   arr    i for i in range 0  arr len   start   time time   for x in batch arr  batch size       tmp   x print time time   - start    ---------chunks------------- def chunks l  n          Yield successive n-sized chunks from l         for i in range 0  len l   n           yield l i i   n  print   r nchunks   arr    i for i in range 0  arr len   start   time time   for x in chunks arr  batch size       tmp   x print time time   - start    -----------grouper-----------  from itertools import zip longest   for Python 3 x  from six moves import zip longest   for both  uses the six compat library   def grouper iterable  n  padvalue None        grouper 3   abcdefg    x   -- gt    a   b   c      d   e   f      g   x   x        return zip longest   iter iterable   n  fillvalue padvalue   arr    i for i in range 0  arr len   print   r ngrouper   start   time time   for x in grouper arr  batch size       tmp   x print time time   - start    Results   slice 31 18285083770752  index 0 02184295654296875  batches 1 0 03503894805908203  batches 2 0 22681021690368652  chunks 0 019841909408569336  grouper 0 006506919860839844

User · Answer

One more solution  def make chunks data  chunk size        while data          chunk  data   data  chunk size   data chunk size           yield chunk   gt  gt  gt  for chunk in make chunks  1  2  3  4  5  6  7   2           print chunk       1  2   3  4   5  6   7   gt  gt  gt

User · Answer

An abstraction would be l    1 2 3 4 5 6 7 8 9  n   3 outList      for i in range n  len l    n  n       outList append l i-n i    print outList   This will print     1  2  3    4  5  6    7  8  9

User · Answer

def main      print chunkify  1 2 3 4 5 6  2    def chunkify list  n     chunks        for i in range 0  len list   n       chunks append list i i n     return chunks  main     I think that it s simple and can give you a chunk of an array

User · Answer

This works in v2 v3  is inlineable  generator-based and uses only the standard library   import itertools def split groups iter in  group size       return   x for    x in item  for    item in itertools groupby enumerate iter in   key lambda x  x 0     group size

User · Answer

You could use numpy s array split function e g   np array split np array data   20  to split into 20 nearly equal size chunks   To make sure chunks are exactly equal in size use np split

User · Answer

def chunk input  size       return map None     iter input     size

User · Answer

def chunks iterable n          assumes n is an integer gt 0             iterable iter iterable      while True          result            for i in range n               try                  a next iterable              except StopIteration                  break             else                  result append a          if result              yield result         else              break  g1  i i for i in range 10   g2 chunks g1 3  print g2   lt generator object chunks at 0x0337B9B8 gt   print list g2     0  1  4    9  16  25    36  49  64    81

User · Answer

Here is a bit of code written in Python3 that does the same as np array split  list map list  map functools partial filter  None   itertools zip longest  iter lambda  tuple itertools islice a  n             It s quite a long one-liner but it does divide the items evenly amongst the resulting sublists

User · Answer

I know this is kind of old but nobody yet mentioned numpy array split   import numpy as np  lst   range 50  np array split lst  5     array  0  1  2  3  4  5  6  7  8  9       array  10  11  12  13  14  15  16  17  18  19       array  20  21  22  23  24  25  26  27  28  29       array  30  31  32  33  34  35  36  37  38  39       array  40  41  42  43  44  45  46  47  48  49

User · Answer

Here is a generator that work on arbitrary iterables   def split seq iterable  size       it   iter iterable      item   list itertools islice it  size       while item          yield item         item   list itertools islice it  size     Example    gt  gt  gt  import pprint  gt  gt  gt  pprint pprint list split seq xrange 75   10      0  1  2  3  4  5  6  7  8  9     10  11  12  13  14  15  16  17  18  19     20  21  22  23  24  25  26  27  28  29     30  31  32  33  34  35  36  37  38  39     40  41  42  43  44  45  46  47  48  49     50  51  52  53  54  55  56  57  58  59     60  61  62  63  64  65  66  67  68  69     70  71  72  73  74

User · Answer

Simple yet elegant  l   range 1  1000  print  l x x 10  for x in xrange 0  len l   10     or if you prefer   def chunks l  n   return  l x  x n  for x in xrange 0  len l   n   chunks l  10

User · Answer

Here s a generator that yields the chunks you want   def chunks lst  n          Yield successive n-sized chunks from lst         for i in range 0  len lst   n           yield lst i i   n      import pprint pprint pprint list chunks range 10  75   10      10  11  12  13  14  15  16  17  18  19     20  21  22  23  24  25  26  27  28  29     30  31  32  33  34  35  36  37  38  39     40  41  42  43  44  45  46  47  48  49     50  51  52  53  54  55  56  57  58  59     60  61  62  63  64  65  66  67  68  69     70  71  72  73  74       If you re using Python 2  you should use xrange   instead of range     def chunks lst  n          Yield successive n-sized chunks from lst         for i in xrange 0  len lst   n           yield lst i i   n      Also you can simply use list comprehension instead of writing a function  though it s a good idea to encapsulate operations like this in named functions so that your code is easier to understand  Python 3    lst i i   n  for i in range 0  len lst   n     Python 2 version    lst i i   n  for i in xrange 0  len lst   n

User · Answer

I don t think I saw this option  so just to add another one        def chunks iterable  chunk size     i   0    while i  lt  len iterable       yield iterable i i chunk size      i    chunk size

User · Answer

gt  gt  gt  def f x  n  acc      return f x n    n  acc   x  n     if x else acc  gt  gt  gt  f  Hallo Welt   3    Hal    lo     Wel    t    gt  gt  gt     If you are into brackets - I picked up a book on Erlang

User · Answer

def chunks iterable n          assumes n is an integer gt 0             iterable iter iterable      while True          result            for i in range n               try                  a next iterable              except StopIteration                  break             else                  result append a          if result              yield result         else              break  g1  i i for i in range 10   g2 chunks g1 3  print g2   lt generator object chunks at 0x0337B9B8 gt   print list g2     0  1  4    9  16  25    36  49  64    81

User · Answer

def chunk input  size       return map None     iter input     size

User · Answer

Here is a list of additional approaches   Given  import itertools as it import collections as ct  import more itertools as mit   iterable   range 11  n   3   Code  The Standard Library  list it zip longest   iter iterable     n       0  1  2    3  4  5    6  7  8    9  10  None       d      for i  x in enumerate iterable       d setdefault i  n      append x   list d values        0  1  2    3  4  5    6  7  8    9  10       dd   ct defaultdict list  for i  x in enumerate iterable       dd i  n  append x   list dd values        0  1  2    3  4  5    6  7  8    9  10     more itertools   list mit chunked iterable  n       0  1  2    3  4  5    6  7  8    9  10    list mit sliced iterable  n      range 0  3   range 3  6   range 6  9   range 9  11    list mit grouper n  iterable       0  1  2    3  4  5    6  7  8    9  10  None    list mit windowed iterable  len iterable   n  step n       0  1  2    3  4  5    6  7  8    9  10  None     References   zip longest  related post  related post  setdefault  ordered results requires Python 3 6   collections defaultdict   ordered results requires Python 3 6   more itertools chunked  related posted  more itertools sliced more itertools grouper  related post  more itertools windowed  see also stagger  zip offset      A third-party library that implements itertools recipes and more   gt  pip install more itertools

User · Answer

If you know list size   def SplitList mylist  chunk size       return  mylist offs offs chunk size  for offs in range 0  len mylist   chunk size     If you don t  an iterator    def IterChunks sequence  chunk size       res          for item in sequence          res append item          if len res   gt   chunk size              yield res             res          if res          yield res    yield the last  incomplete  portion   In the latter case  it can be rephrased in a more beautiful way if you can be sure that the sequence always contains a whole number of chunks of given size  i e  there is no incomplete last chunk

User · Answer

At this point  I think we need the obligatory anonymous-recursive function   Y   lambda f   lambda x  x x   lambda y  f lambda  args  y y   args    chunks   Y lambda f  lambda n   n 0   n 1      f  n 0  n 1     n 1    if len n 0    gt  0 else

User · Answer

Here is a generator that work on arbitrary iterables   def split seq iterable  size       it   iter iterable      item   list itertools islice it  size       while item          yield item         item   list itertools islice it  size     Example    gt  gt  gt  import pprint  gt  gt  gt  pprint pprint list split seq xrange 75   10      0  1  2  3  4  5  6  7  8  9     10  11  12  13  14  15  16  17  18  19     20  21  22  23  24  25  26  27  28  29     30  31  32  33  34  35  36  37  38  39     40  41  42  43  44  45  46  47  48  49     50  51  52  53  54  55  56  57  58  59     60  61  62  63  64  65  66  67  68  69     70  71  72  73  74

User · Answer

This question reminds me of the Raku  formerly Perl 6   comb n  method   It breaks up strings into n-sized chunks    There s more to it than that  but I ll leave out the details   It s easy enough to implement a similar function in Python3 as a lambda expression  comb   lambda s n   s i i n  for i in range 0 len s  n    Then you can call it like this  some list   list range 0  20      creates a list of 20 elements generator   comb some list  4     creates a generator that will generate lists of 4 elements for sublist in generator      print sublist     prints a sublist of four elements  as it s generated  Of course  you don t have to assign the generator to a variable  you can just loop over it directly like this  for sublist in comb some list  4       print sublist     prints a sublist of four elements  as it s generated  As a bonus  this comb   function also operates on strings  list  comb  catdogant   3       returns   cat    dog    ant

User · Answer

I know this is kind of old but nobody yet mentioned numpy array split   import numpy as np  lst   range 50  np array split lst  5     array  0  1  2  3  4  5  6  7  8  9       array  10  11  12  13  14  15  16  17  18  19       array  20  21  22  23  24  25  26  27  28  29       array  30  31  32  33  34  35  36  37  38  39       array  40  41  42  43  44  45  46  47  48  49

User · Answer

If you had a chunk size of 3 for example  you could do   zip   iterable i  3  for i in range 3       source  http   code activestate com recipes 303060-group-a-list-into-sequential-n-tuples   I would use this when my chunk size is fixed number I can type  e g   3   and would never change

User · Answer

Here is a bit of code written in Python3 that does the same as np array split  list map list  map functools partial filter  None   itertools zip longest  iter lambda  tuple itertools islice a  n             It s quite a long one-liner but it does divide the items evenly amongst the resulting sublists

User · Answer

Since I had to do something like this  here s my solution given a generator and a batch size   def pop n elems from generator g  n       elems          try          for idx in xrange 0  n               elems append g next            return elems     except StopIteration          return elems

User · Answer

I realise this question is old  stumbled over it on Google   but surely something like the following is far simpler and clearer than any of the huge complex suggestions and only uses slicing   def chunker iterable  chunksize       for i c in enumerate iterable   chunksize            yield iterable i chunksize  i 1  chunksize    gt  gt  gt  for chunk in chunker range 0 100   10           print list chunk        0  1  2  3  4  5  6  7  8  9   10  11  12  13  14  15  16  17  18  19   20  21  22  23  24  25  26  27  28  29      etc

User · Answer

One more solution  def make chunks data  chunk size        while data          chunk  data   data  chunk size   data chunk size           yield chunk   gt  gt  gt  for chunk in make chunks  1  2  3  4  5  6  7   2           print chunk       1  2   3  4   5  6   7   gt  gt  gt

User · Answer

heh  one line version  In  48   chunk   lambda ulist  step   map lambda i  ulist i i step    xrange 0  len ulist   step    In  49   chunk range 1 100   10  Out 49      1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20     21  22  23  24  25  26  27  28  29  30     31  32  33  34  35  36  37  38  39  40     41  42  43  44  45  46  47  48  49  50     51  52  53  54  55  56  57  58  59  60     61  62  63  64  65  66  67  68  69  70     71  72  73  74  75  76  77  78  79  80     81  82  83  84  85  86  87  88  89  90     91  92  93  94  95  96  97  98  99

User · Answer

using List Comprehensions of python   range t t 10  for t in range 1 1000 10      1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20     21  22  23  24  25  26  27  28  29  30             981  982  983  984  985  986  987  988  989  990     991  992  993  994  995  996  997  998  999  1000     visit this link to know about List Comprehensions

User · Answer

Here s a generator that yields the chunks you want   def chunks lst  n          Yield successive n-sized chunks from lst         for i in range 0  len lst   n           yield lst i i   n      import pprint pprint pprint list chunks range 10  75   10      10  11  12  13  14  15  16  17  18  19     20  21  22  23  24  25  26  27  28  29     30  31  32  33  34  35  36  37  38  39     40  41  42  43  44  45  46  47  48  49     50  51  52  53  54  55  56  57  58  59     60  61  62  63  64  65  66  67  68  69     70  71  72  73  74       If you re using Python 2  you should use xrange   instead of range     def chunks lst  n          Yield successive n-sized chunks from lst         for i in xrange 0  len lst   n           yield lst i i   n      Also you can simply use list comprehension instead of writing a function  though it s a good idea to encapsulate operations like this in named functions so that your code is easier to understand  Python 3    lst i i   n  for i in range 0  len lst   n     Python 2 version    lst i i   n  for i in xrange 0  len lst   n

User · Answer

a    1  2  3  4  5  6  7  8  9  CHUNK   4  a i CHUNK  i 1  CHUNK  for i in xrange  len a    CHUNK - 1    CHUNK

User · Answer

I saw the most awesome Python-ish answer in a duplicate of this question   from itertools import zip longest  a   range 1  16  i   iter a  r   list zip longest i  i  i    gt  gt  gt  print r    1  2  3    4  5  6    7  8  9    10  11  12    13  14  15     You can create n-tuple for any n  If a   range 1  15   then the result will be     1  2  3    4  5  6    7  8  9    10  11  12    13  14  None     If the list is divided evenly  then you can replace zip longest with zip  otherwise the triplet  13  14  None  would be lost  Python 3 is used above  For Python 2  use izip longest

User · Answer

I realise this question is old  stumbled over it on Google   but surely something like the following is far simpler and clearer than any of the huge complex suggestions and only uses slicing   def chunker iterable  chunksize       for i c in enumerate iterable   chunksize            yield iterable i chunksize  i 1  chunksize    gt  gt  gt  for chunk in chunker range 0 100   10           print list chunk        0  1  2  3  4  5  6  7  8  9   10  11  12  13  14  15  16  17  18  19   20  21  22  23  24  25  26  27  28  29      etc

User · Answer

See this reference   gt  gt  gt  orange   range 1  1001   gt  gt  gt  otuples   list  zip   iter orange   10    gt  gt  gt  print otuples    1  2  3  4  5  6  7  8  9  10        991  992  993  994  995  996  997  998  999  1000    gt  gt  gt  olist    list i  for i in otuples   gt  gt  gt  print olist    1  2  3  4  5  6  7  8  9  10         991  992  993  994  995  996  997  998  999  1000    gt  gt  gt     Python3

User · Answer

I like the Python doc s version proposed by tzot and J F Sebastian a lot   but it has two shortcomings    it is not very explicit I usually don t want a fill value in the last chunk   I m using this one a lot in my code   from itertools import islice  def chunks n  iterable       iterable   iter iterable      while True          yield tuple islice iterable  n   or iterable next     UPDATE  A lazy chunks version   from itertools import chain  islice  def chunks n  iterable      iterable   iter iterable     while True         yield chain  next iterable    islice iterable  n-1

User · Answer

I have come up to following solution without creation temorary list object, which should work with any iterable object. Please note that this version for Python 2.x:

def chunked(iterable, size):
    stop = []
    it = iter(iterable)
    def _next_chunk():
        try:
            for _ in xrange(size):
                yield next(it)
        except StopIteration:
            stop.append(True)
            return

    while not stop:
        yield _next_chunk()

for it in chunked(xrange(16), 4):
   print list(it)

Output:

[0, 1, 2, 3]
[4, 5, 6, 7]
[8, 9, 10, 11]
[12, 13, 14, 15] 
[]

As you can see if len(iterable) % size == 0 then we have additional empty iterator object. But I do not think that it is big problem.

User · Answer

AA i i SS  for i in range len AA     SS     Where AA is array  SS is chunk size  For example    gt  gt  gt  AA range 10 21  SS 3  gt  gt  gt   AA i i SS  for i in range len AA     SS     10  11  12    13  14  15    16  17  18    19  20     or  range 10  13   range 13  16   range 16  19   range 19  21   in py3

User · Answer

An old school approach that does not require itertools but still works with arbitrary generators   def chunks g  n        divide a generator  g  into small chunks   Yields      a chunk that has  n  or less items         n   max 1  n    buff        for item in g      buff append item      if len buff     n        yield buff       buff        if buff      yield buff

User · Answer

I have one solution below which does work but more important than that solution is a few comments on other approaches   First  a good solution shouldn t require that one loop through the sub-iterators in order   If I run  g   paged iter list range 50    11   i0   next g  i1   next g  list i1  list i0    The appropriate output for the last command is       0  1  2  3  4  5  6  7  8  9  10    not        As most of the itertools based solutions here return   This isn t just the usual boring restriction about accessing iterators in order   Imagine a consumer trying to clean up poorly entered data which reversed the appropriate order of blocks of 5  i e   the data looks like  B5  A5  D5  C5  and should look like  A5  B5  C5  D5   where A5 is just five elements not a sublist    This consumer would look at the claimed behavior of the grouping function and not hesitate to write a loop like  i   0 out      for it in paged iter data 5      if  i   2    0            swapped   it     else            out    list it           out    list swapped      i   i   1   This will produce mysteriously wrong results if you sneakily assume that sub-iterators are always fully used in order   It gets even worse if you want to interleave elements from the chunks    Second  a decent number of the suggested solutions implicitly rely on the fact that iterators have a deterministic order  they don t e g  set  and while some of the solutions using islice may be ok it worries me   Third  the itertools grouper approach works but the recipe relies on internal behavior of the zip longest  or zip  functions that isn t part of their published behavior   In particular  the grouper function only works because in zip longest i0   in  the next function is always called in order next i0   next i1       next in  before starting over   As grouper passes n copies of the same iterator object it relies on this behavior   Finally  while the solution below can be improved if you make the assumption criticized above that sub-iterators are accessed in order and fully perused without this assumption one MUST implicitly  via call chain  or explicitly  via deques or other data structure  store elements for each subiterator somewhere   So don t bother wasting time  as I did  assuming one could get around this with some clever trick   def paged iter iterat  n       itr   iter iterat      deq   None     try          while True               deq   collections deque maxlen n              for q in range n                   deq append next itr               yield  i for i in deq      except StopIteration          yield  i for i in deq

User · Answer

If you want something super simple   def chunks l  n       n   max 1  n      return  l i i n  for i in range 0  len l   n     Use xrange   instead of range   in the case of Python 2 x

User · Answer

def chunk lst       out          for x in xrange 2  len lst    1           if not len lst    x              factor   len lst    x             break     while lst          out append  lst pop 0  for x in xrange factor        return out

User · Answer

using List Comprehensions of python   range t t 10  for t in range 1 1000 10      1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20     21  22  23  24  25  26  27  28  29  30             981  982  983  984  985  986  987  988  989  990     991  992  993  994  995  996  997  998  999  1000     visit this link to know about List Comprehensions

User · Answer

If you know list size   def SplitList mylist  chunk size       return  mylist offs offs chunk size  for offs in range 0  len mylist   chunk size     If you don t  an iterator    def IterChunks sequence  chunk size       res          for item in sequence          res append item          if len res   gt   chunk size              yield res             res          if res          yield res    yield the last  incomplete  portion   In the latter case  it can be rephrased in a more beautiful way if you can be sure that the sequence always contains a whole number of chunks of given size  i e  there is no incomplete last chunk

User · Answer

python pydash package could be a good choice    from pydash arrays import chunk ids     22    89    2    3    4    5    6    7    8    9    10    11    1   chunk ids   chunk ids 5  print chunk ids    output     22    89    2    3    4      5    6    7    8    9      10    11    1      for more checkout pydash chunk list

User · Answer

No magic  but simple and correct   def chunks iterable  n          Yield successive n-sized chunks from iterable         values          for i  item in enumerate iterable  1           values append item          if i   n    0              yield values             values          if values          yield values

User · Answer

Since everybody here talking about iterators  boltons has perfect method for that  called iterutils chunked iter   from boltons import iterutils  list iterutils chunked iter list range 50    11     Output     0  1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20  21     22  23  24  25  26  27  28  29  30  31  32     33  34  35  36  37  38  39  40  41  42  43     44  45  46  47  48  49     But if you don t want to be mercy on memory  you can use old-way and store the full list in the first place with iterutils chunked

User · Answer

A generic chunker for any iterable  which gives the user a choice of how to handle a partial chunk at the end   Tested on Python 3   chunker py  from enum import Enum  class PartialChunkOptions Enum       INCLUDE   0     EXCLUDE   1     PAD   2     ERROR   3  class PartialChunkException Exception       pass  def chunker iterable  n  on partial PartialChunkOptions INCLUDE  pad None               A chunker yielding n-element lists from an iterable  with various options     about what to do about a partial chunk at the end       on partial PartialChunkOptions INCLUDE  the default                        include the partial chunk as a short   lt n  element list      on partial PartialChunkOptions EXCLUDE                      do not include the partial chunk      on partial PartialChunkOptions PAD                      pad to an n-element list                        also pass pad  lt pad value gt   default None       on partial PartialChunkOptions ERROR                      raise a RuntimeError if a partial chunk is encountered              on partial   PartialChunkOptions on partial               iterator   iter iterable      while True          vals              for i in range n               try                  vals append next iterator               except StopIteration                  if vals                      if on partial    PartialChunkOptions INCLUDE                          yield vals                     elif on partial    PartialChunkOptions EXCLUDE                          pass                     elif on partial    PartialChunkOptions PAD                          yield vals    pad     n - len vals                       elif on partial    PartialChunkOptions ERROR                          raise PartialChunkException                     return                 return         yield vals   test py  import chunker  chunk size   3  for it in  range 100  107             range 100  109         print   nITERABLE TO CHUNK      format it       print  CHUNK SIZE      format chunk size        for option in chunker PartialChunkOptions   members   values            print   noption    used  format option           try              for chunk in chunker chunker it  chunk size  on partial option                   print chunk          except chunker PartialChunkException              print  PartialChunkException was raised       print       output of test py   ITERABLE TO CHUNK  range 100  107  CHUNK SIZE  3  option PartialChunkOptions INCLUDE used  100  101  102   103  104  105   106   option PartialChunkOptions EXCLUDE used  100  101  102   103  104  105   option PartialChunkOptions PAD used  100  101  102   103  104  105   106  None  None   option PartialChunkOptions ERROR used  100  101  102   103  104  105  PartialChunkException was raised   ITERABLE TO CHUNK  range 100  109  CHUNK SIZE  3  option PartialChunkOptions INCLUDE used  100  101  102   103  104  105   106  107  108   option PartialChunkOptions EXCLUDE used  100  101  102   103  104  105   106  107  108   option PartialChunkOptions PAD used  100  101  102   103  104  105   106  107  108   option PartialChunkOptions ERROR used  100  101  102   103  104  105   106  107  108

User · Answer

Since I had to do something like this  here s my solution given a generator and a batch size   def pop n elems from generator g  n       elems          try          for idx in xrange 0  n               elems append g next            return elems     except StopIteration          return elems

User · Answer

With Assignment Expressions in Python 3 8 it becomes quite nice   import itertools  def batch iterable  size       it   iter iterable      while item    list itertools islice it  size            yield item   This works on an arbitrary iterable  not just a list    gt  gt  gt  import pprint  gt  gt  gt  pprint pprint list batch range 75   10      0  1  2  3  4  5  6  7  8  9     10  11  12  13  14  15  16  17  18  19     20  21  22  23  24  25  26  27  28  29     30  31  32  33  34  35  36  37  38  39     40  41  42  43  44  45  46  47  48  49     50  51  52  53  54  55  56  57  58  59     60  61  62  63  64  65  66  67  68  69     70  71  72  73  74

User · Answer

Yes  it is an old question  but I had to post this one  because it is even a little shorter than the similar ones  Yes  the result looks scrambled  but if it is just about even length      gt  gt  gt  n   3   number of groups  gt  gt  gt  biglist   range 30   gt  gt  gt   gt  gt  gt    biglist i  n  for i in xrange n      0  3  6  9  12  15  18  21  24  27     1  4  7  10  13  16  19  22  25  28     2  5  8  11  14  17  20  23  26  29

User · Answer

I m surprised nobody has thought of using iter s two-argument form   from itertools import islice  def chunk it  size       it   iter it      return iter lambda  tuple islice it  size          Demo    gt  gt  gt  list chunk range 14   3     0  1  2    3  4  5    6  7  8    9  10  11    12  13     This works with any iterable and produces output lazily  It returns tuples rather than iterators  but I think it has a certain elegance nonetheless  It also doesn t pad  if you want padding  a simple variation on the above will suffice   from itertools import islice  chain  repeat  def chunk pad it  size  padval None       it   chain iter it   repeat padval       return iter lambda  tuple islice it  size     padval     size    Demo    gt  gt  gt  list chunk pad range 14   3     0  1  2    3  4  5    6  7  8    9  10  11    12  13  None    gt  gt  gt  list chunk pad range 14   3   a      0  1  2    3  4  5    6  7  8    9  10  11    12  13   a      Like the izip longest-based solutions  the above always pads  As far as I know  there s no one- or two-line itertools recipe for a function that optionally pads  By combining the above two approaches  this one comes pretty close    no padding   object    def chunk it  size  padval  no padding       if padval     no padding          it   iter it          sentinel          else          it   chain iter it   repeat padval           sentinel    padval     size     return iter lambda  tuple islice it  size    sentinel    Demo    gt  gt  gt  list chunk range 14   3     0  1  2    3  4  5    6  7  8    9  10  11    12  13    gt  gt  gt  list chunk range 14   3  None     0  1  2    3  4  5    6  7  8    9  10  11    12  13  None    gt  gt  gt  list chunk range 14   3   a      0  1  2    3  4  5    6  7  8    9  10  11    12  13   a      I believe this is the shortest chunker proposed that offers optional padding   As Tomasz Gandor observed  the two padding chunkers will stop unexpectedly if they encounter a long sequence of pad values  Here s a final variation that works around that problem in a reasonable way    no padding   object   def chunk it  size  padval  no padding       it   iter it      chunker   iter lambda  tuple islice it  size            if padval     no padding          yield from chunker     else          for ch in chunker              yield ch if len ch     size else ch    padval      size - len ch     Demo    gt  gt  gt  list chunk  1  2          5   2     1  2              5     gt  gt  gt  list chunk  1  2  None  None  5   2  None     1  2    None  None    5  None

User · Answer

I wrote a small library expressly for this purpose  available here  The library s chunked function is particularly efficient because it s implemented as a generator  so a substantial amount of memory can be saved in certain situations  It also doesn t rely on the slice notation  so any arbitrary iterator can be used   import iterlib  print list iterlib chunked xrange 1  1000   10     prints   1  2  3  4  5  6  7  8  9  10    11  12  13  14  15  16  17  18  19  20

User · Answer

If you know list size   def SplitList mylist  chunk size       return  mylist offs offs chunk size  for offs in range 0  len mylist   chunk size     If you don t  an iterator    def IterChunks sequence  chunk size       res          for item in sequence          res append item          if len res   gt   chunk size              yield res             res          if res          yield res    yield the last  incomplete  portion   In the latter case  it can be rephrased in a more beautiful way if you can be sure that the sequence always contains a whole number of chunks of given size  i e  there is no incomplete last chunk

User · Answer

def split seq seq  num pieces       start   0     for i in xrange num pieces           stop   start   len seq i  num pieces           yield seq start stop          start   stop   usage   seq    1  2  3  4  5  6  7  8  9  10   for seq in split seq seq  3       print seq

User · Answer

I ve created these two fancy one-liners which are efficient and lazy  both input and output are iterables  also they doen t depend on any module  First one-liner is totally lazy meaning that it returns iterator producing iterators  i e  each chunk produced is iterator iterating over chunk s elements   this version is good for the case if chunks are very large or elements are produced slowly one by one and should become available immediately as they are produced  Try it online  chunk iters   lambda it  n    e for i  g in enumerate   f    cit   for j  e in zip range  1  n - 1  i    g   for cit in  iter it    for f in cit   Second one-liner returns iterator that produces lists  Each list is produced as soon as elements of whole chunk become available through input iterator or if very last element of last chunk is reached  This version should be used if input elements are produced fast or all available immediately  Other wise first more-lazy one-liner version should be used  Try it online  chunk lists   lambda it  n   l for l in       for i  g in enumerate  it          for e in g for l in  l  len l    n     e   1 - i    if  len l    n    0     i   Also I provide multi-line version of first chunk iters one-liner  which returns iterator producing another iterators  going through each chunk s elements   Try it online  def chunk iters it  n       cit   iter it      def one chunk f           yield f         for i  e in zip range n - 1   cit               yield e     for f in cit          yield one chunk f

User · Answer

This question reminds me of the Raku  formerly Perl 6   comb n  method   It breaks up strings into n-sized chunks    There s more to it than that  but I ll leave out the details   It s easy enough to implement a similar function in Python3 as a lambda expression  comb   lambda s n   s i i n  for i in range 0 len s  n    Then you can call it like this  some list   list range 0  20      creates a list of 20 elements generator   comb some list  4     creates a generator that will generate lists of 4 elements for sublist in generator      print sublist     prints a sublist of four elements  as it s generated  Of course  you don t have to assign the generator to a variable  you can just loop over it directly like this  for sublist in comb some list  4       print sublist     prints a sublist of four elements  as it s generated  As a bonus  this comb   function also operates on strings  list  comb  catdogant   3       returns   cat    dog    ant

User · Answer

Another more explicit version   def chunkList initialList  chunkSize               This function chunks a list into sub lists      that have a length equals to chunkSize       Example      lst    3  4  9  7  1  1  2  3      print chunkList lst  3        returns       3  4  9    7  1  1    2  3               finalList          for i in range 0  len initialList   chunkSize           finalList append initialList i i chunkSize       return finalList

User · Answer

heh  one line version  In  48   chunk   lambda ulist  step   map lambda i  ulist i i step    xrange 0  len ulist   step    In  49   chunk range 1 100   10  Out 49      1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20     21  22  23  24  25  26  27  28  29  30     31  32  33  34  35  36  37  38  39  40     41  42  43  44  45  46  47  48  49  50     51  52  53  54  55  56  57  58  59  60     61  62  63  64  65  66  67  68  69  70     71  72  73  74  75  76  77  78  79  80     81  82  83  84  85  86  87  88  89  90     91  92  93  94  95  96  97  98  99

User · Answer

Directly from the  old  Python documentation  recipes for itertools    from itertools import izip  chain  repeat  def grouper n  iterable  padvalue None        grouper 3   abcdefg    x   -- gt    a   b   c      d   e   f      g   x   x        return izip   chain iterable  repeat padvalue  n-1    n    The current version  as suggested by J F Sebastian    from itertools import izip longest as zip longest   for Python 2 x from itertools import zip longest   for Python 3 x  from six moves import zip longest   for both  uses the six compat library   def grouper n  iterable  padvalue None        grouper 3   abcdefg    x   -- gt    a   b   c      d   e   f      g   x   x        return zip longest   iter iterable   n  fillvalue padvalue    I guess Guido s time machine works   worked   will work   will have worked   was working again   These solutions work because  iter iterable   n  or the equivalent in the earlier version  creates one iterator  repeated n times in the list  izip longest then effectively performs a round-robin of  each  iterator  because this is the same iterator  it is advanced by each such call  resulting in each such zip-roundrobin generating one tuple of n items

User · Answer

Works with any iterable Inner data is generator object  not a list  One liner    In  259   get in chunks   lambda itr n     v for   v in g  for   g in itertools groupby enumerate itr  lambda  ind     ind n    In  260   list list x  for x in get in chunks range 30  7   Out 260     0  1  2  3  4  5  6     7  8  9  10  11  12  13     14  15  16  17  18  19  20     21  22  23  24  25  26  27     28  29

User · Answer

def chunk lst       out          for x in xrange 2  len lst    1           if not len lst    x              factor   len lst    x             break     while lst          out append  lst pop 0  for x in xrange factor        return out

User · Answer

Use list comprehensions   l    1 2 3 4 5 6 7 8 9 10 11 12  k   5  chunk size print  tuple l x y   for  x  y  in   x  x k  for x in range 0  len l   k

User · Answer

At this point  I think we need the obligatory anonymous-recursive function   Y   lambda f   lambda x  x x   lambda y  f lambda  args  y y   args    chunks   Y lambda f  lambda n   n 0   n 1      f  n 0  n 1     n 1    if len n 0    gt  0 else

User · Answer

See this reference   gt  gt  gt  orange   range 1  1001   gt  gt  gt  otuples   list  zip   iter orange   10    gt  gt  gt  print otuples    1  2  3  4  5  6  7  8  9  10        991  992  993  994  995  996  997  998  999  1000    gt  gt  gt  olist    list i  for i in otuples   gt  gt  gt  print olist    1  2  3  4  5  6  7  8  9  10         991  992  993  994  995  996  997  998  999  1000    gt  gt  gt     Python3

User · Answer

You could use numpy s array split function e g   np array split np array data   20  to split into 20 nearly equal size chunks   To make sure chunks are exactly equal in size use np split

User · Answer

Simple yet elegant  l   range 1  1000  print  l x x 10  for x in xrange 0  len l   10     or if you prefer   def chunks l  n   return  l x  x n  for x in xrange 0  len l   n   chunks l  10

User · Answer

I have one solution below which does work but more important than that solution is a few comments on other approaches   First  a good solution shouldn t require that one loop through the sub-iterators in order   If I run  g   paged iter list range 50    11   i0   next g  i1   next g  list i1  list i0    The appropriate output for the last command is       0  1  2  3  4  5  6  7  8  9  10    not        As most of the itertools based solutions here return   This isn t just the usual boring restriction about accessing iterators in order   Imagine a consumer trying to clean up poorly entered data which reversed the appropriate order of blocks of 5  i e   the data looks like  B5  A5  D5  C5  and should look like  A5  B5  C5  D5   where A5 is just five elements not a sublist    This consumer would look at the claimed behavior of the grouping function and not hesitate to write a loop like  i   0 out      for it in paged iter data 5      if  i   2    0            swapped   it     else            out    list it           out    list swapped      i   i   1   This will produce mysteriously wrong results if you sneakily assume that sub-iterators are always fully used in order   It gets even worse if you want to interleave elements from the chunks    Second  a decent number of the suggested solutions implicitly rely on the fact that iterators have a deterministic order  they don t e g  set  and while some of the solutions using islice may be ok it worries me   Third  the itertools grouper approach works but the recipe relies on internal behavior of the zip longest  or zip  functions that isn t part of their published behavior   In particular  the grouper function only works because in zip longest i0   in  the next function is always called in order next i0   next i1       next in  before starting over   As grouper passes n copies of the same iterator object it relies on this behavior   Finally  while the solution below can be improved if you make the assumption criticized above that sub-iterators are accessed in order and fully perused without this assumption one MUST implicitly  via call chain  or explicitly  via deques or other data structure  store elements for each subiterator somewhere   So don t bother wasting time  as I did  assuming one could get around this with some clever trick   def paged iter iterat  n       itr   iter iterat      deq   None     try          while True               deq   collections deque maxlen n              for q in range n                   deq append next itr               yield  i for i in deq      except StopIteration          yield  i for i in deq

User · Answer

def chunk lst       out          for x in xrange 2  len lst    1           if not len lst    x              factor   len lst    x             break     while lst          out append  lst pop 0  for x in xrange factor        return out

User · Answer

Like  AaronHall I got here looking for roughly evenly sized chunks  There are different interpretations of that  In my case  if the desired size is N  I would like each group to be of size  N  Thus  the orphans which are created in most of the above should be redistributed to other groups   This can be done using   def nChunks l  n           Yield n successive chunks from l      Works for lists   pandas dataframes  etc             newn   int 1 0   len l    n   0 5      for i in xrange 0  n-1           yield l i newn i newn newn      yield l n newn-newn      from Splitting a list of into N parts of approximately equal length  by simply calling it as nChunks l l n   or nChunks l floor l n

User · Answer

def split seq seq  num pieces       start   0     for i in xrange num pieces           stop   start   len seq i  num pieces           yield seq start stop          start   stop   usage   seq    1  2  3  4  5  6  7  8  9  10   for seq in split seq seq  3       print seq

User · Answer

Another more explicit version   def chunkList initialList  chunkSize               This function chunks a list into sub lists      that have a length equals to chunkSize       Example      lst    3  4  9  7  1  1  2  3      print chunkList lst  3        returns       3  4  9    7  1  1    2  3               finalList          for i in range 0  len initialList   chunkSize           finalList append initialList i i chunkSize       return finalList

User · Answer

At this point  I think we need a recursive generator  just in case     In python 2   def chunks li  n       if li                return     yield li  n      for e in chunks li n    n           yield e   In python 3   def chunks li  n       if li                return     yield li  n      yield from chunks li n    n    Also  in case of massive Alien invasion  a decorated recursive generator might become handy   def dec gen       def new gen li  n           for e in gen li  n               if e                        return             yield e     return new gen   dec def chunks li  n       yield li  n      for e in chunks li n    n           yield e

User · Answer

Directly from the  old  Python documentation  recipes for itertools    from itertools import izip  chain  repeat  def grouper n  iterable  padvalue None        grouper 3   abcdefg    x   -- gt    a   b   c      d   e   f      g   x   x        return izip   chain iterable  repeat padvalue  n-1    n    The current version  as suggested by J F Sebastian    from itertools import izip longest as zip longest   for Python 2 x from itertools import zip longest   for Python 3 x  from six moves import zip longest   for both  uses the six compat library   def grouper n  iterable  padvalue None        grouper 3   abcdefg    x   -- gt    a   b   c      d   e   f      g   x   x        return zip longest   iter iterable   n  fillvalue padvalue    I guess Guido s time machine works   worked   will work   will have worked   was working again   These solutions work because  iter iterable   n  or the equivalent in the earlier version  creates one iterator  repeated n times in the list  izip longest then effectively performs a round-robin of  each  iterator  because this is the same iterator  it is advanced by each such call  resulting in each such zip-roundrobin generating one tuple of n items

User · Answer

At this point  I think we need a recursive generator  just in case     In python 2   def chunks li  n       if li                return     yield li  n      for e in chunks li n    n           yield e   In python 3   def chunks li  n       if li                return     yield li  n      yield from chunks li n    n    Also  in case of massive Alien invasion  a decorated recursive generator might become handy   def dec gen       def new gen li  n           for e in gen li  n               if e                        return             yield e     return new gen   dec def chunks li  n       yield li  n      for e in chunks li n    n           yield e

User · Answer

Yes  it is an old question  but I had to post this one  because it is even a little shorter than the similar ones  Yes  the result looks scrambled  but if it is just about even length      gt  gt  gt  n   3   number of groups  gt  gt  gt  biglist   range 30   gt  gt  gt   gt  gt  gt    biglist i  n  for i in xrange n      0  3  6  9  12  15  18  21  24  27     1  4  7  10  13  16  19  22  25  28     2  5  8  11  14  17  20  23  26  29

User · Answer

def split seq seq  num pieces       start   0     for i in xrange num pieces           stop   start   len seq i  num pieces           yield seq start stop          start   stop   usage   seq    1  2  3  4  5  6  7  8  9  10   for seq in split seq seq  3       print seq

User · Answer

If you don t care about the order    gt  from itertools import groupby  gt  batch no   3  gt  data    abcdefgh    gt         x 1  for x in x 1        for x in      groupby        sorted           x 0    batch no  x 1            for x in          enumerate data                 key lambda x  x 0                a    d    g      b    e    h      c    f       This solution doesn t generates sets of same size  but distributes values so batches are as big as possible while keeping the number of generated batches

User · Answer

gt  gt  gt  def f x  n  acc      return f x n    n  acc   x  n     if x else acc  gt  gt  gt  f  Hallo Welt   3    Hal    lo     Wel    t    gt  gt  gt     If you are into brackets - I picked up a book on Erlang

User · Answer

def main      print chunkify  1 2 3 4 5 6  2    def chunkify list  n     chunks        for i in range 0  len list   n       chunks append list i i n     return chunks  main     I think that it s simple and can give you a chunk of an array

User · Answer

An abstraction would be l    1 2 3 4 5 6 7 8 9  n   3 outList      for i in range n  len l    n  n       outList append l i-n i    print outList   This will print     1  2  3    4  5  6    7  8  9

User · Answer

I was curious about the performance of different approaches and here it is   Tested on Python 3 5 1  import time batch size   7 arr len   298937   ---------slice-------------  print   r nslice   start   time time   arr    i for i in range 0  arr len   while True      if not arr          break      tmp   arr 0 batch size      arr   arr batch size -1  print time time   - start    -----------index-----------  print   r nindex   arr    i for i in range 0  arr len   start   time time   for i in range 0  round len arr    batch size   1        tmp   arr batch size   i   batch size    i   1   print time time   - start    ----------batches 1------------  def batch iterable  n 1       l   len iterable      for ndx in range 0  l  n           yield iterable ndx min ndx   n  l    print   r nbatches 1   arr    i for i in range 0  arr len   start   time time   for x in batch arr  batch size       tmp   x print time time   - start    ----------batches 2------------  from itertools import islice  chain  def batch iterable  size       sourceiter   iter iterable      while True          batchiter   islice sourceiter  size          yield chain  next batchiter    batchiter    print   r nbatches 2   arr    i for i in range 0  arr len   start   time time   for x in batch arr  batch size       tmp   x print time time   - start    ---------chunks------------- def chunks l  n          Yield successive n-sized chunks from l         for i in range 0  len l   n           yield l i i   n  print   r nchunks   arr    i for i in range 0  arr len   start   time time   for x in chunks arr  batch size       tmp   x print time time   - start    -----------grouper-----------  from itertools import zip longest   for Python 3 x  from six moves import zip longest   for both  uses the six compat library   def grouper iterable  n  padvalue None        grouper 3   abcdefg    x   -- gt    a   b   c      d   e   f      g   x   x        return zip longest   iter iterable   n  fillvalue padvalue   arr    i for i in range 0  arr len   print   r ngrouper   start   time time   for x in grouper arr  batch size       tmp   x print time time   - start    Results   slice 31 18285083770752  index 0 02184295654296875  batches 1 0 03503894805908203  batches 2 0 22681021690368652  chunks 0 019841909408569336  grouper 0 006506919860839844

User · Answer

I don t think I saw this option  so just to add another one        def chunks iterable  chunk size     i   0    while i  lt  len iterable       yield iterable i i chunk size      i    chunk size

User · Answer

Directly from the  old  Python documentation  recipes for itertools    from itertools import izip  chain  repeat  def grouper n  iterable  padvalue None        grouper 3   abcdefg    x   -- gt    a   b   c      d   e   f      g   x   x        return izip   chain iterable  repeat padvalue  n-1    n    The current version  as suggested by J F Sebastian    from itertools import izip longest as zip longest   for Python 2 x from itertools import zip longest   for Python 3 x  from six moves import zip longest   for both  uses the six compat library   def grouper n  iterable  padvalue None        grouper 3   abcdefg    x   -- gt    a   b   c      d   e   f      g   x   x        return zip longest   iter iterable   n  fillvalue padvalue    I guess Guido s time machine works   worked   will work   will have worked   was working again   These solutions work because  iter iterable   n  or the equivalent in the earlier version  creates one iterator  repeated n times in the list  izip longest then effectively performs a round-robin of  each  iterator  because this is the same iterator  it is advanced by each such call  resulting in each such zip-roundrobin generating one tuple of n items

User · Answer

I wrote a small library expressly for this purpose  available here  The library s chunked function is particularly efficient because it s implemented as a generator  so a substantial amount of memory can be saved in certain situations  It also doesn t rely on the slice notation  so any arbitrary iterator can be used   import iterlib  print list iterlib chunked xrange 1  1000   10     prints   1  2  3  4  5  6  7  8  9  10    11  12  13  14  15  16  17  18  19  20

User · Answer

Here is a generator that work on arbitrary iterables   def split seq iterable  size       it   iter iterable      item   list itertools islice it  size       while item          yield item         item   list itertools islice it  size     Example    gt  gt  gt  import pprint  gt  gt  gt  pprint pprint list split seq xrange 75   10      0  1  2  3  4  5  6  7  8  9     10  11  12  13  14  15  16  17  18  19     20  21  22  23  24  25  26  27  28  29     30  31  32  33  34  35  36  37  38  39     40  41  42  43  44  45  46  47  48  49     50  51  52  53  54  55  56  57  58  59     60  61  62  63  64  65  66  67  68  69     70  71  72  73  74

User · Answer

python pydash package could be a good choice    from pydash arrays import chunk ids     22    89    2    3    4    5    6    7    8    9    10    11    1   chunk ids   chunk ids 5  print chunk ids    output     22    89    2    3    4      5    6    7    8    9      10    11    1      for more checkout pydash chunk list

User · Answer

a    1  2  3  4  5  6  7  8  9  CHUNK   4  a i CHUNK  i 1  CHUNK  for i in xrange  len a    CHUNK - 1    CHUNK

User · Answer

Without calling len   which is good for large lists   def splitter l  n       i   0     chunk   l  n      while chunk          yield chunk         i    n         chunk   l i i n    And this is for iterables   def isplitter l  n       l   iter l      chunk   list islice l  n       while chunk          yield chunk         chunk   list islice l  n     The functional flavour of the above   def isplitter2 l  n       return takewhile bool                        tuple islice start  n                               for start in repeat iter l       OR   def chunks gen sentinel n  seq       continuous slices   imap islice  repeat iter seq    repeat 0   repeat n       return iter imap tuple  continuous slices  next       OR   def chunks gen filter n  seq       continuous slices   imap islice  repeat iter seq    repeat 0   repeat n       return takewhile bool imap tuple  continuous slices

User · Answer

If you want something super simple   def chunks l  n       n   max 1  n      return  l i i n  for i in range 0  len l   n     Use xrange   instead of range   in the case of Python 2 x

User · Answer

An old school approach that does not require itertools but still works with arbitrary generators   def chunks g  n        divide a generator  g  into small chunks   Yields      a chunk that has  n  or less items         n   max 1  n    buff        for item in g      buff append item      if len buff     n        yield buff       buff        if buff      yield buff

User · Answer

Here s a generator that yields the chunks you want   def chunks lst  n          Yield successive n-sized chunks from lst         for i in range 0  len lst   n           yield lst i i   n      import pprint pprint pprint list chunks range 10  75   10      10  11  12  13  14  15  16  17  18  19     20  21  22  23  24  25  26  27  28  29     30  31  32  33  34  35  36  37  38  39     40  41  42  43  44  45  46  47  48  49     50  51  52  53  54  55  56  57  58  59     60  61  62  63  64  65  66  67  68  69     70  71  72  73  74       If you re using Python 2  you should use xrange   instead of range     def chunks lst  n          Yield successive n-sized chunks from lst         for i in xrange 0  len lst   n           yield lst i i   n      Also you can simply use list comprehension instead of writing a function  though it s a good idea to encapsulate operations like this in named functions so that your code is easier to understand  Python 3    lst i i   n  for i in range 0  len lst   n     Python 2 version    lst i i   n  for i in xrange 0  len lst   n

User · Answer

Here is a list of additional approaches   Given  import itertools as it import collections as ct  import more itertools as mit   iterable   range 11  n   3   Code  The Standard Library  list it zip longest   iter iterable     n       0  1  2    3  4  5    6  7  8    9  10  None       d      for i  x in enumerate iterable       d setdefault i  n      append x   list d values        0  1  2    3  4  5    6  7  8    9  10       dd   ct defaultdict list  for i  x in enumerate iterable       dd i  n  append x   list dd values        0  1  2    3  4  5    6  7  8    9  10     more itertools   list mit chunked iterable  n       0  1  2    3  4  5    6  7  8    9  10    list mit sliced iterable  n      range 0  3   range 3  6   range 6  9   range 9  11    list mit grouper n  iterable       0  1  2    3  4  5    6  7  8    9  10  None    list mit windowed iterable  len iterable   n  step n       0  1  2    3  4  5    6  7  8    9  10  None     References   zip longest  related post  related post  setdefault  ordered results requires Python 3 6   collections defaultdict   ordered results requires Python 3 6   more itertools chunked  related posted  more itertools sliced more itertools grouper  related post  more itertools windowed  see also stagger  zip offset      A third-party library that implements itertools recipes and more   gt  pip install more itertools

User · Answer

The toolz library has the partition function for this   from toolz itertoolz core import partition  list partition 2   1  2  3  4      1  2    3  4

User · Answer

def chunk lst       out          for x in xrange 2  len lst    1           if not len lst    x              factor   len lst    x             break     while lst          out append  lst pop 0  for x in xrange factor        return out

User · Answer

letting r be the chunk size  and L be the initial list  you can do    chunkL      i for i in L r k r  k 1     for k in range len L  r

User · Answer

If you know list size   def SplitList mylist  chunk size       return  mylist offs offs chunk size  for offs in range 0  len mylist   chunk size     If you don t  an iterator    def IterChunks sequence  chunk size       res          for item in sequence          res append item          if len res   gt   chunk size              yield res             res          if res          yield res    yield the last  incomplete  portion   In the latter case  it can be rephrased in a more beautiful way if you can be sure that the sequence always contains a whole number of chunks of given size  i e  there is no incomplete last chunk

User · Answer

Here s an idea using itertools groupby   def chunks l  n       c   itertools count       return  it for    it in itertools groupby l  lambda x  next c   n     This returns a generator of generators  If you want a list of lists  just replace the last line with      return  list it  for    it in itertools groupby l  lambda x  next c   n     Example returning list of lists    gt  gt  gt  chunks  abcdefghij   4     a    b    c    d      e    f    g    h      i    j       So yes  this suffers form the  runt problem   which may or may not be a problem in a given situation

User · Answer

Here is a generator that work on arbitrary iterables   def split seq iterable  size       it   iter iterable      item   list itertools islice it  size       while item          yield item         item   list itertools islice it  size     Example    gt  gt  gt  import pprint  gt  gt  gt  pprint pprint list split seq xrange 75   10      0  1  2  3  4  5  6  7  8  9     10  11  12  13  14  15  16  17  18  19     20  21  22  23  24  25  26  27  28  29     30  31  32  33  34  35  36  37  38  39     40  41  42  43  44  45  46  47  48  49     50  51  52  53  54  55  56  57  58  59     60  61  62  63  64  65  66  67  68  69     70  71  72  73  74

User · Answer

As per this answer  the top-voted answer leaves a  runt  at the end  Here s my solution to really get about as evenly-sized chunks as you can  with no runts  It basically tries to pick exactly the fractional spot where it should split the list  but just rounds it off to the nearest integer   from   future   import division    not needed in Python 3 def n even chunks l  n          Yield n as even chunks as possible from l         last   0     for i in range 1  n 1           cur   int round i    len l    n            yield l last cur          last   cur   Demonstration    gt  gt  gt  pprint pprint list n even chunks list range 100    9      0  1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20  21     22  23  24  25  26  27  28  29  30  31  32     33  34  35  36  37  38  39  40  41  42  43     44  45  46  47  48  49  50  51  52  53  54  55     56  57  58  59  60  61  62  63  64  65  66     67  68  69  70  71  72  73  74  75  76  77     78  79  80  81  82  83  84  85  86  87  88     89  90  91  92  93  94  95  96  97  98  99    gt  gt  gt  pprint pprint list n even chunks list range 100    11      0  1  2  3  4  5  6  7  8     9  10  11  12  13  14  15  16  17     18  19  20  21  22  23  24  25  26     27  28  29  30  31  32  33  34  35     36  37  38  39  40  41  42  43  44     45  46  47  48  49  50  51  52  53  54     55  56  57  58  59  60  61  62  63     64  65  66  67  68  69  70  71  72     73  74  75  76  77  78  79  80  81     82  83  84  85  86  87  88  89  90     91  92  93  94  95  96  97  98  99     Compare to the top-voted chunks answer    gt  gt  gt  pprint pprint list chunks list range 100    100  9      0  1  2  3  4  5  6  7  8  9  10     11  12  13  14  15  16  17  18  19  20  21     22  23  24  25  26  27  28  29  30  31  32     33  34  35  36  37  38  39  40  41  42  43     44  45  46  47  48  49  50  51  52  53  54     55  56  57  58  59  60  61  62  63  64  65     66  67  68  69  70  71  72  73  74  75  76     77  78  79  80  81  82  83  84  85  86  87     88  89  90  91  92  93  94  95  96  97  98     99    gt  gt  gt  pprint pprint list chunks list range 100    100  11      0  1  2  3  4  5  6  7  8     9  10  11  12  13  14  15  16  17     18  19  20  21  22  23  24  25  26     27  28  29  30  31  32  33  34  35     36  37  38  39  40  41  42  43  44     45  46  47  48  49  50  51  52  53     54  55  56  57  58  59  60  61  62     63  64  65  66  67  68  69  70  71     72  73  74  75  76  77  78  79  80     81  82  83  84  85  86  87  88  89     90  91  92  93  94  95  96  97  98     99

User · Answer

How do you split a list into evenly sized chunks   quot Evenly sized chunks quot   to me  implies that they are all the same length  or barring that option  at minimal variance in length  E g  5 baskets for 21 items could have the following results   gt  gt  gt  import statistics  gt  gt  gt  statistics variance  5 5 5 5 1    3 2  gt  gt  gt  statistics variance  5 4 4 4 4    0 19999999999999998  A practical reason to prefer the latter result  if you were using these functions to distribute work  you ve built-in the prospect of one likely finishing well before the others  so it would sit around doing nothing while the others continued working hard  Critique of other answers here When I originally wrote this answer  none of the other answers were evenly sized chunks - they all leave a runt chunk at the end  so they re not well balanced  and have a higher than necessary variance of lengths  For example  the current top answer ends with   60  61  62  63  64  65  66  67  68  69    70  71  72  73  74    Others  like list grouper 3  range 7     and chunk range 7   3  both return    0  1  2    3  4  5    6  None  None    The None s are just padding  and rather inelegant in my opinion  They are NOT evenly chunking the iterables  Why can t we divide these better  Cycle Solution A high-level balanced solution using itertools cycle  which is the way I might do it today  Here s the setup  from itertools import cycle items   range 10  75  number of baskets   10  Now we need our lists into which to populate the elements  baskets       for   in range number of baskets    Finally  we zip the elements we re going to allocate together with a cycle of the baskets until we run out of elements  which  semantically  it exactly what we want  for element  basket in zip items  cycle baskets        basket append element   Here s the result   gt  gt  gt  from pprint import pprint  gt  gt  gt  pprint baskets    10  20  30  40  50  60  70     11  21  31  41  51  61  71     12  22  32  42  52  62  72     13  23  33  43  53  63  73     14  24  34  44  54  64  74     15  25  35  45  55  65     16  26  36  46  56  66     17  27  37  47  57  67     18  28  38  48  58  68     19  29  39  49  59  69    To productionize this solution  we write a function  and provide the type annotations  from itertools import cycle from typing import List  Any  def cycle baskets items  List Any   maxbaskets  int  - gt  List List Any        baskets       for   in range min maxbaskets  len items         for item  basket in zip items  cycle baskets            basket append item      return baskets  In the above  we take our list of items  and the max number of baskets  We create a list of empty lists  in which to append each element  in a round-robin style  Slices Another elegant solution is to use slices - specifically the less-commonly used step argument to slices  i e   start   0 stop   None step   number of baskets  first basket   items start stop step   This is especially elegant in that slices don t care how long the data are - the result  our first basket  is only as long as it needs to be  We ll only need to increment the starting point for each basket  In fact this could be a one-liner  but we ll go multiline for readability and to avoid an overlong line of code  from typing import List  Any  def slice baskets items  List Any   maxbaskets  int  - gt  List List Any        n baskets   min maxbaskets  len items       return  items i  n baskets  for i in range n baskets    And islice from the itertools module will provide a lazily iterating approach  like that which was originally asked for in the question  I don t expect most use-cases to benefit very much  as the original data is already fully materialized in a list  but for large datasets  it could save nearly half the memory usage  from itertools import islice from typing import List  Any  Generator      def yield islice baskets items  List Any   maxbaskets  int  - gt  Generator List Any   None  None       n baskets   min maxbaskets  len items       for i in range n baskets           yield islice items  i  None  n baskets   View results with  from pprint import pprint  items   list range 10  75   pprint cycle baskets items  10   pprint slice baskets items  10   pprint  list s  for s in yield islice baskets items  10     Updated prior solutions Here s another balanced solution  adapted from a function I ve used in production in the past  that uses the modulo operator  def baskets from items  maxbaskets 25       baskets       for   in range maxbaskets       for i  item in enumerate items           baskets i   maxbaskets  append item      return filter None  baskets    And I created a generator that does the same if you put it into a list  def iter baskets from items  maxbaskets 3          generates evenly balanced baskets from indexable iterable        item count   len items      baskets   min item count  maxbaskets      for x i in range baskets           yield  items y i  for y i in range x i  item count  baskets         And finally  since I see that all of the above functions return elements in a contiguous order  as they were given   def iter baskets contiguous items  maxbaskets 3  item count None               generates balanced baskets from iterable  contiguous contents     provide item count if providing a iterator that doesn t support len               item count   item count or len items      baskets   min item count  maxbaskets      items   iter items      floor   item count    baskets      ceiling   floor   1     stepdown   item count   baskets     for x i in range baskets           length   ceiling if x i  lt  stepdown else floor         yield  items next   for   in range length    Output To test them out  print baskets from range 6   8   print list iter baskets from range 6   8    print list iter baskets contiguous range 6   8    print baskets from range 22   8   print list iter baskets from range 22   8    print list iter baskets contiguous range 22   8    print baskets from  ABCDEFG   3   print list iter baskets from  ABCDEFG   3    print list iter baskets contiguous  ABCDEFG   3    print baskets from range 26   5   print list iter baskets from range 26   5    print list iter baskets contiguous range 26   5     Which prints out    0    1    2    3    4    5     0    1    2    3    4    5     0    1    2    3    4    5     0  8  16    1  9  17    2  10  18    3  11  19    4  12  20    5  13  21    6  14    7  15     0  8  16    1  9  17    2  10  18    3  11  19    4  12  20    5  13  21    6  14    7  15     0  1  2    3  4  5    6  7  8    9  10  11    12  13  14    15  16  17    18  19    20  21      A    D    G      B    E      C    F       A    D    G      B    E      C    F       A    B    C      D    E      F    G      0  5  10  15  20  25    1  6  11  16  21    2  7  12  17  22    3  8  13  18  23    4  9  14  19  24     0  5  10  15  20  25    1  6  11  16  21    2  7  12  17  22    3  8  13  18  23    4  9  14  19  24     0  1  2  3  4  5    6  7  8  9  10    11  12  13  14  15    16  17  18  19  20    21  22  23  24  25    Notice that the contiguous generator provide chunks in the same length patterns as the other two  but the items are all in order  and they are as evenly divided as one may divide a list of discrete elements

User · Answer

With Assignment Expressions in Python 3 8 it becomes quite nice   import itertools  def batch iterable  size       it   iter iterable      while item    list itertools islice it  size            yield item   This works on an arbitrary iterable  not just a list    gt  gt  gt  import pprint  gt  gt  gt  pprint pprint list batch range 75   10      0  1  2  3  4  5  6  7  8  9     10  11  12  13  14  15  16  17  18  19     20  21  22  23  24  25  26  27  28  29     30  31  32  33  34  35  36  37  38  39     40  41  42  43  44  45  46  47  48  49     50  51  52  53  54  55  56  57  58  59     60  61  62  63  64  65  66  67  68  69     70  71  72  73  74

User · Answer

Without calling len   which is good for large lists   def splitter l  n       i   0     chunk   l  n      while chunk          yield chunk         i    n         chunk   l i i n    And this is for iterables   def isplitter l  n       l   iter l      chunk   list islice l  n       while chunk          yield chunk         chunk   list islice l  n     The functional flavour of the above   def isplitter2 l  n       return takewhile bool                        tuple islice start  n                               for start in repeat iter l       OR   def chunks gen sentinel n  seq       continuous slices   imap islice  repeat iter seq    repeat 0   repeat n       return iter imap tuple  continuous slices  next       OR   def chunks gen filter n  seq       continuous slices   imap islice  repeat iter seq    repeat 0   repeat n       return takewhile bool imap tuple  continuous slices

User · Answer

Works with any iterable Inner data is generator object  not a list  One liner    In  259   get in chunks   lambda itr n     v for   v in g  for   g in itertools groupby enumerate itr  lambda  ind     ind n    In  260   list list x  for x in get in chunks range 30  7   Out 260     0  1  2  3  4  5  6     7  8  9  10  11  12  13     14  15  16  17  18  19  20     21  22  23  24  25  26  27     28  29

[python] How do you split a list into evenly sized chunks?

Examples related to python

Examples related to list

Examples related to split

Examples related to chunks