What is the most pythonic way to iterate over a list in chunks

Question

I have a Python script which takes as input a list of integers  which I need to work with four integers at a time   Unfortunately  I don t have control of the input  or I d have it passed in as a list of four-element tuples   Currently  I m iterating over it this way   for i in xrange 0  len ints   4         dummy op for example code     foo    ints i    ints i   1    ints i   2    ints i   3    It looks a lot like  C-think   though  which makes me suspect there s a more pythonic way of dealing with this situation   The list is discarded after iterating  so it needn t be preserved   Perhaps something like this would be better   while ints      foo    ints 0    ints 1    ints 2    ints 3      ints 0 4         Still doesn t quite  feel  right  though    -   Related question  How do you split a list into evenly sized chunks in Python

User · Answer

Similar to other proposals  but not exactly identical  I like doing it this way  because it s simple and easy to read   it   iter  1  2  3  4  5  6  7  8  9   for chunk in zip it  it  it  it       print chunk   gt  gt  gt   1  2  3  4   gt  gt  gt   5  6  7  8    This way you won t get the last partial chunk  If you want to get  9  None  None  None  as last chunk  just use izip longest from itertools

User · Answer

Quite pythonic here  you may also inline the body of the split groups function   import itertools def split groups iter in  group size       return   x for    x in item  for    item in itertools groupby enumerate iter in   key lambda x  x 0     group size    for x  y  z  w in split groups range 16   4       foo    x   y   z   w

User · Answer

from itertools import izip longest  def chunker iterable  chunksize  filler       return izip longest   iter iterable   chunksize  fillvalue filler

User · Answer

Modified from the recipes section of Python s itertools docs   from itertools import zip longest  def grouper iterable  n  fillvalue None       args    iter iterable     n     return zip longest  args  fillvalue fillvalue    Example In pseudocode to keep the example terse   grouper  ABCDEFG   3   x   -- gt   ABC   DEF   Gxx    Note  on Python 2 use izip longest instead of zip longest

User · Answer

You can use partition or chunks function from funcy library   from funcy import partition  for a  b  c  d in partition 4  ints       foo    a   b   c   d   These functions also has iterator versions ipartition and ichunks  which will be more efficient in this case   You can also peek at their implementation

User · Answer

Here is a chunker without imports that supports generators   def chunks seq  size       it   iter seq      while True          ret   tuple next it  for   in range size           if len ret     size              yield ret         else              raise StopIteration     Example of use    gt  gt  gt  def foo            i   0         while True              i    1             yield i      gt  gt  gt  c   chunks foo    3   gt  gt  gt  c next    1  2  3   gt  gt  gt  c next    4  5  6   gt  gt  gt  list chunks  abcdefg   2      a    b      c    d      e    f

User · Answer

from itertools import izip longest  def chunker iterable  chunksize  filler       return izip longest   iter iterable   chunksize  fillvalue filler

User · Answer

At first  I designed it to split strings into substrings to parse string containing hex  Today I turned it into complex  but still simple generator   def chunker iterable  size  reductor  condition       it   iter iterable      def chunk generator            return  next it  for   in range size       chunk   reductor chunk generator        while condition chunk           yield chunk         chunk   reductor chunk generator      Arguments   Obvious ones   iterable is any iterable   iterator   generator containg   generating   iterating over input data  size is  of course  size of chunk you want get    More interesting   reductor is a callable  which receives generator iterating over content of chunk  I d expect it to return sequence or string  but I don t demand that   You can pass as this argument for example list  tuple  set  frozenset  or anything fancier  I d pass this function  returning string  provided that iterable contains   generates   iterates over strings    def concatenate iterable       return    join iterable    Note that reductor can cause closing generator by raising exception  condition is a callable which receives anything what reductor returned  It decides to approve  amp  yield it  by returning anything evaluating to True   or to decline it  amp  finish generator s work  by returning anything other or raising exception    When number of elements in iterable is not divisible by size  when it gets exhausted  reductor will receive generator generating less elements than size  Let s call these elements lasts elements   I invited two functions to pass as this argument      lambda x x - the lasts elements will be yielded  lambda x  len x    lt size gt  - the lasts elements will be rejected  replace  lt size gt  using number equal to size

User · Answer

Yet another answer  the advantages of which are   1  Easily understandable 2  Works on any iterable  not just sequences  some of the above answers will choke on filehandles  3  Does not load the chunk into memory all at once 4  Does not make a chunk-long list of references to the same iterator in memory 5  No padding of fill values at the end of the list  That being said  I haven t timed it so it might be slower than some of the more clever methods  and some of the advantages may be irrelevant given the use case   def chunkiter iterable  size     def inneriter first  iterator  size       yield first     for   in xrange size - 1          yield iterator next     it   iter iterable    while True      yield inneriter it next    it  size   In  2   i   chunkiter  abcdefgh   3  In  3   for ii in i                                                            for c in ii              print c            print                              a b c          d e f          g h    Update  A couple of drawbacks due to the fact the inner and outer loops are pulling values from the same iterator  1  continue doesn t work as expected in the outer loop - it just continues on to the next item rather than skipping a chunk  However  this doesn t seem like a problem as there s nothing to test in the outer loop  2  break doesn t work as expected in the inner loop - control will wind up in the inner loop again with the next item in the iterator  To skip whole chunks  either wrap the inner iterator  ii above  in a tuple  e g  for c in tuple ii   or set a flag and exhaust the iterator

User · Answer

The more-itertools package has chunked method which does exactly that  import more itertools for s in more itertools chunked range 9   4       print s   Prints  0  1  2  3   4  5  6  7   8   chunked returns the items in a list  If you d prefer iterables  use ichunked

User · Answer

def group by iterable  size          Group an iterable into lists that don t exceed the size given        gt  gt  gt  group by  1 2 3 4 5   2        1  2    3  4    5                sublist           for index  item in enumerate iterable           if index  gt  0 and index   size    0              yield sublist             sublist               sublist append item       if sublist          yield sublist

User · Answer

def chunker iterable  n          Yield iterable in chunk sizes        gt  gt  gt  chunks   chunker  ABCDEF   n 4       gt  gt  gt  chunks next         A    B    C    D        gt  gt  gt  chunks next         E    F               it   iter iterable      while True          chunk              for i in range n               try                  chunk append next it               except StopIteration                  yield chunk                 raise StopIteration         yield chunk  if   name         main         import doctest      doctest testmod

User · Answer

In your second method  I would advance to the next group of 4 by doing this   ints   ints 4     However  I haven t done any performance measurement so I don t know which one might be more efficient   Having said that  I would usually choose the first method  It s not pretty  but that s often a consequence of interfacing with the outside world

User · Answer

In your second method  I would advance to the next group of 4 by doing this   ints   ints 4     However  I haven t done any performance measurement so I don t know which one might be more efficient   Having said that  I would usually choose the first method  It s not pretty  but that s often a consequence of interfacing with the outside world

User · Answer

Using map   instead of zip   fixes the padding issue in J F  Sebastian s answer    gt  gt  gt  def chunker iterable  chunksize         return map None   iter iterable   chunksize    Example    gt  gt  gt  s    1234567890   gt  gt  gt  chunker s  3     1    2    3      4    5    6      7    8    9      0   None  None    gt  gt  gt  chunker s  4     1    2    3    4      5    6    7    8      9    0   None  None    gt  gt  gt  chunker s  5     1    2    3    4    5      6    7    8    9    0

User · Answer

def chunker seq  size       return  seq pos pos   size  for pos in range 0  len seq   size      in python 2 use xrange   instead of range   to avoid allocating a list   Works with any sequence  text    quot I am a very  very helpful text quot   for group in chunker text  7      print repr group       I am a    very  v   ery hel   pful te   xt   print     join chunker text  10     I am a ver y  very he lpful text  animals     cat    dog    rabbit    duck    bird    cow    gnu    fish    for group in chunker animals  3       print group      cat    dog    rabbit       duck    bird    cow       gnu    fish

User · Answer

def chunker seq  size       return  seq pos pos   size  for pos in range 0  len seq   size      in python 2 use xrange   instead of range   to avoid allocating a list   Works with any sequence  text    quot I am a very  very helpful text quot   for group in chunker text  7      print repr group       I am a    very  v   ery hel   pful te   xt   print     join chunker text  10     I am a ver y  very he lpful text  animals     cat    dog    rabbit    duck    bird    cow    gnu    fish    for group in chunker animals  3       print group      cat    dog    rabbit       duck    bird    cow       gnu    fish

User · Answer

I needed a solution that would also work with sets and generators  I couldn t come up with anything very short and pretty  but it s quite readable at least   def chunker seq  size       res          for el in seq          res append el          if len res     size              yield res             res          if res          yield res   List    gt  gt  gt  list chunker  i for i in range 10    3     0  1  2    3  4  5    6  7  8    9     Set    gt  gt  gt  list chunker set  i for i in range 10     3     0  1  2    3  4  5    6  7  8    9     Generator    gt  gt  gt  list chunker  i for i in range 10    3     0  1  2    3  4  5    6  7  8    9

User · Answer

If you don t mind using an external package you could use iteration utilities grouper from iteration utilties 1  It supports all iterables  not just sequences    from iteration utilities import grouper seq   list range 20   for group in grouper seq  4       print group    which prints    0  1  2  3   4  5  6  7   8  9  10  11   12  13  14  15   16  17  18  19    In case the length isn t a multiple of the groupsize it also supports filling  the incomplete last group  or truncating  discarding the incomplete last group  the last one   from iteration utilities import grouper seq   list range 17   for group in grouper seq  4       print group     0  1  2  3     4  5  6  7     8  9  10  11     12  13  14  15     16    for group in grouper seq  4  fillvalue None       print group     0  1  2  3     4  5  6  7     8  9  10  11     12  13  14  15     16  None  None  None   for group in grouper seq  4  truncate True       print group     0  1  2  3     4  5  6  7     8  9  10  11     12  13  14  15      Benchmarks  I also decided to compare the run-time of a few of the mentioned approaches  It s a log-log plot grouping into groups of  10  elements based on a list of varying size  For qualitative results  Lower means faster     At least in this benchmark the iteration utilities grouper performs best  Followed by the approach of Craz    The benchmark was created with simple benchmark1  The code used to run this benchmark was   import iteration utilities import itertools from itertools import zip longest  def consume all it       return iteration utilities consume it  None   import simple benchmark b   simple benchmark BenchmarkBuilder     b add function   def grouper l  n       return consume all iteration utilities grouper l  n    def Craz inner iterable  n  fillvalue None       args    iter iterable     n     return zip longest  args  fillvalue fillvalue    b add function   def Craz iterable  n  fillvalue None       return consume all Craz inner iterable  n  fillvalue    def nosklo inner seq  size       return  seq pos pos   size  for pos in range 0  len seq   size     b add function   def nosklo seq  size       return consume all nosklo inner seq  size    def SLott inner ints  chunk size       for i in range 0  len ints   chunk size           yield ints i i chunk size    b add function   def SLott ints  chunk size       return consume all SLott inner ints  chunk size    def MarkusJarderot1 inner iterable size       it   iter iterable      chunk   tuple itertools islice it size       while chunk          yield chunk         chunk   tuple itertools islice it size     b add function   def MarkusJarderot1 iterable size       return consume all MarkusJarderot1 inner iterable size    def MarkusJarderot2 inner iterable size filler None       it   itertools chain iterable itertools repeat filler size-1       chunk   tuple itertools islice it size       while len chunk     size          yield chunk         chunk   tuple itertools islice it size     b add function   def MarkusJarderot2 iterable size       return consume all MarkusJarderot2 inner iterable size     b add arguments   def argument provider        for exp in range 2  20           size   2  exp         yield size  simple benchmark MultiArgument   0    size  10    r   b run       1 Disclaimer  I m the author of the libraries iteration utilities and simple benchmark

User · Answer

If the lists are the same size  you can combine them into lists of 4-tuples with zip    For example     Four lists of four elements each   l1   range 0  4  l2   range 4  8  l3   range 8  12  l4   range 12  16   for i1  i2  i3  i4 in zip l1  l2  l3  l4             Here s what the zip   function produces    gt  gt  gt  print l1  0  1  2  3   gt  gt  gt  print l2  4  5  6  7   gt  gt  gt  print l3  8  9  10  11   gt  gt  gt  print l4  12  13  14  15   gt  gt  gt  print zip l1  l2  l3  l4    0  4  8  12    1  5  9  13    2  6  10  14    3  7  11  15     If the lists are large  and you don t want to combine them into a bigger list  use itertools izip    which produces an iterator  rather than a list   from itertools import izip  for i1  i2  i3  i4 in izip l1  l2  l3  l4

User · Answer

With Python 3 8 you can use the walrus operator and itertools islice   from itertools import islice  list     i for i in range 10  100    def chunker it  size       iterator   iter it      while chunk    list islice iterator  size            print chunk    In  2   chunker list   10                                                            10  11  12  13  14  15  16  17  18  19   20  21  22  23  24  25  26  27  28  29   30  31  32  33  34  35  36  37  38  39   40  41  42  43  44  45  46  47  48  49   50  51  52  53  54  55  56  57  58  59   60  61  62  63  64  65  66  67  68  69   70  71  72  73  74  75  76  77  78  79   80  81  82  83  84  85  86  87  88  89   90  91  92  93  94  95  96  97  98  99

User · Answer

import itertools def chunks iterable size       it   iter iterable      chunk   tuple itertools islice it size       while chunk          yield chunk         chunk   tuple itertools islice it size      though this will throw ValueError if the length of ints   isn t a multiple of four  for x1 x2 x3 x4 in chunks ints 4       foo    x1   x2   x3   x4  for chunk in chunks ints 4       foo    sum chunk    Another way   import itertools def chunks2 iterable size filler None       it   itertools chain iterable itertools repeat filler size-1       chunk   tuple itertools islice it size       while len chunk     size          yield chunk         chunk   tuple itertools islice it size      x2  x3 and x4 could get the value 0 if the length is not   a multiple of 4  for x1 x2 x3 x4 in chunks2 ints 4 0       foo    x1   x2   x3   x4

User · Answer

You can use partition or chunks function from funcy library   from funcy import partition  for a  b  c  d in partition 4  ints       foo    a   b   c   d   These functions also has iterator versions ipartition and ichunks  which will be more efficient in this case   You can also peek at their implementation

User · Answer

Similar to other proposals  but not exactly identical  I like doing it this way  because it s simple and easy to read   it   iter  1  2  3  4  5  6  7  8  9   for chunk in zip it  it  it  it       print chunk   gt  gt  gt   1  2  3  4   gt  gt  gt   5  6  7  8    This way you won t get the last partial chunk  If you want to get  9  None  None  None  as last chunk  just use izip longest from itertools

User · Answer

Using little functions and things really doesn t appeal to me  I prefer to just use slices   data         chunk size   10000   or whatever chunks    data i i chunk size  for i in xrange 0 len data  chunk size   for chunk in chunks

User · Answer

def chunker seq  size       return  seq pos pos   size  for pos in range 0  len seq   size      in python 2 use xrange   instead of range   to avoid allocating a list   Works with any sequence  text    quot I am a very  very helpful text quot   for group in chunker text  7      print repr group       I am a    very  v   ery hel   pful te   xt   print     join chunker text  10     I am a ver y  very he lpful text  animals     cat    dog    rabbit    duck    bird    cow    gnu    fish    for group in chunker animals  3       print group      cat    dog    rabbit       duck    bird    cow       gnu    fish

User · Answer

The ideal solution for this problem works with iterators  not just sequences   It should also be fast   This is the solution provided by the documentation for itertools   def grouper n  iterable  fillvalue None         grouper 3   ABCDEFG    x   -- gt  ABC DEF Gxx      args    iter iterable     n     return itertools izip longest fillvalue fillvalue   args    Using ipython s  timeit on my mac book air  I get 47 5 us per loop   However  this really doesn t work for me since the results are padded to be even sized groups  A solution without the padding is slightly more complicated  The most naive solution might be   def grouper size  iterable       i   iter iterable      while True          out              try              for   in range size                   out append i next            except StopIteration              yield out             break          yield out   Simple  but pretty slow  693 us per loop  The best solution I could come up with uses islice for the inner loop   def grouper size  iterable       it   iter iterable      while True          group   tuple itertools islice it  None  size           if not group              break         yield group   With the same dataset  I get 305 us per loop   Unable to get a pure solution any faster than that  I provide the following solution with an important caveat  If your input data has instances of filldata in it  you could get wrong answer   def grouper n  iterable  fillvalue None         grouper 3   ABCDEFG    x   -- gt  ABC DEF Gxx      args    iter iterable     n     for i in itertools izip longest fillvalue fillvalue   args           if tuple i  -1     fillvalue              yield tuple v for v in i if v    fillvalue          else              yield i   I really don t like this answer  but it is significantly faster  124 us per loop

User · Answer

The ideal solution for this problem works with iterators  not just sequences   It should also be fast   This is the solution provided by the documentation for itertools   def grouper n  iterable  fillvalue None         grouper 3   ABCDEFG    x   -- gt  ABC DEF Gxx      args    iter iterable     n     return itertools izip longest fillvalue fillvalue   args    Using ipython s  timeit on my mac book air  I get 47 5 us per loop   However  this really doesn t work for me since the results are padded to be even sized groups  A solution without the padding is slightly more complicated  The most naive solution might be   def grouper size  iterable       i   iter iterable      while True          out              try              for   in range size                   out append i next            except StopIteration              yield out             break          yield out   Simple  but pretty slow  693 us per loop  The best solution I could come up with uses islice for the inner loop   def grouper size  iterable       it   iter iterable      while True          group   tuple itertools islice it  None  size           if not group              break         yield group   With the same dataset  I get 305 us per loop   Unable to get a pure solution any faster than that  I provide the following solution with an important caveat  If your input data has instances of filldata in it  you could get wrong answer   def grouper n  iterable  fillvalue None         grouper 3   ABCDEFG    x   -- gt  ABC DEF Gxx      args    iter iterable     n     for i in itertools izip longest fillvalue fillvalue   args           if tuple i  -1     fillvalue              yield tuple v for v in i if v    fillvalue          else              yield i   I really don t like this answer  but it is significantly faster  124 us per loop

User · Answer

If the list is large  the highest-performing way to do this will be to use a generator   def get chunk iterable  chunk size       result          for item in iterable          result append item          if len result     chunk size              yield tuple result              result          if len result   gt  0          yield tuple result   for x in get chunk  1 2 3 4 5 6 7 8 9 10   3       print x   1  2  3   4  5  6   7  8  9   10

User · Answer

def chunker seq  size       return  seq pos pos   size  for pos in range 0  len seq   size      in python 2 use xrange   instead of range   to avoid allocating a list   Works with any sequence  text    quot I am a very  very helpful text quot   for group in chunker text  7      print repr group       I am a    very  v   ery hel   pful te   xt   print     join chunker text  10     I am a ver y  very he lpful text  animals     cat    dog    rabbit    duck    bird    cow    gnu    fish    for group in chunker animals  3       print group      cat    dog    rabbit       duck    bird    cow       gnu    fish

User · Answer

About solution gave by J F  Sebastian here   def chunker iterable  chunksize       return zip   iter iterable   chunksize    It s clever  but has one disadvantage - always return tuple  How to get string instead  Of course you can write    join chunker        but the temporary tuple is constructed anyway   You can get rid of the temporary tuple by writing own zip  like this   class IteratorExhausted Exception       pass  def translate StopIteration iterable  to IteratorExhausted       for i in iterable          yield i     raise to   StopIteration would get ignored because this is generator                 but custom exception can leave the generator   def custom zip  iterables  reductor tuple       iterators   tuple map translate StopIteration  iterables       while True          try              yield reductor next i  for i in iterators          except IteratorExhausted    when any of iterators get exhausted              break   Then  def chunker data  size  reductor tuple       return custom zip   iter data   size  reductor reductor    Example usage    gt  gt  gt  for i in chunker  12345   2           print repr i         1    2     3    4    gt  gt  gt  for i in chunker  12345   2     join           print repr i        12   34

User · Answer

To avoid all conversions to a list import itertools and    gt  gt  gt  for k  g in itertools groupby xrange 35   lambda x  x 10           list g    Produces        0  0  1  2  3  4  5  6  7  8  9  1  10  11  12  13  14  15  16  17  18  19  2  20  21  22  23  24  25  26  27  28  29  3  30  31  32  33  34   gt  gt  gt     I checked groupby and it doesn t convert to list or use len so I  think  this will delay resolution of each value until it is actually used   Sadly none of the available answers  at this time  seemed to offer this variation   Obviously if you need to handle each item in turn nest a for loop over g   for k g in itertools groupby xrange 35   lambda x  x 10       for i in g           do what you need to do with individual items       now do what you need to do with the whole group   My specific interest in this was the need to consume a generator to submit changes in batches of up to 1000 to the gmail API       messages   a generator which would not be smart as a list     for idx  batch in groupby messages  lambda x  x 1000           batch request   BatchHttpRequest           for message in batch              batch request add self service users   messages   modify userId  me   id message  id    body msg labels           http   httplib2 Http           self credentials authorize http          batch request execute http http

User · Answer

Here is a chunker without imports that supports generators   def chunks seq  size       it   iter seq      while True          ret   tuple next it  for   in range size           if len ret     size              yield ret         else              raise StopIteration     Example of use    gt  gt  gt  def foo            i   0         while True              i    1             yield i      gt  gt  gt  c   chunks foo    3   gt  gt  gt  c next    1  2  3   gt  gt  gt  c next    4  5  6   gt  gt  gt  list chunks  abcdefg   2      a    b      c    d      e    f

User · Answer

import itertools def chunks iterable size       it   iter iterable      chunk   tuple itertools islice it size       while chunk          yield chunk         chunk   tuple itertools islice it size      though this will throw ValueError if the length of ints   isn t a multiple of four  for x1 x2 x3 x4 in chunks ints 4       foo    x1   x2   x3   x4  for chunk in chunks ints 4       foo    sum chunk    Another way   import itertools def chunks2 iterable size filler None       it   itertools chain iterable itertools repeat filler size-1       chunk   tuple itertools islice it size       while len chunk     size          yield chunk         chunk   tuple itertools islice it size      x2  x3 and x4 could get the value 0 if the length is not   a multiple of 4  for x1 x2 x3 x4 in chunks2 ints 4 0       foo    x1   x2   x3   x4

User · Answer

If the list is large  the highest-performing way to do this will be to use a generator   def get chunk iterable  chunk size       result          for item in iterable          result append item          if len result     chunk size              yield tuple result              result          if len result   gt  0          yield tuple result   for x in get chunk  1 2 3 4 5 6 7 8 9 10   3       print x   1  2  3   4  5  6   7  8  9   10

User · Answer

It is easy to make itertools groupby work for you to get an iterable of iterables  without creating any temporary lists   groupby iterable   lambda x y   lambda z  x next   y   count   100     Don t get put off by the nested lambdas  outer lambda runs just once to put count   generator and the constant 100 into the scope of the inner lambda   I use this to send chunks of rows to mysql   for k v in groupby bigdata   lambda x y   lambda z  x next   y   count   100         cursor executemany sql  v

User · Answer

Quite pythonic here  you may also inline the body of the split groups function   import itertools def split groups iter in  group size       return   x for    x in item  for    item in itertools groupby enumerate iter in   key lambda x  x 0     group size    for x  y  z  w in split groups range 16   4       foo    x   y   z   w

User · Answer

Yet another answer  the advantages of which are   1  Easily understandable 2  Works on any iterable  not just sequences  some of the above answers will choke on filehandles  3  Does not load the chunk into memory all at once 4  Does not make a chunk-long list of references to the same iterator in memory 5  No padding of fill values at the end of the list  That being said  I haven t timed it so it might be slower than some of the more clever methods  and some of the advantages may be irrelevant given the use case   def chunkiter iterable  size     def inneriter first  iterator  size       yield first     for   in xrange size - 1          yield iterator next     it   iter iterable    while True      yield inneriter it next    it  size   In  2   i   chunkiter  abcdefgh   3  In  3   for ii in i                                                            for c in ii              print c            print                              a b c          d e f          g h    Update  A couple of drawbacks due to the fact the inner and outer loops are pulling values from the same iterator  1  continue doesn t work as expected in the outer loop - it just continues on to the next item rather than skipping a chunk  However  this doesn t seem like a problem as there s nothing to test in the outer loop  2  break doesn t work as expected in the inner loop - control will wind up in the inner loop again with the next item in the iterator  To skip whole chunks  either wrap the inner iterator  ii above  in a tuple  e g  for c in tuple ii   or set a flag and exhaust the iterator

User · Answer

Unless I misses something  the following simple solution with generator expressions has not been mentioned   It assumes that both the size and the number of chunks are known  which is often the case   and that no padding is required   def chunks it  n  m          Make an iterator over m first chunks of size n              it   iter it        Chunks are presented as tuples      return  tuple next it  for   in range n   for   in range m

User · Answer

Using little functions and things really doesn t appeal to me  I prefer to just use slices   data         chunk size   10000   or whatever chunks    data i i chunk size  for i in xrange 0 len data  chunk size   for chunk in chunks

User · Answer

Another approach would be to use the two-argument form of iter    from itertools import islice  def group it  size       it   iter it      return iter lambda  tuple islice it  size          This can be adapted easily to use padding  this is similar to Markus Jarderot   s answer    from itertools import islice  chain  repeat  def group pad it  size  pad None       it   chain iter it   repeat pad       return iter lambda  tuple islice it  size     pad     size    These can even be combined for optional padding    no pad   object   def group it  size  pad  no pad       if pad     no pad          it   iter it          sentinel          else          it   chain iter it   repeat pad           sentinel    pad     size     return iter lambda  tuple islice it  size    sentinel

User · Answer

If the lists are the same size  you can combine them into lists of 4-tuples with zip    For example     Four lists of four elements each   l1   range 0  4  l2   range 4  8  l3   range 8  12  l4   range 12  16   for i1  i2  i3  i4 in zip l1  l2  l3  l4             Here s what the zip   function produces    gt  gt  gt  print l1  0  1  2  3   gt  gt  gt  print l2  4  5  6  7   gt  gt  gt  print l3  8  9  10  11   gt  gt  gt  print l4  12  13  14  15   gt  gt  gt  print zip l1  l2  l3  l4    0  4  8  12    1  5  9  13    2  6  10  14    3  7  11  15     If the lists are large  and you don t want to combine them into a bigger list  use itertools izip    which produces an iterator  rather than a list   from itertools import izip  for i1  i2  i3  i4 in izip l1  l2  l3  l4

User · Answer

If the lists are the same size  you can combine them into lists of 4-tuples with zip    For example     Four lists of four elements each   l1   range 0  4  l2   range 4  8  l3   range 8  12  l4   range 12  16   for i1  i2  i3  i4 in zip l1  l2  l3  l4             Here s what the zip   function produces    gt  gt  gt  print l1  0  1  2  3   gt  gt  gt  print l2  4  5  6  7   gt  gt  gt  print l3  8  9  10  11   gt  gt  gt  print l4  12  13  14  15   gt  gt  gt  print zip l1  l2  l3  l4    0  4  8  12    1  5  9  13    2  6  10  14    3  7  11  15     If the lists are large  and you don t want to combine them into a bigger list  use itertools izip    which produces an iterator  rather than a list   from itertools import izip  for i1  i2  i3  i4 in izip l1  l2  l3  l4

User · Answer

There doesn t seem to be a pretty way to do this   Here is a page that has a number of methods  including   def split seq seq  size       newseq          splitsize   1 0 size len seq      for i in range size           newseq append seq int round i splitsize   int round  i 1  splitsize         return newseq

User · Answer

With NumPy it s simple   ints   array  1  2  3  4  5  6  7  8   for int1  int2 in ints reshape -1  2       print int1  int2    output   1 2 3 4 5 6 7 8

User · Answer

I needed a solution that would also work with sets and generators  I couldn t come up with anything very short and pretty  but it s quite readable at least   def chunker seq  size       res          for el in seq          res append el          if len res     size              yield res             res          if res          yield res   List    gt  gt  gt  list chunker  i for i in range 10    3     0  1  2    3  4  5    6  7  8    9     Set    gt  gt  gt  list chunker set  i for i in range 10     3     0  1  2    3  4  5    6  7  8    9     Generator    gt  gt  gt  list chunker  i for i in range 10    3     0  1  2    3  4  5    6  7  8    9

User · Answer

I never want my chunks padded  so that requirement is essential   I find that the ability to work on any iterable is also requirement   Given that  I decided to extend on the accepted answer  https   stackoverflow com a 434411 1074659     Performance takes a slight hit in this approach if padding is not wanted due to the need to compare and filter the padded values   However  for large chunk sizes  this utility is very performant      usr bin env python3 from itertools import zip longest    UNDEFINED   object     def chunker iterable  chunksize  fillvalue  UNDEFINED               Collect data into chunks and optionally pad it       Performance worsens as  chunksize  approaches 1       Inspired by          https   docs python org 3 library itertools html itertools-recipes              args    iter iterable     chunksize     chunks   zip longest  args  fillvalue fillvalue      yield from           filter lambda val  val is not  UNDEFINED  chunk          if chunk -1  is  UNDEFINED         else chunk         for chunk in chunks       if fillvalue is  UNDEFINED else chunks

User · Answer

import itertools def chunks iterable size       it   iter iterable      chunk   tuple itertools islice it size       while chunk          yield chunk         chunk   tuple itertools islice it size      though this will throw ValueError if the length of ints   isn t a multiple of four  for x1 x2 x3 x4 in chunks ints 4       foo    x1   x2   x3   x4  for chunk in chunks ints 4       foo    sum chunk    Another way   import itertools def chunks2 iterable size filler None       it   itertools chain iterable itertools repeat filler size-1       chunk   tuple itertools islice it size       while len chunk     size          yield chunk         chunk   tuple itertools islice it size      x2  x3 and x4 could get the value 0 if the length is not   a multiple of 4  for x1 x2 x3 x4 in chunks2 ints 4 0       foo    x1   x2   x3   x4

User · Answer

Since nobody s mentioned it yet here s a zip   solution    gt  gt  gt  def chunker iterable  chunksize           return zip   iter iterable   chunksize    It works only if your sequence s length is always divisible by the chunk size or you don t care about a trailing chunk if it isn t   Example    gt  gt  gt  s    1234567890   gt  gt  gt  chunker s  3     1    2    3      4    5    6      7    8    9     gt  gt  gt  chunker s  4     1    2    3    4      5    6    7    8     gt  gt  gt  chunker s  5     1    2    3    4    5      6    7    8    9    0      Or using itertools izip to return an iterator instead of a list    gt  gt  gt  from itertools import izip  gt  gt  gt  def chunker iterable  chunksize           return izip   iter iterable   chunksize    Padding can be fixed using    O      s answer    gt  gt  gt  from itertools import chain  izip  repeat  gt  gt  gt  def chunker iterable  chunksize  fillvalue None           it     chain iterable  repeat fillvalue  chunksize-1           args    it    chunksize         return izip  args

User · Answer

If the lists are the same size  you can combine them into lists of 4-tuples with zip    For example     Four lists of four elements each   l1   range 0  4  l2   range 4  8  l3   range 8  12  l4   range 12  16   for i1  i2  i3  i4 in zip l1  l2  l3  l4             Here s what the zip   function produces    gt  gt  gt  print l1  0  1  2  3   gt  gt  gt  print l2  4  5  6  7   gt  gt  gt  print l3  8  9  10  11   gt  gt  gt  print l4  12  13  14  15   gt  gt  gt  print zip l1  l2  l3  l4    0  4  8  12    1  5  9  13    2  6  10  14    3  7  11  15     If the lists are large  and you don t want to combine them into a bigger list  use itertools izip    which produces an iterator  rather than a list   from itertools import izip  for i1  i2  i3  i4 in izip l1  l2  l3  l4

User · Answer

If the list is large  the highest-performing way to do this will be to use a generator   def get chunk iterable  chunk size       result          for item in iterable          result append item          if len result     chunk size              yield tuple result              result          if len result   gt  0          yield tuple result   for x in get chunk  1 2 3 4 5 6 7 8 9 10   3       print x   1  2  3   4  5  6   7  8  9   10

User · Answer

Modified from the recipes section of Python s itertools docs   from itertools import zip longest  def grouper iterable  n  fillvalue None       args    iter iterable     n     return zip longest  args  fillvalue fillvalue    Example In pseudocode to keep the example terse   grouper  ABCDEFG   3   x   -- gt   ABC   DEF   Gxx    Note  on Python 2 use izip longest instead of zip longest

User · Answer

I like this approach  It feels simple and not magical and supports all iterable types and doesn t require imports   def chunk iter iterable  chunk size   it   iter iterable  while True      chunk   tuple next it  for   in range chunk size       if not chunk          break     yield chunk

User · Answer

One-liner  adhoc solution to iterate over a list x in chunks of size 4 -  for a  b  c  d in zip x 0  4   x 1  4   x 2  4   x 3  4            do something with a  b  c and d

User · Answer

Unless I misses something  the following simple solution with generator expressions has not been mentioned   It assumes that both the size and the number of chunks are known  which is often the case   and that no padding is required   def chunks it  n  m          Make an iterator over m first chunks of size n              it   iter it        Chunks are presented as tuples      return  tuple next it  for   in range n   for   in range m

User · Answer

Using map   instead of zip   fixes the padding issue in J F  Sebastian s answer    gt  gt  gt  def chunker iterable  chunksize         return map None   iter iterable   chunksize    Example    gt  gt  gt  s    1234567890   gt  gt  gt  chunker s  3     1    2    3      4    5    6      7    8    9      0   None  None    gt  gt  gt  chunker s  4     1    2    3    4      5    6    7    8      9    0   None  None    gt  gt  gt  chunker s  5     1    2    3    4    5      6    7    8    9    0

User · Answer

There doesn t seem to be a pretty way to do this   Here is a page that has a number of methods  including   def split seq seq  size       newseq          splitsize   1 0 size len seq      for i in range size           newseq append seq int round i splitsize   int round  i 1  splitsize         return newseq

User · Answer

I m a fan of   chunk size  4 for i in range 0  len ints   chunk size       chunk   ints i i chunk size        process chunk of size  lt   chunk size

User · Answer

About solution gave by J F  Sebastian here   def chunker iterable  chunksize       return zip   iter iterable   chunksize    It s clever  but has one disadvantage - always return tuple  How to get string instead  Of course you can write    join chunker        but the temporary tuple is constructed anyway   You can get rid of the temporary tuple by writing own zip  like this   class IteratorExhausted Exception       pass  def translate StopIteration iterable  to IteratorExhausted       for i in iterable          yield i     raise to   StopIteration would get ignored because this is generator                 but custom exception can leave the generator   def custom zip  iterables  reductor tuple       iterators   tuple map translate StopIteration  iterables       while True          try              yield reductor next i  for i in iterators          except IteratorExhausted    when any of iterators get exhausted              break   Then  def chunker data  size  reductor tuple       return custom zip   iter data   size  reductor reductor    Example usage    gt  gt  gt  for i in chunker  12345   2           print repr i         1    2     3    4    gt  gt  gt  for i in chunker  12345   2     join           print repr i        12   34

User · Answer

from itertools import izip longest  def chunker iterable  chunksize  filler       return izip longest   iter iterable   chunksize  fillvalue filler

User · Answer

To avoid all conversions to a list import itertools and    gt  gt  gt  for k  g in itertools groupby xrange 35   lambda x  x 10           list g    Produces        0  0  1  2  3  4  5  6  7  8  9  1  10  11  12  13  14  15  16  17  18  19  2  20  21  22  23  24  25  26  27  28  29  3  30  31  32  33  34   gt  gt  gt     I checked groupby and it doesn t convert to list or use len so I  think  this will delay resolution of each value until it is actually used   Sadly none of the available answers  at this time  seemed to offer this variation   Obviously if you need to handle each item in turn nest a for loop over g   for k g in itertools groupby xrange 35   lambda x  x 10       for i in g           do what you need to do with individual items       now do what you need to do with the whole group   My specific interest in this was the need to consume a generator to submit changes in batches of up to 1000 to the gmail API       messages   a generator which would not be smart as a list     for idx  batch in groupby messages  lambda x  x 1000           batch request   BatchHttpRequest           for message in batch              batch request add self service users   messages   modify userId  me   id message  id    body msg labels           http   httplib2 Http           self credentials authorize http          batch request execute http http

User · Answer

In your second method  I would advance to the next group of 4 by doing this   ints   ints 4     However  I haven t done any performance measurement so I don t know which one might be more efficient   Having said that  I would usually choose the first method  It s not pretty  but that s often a consequence of interfacing with the outside world

User · Answer

I am hoping that by turning an iterator out of a list i am not simply copying a slice of the list  Generators can be sliced and they will automatically still be a generator  while lists will be sliced into huge chunks of 1000 entries  which is less efficient       def iter group iterable  batch size int       length   len iterable      start   batch size -1     end   0     while end  lt  length           start    batch size         end    batch size         if type iterable     list              yield  iterable i  for i in range start min length-1 end            else              yield iterable start end    Usage   items   list range 1 1251    for item group in iter group items  1000       for item in item group          print item

User · Answer

With NumPy it s simple   ints   array  1  2  3  4  5  6  7  8   for int1  int2 in ints reshape -1  2       print int1  int2    output   1 2 3 4 5 6 7 8

User · Answer

Modified from the recipes section of Python s itertools docs   from itertools import zip longest  def grouper iterable  n  fillvalue None       args    iter iterable     n     return zip longest  args  fillvalue fillvalue    Example In pseudocode to keep the example terse   grouper  ABCDEFG   3   x   -- gt   ABC   DEF   Gxx    Note  on Python 2 use izip longest instead of zip longest

User · Answer

Since nobody s mentioned it yet here s a zip   solution    gt  gt  gt  def chunker iterable  chunksize           return zip   iter iterable   chunksize    It works only if your sequence s length is always divisible by the chunk size or you don t care about a trailing chunk if it isn t   Example    gt  gt  gt  s    1234567890   gt  gt  gt  chunker s  3     1    2    3      4    5    6      7    8    9     gt  gt  gt  chunker s  4     1    2    3    4      5    6    7    8     gt  gt  gt  chunker s  5     1    2    3    4    5      6    7    8    9    0      Or using itertools izip to return an iterator instead of a list    gt  gt  gt  from itertools import izip  gt  gt  gt  def chunker iterable  chunksize           return izip   iter iterable   chunksize    Padding can be fixed using    O      s answer    gt  gt  gt  from itertools import chain  izip  repeat  gt  gt  gt  def chunker iterable  chunksize  fillvalue None           it     chain iterable  repeat fillvalue  chunksize-1           args    it    chunksize         return izip  args

User · Answer

Why not use list comprehension   l    1   2  3  4  5  6  7  8  9  10  11  n   4 filler   0 fills   len l    n chunks     l    filler    fills  x   n x   n   n  for x in range int  len l    n - 1  n    print chunks     1  2  3  4    5  6  7  8    9  10  11  0

User · Answer

Since nobody s mentioned it yet here s a zip   solution    gt  gt  gt  def chunker iterable  chunksize           return zip   iter iterable   chunksize    It works only if your sequence s length is always divisible by the chunk size or you don t care about a trailing chunk if it isn t   Example    gt  gt  gt  s    1234567890   gt  gt  gt  chunker s  3     1    2    3      4    5    6      7    8    9     gt  gt  gt  chunker s  4     1    2    3    4      5    6    7    8     gt  gt  gt  chunker s  5     1    2    3    4    5      6    7    8    9    0      Or using itertools izip to return an iterator instead of a list    gt  gt  gt  from itertools import izip  gt  gt  gt  def chunker iterable  chunksize           return izip   iter iterable   chunksize    Padding can be fixed using    O      s answer    gt  gt  gt  from itertools import chain  izip  repeat  gt  gt  gt  def chunker iterable  chunksize  fillvalue None           it     chain iterable  repeat fillvalue  chunksize-1           args    it    chunksize         return izip  args

User · Answer

Why not use list comprehension   l    1   2  3  4  5  6  7  8  9  10  11  n   4 filler   0 fills   len l    n chunks     l    filler    fills  x   n x   n   n  for x in range int  len l    n - 1  n    print chunks     1  2  3  4    5  6  7  8    9  10  11  0

User · Answer

def group by iterable  size          Group an iterable into lists that don t exceed the size given        gt  gt  gt  group by  1 2 3 4 5   2        1  2    3  4    5                sublist           for index  item in enumerate iterable           if index  gt  0 and index   size    0              yield sublist             sublist               sublist append item       if sublist          yield sublist

User · Answer

If the list is large  the highest-performing way to do this will be to use a generator   def get chunk iterable  chunk size       result          for item in iterable          result append item          if len result     chunk size              yield tuple result              result          if len result   gt  0          yield tuple result   for x in get chunk  1 2 3 4 5 6 7 8 9 10   3       print x   1  2  3   4  5  6   7  8  9   10

User · Answer

import itertools def chunks iterable size       it   iter iterable      chunk   tuple itertools islice it size       while chunk          yield chunk         chunk   tuple itertools islice it size      though this will throw ValueError if the length of ints   isn t a multiple of four  for x1 x2 x3 x4 in chunks ints 4       foo    x1   x2   x3   x4  for chunk in chunks ints 4       foo    sum chunk    Another way   import itertools def chunks2 iterable size filler None       it   itertools chain iterable itertools repeat filler size-1       chunk   tuple itertools islice it size       while len chunk     size          yield chunk         chunk   tuple itertools islice it size      x2  x3 and x4 could get the value 0 if the length is not   a multiple of 4  for x1 x2 x3 x4 in chunks2 ints 4 0       foo    x1   x2   x3   x4

User · Answer

With Python 3 8 you can use the walrus operator and itertools islice   from itertools import islice  list     i for i in range 10  100    def chunker it  size       iterator   iter it      while chunk    list islice iterator  size            print chunk    In  2   chunker list   10                                                            10  11  12  13  14  15  16  17  18  19   20  21  22  23  24  25  26  27  28  29   30  31  32  33  34  35  36  37  38  39   40  41  42  43  44  45  46  47  48  49   50  51  52  53  54  55  56  57  58  59   60  61  62  63  64  65  66  67  68  69   70  71  72  73  74  75  76  77  78  79   80  81  82  83  84  85  86  87  88  89   90  91  92  93  94  95  96  97  98  99

User · Answer

In your second method  I would advance to the next group of 4 by doing this   ints   ints 4     However  I haven t done any performance measurement so I don t know which one might be more efficient   Having said that  I would usually choose the first method  It s not pretty  but that s often a consequence of interfacing with the outside world

User · Answer

I am hoping that by turning an iterator out of a list i am not simply copying a slice of the list  Generators can be sliced and they will automatically still be a generator  while lists will be sliced into huge chunks of 1000 entries  which is less efficient       def iter group iterable  batch size int       length   len iterable      start   batch size -1     end   0     while end  lt  length           start    batch size         end    batch size         if type iterable     list              yield  iterable i  for i in range start min length-1 end            else              yield iterable start end    Usage   items   list range 1 1251    for item group in iter group items  1000       for item in item group          print item

User · Answer

I m a fan of   chunk size  4 for i in range 0  len ints   chunk size       chunk   ints i i chunk size        process chunk of size  lt   chunk size

User · Answer

There doesn t seem to be a pretty way to do this   Here is a page that has a number of methods  including   def split seq seq  size       newseq          splitsize   1 0 size len seq      for i in range size           newseq append seq int round i splitsize   int round  i 1  splitsize         return newseq

User · Answer

Modified from the recipes section of Python s itertools docs   from itertools import zip longest  def grouper iterable  n  fillvalue None       args    iter iterable     n     return zip longest  args  fillvalue fillvalue    Example In pseudocode to keep the example terse   grouper  ABCDEFG   3   x   -- gt   ABC   DEF   Gxx    Note  on Python 2 use izip longest instead of zip longest

User · Answer

There doesn t seem to be a pretty way to do this   Here is a page that has a number of methods  including   def split seq seq  size       newseq          splitsize   1 0 size len seq      for i in range size           newseq append seq int round i splitsize   int round  i 1  splitsize         return newseq

User · Answer

from itertools import izip longest  def chunker iterable  chunksize  filler       return izip longest   iter iterable   chunksize  fillvalue filler

User · Answer

I like this approach  It feels simple and not magical and supports all iterable types and doesn t require imports   def chunk iter iterable  chunk size   it   iter iterable  while True      chunk   tuple next it  for   in range chunk size       if not chunk          break     yield chunk

User · Answer

One-liner  adhoc solution to iterate over a list x in chunks of size 4 -  for a  b  c  d in zip x 0  4   x 1  4   x 2  4   x 3  4            do something with a  b  c and d

User · Answer

Since nobody s mentioned it yet here s a zip   solution    gt  gt  gt  def chunker iterable  chunksize           return zip   iter iterable   chunksize    It works only if your sequence s length is always divisible by the chunk size or you don t care about a trailing chunk if it isn t   Example    gt  gt  gt  s    1234567890   gt  gt  gt  chunker s  3     1    2    3      4    5    6      7    8    9     gt  gt  gt  chunker s  4     1    2    3    4      5    6    7    8     gt  gt  gt  chunker s  5     1    2    3    4    5      6    7    8    9    0      Or using itertools izip to return an iterator instead of a list    gt  gt  gt  from itertools import izip  gt  gt  gt  def chunker iterable  chunksize           return izip   iter iterable   chunksize    Padding can be fixed using    O      s answer    gt  gt  gt  from itertools import chain  izip  repeat  gt  gt  gt  def chunker iterable  chunksize  fillvalue None           it     chain iterable  repeat fillvalue  chunksize-1           args    it    chunksize         return izip  args

User · Answer

This answer splits a list of strings  f ex  to achieve PEP8-line length compliance   def split what  target length 79          splits list of strings into sublists  each      having string length at most 79        out            while what          if len        join out -1      len what 0    lt  target length              out -1  append what pop 0           else              if not out -1     string longer than target length                 out -1     what pop 0               out append         return out   Use as   gt  gt  gt  split   deferred income    long term incentive    restricted stock deferred    shared receipt with poi    loan advances    from messages    other    director fees    bonus    total stock value    from poi to this person    from this person to poi    restricted stock    salary    total payments    exercised stock options    75     deferred income    long term incentive    restricted stock deferred      shared receipt with poi    loan advances    from messages    other      director fees    bonus    total stock value    from poi to this person      from this person to poi    restricted stock    salary    total payments      exercised stock options

User · Answer

I m a fan of   chunk size  4 for i in range 0  len ints   chunk size       chunk   ints i i chunk size        process chunk of size  lt   chunk size

User · Answer

The more-itertools package has chunked method which does exactly that  import more itertools for s in more itertools chunked range 9   4       print s   Prints  0  1  2  3   4  5  6  7   8   chunked returns the items in a list  If you d prefer iterables  use ichunked

User · Answer

I m a fan of   chunk size  4 for i in range 0  len ints   chunk size       chunk   ints i i chunk size        process chunk of size  lt   chunk size

User · Answer

Another approach would be to use the two-argument form of iter    from itertools import islice  def group it  size       it   iter it      return iter lambda  tuple islice it  size          This can be adapted easily to use padding  this is similar to Markus Jarderot   s answer    from itertools import islice  chain  repeat  def group pad it  size  pad None       it   chain iter it   repeat pad       return iter lambda  tuple islice it  size     pad     size    These can even be combined for optional padding    no pad   object   def group it  size  pad  no pad       if pad     no pad          it   iter it          sentinel          else          it   chain iter it   repeat pad           sentinel    pad     size     return iter lambda  tuple islice it  size    sentinel

User · Answer

If you don t mind using an external package you could use iteration utilities grouper from iteration utilties 1  It supports all iterables  not just sequences    from iteration utilities import grouper seq   list range 20   for group in grouper seq  4       print group    which prints    0  1  2  3   4  5  6  7   8  9  10  11   12  13  14  15   16  17  18  19    In case the length isn t a multiple of the groupsize it also supports filling  the incomplete last group  or truncating  discarding the incomplete last group  the last one   from iteration utilities import grouper seq   list range 17   for group in grouper seq  4       print group     0  1  2  3     4  5  6  7     8  9  10  11     12  13  14  15     16    for group in grouper seq  4  fillvalue None       print group     0  1  2  3     4  5  6  7     8  9  10  11     12  13  14  15     16  None  None  None   for group in grouper seq  4  truncate True       print group     0  1  2  3     4  5  6  7     8  9  10  11     12  13  14  15      Benchmarks  I also decided to compare the run-time of a few of the mentioned approaches  It s a log-log plot grouping into groups of  10  elements based on a list of varying size  For qualitative results  Lower means faster     At least in this benchmark the iteration utilities grouper performs best  Followed by the approach of Craz    The benchmark was created with simple benchmark1  The code used to run this benchmark was   import iteration utilities import itertools from itertools import zip longest  def consume all it       return iteration utilities consume it  None   import simple benchmark b   simple benchmark BenchmarkBuilder     b add function   def grouper l  n       return consume all iteration utilities grouper l  n    def Craz inner iterable  n  fillvalue None       args    iter iterable     n     return zip longest  args  fillvalue fillvalue    b add function   def Craz iterable  n  fillvalue None       return consume all Craz inner iterable  n  fillvalue    def nosklo inner seq  size       return  seq pos pos   size  for pos in range 0  len seq   size     b add function   def nosklo seq  size       return consume all nosklo inner seq  size    def SLott inner ints  chunk size       for i in range 0  len ints   chunk size           yield ints i i chunk size    b add function   def SLott ints  chunk size       return consume all SLott inner ints  chunk size    def MarkusJarderot1 inner iterable size       it   iter iterable      chunk   tuple itertools islice it size       while chunk          yield chunk         chunk   tuple itertools islice it size     b add function   def MarkusJarderot1 iterable size       return consume all MarkusJarderot1 inner iterable size    def MarkusJarderot2 inner iterable size filler None       it   itertools chain iterable itertools repeat filler size-1       chunk   tuple itertools islice it size       while len chunk     size          yield chunk         chunk   tuple itertools islice it size     b add function   def MarkusJarderot2 iterable size       return consume all MarkusJarderot2 inner iterable size     b add arguments   def argument provider        for exp in range 2  20           size   2  exp         yield size  simple benchmark MultiArgument   0    size  10    r   b run       1 Disclaimer  I m the author of the libraries iteration utilities and simple benchmark

User · Answer

It is easy to make itertools groupby work for you to get an iterable of iterables  without creating any temporary lists   groupby iterable   lambda x y   lambda z  x next   y   count   100     Don t get put off by the nested lambdas  outer lambda runs just once to put count   generator and the constant 100 into the scope of the inner lambda   I use this to send chunks of rows to mysql   for k v in groupby bigdata   lambda x y   lambda z  x next   y   count   100         cursor executemany sql  v

User · Answer

At first  I designed it to split strings into substrings to parse string containing hex  Today I turned it into complex  but still simple generator   def chunker iterable  size  reductor  condition       it   iter iterable      def chunk generator            return  next it  for   in range size       chunk   reductor chunk generator        while condition chunk           yield chunk         chunk   reductor chunk generator      Arguments   Obvious ones   iterable is any iterable   iterator   generator containg   generating   iterating over input data  size is  of course  size of chunk you want get    More interesting   reductor is a callable  which receives generator iterating over content of chunk  I d expect it to return sequence or string  but I don t demand that   You can pass as this argument for example list  tuple  set  frozenset  or anything fancier  I d pass this function  returning string  provided that iterable contains   generates   iterates over strings    def concatenate iterable       return    join iterable    Note that reductor can cause closing generator by raising exception  condition is a callable which receives anything what reductor returned  It decides to approve  amp  yield it  by returning anything evaluating to True   or to decline it  amp  finish generator s work  by returning anything other or raising exception    When number of elements in iterable is not divisible by size  when it gets exhausted  reductor will receive generator generating less elements than size  Let s call these elements lasts elements   I invited two functions to pass as this argument      lambda x x - the lasts elements will be yielded  lambda x  len x    lt size gt  - the lasts elements will be rejected  replace  lt size gt  using number equal to size

User · Answer

def chunker iterable  n          Yield iterable in chunk sizes        gt  gt  gt  chunks   chunker  ABCDEF   n 4       gt  gt  gt  chunks next         A    B    C    D        gt  gt  gt  chunks next         E    F               it   iter iterable      while True          chunk              for i in range n               try                  chunk append next it               except StopIteration                  yield chunk                 raise StopIteration         yield chunk  if   name         main         import doctest      doctest testmod

User · Answer

This answer splits a list of strings  f ex  to achieve PEP8-line length compliance   def split what  target length 79          splits list of strings into sublists  each      having string length at most 79        out            while what          if len        join out -1      len what 0    lt  target length              out -1  append what pop 0           else              if not out -1     string longer than target length                 out -1     what pop 0               out append         return out   Use as   gt  gt  gt  split   deferred income    long term incentive    restricted stock deferred    shared receipt with poi    loan advances    from messages    other    director fees    bonus    total stock value    from poi to this person    from this person to poi    restricted stock    salary    total payments    exercised stock options    75     deferred income    long term incentive    restricted stock deferred      shared receipt with poi    loan advances    from messages    other      director fees    bonus    total stock value    from poi to this person      from this person to poi    restricted stock    salary    total payments      exercised stock options

User · Answer

I never want my chunks padded  so that requirement is essential   I find that the ability to work on any iterable is also requirement   Given that  I decided to extend on the accepted answer  https   stackoverflow com a 434411 1074659     Performance takes a slight hit in this approach if padding is not wanted due to the need to compare and filter the padded values   However  for large chunk sizes  this utility is very performant      usr bin env python3 from itertools import zip longest    UNDEFINED   object     def chunker iterable  chunksize  fillvalue  UNDEFINED               Collect data into chunks and optionally pad it       Performance worsens as  chunksize  approaches 1       Inspired by          https   docs python org 3 library itertools html itertools-recipes              args    iter iterable     chunksize     chunks   zip longest  args  fillvalue fillvalue      yield from           filter lambda val  val is not  UNDEFINED  chunk          if chunk -1  is  UNDEFINED         else chunk         for chunk in chunks       if fillvalue is  UNDEFINED else chunks

[python] What is the most "pythonic" way to iterate over a list in chunks?

Examples related to python

Examples related to list

Examples related to loops

Examples related to optimization

Examples related to chunks