Difference between Python s Generators and Iterators

Question

What is the difference between iterators and generators  Some examples for when you would use each case would be helpful

User · Answer

You can compare both approaches for the same data:

def myGeneratorList(n):
    for i in range(n):
        yield i

def myIterableList(n):
    ll = n*[None]
    for i in range(n):
        ll[i] = i
    return ll

# Same values
ll1 = myGeneratorList(10)
ll2 = myIterableList(10)
for i1, i2 in zip(ll1, ll2):
    print("{} {}".format(i1, i2))

# Generator can only be read once
ll1 = myGeneratorList(10)
ll2 = myIterableList(10)

print("{} {}".format(len(list(ll1)), len(ll2)))
print("{} {}".format(len(list(ll1)), len(ll2)))

# Generator can be read several times if converted into iterable
ll1 = list(myGeneratorList(10))
ll2 = myIterableList(10)

print("{} {}".format(len(list(ll1)), len(ll2)))
print("{} {}".format(len(list(ll1)), len(ll2)))

Besides, if you check the memory footprint, the generator takes much less memory as it doesn't need to store all the values in memory at the same time.

User · Answer

Generator Function  Generator Object  Generator    A Generator function is just like a regular function in Python but it contains one or more yield statements  Generator functions is a great tool to create  Iterator objects as easy as possible   The Iterator object returend by generator function is also called Generator object or Generator    In this example I have created a Generator function which returns a Generator object  lt generator object fib at 0x01342480 gt   Just like other iterators  Generator objects can be used in a  for loop  or with   the built-in function next   which returns the next value from generator    def fib max       a  b   0  1     for i in range max           yield a         a  b   b  a   b print fib 10                 lt generator object fib at 0x01342480 gt   for i in fib 10       print i                  0 1 1 2 3 5 8 13 21 34   print next myfib            0 print next myfib            1 print next myfib            1 print next myfib            2   So a generator function is the easiest way to create an Iterator object      Iterator    Every generator object is an iterator but not vice versa   A custom  iterator object  can  be created if its class implements    iter   and   next   method  also called iterator protocol     However  it is  much easier to use generators function to create iterators because they simplify their creation   but a custom Iterator gives you more freedom and you can also implement other methods according to your requirements as shown in the  below example    class Fib      def   init   self max           self current 0         self next 1         self max max         self count 0      def   iter   self           return self      def   next   self           if self count gt self max              raise StopIteration         else              self current self next self next  self current self next              self count  1             return self next-self current      def   str   self           return  Generator object   itobj Fib 4  print itobj                 Generator object  for i in Fib 4         print i                 0 1 1 2  print next itobj            0 print next itobj            1 print next itobj            1

User · Answer

Examples from  Ned Batchelder highly recommended for iterators and generators  A method without generators that do something to even numbers  def evens stream      them         for n in stream        if n   2    0           them append n     return them   while by using a generator  def evens stream       for n in stream          if n   2    0              yield n    We don t need any list nor a return statement Efficient for large  infinite length stream     it just walks and yield the value   Calling the evens method  generator  is as usual  num         for n in evens num      do smth n     Generator also used to Break double loop   Iterator     A book full of pages is an iterable   A bookmark is an   iterator   and this bookmark has nothing to do except to move next  litr   iter  1 2 3   next litr     1 next litr     2 next litr     3 next litr     StopIteration   Exception  as we got end of the iterator   To use Generator     we need a function  To use Iterator       we need next and iter  As been said      A Generator function returns an iterator object    The Whole benefit of Iterator      Store one element a time in memory

User · Answer

Everybody has a really nice and verbose answer with examples and I really appreciate it  I just wanted to give a short few lines answer for people who are still not quite clear conceptually   If you create your own iterator  it is a little bit involved - you have  to create a class and at least implement the iter and the next methods  But what if you don t want to go through this hassle and want to quickly create an iterator  Fortunately  Python provides a short-cut way to defining an iterator  All you need to do is define a function with at least 1 call to yield and now when you call that function it will return  something  which will act like an iterator  you can call next method and use it in a for loop   This something has a name in Python called Generator  Hope that clarifies a bit

User · Answer

I am writing specifically for Python newbies in a very simple way  though deep down Python does so many things   Let   s start with the very basic   Consider a list   l    1 2 3    Let   s write an equivalent function   def f        return  1 2 3    o p of print l    1 2 3   amp  o p of print f       1 2 3   Let   s make list l iterable  In python list is always iterable that means you can apply iterator whenever you want   Let   s apply iterator on list   iter l   iter l    iterator applied explicitly   Let   s make a function iterable  i e  write an equivalent generator function  In python as soon as you introduce the keyword yield  it becomes a generator function and iterator will be applied implicitly   Note  Every generator is always iterable with implicit iterator applied and here implicit iterator is the crux So the generator function will be   def f      yield 1    yield 2   yield 3  iter f   f     which is iter f  as iterator is already applied implicitly   So if you have observed  as soon as you made function f a generator  it is already iter f   Now      l is the list  after applying iterator method  iter  it becomes    iter l       f is already iter f   after applying iterator method  iter  it   becomes  iter iter f    which is again iter f    It s kinda you are casting int to int x  which is already int and it will remain int x    For example o p of    print type iter iter l       is   lt class  list iterator  gt    Never forget this is Python and not C or C    Hence the conclusion from above explanation is      list l    iter l       generator function f    iter f

User · Answer

Previous answers missed this addition  a generator has a close method  while typical iterators don   t  The close method triggers a StopIteration exception in the generator  which may be caught in a finally clause in that iterator  to get a chance to run some clean-up  This abstraction makes it most usable in the large than simple iterators  One can close a generator as one could close a file  without having to bother about what   s underneath   That said  my personal answer to the first question would be  iteratable has an   iter   method only  typical iterators have a   next   method only  generators has both an   iter   and a   next   and an additional close   For the second question  my personal answer would be  in a public interface  I tend to favor generators a lot  since it   s more resilient   the close method an a greater composability with yield from  Locally  I may use iterators  but only if it   s a flat and simple structure  iterators does not compose easily  and if there are reasons to believe the sequence is rather short especially if it may be stopped before it reach the end  I tend to look at iterators as a low level primitive  except as literals   For control flow matters  generators are an as much important concept as promises  both are abstract and composable

User · Answer

It s difficult to answer the question without 2 other concepts  iterable and iterator protocol   What is difference between iterator and iterable  Conceptually you iterate over iterable with the help of corresponding iterator  There are a few differences that can help to distinguish iterator and iterable in practice   One difference is that iterator has   next   method  iterable does not  Another difference - both of them contain   iter   method  In case of iterable it returns the corresponding iterator  In case of iterator it returns itself  This can help to distinguish iterator and iterable in practice      gt  gt  gt  x    1  2  3   gt  gt  gt  dir x          iter         gt  gt  gt  x iter   iter x   gt  gt  gt  dir x iter         iter         next         gt  gt  gt  type x iter  list iterator   What are iterables in python  list  string  range etc  What are iterators  enumerate  zip  reversed etc  We may check this using the approach above  It s kind of confusing  Probably it would be easier if we have only one type  Is there any difference between range and zip  One of the reasons to do this - range has a lot of additional functionality - we may index it or check if it contains some number etc   see details here    How can we create an iterator ourselves  Theoretically we may implement Iterator Protocol  see here   We need to write   next   and   iter   methods and raise StopIteration exception and so on  see Alex Martelli s answer for an example and possible motivation  see also here   But in practice we use generators  It seems to be by far the main method to create iterators in python    I can give you a few more interesting examples that show somewhat confusing usage of those concepts in practice   in keras we have tf keras preprocessing image ImageDataGenerator  this class doesn t have   next   and   iter   methods  so it s not an iterator  or generator   if you call its flow from dataframe   method you ll get DataFrameIterator that has those methods  but it doesn t implement StopIteration  which is not common in build-in iterators in python   in documentation we may read that  quot A DataFrameIterator yielding tuples of  x  y  quot  - again confusing usage of terminology  we also have Sequence class in keras and that s custom implementation of a generator functionality  regular generators are not suitable for multithreading  but it doesn t implement   next   and   iter    rather it s a wrapper around generators  it uses yield statement

User · Answer

What is the difference between iterators and generators  Some examples for when you would use each case would be helpful    In summary  Iterators are objects that have an   iter   and a   next    next in Python 2  method  Generators provide an easy  built-in way to create instances of Iterators   A function with yield in it is still a function  that  when called  returns an instance of a generator object   def a function         when called  returns generator object      yield   A generator expression also returns a generator   a generator    i for i in range 0     For a more in-depth exposition and examples  keep reading   A Generator is an Iterator  Specifically  generator is a subtype of iterator    gt  gt  gt  import collections  types  gt  gt  gt  issubclass types GeneratorType  collections Iterator  True   We can create a generator several ways  A very common and simple way to do so is with a function   Specifically  a function with yield in it is a function  that  when called  returns a generator    gt  gt  gt  def a function             just a function definition with yield in it          yield  gt  gt  gt  type a function   lt class  function  gt   gt  gt  gt  a generator   a function      when called  gt  gt  gt  type a generator              returns a generator  lt class  generator  gt    And a generator  again  is an Iterator    gt  gt  gt  isinstance a generator  collections Iterator  True   An Iterator is an Iterable  An Iterator is an Iterable     gt  gt  gt  issubclass collections Iterator  collections Iterable  True   which requires an   iter   method that returns an Iterator    gt  gt  gt  collections Iterable   Traceback  most recent call last     File   lt pyshell 79 gt    line 1  in  lt module gt      collections Iterable   TypeError  Can t instantiate abstract class Iterable with abstract methods   iter     Some examples of iterables are the built-in tuples  lists  dictionaries  sets  frozen sets  strings  byte strings  byte arrays  ranges and memoryviews    gt  gt  gt  all isinstance element  collections Iterable  for element in                       set    frozenset        b    bytearray    range 0   memoryview b      True   Iterators require a next or   next   method  In Python 2     gt  gt  gt  collections Iterator   Traceback  most recent call last     File   lt pyshell 80 gt    line 1  in  lt module gt      collections Iterator   TypeError  Can t instantiate abstract class Iterator with abstract methods next   And in Python 3    gt  gt  gt  collections Iterator   Traceback  most recent call last     File   lt stdin gt    line 1  in  lt module gt  TypeError  Can t instantiate abstract class Iterator with abstract methods   next     We can get the iterators from the built-in objects  or custom objects  with the iter function    gt  gt  gt  all isinstance iter element   collections Iterator  for element in                       set    frozenset        b    bytearray    range 0   memoryview b      True   The   iter   method is called when you attempt to use an object with a for-loop  Then the   next   method is called on the iterator object to get each item out for the loop  The iterator raises StopIteration when you have exhausted it  and it cannot be reused at that point   From the documentation  From the Generator Types section of the Iterator Types section of the Built-in Types documentation      Python   s generators provide a convenient way to implement the iterator protocol  If a container object   s   iter     method is implemented as a generator  it will automatically return an iterator object  technically  a generator object  supplying the   iter     and next      next     in Python 3  methods  More information about generators can be found in the documentation for the yield expression     Emphasis added    So from this we learn that Generators are a  convenient  type of Iterator    Example Iterator Objects  You might create object that implements the Iterator protocol by creating or extending your own object   class Yes collections Iterator        def   init   self  stop           self x   0         self stop   stop      def   iter   self           return self      def next self           if self x  lt  self stop              self x    1             return  yes          else                Iterators must raise when done  else considered broken             raise StopIteration        next     next   Python 3 compatibility   But it s easier to simply use a Generator to do this   def yes stop       for   in range stop           yield  yes    Or perhaps simpler  a Generator Expression  works similarly to list comprehensions    yes expr     yes  for   in range stop     They can all be used in the same way    gt  gt  gt  stop   4               gt  gt  gt  for i  y1  y2  y3 in zip range stop   Yes stop   yes stop                                   yes  for   in range stop             print   0    1      2      3   format i  y1  y2  y3            0  yes    yes    yes 1  yes    yes    yes 2  yes    yes    yes 3  yes    yes    yes   Conclusion  You can use the Iterator protocol directly when you need to extend a Python object as an object that can be iterated over    However  in the vast majority of cases  you are best suited to use yield to define a function that returns a Generator Iterator or consider Generator Expressions   Finally  note that generators provide even more functionality as coroutines  I explain Generators  along with the yield statement  in depth on my answer to  What does the    yield    keyword do

User · Answer

Iterators   Iterator are objects which uses next   method to get next value of sequence   Generators   A generator is a function that produces or yields a sequence of values using yield method    Every next   method call on generator object for ex  f as in below example  returned by generator function for ex  foo   function in below example   generates next value in sequence   When a generator function is called  it returns an generator object without even beginning execution of the function  When next   method is called for the first time  the function starts executing until it reaches yield statement which returns the yielded value  The yield keeps track of i e  remembers last execution  And second next   call continues from previous value   The following example demonstrates the interplay between yield and call to next method on generator object    gt  gt  gt  def foo            print  begin          for i in range 3               print  before yield   i             yield i             print  after yield   i         print  end       gt  gt  gt  f   foo    gt  gt  gt  f next   begin before yield 0              Control is in for loop 0  gt  gt  gt  f next   after yield 0              before yield 1              Continue for loop 1  gt  gt  gt  f next   after yield 1 before yield 2 2  gt  gt  gt  f next   after yield 2 end Traceback  most recent call last     File   lt stdin gt    line 1  in  lt module gt  StopIteration  gt  gt  gt

User · Answer

Adding an answer because none of the existing answers specifically address the confusion in the official literature  Generator functions are ordinary functions defined using yield instead of return  When called  a generator function returns a generator object  which is a kind of iterator - it has a next   method  When you call next    the next value yielded by the generator function is returned  Either the function or the object may be called the  quot generator quot  depending on which Python source document you read  The Python glossary says generator functions  while the Python wiki implies generator objects  The Python tutorial remarkably manages to imply both usages in the space of three sentences   Generators are a simple and powerful tool for creating iterators  They are written like regular functions but use the yield statement whenever they want to return data  Each time next   is called on it  the generator resumes where it left off  it remembers all the data values and which statement was last executed    The first two sentences identify generators with generator functions  while the third sentence identifies them with generator objects  Despite all this confusion  one can seek out the Python language reference for the clear and final word   The yield expression is only used when defining a generator function  and can only be used in the body of a function definition  Using a yield expression in a function definition is sufficient to cause that definition to create a generator function instead of a normal function  When a generator function is called  it returns an iterator known as a generator  That generator then controls the execution of a generator function   So  in formal and precise usage   quot generator quot  unqualified means generator object  not generator function  The above references are for Python 2 but Python 3 language reference says the same thing  However  the Python 3 glossary states that  generator     Usually refers to a generator function  but may refer to a generator iterator in some contexts  In cases where the intended meaning isn   t clear  using the full terms avoids ambiguity

User · Answer

iterator is a more general concept  any object whose class has a   next   method  next in Python 2  and an   iter   method that does return self  Every generator is an iterator  but not vice versa   A generator is built by calling a function that has one or more yield expressions  yield statements  in Python 2 5 and earlier   and is an object that meets the previous paragraph s definition of an iterator  You may want to use a custom iterator  rather than a generator  when you need a class with somewhat complex state-maintaining behavior  or want to expose other methods besides   next    and   iter   and   init      Most often  a generator  sometimes  for sufficiently simple needs  a generator expression  is sufficient  and it s simpler to code because state maintenance  within reasonable limits  is basically  quot done for you quot  by the frame getting suspended and resumed  For example  a generator such as  def squares start  stop       for i in range start  stop           yield i   i  generator   squares a  b   or the equivalent generator expression  genexp  generator    i i for i in range a  b    would take more code to build as a custom iterator  class Squares object       def   init   self  start  stop          self start   start        self stop   stop     def   iter   self   return self     def   next   self     next in Python 2        if self start  gt   self stop             raise StopIteration        current   self start   self start        self start    1        return current  iterator   Squares a  b   But  of course  with class Squares you could easily offer extra methods  i e      def current self          return self start  if you have any actual need for such extra functionality in your application

[python] Difference between Python's Generators and Iterators

Examples related to python

Examples related to iterator

Examples related to generator