In practice what are the main uses for the new yield from syntax in Python 3 3

Question

I m having a hard time wrapping my brain around PEP 380    What are the situations where  yield from  is useful   What is the classic use case  Why is it compared to micro-threads      update    Now I understand the cause of my difficulties  I ve used generators  but never really used coroutines  introduced by PEP-342   Despite some similarities  generators and coroutines are basically two different concepts  Understanding coroutines  not only generators  is the key to understanding the new syntax   IMHO coroutines are the most obscure Python feature  most books make it look useless and uninteresting   Thanks for the great answers  but special thanks to agf and his comment linking to David Beazley presentations  David rocks

User · Answer

A short example will help you understand one of yield from s use case  get value from another generator  def flatten sequence          flatten a multi level list or something      gt  gt  gt  list flatten  1   2   3         1  2  3       gt  gt  gt  list flatten  1   2    3   4           1  2  3  4              for element in sequence          if hasattr element     iter                  yield from flatten element          else              yield element  print list flatten  1   2    3   4

User · Answer

What are the situations where  yield from  is useful    Every situation where you have a loop like this   for x in subgenerator    yield x   As the PEP describes  this is a rather naive attempt at using the subgenerator  it s missing several aspects  especially the proper handling of the  throw    send    close   mechanisms introduced by PEP 342  To do this properly  rather complicated code is necessary      What is the classic use case    Consider that you want to extract information from a recursive data structure  Let s say we want to get all leaf nodes in a tree   def traverse tree node     if not node children      yield node   for child in node children      yield from traverse tree child    Even more important is the fact that until the yield from  there was no simple method of refactoring the generator code  Suppose you have a  senseless  generator like this   def get list values lst     for item in lst      yield int item    for item in lst      yield str item    for item in lst      yield float item    Now you decide to factor out these loops into separate generators  Without yield from  this is ugly  up to the point where you will think twice whether you actually want to do it  With yield from  it s actually nice to look at   def get list values lst     for sub in  get list values as int                 get list values as str                 get list values as float       yield from sub lst       Why is it compared to micro-threads    I think what this section in the PEP is talking about is that every generator does have its own isolated execution context  Together with the fact that execution is switched between the generator-iterator and the caller using yield and   next      respectively  this is similar to threads  where the operating system switches the executing thread from time to time  along with the execution context  stack  registers         The effect of this is also comparable  Both the generator-iterator and the caller progress in their execution state at the same time  their executions are interleaved  For example  if the generator does some kind of computation and the caller prints out the results  you ll see the results as soon as they re available  This is a form of concurrency   That analogy isn t anything specific to yield from  though - it s rather a general property of generators in Python

User · Answer

Simply put  yield from provides tail recursion for iterator functions

User · Answer

Wherever you invoke a generator from within a generator you need a  pump  to re-yield the values   for v in inner generator  yield v   As the PEP points out there are subtle complexities to this which most people ignore   Non-local flow-control like throw   is one example given in the PEP   The new syntax yield from inner generator is used wherever you would have written the explicit for loop before   It s not merely syntactic sugar  though   It handles all of the corner cases that are ignored by the for loop   Being  sugary  encourages people to use it and thus get the right behaviors   This message in the discussion thread talks about these complexities      With the additional generator features introduced by PEP 342  that is no   longer the case  as described in Greg s PEP  simple iteration doesn t   support send   and throw   correctly  The gymnastics needed to support   send   and throw   actually aren t that complex when you break them   down  but they aren t trivial either    I can t speak to a comparison with micro-threads  other than to observe that generators are a type of paralellism   You can consider the suspended generator to be a thread which sends values via yield to a consumer thread   The actual implementation may be nothing like this  and the actual implementation is obviously of great interest to the Python developers  but this does not concern the users   The new yield from syntax does not add any additional capability to the language in terms of threading  it just makes it easier to use existing features correctly   Or more precisely it makes it easier for a novice consumer of a complex inner generator written by an expert to pass through that generator without breaking any of its complex features

User · Answer

This code defines a function fixed sum digits returning a generator enumerating all six digits numbers such that the sum of digits is 20    def iter fun sum  deepness  myString  Total       if deepness    0          if sum    Total              yield myString     else            for i in range min 10  Total - sum   1                yield from iter fun sum   i deepness - 1 myString   str i  Total   def fixed sum digits digits  Tot       return iter fun 0 digits    Tot     Try to write it without yield from  If you find an effective way to do it let me know    I think that for cases like this one  visiting trees  yield from makes the code simpler and cleaner

User · Answer

yield from basically chains iterators in a efficient way     chain from itertools  def chain  iters       for it in iters          for item in it              yield item    with the new keyword def chain  iters       for it in iters          yield from it   As you can see it removes one pure Python loop  That s pretty much all it does  but chaining iterators is a pretty common pattern in Python    Threads are basically a feature that allow you to jump out of functions at completely random points and jump back into the state of another function  The thread supervisor does this very often  so the program appears to run all these functions at the same time  The problem is that the points are random  so you need to use locking to prevent the supervisor from stopping the function at a problematic point   Generators are pretty similar to threads in this sense  They allow you to specify specific points  whenever they yield  where you can jump in and out  When used this way  generators are called coroutines    Read this excellent tutorials about coroutines in Python for more details

User · Answer

In applied usage for the Asynchronous IO coroutine  yield from has a similar behavior as await in a coroutine function  Both of which is used to suspend the execution of coroutine     yield from is used by the generator-based coroutine  await is used for async def  coroutine   since Python 3 5     For Asyncio  if there s no need to support an older Python version  i e   3 5   async def await is the recommended syntax to define a coroutine  Thus yield from is no longer needed in a coroutine    But in general outside of asyncio  yield from  lt sub-generator gt  has still some other usage in iterating the sub-generator as mentioned in the earlier answer

User · Answer

Let s get one thing out of the way first  The explanation that yield from g is equivalent to for v in g  yield v does not even begin to do justice to what yield from is all about  Because  let s face it  if all yield from does is expand the for loop  then it does not warrant adding yield from to the language and preclude a whole bunch of new features from being implemented in Python 2 x   What yield from does is it establishes a transparent bidirectional connection between the caller and the sub-generator    The connection is  transparent  in the sense that it will propagate everything correctly too  not just the elements being generated  e g  exceptions are propagated   The connection is  bidirectional  in the sense that data can be both sent from and to a generator     If we were talking about TCP  yield from g might mean  now temporarily disconnect my client s socket and reconnect it to this other server socket     BTW  if you are not sure what sending data to a generator even means  you need to drop everything and read about coroutines first   they re very useful  contrast them with subroutines   but unfortunately lesser-known in Python  Dave Beazley s Curious Course on Coroutines is an excellent start  Read slides 24-33 for a quick primer   Reading data from a generator using yield from  def reader           A generator that fakes a read from a file  socket  etc         for i in range 4           yield   lt  lt   s    i  def reader wrapper g         Manually iterate over data produced by reader     for v in g          yield v  wrap   reader wrapper reader    for i in wrap      print i     Result  lt  lt  0  lt  lt  1  lt  lt  2  lt  lt  3   Instead of manually iterating over reader    we can just yield from it   def reader wrapper g       yield from g   That works  and we eliminated one line of code  And probably the intent is a little bit clearer  or not   But nothing life changing   Sending data to a generator  coroutine  using yield from - Part 1  Now let s do something more interesting  Let s create a coroutine called writer that accepts data sent to it and writes to a socket  fd  etc   def writer           A coroutine that writes data  sent  to it to fd  socket  etc         while True          w    yield          print   gt  gt     w    Now the question is  how should the wrapper function handle sending data to the writer  so that any data that is sent to the wrapper is transparently sent to the writer     def writer wrapper coro         TBD     pass  w   writer   wrap   writer wrapper w  wrap send None      prime  the coroutine for i in range 4       wrap send i     Expected result  gt  gt   0  gt  gt   1  gt  gt   2  gt  gt   3   The wrapper needs to accept the data that is sent to it  obviously  and should also handle the StopIteration when the for loop is exhausted  Evidently just doing for x in coro  yield x won t do  Here is a version that works   def writer wrapper coro       coro send None     prime the coro     while True          try              x    yield     Capture the value that s sent             coro send x     and pass it to the writer         except StopIteration              pass   Or  we could do this   def writer wrapper coro       yield from coro   That saves 6 lines of code  make it much much more readable and it just works  Magic   Sending data to a generator yield from - Part 2 - Exception handling  Let s make it more complicated  What if our writer needs to handle exceptions  Let s say the writer handles a SpamException and it prints     if it encounters one   class SpamException Exception       pass  def writer        while True          try              w    yield          except SpamException              print                else              print   gt  gt     w    What if we don t change writer wrapper  Does it work  Let s try    writer wrapper same as above  w   writer   wrap   writer wrapper w  wrap send None      prime  the coroutine for i in  0  1  2   spam   4       if i     spam           wrap throw SpamException      else          wrap send i     Expected Result  gt  gt   0  gt  gt   1  gt  gt   2      gt  gt   4    Actual Result  gt  gt   0  gt  gt   1  gt  gt   2 Traceback  most recent call last         redacted       File     in writer wrapper     x    yield    main   SpamException   Um  it s not working because x    yield  just raises the exception and everything comes to a crashing halt  Let s make it work  but manually handling exceptions and sending them or throwing them into the sub-generator  writer   def writer wrapper coro          Works  Manually catches exceptions and throws them        coro send None     prime the coro     while True          try              try                  x    yield              except Exception as e      This catches the SpamException                 coro throw e              else                  coro send x          except StopIteration              pass   This works     Result  gt  gt   0  gt  gt   1  gt  gt   2      gt  gt   4   But so does this   def writer wrapper coro       yield from coro   The yield from transparently handles sending the values or throwing values into the sub-generator   This still does not cover all the corner cases though  What happens if the outer generator is closed  What about the case when the sub-generator returns a value  yes  in Python 3 3   generators can return values   how should the return value be propagated  That yield from transparently handles all the corner cases is really impressive  yield from just magically works and handles all those cases   I personally feel yield from is a poor keyword choice because it does not make the two-way nature apparent  There were other keywords proposed  like delegate but were rejected because adding a new keyword to the language is much more difficult than combining existing ones   In summary  it s best to think of yield from as a transparent two way channel between the caller and the sub-generator   References    PEP 380 - Syntax for delegating to a sub-generator  Ewing   v3 3  2009-02-13  PEP 342 - Coroutines via Enhanced Generators  GvR  Eby   v2 5  2005-05-10

[python] In practice, what are the main uses for the new "yield from" syntax in Python 3.3?

Examples related to python

Examples related to yield