Asyncio gather vs asyncio wait

Question

asyncio gather and asyncio wait seem to have similar uses  I have a bunch of async things that I want to execute wait for  not necessarily waiting for one to finish before the next one starts   They use a different syntax  and differ in some details  but it seems very un-pythonic to me to have 2 functions that have such a huge overlap in functionality  What am I missing

User · Answer

asyncio wait is more low level than asyncio gather  As the name suggests  asyncio gather mainly focuses on gathering the results  It waits on a bunch of futures and returns their results in a given order  asyncio wait just waits on the futures  And instead of giving you the results directly  it gives done and pending tasks  You have to manually collect the values  Moreover  you could specify to wait for all futures to finish or just the first one with wait

User · Answer

A very important distinction  which is easy to miss  is the default bheavior of these two functions  when it comes to exceptions   I ll use this example to simulate a coroutine that will raise exceptions  sometimes - import asyncio import random   async def a flaky tsk i       await asyncio sleep i     bit of fuzz to simulate a real-world example      if i   2    0          print i   quot ok quot       else          print i   quot crashed  quot           raise ValueError  coros    a flaky tsk i  for i in range 10     await asyncio gather  coros  outputs - 0 ok 1 crashed  Traceback  most recent call last     File  quot  Users dev PycharmProjects trading xxx py quot   line 20  in  lt module gt      asyncio run main      File  quot  Users dev  pyenv versions 3 8 2 lib python3 8 asyncio runners py quot   line 43  in run     return loop run until complete main    File  quot  Users dev  pyenv versions 3 8 2 lib python3 8 asyncio base events py quot   line 616  in run until complete     return future result     File  quot  Users dev PycharmProjects trading xxx py quot   line 17  in main     await asyncio gather  coros    File  quot  Users dev PycharmProjects trading xxx py quot   line 12  in a flaky tsk     raise ValueError ValueError  As you can see  the coros after index 1 never got to execute   But await asyncio wait coros  continues to execute tasks  even if some of them fail - 0 ok 1 crashed  2 ok 3 crashed  4 ok 5 crashed  6 ok 7 crashed  8 ok 9 crashed  Task exception was never retrieved future   lt Task finished name  Task-10  coro  lt a flaky tsk   done  defined at  Users dev PycharmProjects trading xxx py 6 gt  exception ValueError   gt  Traceback  most recent call last     File  quot  Users dev PycharmProjects trading xxx py quot   line 12  in a flaky tsk     raise ValueError ValueError Task exception was never retrieved future   lt Task finished name  Task-8  coro  lt a flaky tsk   done  defined at  Users dev PycharmProjects trading xxx py 6 gt  exception ValueError   gt  Traceback  most recent call last     File  quot  Users dev PycharmProjects trading xxx py quot   line 12  in a flaky tsk     raise ValueError ValueError Task exception was never retrieved future   lt Task finished name  Task-2  coro  lt a flaky tsk   done  defined at  Users dev PycharmProjects trading xxx py 6 gt  exception ValueError   gt  Traceback  most recent call last     File  quot  Users dev PycharmProjects trading xxx py quot   line 12  in a flaky tsk     raise ValueError ValueError Task exception was never retrieved future   lt Task finished name  Task-9  coro  lt a flaky tsk   done  defined at  Users dev PycharmProjects trading xxx py 6 gt  exception ValueError   gt  Traceback  most recent call last     File  quot  Users dev PycharmProjects trading xxx py quot   line 12  in a flaky tsk     raise ValueError ValueError Task exception was never retrieved future   lt Task finished name  Task-3  coro  lt a flaky tsk   done  defined at  Users dev PycharmProjects trading xxx py 6 gt  exception ValueError   gt  Traceback  most recent call last     File  quot  Users dev PycharmProjects trading xxx py quot   line 12  in a flaky tsk     raise ValueError ValueError   Ofcourse  this behavior can be changed for both by using - asyncio gather      return exceptions True  or  asyncio wait        return when asyncio FIRST EXCEPTION   But it doesn t end here  Notice  Task exception was never retrieved in the logs above  asyncio wait   won t re-raise exceptions from the child tasks until you await them individually   The stacktrace in the logs are just messages  they cannot be caught   done  pending   await asyncio wait coros  for tsk in done      try          await tsk     except Exception as e          print  quot I caught  quot   repr e    Output - 0 ok 1 crashed  2 ok 3 crashed  4 ok 5 crashed  6 ok 7 crashed  8 ok 9 crashed  I caught  ValueError   I caught  ValueError   I caught  ValueError   I caught  ValueError   I caught  ValueError     On the other hand  to catch exceptions with asyncio gather    you must - results   await asyncio gather  coros  return exceptions True  for result or exc in results      if isinstance result or exc  Exception           print  quot I caught  quot   repr result or exc     Same output as before

User · Answer

In addition to all the previous answers  I would like to tell about the different behavior of gather   and wait   in case they are cancelled  Gather cancellation If gather   is cancelled  all submitted awaitables  that have not completed yet  are also cancelled  Wait cancellation If the wait   task is cancelled  it simply throws an CancelledError and the waited tasks remain intact  Simple example  import asyncio   async def task arg       await asyncio sleep 5      return arg   async def cancel waiting task work task  waiting task       await asyncio sleep 2      waiting task cancel       try          await waiting task         print  quot Waiting done quot       except asyncio CancelledError          print  quot Waiting task cancelled quot        try          res   await work task         print f quot Work result   res  quot       except asyncio CancelledError          print  quot Work task cancelled quot     async def main        work task   asyncio create task task  quot done quot        waiting   asyncio create task asyncio wait  work task        await cancel waiting task work task  waiting       work task   asyncio create task task  quot done quot        waiting   asyncio gather work task      await cancel waiting task work task  waiting    asyncio run main     Output  asyncio wait   Waiting task cancelled Work result  done ---------------- asyncio gather   Waiting task cancelled Work task cancelled   Sometimes it becomes necessary to combine wait   and gather   functionality  For example  we want to wait for the completion of at least one task and cancel the rest pending tasks after that  and if the waiting itself was canceled  then also cancel all pending tasks  As real examples  let s say we have a disconnect event and a work task  And we want to wait for the results of the work task  but if the connection was lost  then cancel it  Or we will make several parallel requests  but upon completion of at least one response  cancel all others  It could be done this way  import asyncio from typing import Optional  Tuple  Set   async def wait any          tasks  Set asyncio Future      timeout  Optional int    None    - gt  Tuple Set asyncio Future   Set asyncio Future        tasks to cancel  Set asyncio Future    set       try          done  tasks to cancel   await asyncio wait              tasks  timeout timeout  return when asyncio FIRST COMPLETED                   return done  tasks to cancel     except asyncio CancelledError          tasks to cancel   tasks         raise     finally          for task in tasks to cancel              task cancel     async def task        await asyncio sleep 5    async def cancel waiting task work task  waiting task       await asyncio sleep 2      waiting task cancel       try          await waiting task         print  quot Waiting done quot       except asyncio CancelledError          print  quot Waiting task cancelled quot        try          res   await work task         print f quot Work result   res  quot       except asyncio CancelledError          print  quot Work task cancelled quot     async def check tasks waiting task  working task  waiting conn lost task       try          await waiting task         print  quot waiting is done quot       except asyncio CancelledError          print  quot waiting is cancelled quot        try          await waiting conn lost task         print  quot connection is lost quot       except asyncio CancelledError          print  quot waiting connection lost is cancelled quot        try          await working task         print  quot work is done quot       except asyncio CancelledError          print  quot work is cancelled quot     async def work done case        working task   asyncio create task task        connection lost event   asyncio Event       waiting conn lost task   asyncio create task connection lost event wait        waiting task   asyncio create task wait any  working task  waiting conn lost task        await check tasks waiting task  working task  waiting conn lost task    async def conn lost case        working task   asyncio create task task        connection lost event   asyncio Event       waiting conn lost task   asyncio create task connection lost event wait        waiting task   asyncio create task wait any  working task  waiting conn lost task        await asyncio sleep 2      connection lost event set       lt ---     await check tasks waiting task  working task  waiting conn lost task    async def cancel waiting case        working task   asyncio create task task        connection lost event   asyncio Event       waiting conn lost task   asyncio create task connection lost event wait        waiting task   asyncio create task wait any  working task  waiting conn lost task        await asyncio sleep 2      waiting task cancel       lt ---     await check tasks waiting task  working task  waiting conn lost task    async def main        print  quot Work done quot       print  quot ------------------- quot       await work done case       print  quot  nConnection lost quot       print  quot ------------------- quot       await conn lost case       print  quot  nCancel waiting quot       print  quot ------------------- quot       await cancel waiting case     asyncio run main     Output  Work done ------------------- waiting is done waiting connection lost is cancelled work is done  Connection lost ------------------- waiting is done connection is lost work is cancelled  Cancel waiting ------------------- waiting is cancelled waiting connection lost is cancelled work is cancelled

User · Answer

Although similar in general cases   run and get results for many tasks    each function has some specific functionality for other cases   asyncio gather    Returns a Future instance  allowing high level grouping of tasks   import asyncio from pprint import pprint  import random   async def coro tag       print   gt    tag      await asyncio sleep random uniform 1  3       print   lt    tag      return tag   loop   asyncio get event loop    group1   asyncio gather   coro  group 1     format i   for i in range 1  6    group2   asyncio gather   coro  group 2     format i   for i in range 1  4    group3   asyncio gather   coro  group 3     format i   for i in range 1  10     all groups   asyncio gather group1  group2  group3   results   loop run until complete all groups   loop close    pprint results    All tasks in a group can be cancelled by calling group2 cancel   or even all groups cancel     See also  gather      return exceptions True    asyncio wait    Supports waiting to be stopped after the first task is done  or after a specified timeout  allowing lower level precision of operations   import asyncio import random   async def coro tag       print   gt    tag      await asyncio sleep random uniform 0 5  5       print   lt    tag      return tag   loop   asyncio get event loop    tasks    coro i  for i in range 1  11    print  Get first result    finished  unfinished   loop run until complete      asyncio wait tasks  return when asyncio FIRST COMPLETED    for task in finished      print task result    print  unfinished    len unfinished    print  Get more results in 2 seconds    finished2  unfinished2   loop run until complete      asyncio wait unfinished  timeout 2    for task in finished2      print task result    print  unfinished2    len unfinished2    print  Get all other results    finished3  unfinished3   loop run until complete asyncio wait unfinished2    for task in finished3      print task result     loop close

User · Answer

I also noticed that you can provide a group of coroutines in wait   by simply specifying the list   result loop run until complete asyncio wait           say  first hello   2           say  second hello   1           say  third hello   4            Whereas grouping in gather   is done by just specifying multiple coroutines   result loop run until complete asyncio gather          say  first hello   2           say  second hello   1           say  third hello   4

[python] Asyncio.gather vs asyncio.wait

Examples related to python

Examples related to python-asyncio