Asynchronous method call in Python

Question

I was wondering if there s any library for asynchronous method calls in Python  It would be great if you could do something like   async def longComputation         lt code gt    token   longComputation   token registerCallback callback function    alternative  polling while not token finished        doSomethingElse       if token finished            result   token result     Or to call a non-async routine asynchronously  def longComputation        lt code gt   token   asynccall longComputation      It would be great to have a more refined strategy as native in the language core  Was this considered

User · Answer

It s not in the language core  but a very mature library that does what you want is Twisted  It introduces the Deferred object  which you can attach callbacks or error handlers   errbacks   to  A Deferred is basically a  promise  that a function will have a result eventually

User · Answer

Something like   import threading  thr   threading Thread target foo  args     kwargs     thr start     Will run  foo       thr is alive     Will return whether foo is running currently      thr join     Will wait till  foo  is done   See the documentation at https   docs python org library threading html for more details

User · Answer

You could use eventlet  It lets you write what appears to be synchronous code  but have it operate asynchronously over the network   Here s an example of a super minimal crawler   urls     http   www google com intl en ALL images logo gif         https   wiki secondlife com w images secondlife jpg         http   us i1 yimg com us yimg com i ww beta y3 gif    import eventlet from eventlet green import urllib2  def fetch url      return urllib2 urlopen url  read    pool   eventlet GreenPool    for body in pool imap fetch  urls     print  got body   len body

User · Answer

Something like this works for me  you can then call the function  and it will dispatch itself onto a new thread   from thread import start new thread  def dowork asynchronous True       if asynchronous          args    False          start new thread dowork args   Call itself on a new thread      else          while True               do something                time sleep 60   sleep for a minute     return

User · Answer

You can use process  If you want to run it forever use while  like networking  in you function   from multiprocessing import Process def foo        while 1            Do something  p   Process target   foo  p start     if you just want to run it one time  do like that   from multiprocessing import Process def foo          Do something  p   Process target   foo  p start   p join

User · Answer

You can use concurrent futures  added in Python 3 2    import time from concurrent futures import ThreadPoolExecutor   def long computation duration       for x in range 0  duration           print x          time sleep 1      return duration   2   print  Use polling   with ThreadPoolExecutor max workers 1  as executor      future   executor submit long computation  5      while not future done            print  waiting              time sleep 0 5       print future result     print  Use callback   executor   ThreadPoolExecutor max workers 1  future   executor submit long computation  5  future add done callback lambda f  print f result      print  waiting for callback    executor shutdown False     non-blocking  print  shutdown invoked

User · Answer

My solution is   import threading  class TimeoutError RuntimeError       pass  class AsyncCall object       def   init   self  fnc  callback   None           self Callable   fnc         self Callback   callback      def   call   self   args    kwargs           self Thread   threading Thread target   self run  name   self Callable   name    args   args  kwargs   kwargs          self Thread start           return self      def wait self  timeout   None           self Thread join timeout          if self Thread isAlive                raise TimeoutError           else              return self Result      def run self   args    kwargs           self Result   self Callable  args    kwargs          if self Callback              self Callback self Result   class AsyncMethod object       def   init   self  fnc  callback None           self Callable   fnc         self Callback   callback      def   call   self   args    kwargs           return AsyncCall self Callable  self Callback   args    kwargs   def Async fnc   None  callback   None       if fnc    None          def AddAsyncCallback fnc               return AsyncMethod fnc  callback          return AddAsyncCallback     else          return AsyncMethod fnc  callback    And works exactly as requested    Async def fnc        pass

User · Answer

Is there any reason not to use threads  You can use the threading class  Instead of finished   function use the isAlive    The result   function could join   the thread and retrieve the result  And  if you can  override the run   and   init   functions to call the function specified in the constructor and save the value somewhere to the instance of the class

User · Answer

You can implement a decorator to make your functions asynchronous  though that s a bit tricky  The multiprocessing module is full of little quirks and seemingly arbitrary restrictions     all the more reason to encapsulate it behind a friendly interface  though   from inspect import getmodule from multiprocessing import Pool   def async decorated       r   Wraps a top-level function around an asynchronous dispatcher           when the decorated function is called  a task is submitted to a         process pool  and a future object is returned  providing access to an         eventual return value           The future object has a blocking get   method to access the task         result  it will return immediately if the job is already done  or block         until it completes           This decorator won t work on methods  due to limitations in Python s         pickling machinery  in principle methods could be made pickleable  but         good luck on that                 Keeps the original function visible from the module global namespace        under a name consistent to its   name   attribute  This is necessary for       the multiprocessing pickling machinery to work properly      module   getmodule decorated      decorated   name        original      setattr module  decorated   name    decorated       def send  args    opts           return async pool apply async decorated  args  opts       return send   The code below illustrates usage of the decorator    async def printsum uid  values       summed   0     for value in values          summed    value      print  Worker  i  sum value is  i     uid  summed        return  uid  summed    if   name         main         from random import sample        The process pool must be created inside   main        async pool   Pool 4       p   range 0  1000      results          for i in range 4           result   printsum i  sample p  100           results append result       for result in results          print  Worker  i  sum value is  i    result get      In a real-world case I would ellaborate a bit more on the decorator  providing some way to turn it off for debugging  while keeping the future interface in place   or maybe a facility for dealing with exceptions  but I think this demonstrates the principle well enough

User · Answer

The native Python way for asynchronous calls in 2021 with Python 3 9 suitable also for Jupyter   Ipython Kernel Camabeh s answer is the way to go since Python 3 3   async def display date loop       end time   loop time     5 0     while True          print datetime datetime now            if  loop time     1 0   gt   end time              break         await asyncio sleep 1    loop   asyncio get event loop     Blocking call which returns when the display date   coroutine is done loop run until complete display date loop   loop close     This will work in Jupyter Notebook   Jupyter Lab but throw an error  RuntimeError  This event loop is already running  Due to Ipython s usage of event loops we need something called nested asynchronous loops which is not yet implemented in Python  Luckily there is nest asyncio to deal with the issue  All you need to do is   pip install nest asyncio   use   within Jupyter Notebook  else pip install in shell import nest asyncio nest asyncio apply     Based on this thread  Only when you call loop close   it throws another error as it probably refers to Ipython s main loop  RuntimeError  Cannot close a running event loop  I ll update this answer as soon as someone answered to this github issue

User · Answer

Just  import threading  time  def f        print  f started      time sleep 3      print  f finished   threading Thread target f  start

User · Answer

You can use the multiprocessing module added in Python 2 6  You can use pools of processes and then get results asynchronously with    apply async func   args   kwds   callback       E g    from multiprocessing import Pool  def f x       return x x  if   name         main         pool   Pool processes 1                 Start a worker processes      result   pool apply async f   10   callback    Evaluate  f 10   asynchronously calling callback when finished    This is only one alternative  This module provides lots of facilities to achieve what you want  Also it will be really easy to make a decorator from this

User · Answer

As of Python 3 5  you can use enhanced generators for async functions   import asyncio import datetime   Enhanced generator syntax    asyncio coroutine def display date loop       end time   loop time     5 0     while True          print datetime datetime now            if  loop time     1 0   gt   end time              break         yield from asyncio sleep 1    loop   asyncio get event loop     Blocking call which returns when the display date   coroutine is done loop run until complete display date loop   loop close     New async await syntax   async def display date loop       end time   loop time     5 0     while True          print datetime datetime now            if  loop time     1 0   gt   end time              break         await asyncio sleep 1    loop   asyncio get event loop     Blocking call which returns when the display date   coroutine is done loop run until complete display date loop   loop close

[python] Asynchronous method call in Python?

Examples related to python

Examples related to asynchronous