Asynchronous Requests with Python requests

Question

I tried the sample provided within the documentation of the requests library for python   With async map rs   I get the response codes  but I want to get the content of each page requested  This  for example  does not work   out   async map rs  print out 0  content

User · Accepted Answer

Note  The below answer is not applicable to requests v0 13 0   The asynchronous functionality was moved to grequests after this question was written  However  you could just replace requests with grequests below and it should work   I ve left this answer as is to reflect the original question which was about using requests  lt  v0 13 0     To do multiple tasks with async map asynchronously you have to    Define a function for what you want to do with each object  your task  Add that function as an event hook in your request Call async map on a list of all the requests   actions   Example   from requests import async   If using requests  gt  v0 13 0  use   from grequests import async  urls          http   python-requests org        http   httpbin org        http   python-guide org        http   kennethreitz com       A simple task to do to each response object def do something response       print response url    A list to hold our things to do via async async list       for u in urls        The  hooks         part is where you define what you want to do              Note the lack of parentheses following do something  this is       because the response will be used as the first argument automatically     action item   async get u  hooks     response    do something          Add the task to our list of things to do via async     async list append action item     Do our list of things to do via async async map async list

User · Answer

Unfortunately  as far as I know  the requests library is not equipped for performing asynchronous requests  You can wrap async await syntax around requests  but that will make the underlying requests no less synchronous  If you want true async requests  you must use other tooling that provides it  One such solution is aiohttp  Python 3 5 3    It works well in my experience using it with the Python 3 7 async await syntax  Below I write three implementations of performing n web requests using  Purely synchronous requests  sync requests get all  using the Python requests library Synchronous requests  async requests get all  using the Python requests library wrapped in Python 3 7 async await syntax and asyncio A truly asynchronous implementation  async aiohttp get all  with the Python aiohttp library wrapped in Python 3 7 async await syntax and asyncio  import time import asyncio import requests import aiohttp  from types import SimpleNamespace  durations        def timed func        quot  quot  quot      records approximate durations of function calls      quot  quot  quot      def wrapper  args    kwargs           start   time time           print f  func   name    lt 30  started           result   func  args    kwargs          duration   f  func   name    lt 30  finished in  time time   - start  2f  seconds          print duration          durations append duration          return result     return wrapper   async def fetch url  session        quot  quot  quot      asynchronous get request      quot  quot  quot      async with session get url  as response          response json   await response json           return SimpleNamespace   response json    async def fetch many loop  urls        quot  quot  quot      many asynchronous get requests  gathered      quot  quot  quot      async with aiohttp ClientSession   as session          tasks    loop create task fetch url  session   for url in urls          return await asyncio gather  tasks    timed def sync requests get all urls        quot  quot  quot      performs synchronous get requests      quot  quot  quot        use session to reduce network overhead     session   requests Session       return  SimpleNamespace   session get url  json    for url in urls     timed def async requests get all urls        quot  quot  quot      asynchronous wrapper around synchronous requests      quot  quot  quot      loop   asyncio get event loop         use session to reduce network overhead     session   requests Session        async def async get url           return session get url       async tasks    loop create task async get url   for url in urls      return loop run until complete asyncio gather  async tasks      timed def asnyc aiohttp get all urls        quot  quot  quot      performs asynchronous get requests      quot  quot  quot      loop   asyncio get event loop       return loop run until complete fetch many loop  urls     if   name         main           this endpoint takes  3 seconds to respond        so a purely synchronous implementation should take       little more than 30 seconds and a purely asynchronous       implementation should take little more than 3 seconds      urls     https   postman-echo com delay 3   10      sync requests get all urls      async requests get all urls      asnyc aiohttp get all urls      print  ----------------------        print duration  for duration in durations   On my machine  this is the output  sync requests get all          started sync requests get all          finished in 30 92 seconds async requests get all         started async requests get all         finished in 30 87 seconds asnyc aiohttp get all          started asnyc aiohttp get all          finished in 3 22 seconds ---------------------- sync requests get all          finished in 30 92 seconds async requests get all         finished in 30 87 seconds asnyc aiohttp get all          finished in 3 22 seconds

User · Answer

I have also tried some things using the asynchronous methods in python  how ever I have had much better luck using twisted for asynchronous programming  It has fewer problems and is well documented  Here is a link of something simmilar to what you are trying in twisted   http   pythonquirks blogspot com 2011 04 twisted-asynchronous-http-request html

User · Answer

I have a lot of issues with most of the answers posted - they either use deprecated libraries that have been ported over with limited features  or provide a solution with too much magic on the execution of the request  making it difficult to error handle  If they do not fall into one of the above categories  they re 3rd party libraries or deprecated   Some of the solutions works alright purely in http requests  but the solutions fall short for any other kind of request  which is ludicrous  A highly customized solution is not necessary here    Simply using the python built-in library asyncio is sufficient enough to perform asynchronous requests of any type  as well as providing enough fluidity for complex and usecase specific error handling    import asyncio  loop   asyncio get event loop    def do thing params       async def get rpc info and do chores id             do things         response   perform grpc call id          do chores response       async def get httpapi info and do chores id             do things         response   requests get URL          do chores response       async tasks          for element in list params list of things          async tasks append loop create task get chan info and do chores id           async tasks append loop create task get httpapi info and do chores ch id         loop run until complete asyncio gather  async tasks     How it works is simple  You re creating a series of tasks you d like to occur asynchronously  and then asking a loop to execute those tasks and exit upon completion  No extra libraries subject to lack of maintenance  no lack of functionality required

User · Answer

You can use httpx for that  import httpx  async def get async url       async with httpx AsyncClient   as client          return await client get url   urls     quot http   google com quot    quot http   wikipedia org quot      Note that you need an async context to use  await   await asyncio gather  map get async  urls    if you want a functional syntax  the gamla lib wraps this into get async  Then you can do  await gamla map gamla get async 10     quot http   google com quot    quot http   wikipedia org quot     The 10 is the timeout in seconds   disclaimer  I am its author

User · Answer

I know this has been closed for a while  but I thought it might be useful to promote another async solution built on the requests library   list of requests     http   moop com    http   doop com         from simple requests import Requests for response in Requests   swarm list of requests       print response content   The docs are here  http   pythonhosted org simple-requests

User · Answer

I have been using python requests for async calls against github s gist API for some time    For an example  see the code here   https   github com davidthewatson flasgist blob master views py L60-72  This style of python may not be the clearest example  but I can assure you that the code works  Let me know if this is confusing to you and I will document it

User · Answer

I tested both requests-futures and grequests  Grequests is faster but brings monkey patching and additional problems with dependencies  requests-futures is several times slower than grequests  I decided to write my own and simply wrapped requests into ThreadPoolExecutor and it was almost as fast as grequests  but without external dependencies  import requests import concurrent futures  def get urls        return   quot url1 quot   quot url2 quot    def load url url  timeout       return requests get url  timeout   timeout   with concurrent futures ThreadPoolExecutor max workers 20  as executor       future to url    executor submit load url  url  10   url for url in     get urls        for future in concurrent futures as completed future to url           url   future to url future          try              data   future result           except Exception as exc              resp err   resp err   1         else              resp ok   resp ok   1

User · Answer

async is now an independent module   grequests   See here   https   github com kennethreitz grequests  And there  Ideal method for sending multiple HTTP requests over Python   installation     pip install grequests   usage   build a stack   import grequests  urls          http   www heroku com        http   tablib org        http   httpbin org        http   python-requests org        http   kennethreitz com     rs    grequests get u  for u in urls    send the stack  grequests map rs    result looks like    lt Response  200  gt    lt Response  200  gt    lt Response  200  gt    lt Response  200  gt    lt Response  200  gt     grequests don t seem to set a limitation for concurrent requests  ie when multiple requests are sent to the same server

User · Answer

DISCLAMER  Following code creates different threads for each function  This might be useful for some of the cases as it is simpler to use  But know that it is not async but gives illusion of async using multiple threads  even though decorator suggests that  You can use the following decorator to give a callback once the execution of function is completed  the callback must handle the processing of data returned by the function  Please note that after the function is decorated it will return a Future object  import asyncio     Decorator implementation of async runner    def run async callback  loop None       if loop is None          loop   asyncio get event loop        def inner func           def wrapper  args    kwargs               def   exec                    out   func  args    kwargs                  callback out                  return out              return loop run in executor None    exec           return wrapper      return inner  Example of implementation  urls     quot https   google com quot    quot https   facebook com quot    quot https   apple com quot    quot https   netflix com quot   loaded urls         OPTIONAL  used for showing realtime  which urls are loaded      def  callback resp       print resp url      print resp      loaded urls append  resp url  resp      OPTIONAL  used for showing realtime  which urls are loaded        Must provide a callback function  callback func will be executed after the func completes execution   Callback function will accept the value returned by the function   run async  callback  def get url       return requests get url    for url in urls      get url   If you wish to see which url are loaded in real-time then  you can add the following code at the end as well  while True      print loaded urls      if len loaded urls     len urls           break

User · Answer

from threading import Thread  threads list    for requestURI in requests      t   Thread target self openURL  args  requestURI        t start       threads append t   for thread in threads      thread join         def openURL self  requestURI       o   urllib2 urlopen requestURI  timeout   600      o

User · Answer

If you want to use asyncio  then requests-async provides async await functionality for requests - https   github com encode requests-async

User · Answer

maybe requests-futures is another choice    from requests futures sessions import FuturesSession  session   FuturesSession     first request is started in background future one   session get  http   httpbin org get     second requests is started immediately future two   session get  http   httpbin org get foo bar     wait for the first request to complete  if it hasn t already response one   future one result   print  response one status   0   format response one status code   print response one content    wait for the second request to complete  if it hasn t already response two   future two result   print  response two status   0   format response two status code   print response two content    It is also recommended in the office document  If you don t want involve gevent  it s a good one

[python] Asynchronous Requests with Python requests

Note

Examples related to python

Examples related to asynchronous

Examples related to python-requests

Examples related to httprequest