How could I use requests in asyncio

Question

I want to do parallel http request tasks in asyncio  but I find that python-requests would block the event loop of asyncio  I ve found aiohttp but it couldn t provide the service of http request using a http proxy   So I want to know if there s a way to do asynchronous http requests with the help of asyncio

User · Answer

The answers above are still using the old Python 3.4 style coroutines. Here is what you would write if you got Python 3.5+.

aiohttp supports http proxy now

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    urls = [
            'http://python.org',
            'https://google.com',
            'http://yifei.me'
        ]
    tasks = []
    async with aiohttp.ClientSession() as session:
        for url in urls:
            tasks.append(fetch(session, url))
        htmls = await asyncio.gather(*tasks)
        for html in htmls:
            print(html[:100])

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

User · Answer

aiohttp can be used with HTTP proxy already   import asyncio import aiohttp    asyncio coroutine def do request        proxy url    http   localhost 8118     your proxy address     response   yield from aiohttp request           GET    http   google com           proxy proxy url            return response  loop   asyncio get event loop   loop run until complete do request

User · Answer

There is a good case of async await loops and threading in an article by Pimin Konstantin Kefaloukos Easy parallel HTTP requests with Python and asyncio      To minimize the total completion time  we could increase the size of the thread pool to match the number of requests we have to make  Luckily  this is easy to do as we will see next  The code listing below is an example of how to make twenty asynchronous HTTP requests with a thread pool of twenty worker threads      Example 3  asynchronous requests with larger thread pool import asyncio import concurrent futures import requests  async def main         with concurrent futures ThreadPoolExecutor max workers 20  as executor           loop   asyncio get event loop           futures                 loop run in executor                  executor                   requests get                    http   example org                             for i in range 20                    for response in await asyncio gather  futures               pass   loop   asyncio get event loop   loop run until complete main

User · Answer

Requests does not currently support asyncio and there are no plans to provide such support  It s likely that you could implement a custom  Transport Adapter   as discussed here  that knows how to use asyncio   If I find myself with some time it s something I might actually look into  but I can t promise anything

User · Answer

To use requests  or any other blocking libraries  with asyncio  you can use BaseEventLoop run in executor to run a function in another thread and yield from it to get the result  For example   import asyncio import requests   asyncio coroutine def main        loop   asyncio get event loop       future1   loop run in executor None  requests get   http   www google com       future2   loop run in executor None  requests get   http   www google co uk       response1   yield from future1     response2   yield from future2     print response1 text      print response2 text   loop   asyncio get event loop   loop run until complete main      This will get both responses in parallel   With python 3 5 you can use the new await async syntax   import asyncio import requests  async def main        loop   asyncio get event loop       future1   loop run in executor None  requests get   http   www google com       future2   loop run in executor None  requests get   http   www google co uk       response1   await future1     response2   await future2     print response1 text      print response2 text   loop   asyncio get event loop   loop run until complete main      See PEP0492 for more

User · Answer

DISCLAMER  Following code creates different threads for each function  This might be useful for some of the cases as it is simpler to use  But know that it is not async but gives illusion of async using multiple threads  even though decorator suggests that  To make any function non blocking  simply copy the decorator and decorate any function with a callback function as parameter  The callback function will receive the data returned from the function  import asyncio import requests   def run async callback       def inner func           def wrapper  args    kwargs               def   exec                    out   func  args    kwargs                  callback out                  return out              return asyncio get event loop   run in executor None    exec           return wrapper      return inner   def  callback  args       print args      Must provide a callback function  callback func will be executed after the func completes execution     run async  callback  def get url       return requests get url    get  quot https   google com quot   print  quot Non blocking code ran    quot

[python] How could I use requests in asyncio?

Examples related to python

Examples related to python-requests

Examples related to python-3.4

Examples related to aiohttp