Making an asynchronous task in Flask

Question

I am writing an application in Flask  which works really well except that WSGI is synchronous and blocking  I have one task in particular which calls out to a third party API and that task can take several minutes to complete  I would like to make that call  it s actually a series of calls  and let it run  while control is returned to Flask   My view looks like    app route   render  lt id gt    methods   POST    def render script id None               data   json loads request data      text list   data get  text list       final file   audio class render audio data text list        do stuff     return Response          mimetype  application json           status 200         Now  what I want to do is have the line  final file   audio class render audio     run and provide a callback to be executed when the method returns  whilst Flask can continue to process requests  This is the only task which I need Flask to run asynchronously  and I would like some advice on how best to implement this   I have looked at Twisted and Klein  but I m not sure they are overkill  as maybe Threading would suffice  Or maybe Celery is a good choice for this

User · Answer

You can also try using multiprocessing Process with daemon True  the process start   method does not block and you can return a response status immediately to the caller while your expensive function executes in the background  I experienced similar problem while working with falcon framework and using daemon process helped  You d need to do the following  from multiprocessing import Process   app route   render  lt id gt    methods   POST    def render script id None               heavy process   Process     Create a daemonic process with heavy  quot my func quot          target my func          daemon True           heavy process start       return Response          mimetype  application json           status 200          Define some heavy function def my func        time sleep 10      print  quot Process finished quot    You should get a response immediately and  after 10s you should see a printed message in the console  NOTE  Keep in mind that daemonic processes are not allowed to spawn any child processes

User · Answer

I would use Celery to handle the asynchronous task for you  You ll need to install a broker to serve as your task queue  RabbitMQ and Redis are recommended    app py   from flask import Flask from celery import Celery  broker url    amqp   guest localhost             Broker URL for RabbitMQ task queue  app   Flask   name        celery   Celery app name  broker broker url  celery config from object  celeryconfig          Your celery configurations in a celeryconfig py   celery task bind True  def some long task self  x  y         Do some long task           app route   render  lt id gt    methods   POST    def render script id None               data   json loads request data      text list   data get  text list       final file   audio class render audio data text list      some long task delay x  y                    Call your async task and pass whatever necessary variables     return Response          mimetype  application json           status 200         Run your Flask app  and start another process to run your celery worker     celery worker -A app celery --loglevel debug   I would also refer to Miguel Gringberg s write up for a more in depth guide to using Celery with Flask

User · Answer

Threading is another possible solution  Although the Celery based solution is better for applications at scale  if you are not expecting too much traffic on the endpoint in question  threading is a viable alternative   This solution is based on Miguel Grinberg s PyCon 2016 Flask at Scale presentation  specifically slide 41 in his slide deck  His code is also available on github for those interested in the original source   From a user perspective the code works as follows    You make a call to the endpoint that performs the long running task  This endpoint returns 202 Accepted with a link to check on the task status  Calls to the status link returns 202 while the taks is still running  and returns 200  and the result  when the task is complete    To convert an api call to a background task  simply add the  async api decorator   Here is a fully contained example   from flask import Flask  g  abort  current app  request  url for from werkzeug exceptions import HTTPException  InternalServerError from flask restful import Resource  Api from datetime import datetime from functools import wraps import threading import time import uuid  tasks       app   Flask   name    api   Api app     app before first request def before first request           Start a background thread that cleans up old tasks         def clean old tasks                        This function cleans up old tasks from our in-memory data structure                      global tasks         while True                Only keep tasks that are running or that finished less than 5               minutes ago              five min ago   datetime timestamp datetime utcnow    - 5   60             tasks    task id  task for task id  task in tasks items                        if  completion timestamp  not in task or task  completion timestamp    gt  five min ago              time sleep 60       if not current app config  TESTING            thread   threading Thread target clean old tasks          thread start     def async api wrapped function        wraps wrapped function      def new function  args    kwargs           def task call flask app  environ                 Create a request context similar to that of the original request               so that the task can have access to flask g  flask request  etc              with flask app request context environ                   try                      tasks task id   return value     wrapped function  args    kwargs                  except HTTPException as e                      tasks task id   return value     current app handle http exception e                  except Exception as e                        The function raised an exception  so we set a 500 error                     tasks task id   return value     InternalServerError                       if current app debug                            We want to find out if something happened so reraise                         raise                 finally                        We record the time of the response  to help in garbage                       collecting old tasks                     tasks task id   completion timestamp     datetime timestamp datetime utcnow                           close the database session  if any             Assign an id to the asynchronous task         task id   uuid uuid4   hex            Record the task  and then launch it         tasks task id      task thread   threading Thread              target task call  args  current app  get current object                                   request environ            tasks task id   task thread   start              Return a 202 response  with a link that the client can use to           obtain task status         print url for  gettaskstatus   task id task id           return  accepted   202    Location   url for  gettaskstatus   task id task id       return new function   class GetTaskStatus Resource       def get self  task id                       Return status about an asynchronous task  If this request returns a 202         status code  it means that task hasn t finished yet  Else  the response         from the task is returned                      task   tasks get task id          if task is None              abort 404          if  return value  not in task              return     202    Location   url for  gettaskstatus   task id task id           return task  return value     class CatchAll Resource        async api     def get self  path                perform some intensive processing         print  starting processing task  path    s     path          time sleep 10          print  completed processing task  path    s     path          return f The answer is   path     api add resource CatchAll     lt path path gt         api add resource GetTaskStatus    status  lt task id gt      if   name         main         app run debug True

[python] Making an asynchronous task in Flask

Examples related to python

Examples related to asynchronous

Examples related to flask