Deleting all pending tasks in celery rabbitmq

Question

How can I delete all pending tasks without knowing the task id for each task

User · Answer

I found that celery purge doesn t work for my more complex celery config   I use multiple named queues for different purposes     sudo rabbitmqctl list queues -p celery name messages consumers Listing queues        Output sorted  whitespaced for readability celery                                          0   2 celery web01 celery pidbox                      0   1 celery web02 celery pidbox                      0   1 apns                                            0   1 apns web01 celery pidbox                        0   1 analytics                                       1   1 analytics web01 celery pidbox                   0   1 bcast 361093f1-de68-46c5-adff-d49ea8f164c0      0   1 bcast a53632b0-c8b8-46d9-bd59-364afe9998c1      0   1 celeryev c27b070d-b07e-4e37-9dca-dbb45d03fd54   0   1 celeryev c66a9bed-84bd-40b0-8fe7-4e4d0c002866   0   1 celeryev b490f71a-be1a-4cd8-ae17-06a713cc2a99   0   1 celeryev 9d023165-ab4a-42cb-86f8-90294b80bd1e   0   1   The first column is the queue name  the second is the number of messages waiting in the queue  and the third is the number of listeners for that queue   The queues are    celery - Queue for standard  idempotent celery tasks apns - Queue for Apple Push Notification Service tasks  not quite as idempotent analytics - Queue for long running nightly analytics   pidbox - Queue for worker commands  such as shutdown and reset  one per worker  2 celery workers  one apns worker  one analytics worker  bcast   - Broadcast queues  for sending messages to all workers listening to a queue  rather than just the first to grab it  celeryev   - Celery event queues  for reporting task analytics   The analytics task is a brute force tasks that worked great on small data sets  but now takes more than 24 hours to process   Occasionally  something will go wrong and it will get stuck waiting on the database   It needs to be re-written  but until then  when it gets stuck I kill the task  empty the queue  and try again   I detect  stuckness  by looking at the message count for the analytics queue  which should be 0  finished analytics  or 1  waiting for last night s analytics to finish    2 or higher is bad  and I get an email   celery purge offers to erase tasks from one of the broadcast queues  and I don t see an option to pick a different named queue   Here s my process     sudo  etc init d celeryd stop    Wait for analytics task to be last one  Ctrl-C   ps -ef   grep analytics    Get the PID of the worker  not the root PID reported by celery   sudo kill  lt PID gt    sudo  etc init d celeryd stop    Confim dead   python manage py celery amqp queue purge analytics   sudo rabbitmqctl list queues -p celery name messages consumers    Confirm messages is 0   sudo  etc init d celeryd start

User · Answer

From the docs     celery -A proj purge   or  from proj celery import app app control purge      EDIT  Updated with current method

User · Answer

In Celery 3    CLI     celery -A proj purge   Programatically    gt  gt  gt  from proj celery import app  gt  gt  gt  app control purge     http   docs celeryproject org en latest faq html how-do-i-purge-all-waiting-tasks

User · Answer

In Celery 3   http   docs celeryproject org en 3 1 faq html how-do-i-purge-all-waiting-tasks  CLI  Purge named queue    celery -A proj amqp queue purge  lt queue name gt    Purge configured queue  celery -A proj purge      I   ve purged messages  but there are still messages left in the queue    Answer  Tasks are acknowledged  removed from the queue  as soon as they are actually executed  After the worker has received a task  it will take some time until it is actually executed  especially if there are a lot of tasks already waiting for execution  Messages that are not acknowledged are held on to by the worker until it closes the connection to the broker  AMQP server   When that connection is closed  e g  because the worker was stopped  the tasks will be re-sent by the broker to the next available worker  or the same worker when it has been restarted   so to properly purge the queue of waiting tasks you have to stop all the workers  and then purge the tasks using celery control purge      So to purge the entire queue workers must be stopped

User · Answer

1  To properly purge the queue of waiting tasks you have to stop all the workers  http   celery readthedocs io en latest faq html i-ve-purged-messages-but-there-are-still-messages-left-in-the-queue      sudo rabbitmqctl stop   or  in case RabbitMQ message broker is managed by Supervisor      sudo supervisorctl stop all   2      and then purge the tasks from a specific queue     cd  lt source dir gt    celery amqp queue purge  lt queue name gt    3  Start RabbitMQ     sudo rabbitmqctl start   or  in case RabbitMQ is managed by Supervisor      sudo supervisorctl start all

User · Answer

For Celery 2 x and 3 x   When using worker with -Q parameter to define queues  for example  celery worker -Q queue1 queue2 queue3   then celery purge will not work  because you cannot pass the queue params to it  It will only delete the default queue  The solution is to start your workers with --purge parameter like this   celery worker -Q queue1 queue2 queue3 --purge   This will however run the worker   Other option is to use the amqp subcommand of celery  celery amqp queue delete queue1 celery amqp queue delete queue2 celery amqp queue delete queue3

User · Answer

For celery 3 0      celery purge   To purge a specific queue     celery -Q queue name purge

User · Answer

If you want to remove all pending tasks and also the active and reserved ones to completely stop Celery  this is what worked for me   from proj celery import app from celery task control import inspect  revoke    remove pending tasks app control purge      remove active tasks i   inspect   jobs   i active   for hostname in jobs      tasks   jobs hostname      for task in tasks          revoke task  id    terminate True     remove reserved tasks jobs   i reserved   for hostname in jobs      tasks   jobs hostname      for task in tasks          revoke task  id    terminate True

User · Answer

celery 4   celery purge command to purge all configured task queues  celery -A  APPNAME  purge   programmatically   from proj celery import app app control purge     all pending task will be purged  Reference  celerydoc

[task] Deleting all pending tasks in celery / rabbitmq

Examples related to task

Examples related to rabbitmq

Examples related to celery

Examples related to celery-task