RabbitMQ AMQP single queue multiple consumers for same message

Question

I am just starting to use RabbitMQ and AMQP in general    I have a queue of messages I have multiple consumers  which I would like to do different things with the same message     Most of the RabbitMQ documentation seems to be focused on round-robin  ie where a single message is consumed by a single consumer  with the load being spread between each consumer  This is indeed the behavior I witness    An example  the producer has a single queue  and send messages every 2 sec   var amqp   require  amqp    var connection   amqp createConnection   host   localhost   port  5672     var count   1   connection on  ready   function        var sendMessage   function connection  queue name  payload        var encoded payload   JSON stringify payload         connection publish queue name  encoded payload          setInterval  function             var test message    TEST   count     sendMessage connection   my queue name   test message        count    1       2000          And here s a consumer   var amqp   require  amqp    var connection   amqp createConnection   host   localhost   port  5672     connection on  ready   function        connection queue  my queue name   function queue       queue bind            queue subscribe function  message          var encoded payload   unescape message data        var payload   JSON parse encoded payload        console log  Recieved a message          console log payload                   If I start the consumer twice  I can see that each consumer is consuming alternate messages in round-robin behavior  Eg  I ll see messages 1  3  5 in one terminal  2  4  6 in the other   My question is     Can I have each consumer receive the same messages  Ie  both consumers get message 1  2  3  4  5  6  What is this called in AMQP RabbitMQ speak  How is it normally configured  Is this commonly done  Should I just have the exchange route the message into two separate queues  with a single consumer  instead

User · Answer

There is one interesting option in this scenario I haven`t found in answers here.

You can Nack messages with "requeue" feature in one consumer to process them in another. Generally speaking it is not a right way, but maybe it will be good enough for someone.

https://www.rabbitmq.com/nack.html

And beware of loops (when all concumers nack+requeue message)!

User · Answer

I think you should check sending your messages using the fan-out exchanger  That way you willl receiving the same message for differents consumers  under the table RabbitMQ is creating differents queues for each one of this new consumers subscribers   This is the link for see the tutorial example in javascript https   www rabbitmq com tutorials tutorial-one-javascript html

User · Answer

Can I have each consumer receive the same messages  Ie  both consumers get message 1  2  3  4  5  6  What is this called in AMQP RabbitMQ speak  How is it normally configured   No  not if the consumers are on the same queue  From RabbitMQ s AMQP Concepts guide      it is important to understand that  in AMQP 0-9-1  messages are load balanced between consumers    This seems to imply that round-robin behavior within a queue is a given  and not configurable  Ie  separate queues are required in order to have the same message ID be handled by multiple consumers   Is this commonly done  Should I just have the exchange route the message into two separate queues  with a single consumer  instead   No it s not  single queue multiple consumers with each each consumer handling the same message ID isn t possible  Having the exchange route the message onto into two separate queues is indeed better   As I don t require too complex routing  a fanout exchange will handle this nicely  I didn t focus too much on Exchanges earlier as node-amqp has the concept of a  default exchange  allowing you to publish messages to a connection directly  however most AMQP messages are published to a specific exchange    Here s my fanout exchange  both sending and receiving    var amqp   require  amqp    var connection   amqp createConnection   host   localhost   port  5672     var count   1   connection on  ready   function        connection exchange  my exchange   options  type  fanout    function exchange            var sendMessage   function exchange  payload          console log  about to publish         var encoded payload   JSON stringify payload         exchange publish     encoded payload                    Recieve messages     connection queue  my queue name   function queue         console log  Created queue         queue bind exchange              queue subscribe function  message            console log  subscribed to queue           var encoded payload   unescape message data          var payload   JSON parse encoded payload          console log  Recieved a message            console log payload                       setInterval  function               var test message    TEST   count       sendMessage exchange  test message          count    1         2000

User · Answer

Just read the rabbitmq tutorial  You publish message to exchange  not to queue  it is then routed to appropriate queues  In your case  you should bind separate queue for each consumer  That way  they can consume messages completely independently

User · Answer

Fan out was clearly what you wanted  fanout read rabbitMQ tutorial  https   www rabbitmq com tutorials tutorial-three-javascript html here s my example  Publisher js  amqp connect  amqp    lt user gt   lt pass gt   lt host gt   lt port gt    async  error0  connection    gt        if  error0          throw error0            console log  RabbitMQ connected       try            Create exchange for queues       channel   await connection createChannel         await channel assertExchange process env EXCHANGE NAME   fanout     durable  false           await channel publish process env EXCHANGE NAME      Buffer from  msg          catch error          console error error            Subscriber js  amqp connect  amqp    lt user gt   lt pass gt   lt host gt   lt port gt    async  error0  connection    gt        if  error0          throw error0            console log  RabbitMQ connected       try            Create Bind a consumer queue for an exchange broker       channel   await connection createChannel         await channel assertExchange process env EXCHANGE NAME   fanout     durable  false           const queue   await channel assertQueue      exclusive  true         channel bindQueue queue queue  process env EXCHANGE NAME             console log  quot      Waiting for messages in  s  To exit press CTRL C quot          channel consume     consumeMessage   noAck  true          catch error          console error error             here is an example i found in the internet  maybe can also help  https   www codota com code javascript functions amqplib Channel assertExchange

User · Answer

Yes each consumer can receive the same messages   have a look at  http   www rabbitmq com tutorials tutorial-three-python html http   www rabbitmq com tutorials tutorial-four-python html http   www rabbitmq com tutorials tutorial-five-python html  for different ways to route messages   I know they are for python and java but its good to understand the principles  decide what you are doing and then find how to do it in JS   Its sounds like you want to do a simple fanout  tutorial 3   which sends the messages to all queues connected to the exchange   The difference with what you are doing and what you want to do is basically that you are going to set up and exchange or type fanout   Fanout excahnges send all messages to all connected queues   Each queue will have a consumer that will have access to all the messages separately   Yes this is commonly done  it is one of the features of AMPQ

User · Answer

The last couple of answers are almost correct - I have tons of apps that generate messages that need to end up with different consumers so the process is very simple   If you want multiple consumers to the same message  do the following procedure   Create multiple queues  one for each app that is to receive the message  in each queue properties   bind  a routing tag with the amq direct exchange  Change you publishing app to send to amq direct and use the routing-tag  not a queue   AMQP will then copy the message into each queue with the same binding  Works like a charm     Example    Lets say I have a JSON string I generate  I publish it to the  amq direct  exchange using the routing tag  new-sales-order   I have a queue for my order printer app that prints order  I have a queue for my billing system that will send a copy of the order and invoice the client and I have a web archive system where I archive orders for historic compliance reasons and I have a client web interface where orders are tracked as other info comes in about an order   So my queues are  order printer  order billing  order archive and order tracking All have the binding tag  new-sales-order  bound to them  all 4 will get the JSON data   This is an ideal way to send data without the publishing app knowing or caring about the receiving apps

User · Answer

RabbitMQ   AMQP  single queue  multiple consumers for same message and page refresh   rabbit on  ready   function                 sockjs chat on  connection   function  conn             conn on  data   function  message                try                   var obj   JSON parse message replace   r g      replace   n g                         if  obj header     register                             Connect to RabbitMQ                     try                           conn exchange   rabbit exchange exchange    type   topic                               autoDelete  false                              durable  false                              exclusive  false                              confirm  true                                                      conn q   rabbit queue  my-queue-  obj agentID                                durable  false                              autoDelete  false                              exclusive  false                            function                                  conn channel    my-queue-  obj agentID                              conn q bind conn exchange  conn channel                                conn q subscribe function  message                                    console log   MSG  --- gt      JSON stringify message                                    conn write JSON stringify message      n                                   addCallback function ok                                    ctag conn channel    ok consumerTag                                                        catch  err                            console log  Could not create connection to RabbitMQ   nStack trace -- gt     err stack                                            else if  obj header     typing                          var reply                             type   chatMsg                           msg  utils escp obj msga                           visitorNick  obj channel                          customField1                              time  utils getDateTime                            channel  obj channel                                             conn exchange publish  my-queue-  obj agentID  reply                                    catch  err                    console log  ERROR ---- gt      err stack                                         When the visitor closes or reloads a page we need to unbind from RabbitMQ          conn on  close   function                  try                       Close the socket                 conn close                        Close RabbitMQ                           conn q unsubscribe ctag conn channel                   catch  er                    console log           EXCEPTION SOCKJS  ON-CLOSE           gt  gt  gt  gt  gt  gt  gt      er stack

User · Answer

As I assess your case is    I have a queue of messages  your source for receiving messages  lets name it q111  I have multiple consumers  which I would like to do different things with the same message    Your problem here is while 3 messages are received by this queue  message 1 is consumed by a consumer A  other consumers B and C consumes message 2 and 3  Where as you are in need of a setup where rabbitmq passes on the same copies of all these three messages 1 2 3  to all three connected consumers  A B C  simultaneously    While many configurations can be made to achieve this  a simple way is to use the following two step concept    Use a dynamic rabbitmq-shovel to pickup messages from the desired queue q111  and publish to a  fanout exchange  exchange exclusively created and dedicated  for this purpose   Now re-configure your consumers A B  amp  C  who were listening to queue q111   to listen from this Fanout exchange directly using a exclusive  amp  anonymous queue for each consumer    Note  While using this concept don t consume directly from the source queue q111   as messages already consumed wont be shovelled to your Fanout exchange   If you think this does not satisfies your exact requirement    feel free to post your suggestions   -

User · Answer

To get the behavior you want  simply have each consumer consume from its own queue   You ll have to use a non-direct exchange type  topic  header  fanout  in order to get the message to all of the queues at once

User · Answer

If you happen to be using the amqplib library as I am  they have a handy example of an implementation of the Publish Subscribe RabbitMQ tutorial which you might find handy

User · Answer

The send pattern is a one-to-one relationship   If you want to  send  to more than one receiver you should be using the pub sub pattern   See http   www rabbitmq com tutorials tutorial-three-python html for more details

[node.js] RabbitMQ / AMQP: single queue, multiple consumers for same message?

Examples related to node.js

Examples related to messaging

Examples related to rabbitmq

Examples related to amqp

Examples related to node-amqp