What is the difference between Amazon SNS and Amazon SQS

Question

When would I use SNS versus SQS  and why are they always coupled together

User · Answer

AWS SNS is a publisher subscriber network  where subscribers can subscribe to topics and will receive messages whenever a publisher publishes to that topic  AWS SQS is a queue service  which stores messages in a queue  SQS cannot deliver any messages  where an external service  lambda  EC2  etc   is needed to poll SQS and grab messages from SQS  SNS and SQS can be used together for multiple reasons   There may be different kinds of subscribers where some need the immediate delivery of messages  where some would require the message to persist  for later usage via polling  See this link   The  quot Fanout Pattern  quot  This is for the asynchronous processing of messages  When a message is published to SNS  it can distribute it to multiple SQS queues in parallel  This can be great when loading thumbnails in an application in parallel  when images are being published  See this link   Persistent storage  When a service that is going to process a message is not reliable  In a case like this  if SNS pushes a notification to a Service  and that service is unavailable  then the notification will be lost  Therefore we can use SQS as a persistent storage and then process it afterwards

User · Answer

SNS is a distributed publish-subscribe system  Messages are pushed to subscribers as and when they are sent by publishers to SNS  SQS is distributed queuing system  Messages are not pushed to receivers  Receivers have to poll or pull messages from SQS  Messages can t be received by multiple receivers at the same time  Any one receiver can receive a message  process and delete it  Other receivers do not receive the same message later  Polling inherently introduces some latency in message delivery in SQS unlike SNS where messages are immediately pushed to subscribers  SNS supports several end points such as email  SMS  HTTP end point and SQS  If you want unknown number and type of subscribers to receive messages  you need SNS  You don t have to couple SNS and SQS always  You can have SNS send messages to email  SMS or HTTP end point apart from SQS  There are advantages to coupling SNS with SQS  You may not want an external service to make connections to your hosts  a firewall may block all incoming connections to your host from outside   Your end point may just die because of heavy volume of messages  Email and SMS maybe not your choice of processing messages quickly  By coupling SNS with SQS  you can receive messages at your pace  It allows clients to be offline  tolerant to network and host failures  You also achieve guaranteed delivery  If you configure SNS to send messages to an HTTP end point or email or SMS  several failures to send message may result in messages being dropped  SQS is mainly used to decouple applications or integrate applications  Messages can be stored in SQS for a short duration of time  maximum 14 days   SNS distributes several copies of messages to several subscribers  For example  let   s say you want to replicate data generated by an application to several storage systems  You could use SNS and send this data to multiple subscribers  each replicating the messages it receives to different storage systems  S3  hard disk on your host  database  etc

User · Answer

You can see SNS as a traditional topic which you can have multiple Subscribers  You can have heterogeneous subscribers for one given SNS topic  including Lambda and SQS  for example  You can also send SMS messages or even e-mails out of the box using SNS  One thing to consider in SNS is only one message  notification  is received at once  so you cannot take advantage from batching  SQS  on the other hand  is nothing but a queue  where you store messages and subscribe one consumer  yes  you can have N consumers to one SQS queue  but it would get messy very quickly and way harder to manage considering all consumers would need to read the message at least once  so one is better off with SNS combined with SQS for this use case  where SNS would push notifications to N SQS queues and every queue would have one subscriber  only  to process these messages  As of Jun 28  2018  AWS Supports Lambda Triggers for SQS  meaning you don t have to poll for messages any more  Furthermore  you can configure a DLQ on your source SQS queue to send messages to in case of failure  In case of success  messages are automatically deleted  this is another great improvement   so you don t have to worry about the already processed messages being read again in case you forgot to delete them manually   I suggest taking a look at Lambda Retry Behaviour to better understand how it works  One great benefit of using SQS is that it enables batch processing  Each batch can contain up to 10 messages  so if 100 messages arrive at once in your SQS queue  then 10 Lambda functions will spin up  considering the default auto-scaling behaviour for Lambda  and they ll process these 100 messages  keep in mind this is the happy path as in practice  a few more Lambda functions could spin up reading less than the 10 messages in the batch  but you get the idea   If you posted these same 100 messages to SNS  however  100 Lambda functions would spin up  unnecessarily increasing costs and using up your Lambda concurrency  However  if you are still running traditional servers  like EC2 instances   you will still need to poll for messages and manage them manually  You also have FIFO SQS queues  which guarantee the delivery order of the messages  SQS FIFO is also supported as an event source for Lambda as of November 2019 Even though there s some overlap in their use cases  both SQS and SNS have their own spotlight  Use SNS if   multiple subscribers is a requirement sending SMS E-mail out of the box is handy  Use SQS if   only one subscriber is needed batching is important

User · Answer

Here s a comparison of the two  Entity Type  SQS  Queue  Similar to JMS  SNS  Topic  Pub Sub system   Message consumption  SQS  Pull Mechanism - Consumers poll and pull messages from SQS SNS  Push Mechanism - SNS Pushes messages to consumers  Use Case  SQS  Decoupling two applications and allowing parallel asynchronous processing SNS  Fanout - Processing the same message in multiple ways  Persistence  SQS  Messages are persisted for some  configurable  duration if no consumer is available  maximum two weeks   so the consumer does not have to be up when messages are added to queue  SNS  No persistence  Whichever consumer is present at the time of message arrival gets the message and the message is deleted  If no consumers are available then the message is lost after a few retries   Consumer Type  SQS  All the consumers are typically identical and hence process the messages in the exact same way  each message is processed once by one consumer  though in rare cases messages may be resent  SNS  The consumers might process the messages in different ways  Sample applications  SQS  Jobs framework  The Jobs are submitted to SQS and the consumers at the other end can process the jobs asynchronously  If the job frequency increases  the number of consumers can simply be increased to achieve better throughput  SNS  Image processing  If someone uploads an image to S3 then watermark that image  create a thumbnail and also send a Thank You email  In that case S3 can publish notifications to an SNS topic with three consumers listening to it  The first one watermarks the image  the second one creates a thumbnail and the third one sends a Thank You email  All of them receive the same message  image URL  and do their processing in parallel

User · Answer

In simple terms   SNS - sends messages to the subscriber using push mechanism and no need of pull   SQS - it is a message queue service used by distributed applications to exchange messages through a polling model  and can be used to decouple sending and receiving components    A common pattern is to use SNS to publish messages to Amazon SQS queues to reliably send messages to one or many system components asynchronously  Reference from Amazon SNS FAQs

User · Answer

From the AWS documentation   Amazon SNS allows applications to send time-critical messages to multiple subscribers through a    push    mechanism  eliminating the need to periodically check or    poll    for updates  Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model  and can be used to decouple sending and receiving components   without requiring each component to be concurrently available   Fanout to Amazon SQS queues

[amazon-web-services] What is the difference between Amazon SNS and Amazon SQS?

Examples related to amazon-web-services

Examples related to amazon-sqs

Examples related to amazon-sns