SNS and SQS integration with CloudWatch

1. SNS and SQS integration with CloudWatch

Welcome back! Last video, we set up alarms that detect problems. But an alarm that fires into the void is useless, someone needs to be notified. In this video, we'll wire up SNS and SQS so that when something breaks, the right people hear about it fast.

2. Connecting CloudWatch alarms to SNS

Here's the connection: set --alarm-actions to an SNS topic ARN, an Amazon Resource Name that uniquely identifies any AWS resource, and every subscriber is notified when the alarm fires. Add --ok-actions for recovery and --insufficient-data-actions for gaps. One alarm can target multiple topics. To make this work, we need SNS topics.

3. Core SNS Concepts

SNS is publisher/subscriber: a topic is a named channel where publishers send and subscribers receive. Publish once, and it's delivered simultaneously to every subscriber, across email, SMS, HTTP, Lambda, SQS, mobile push, and Kinesis Firehose. Before building a topic, choose between Standard and FIFO.

4. SNS topic types

Standard: best-effort ordering, at-least-once, near-unlimited throughput. FIFO: strict ordering, exactly-once, up to 3,000 per second. Standard is right for CloudWatch alarms; reserve FIFO for workflows that require ordering and deduplication. Once you know the type, creating the topic is straightforward.

5. Creating SNS topics: AWS CLI

Creating a topic is one CLI command. For FIFO, append .fifo and set FifoTopic. Optionally add KMS (Key Management Service) encryption for sensitive payloads. The response returns the topic ARN. Now add subscribers.

6. Adding Lambda and SMS subscribers

Use Lambda to receive the alarm payload and act: post to Slack, create tickets, or trigger remediation. For SMS, specify the protocol and phone number. Now let's see the message a CloudWatch Alarm sends through this topic.

7. SNS message format from CloudWatch alarms

Alarm messages are JSON. Key fields: AlarmName, old and new state values, the human-readable reason, timestamp, and trigger configuration, metric, namespace, statistic, threshold, and dimensions. You'll parse this in Lambda or webhook handlers for notifications and remediation. In a busy system, you may not want every subscriber getting every message.

8. SNS message filtering

Filter policies control which messages subscribers receive. Define a JSON policy specifying fields that must match. Critical: CloudWatch Alarm data is in the message body, set FilterPolicyScope to MessageBody or filters won't match. For high-volume systems, raw JSON isn't ideal.

9. Custom notification formatting with Lambda

Raw JSON isn't great for a 2AM page. Chain a Lambda between the raw alarm topic and formatted-alerts to build readable messages, while machine consumers stay on the raw topic. What if multiple systems must process the same alarm independently?

10. Fan out architecture with SQS

Fan-out delivers one SNS message to multiple SQS queues, here logging, ticketing, and metrics, plus a Lambda. Each consumer is independent, giving parallel processing, decoupling, and SQS persistence. Setting it up takes four steps.

11. Setting Up Fan-Out

Create one SQS queue per consumer, add a queue policy granting the SNS service principal sqs:SendMessage, subscribe each queue, and build your consumers. Scope the policy to the specific topic ARN. Once subscribed, consumers can process the messages.

12. Subscribing Queues

Subscribing a queue is the same subscribe command as earlier, with protocol set to SQS and the queue ARN as the endpoint. Next, consumers poll and process those messages.

13. Processing Messages

To process messages, poll with receive_message, parse the SNS envelope to extract the alarm JSON, process it, then delete it. Long polling minimizes empty responses between messages, saving cost. A good production setup adds filters and dead letter queues.

14. Fan-Out with filtering and dead letter queues

Per-subscription filtering means only critical messages create tickets, while logging gets everything. The dead letter queue is like the post office's undeliverable mail bin: messages are set aside after three bounces. Monitor DLQ depth, growth means your pipeline is broken. With all that in place, how do you decide what to use?

15. SNS vs. SQS: When to Use Each

SNS is push-based and one-to-many; it retries delivery and can route exhausted deliveries to a dead-letter queue. SQS is pull-based, like a voicemail box, messages wait until you're ready, with retention and resiliency. Let's map these patterns to real-world scenarios.

16. Architecture examples

Four patterns cover most of what you'll build. Top-left, a simple alert: SNS pushes straight to email. Top-right, async processing: an API drops work onto an SQS queue, and a worker consumes it at its own pace so slow tasks never block the caller. Bottom-left, multi-channel alerting: one alarm fans out to email, chat, and a pager. Bottom-right, an event pipeline: one event to multiple SQS queues, each with its own processor. Match each to your workload.

17. Video summary

To recap: SNS topics deliver alarm notifications across email, SMS, HTTP, Lambda, and SQS. Fan-out sends one message to multiple SQS queues for parallel, reliable processing. Filtering reduces noise per subscriber, and dead letter queues catch failed messages. So use SNS for push, SQS for reliable async processing, and both for fan-out with reliability. Your team is notified, now they need the root cause. Next, Log Insights.

18. Let's practice!

Now let's check what we have learned in this video.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.