CloudWatch Logs

1. CloudWatch Logs

Welcome back! Imagine you've deployed a Lambda function that fails intermittently and you're unsure why. The answer is buried in your logs, if you know where to look. Over the next four videos, we'll capture logs, detect problems with alarms, notify your team, and investigate root causes.

2. Core concepts: log events, streams, and groups

CloudWatch Logs has three core concepts, and a filing system makes them click. A log event is a single page: one record with its timestamp, message, and ingestion time. A log stream is a folder: a sequence of events from one source, like an instance, invocation, or container. A log group is the filing cabinet: related streams sharing retention, encryption, and access settings. You'll recognize groups by their naming patterns: /aws/lambda/ for Lambda, /ecs/ for containers, and /var/log/ for application logs. Now let's see how logs get from your applications into CloudWatch.

3. CloudWatch Logs architecture

Your application sends logs via an agent or SDK to CloudWatch Logs, which organizes them into groups and streams. From there you can query with Logs Insights, create metric filters, set up real-time subscriptions, or export to S3. This moves you beyond collating and analyzing logs manually or with custom scripts.

4. CloudWatch logs vs. traditional logging

Centralized logging replaces connecting to each server: one searchable location, automated retention, and almost unlimited scale. The trade-off is cost, you pay per GB ingested, so be intentional about what you send. So how do logs get into CloudWatch? For EC2, you install and configure the Unified Agent.

5. Unified agent setup: EC2

Setting up the agent on EC2 takes four steps. First, create an IAM (Identity and Access Management) role with CloudWatchAgentServerPolicy and attach it to your instance. Then install the agent. Next, write a JSON config specifying which files to watch, the log group, and each stream's name. Using {instance_id} gives every instance its own stream automatically. Finally, start the agent. Lambda and ECS, the Elastic Container Service for running containers, are simpler: they need no agent at all.

6. Lambda and ECS logging

Lambda needs no agent: it logs to /aws/lambda/function-name automatically, capturing invocations, duration, stdout, stderr, and errors. Use structured JSON for easier querying. For ECS, configure the awslogs log driver with the log group, region, and stream prefix. Everything written to stdout and stderr lands in CloudWatch automatically. Once logs are flowing, we can do more than store them. Metric filters extract value from log data.

7. Introduction to metric filters

Metric filters scan log events for patterns and emit CloudWatch metrics when they match, bridging logs and metrics. You create them on a log group. Count errors, track response times, or monitor business events like orders and signups, then feed alarms from those patterns like any other metric. Writing good metric filters means understanding the pattern syntax.

8. Filter pattern syntax

There are three filter pattern formats: simple text matching, space-delimited positional fields, and JSON using dollar-dot notation with comparisons and boolean operators. JSON tends to be the most reliable for structured logs.

9. Metric filter recommended practices

A few recommended practices. Always test patterns in the console before creating them. Set defaultValue=0 on counters so the metric doesn't disappear during quiet periods. Group namespaces hierarchically, and keep patterns simple, complexity adds up fast. Now let me introduce the other tool for getting value from log data.

10. Introduction to CloudWatch Logs Insights

Finally, CloudWatch Logs Insights lets you interactively query your log data across one or more log groups using three supported languages: Logs Insights Query Language, OpenSearch Piped Processing Language, and OpenSearch SQL. We'll master these in depth in the final video of this chapter.

11. Video summary

To recap: CloudWatch Logs organizes data into events, streams, and groups. EC2 uses the Unified Agent, Lambda logs automatically, ECS uses the awslogs driver. Metric filters bridge logs and metrics, and three query languages cover everything from simple searches to advanced analysis. You've learned to capture logs and create metrics from them. But what happens when those metrics spike at 3 AM? That's where alarms come in.

12. Let's practice!

Let's take a closer look at CloudWatch Logs.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.