Logging and metrics with Amazon CloudWatch

1. Logging and metrics with Amazon CloudWatch

Welcome back! In this video, you'll learn how Amazon CloudWatch gives you visibility through logs, metrics and alarms, let's get started.

2. Is it down?

A customer says your app is broken, but it looks fine to you. Is it slow for everyone or just them? Since when? Without logs, metrics, and alarms you're guessing, and guessing during an incident is expensive. CloudWatch turns those questions into answers.

3. Logging vs monitoring vs observability

Three words get used interchangeably, but they answer different questions. Logging records discrete events: a request arrived, an error was thrown. Monitoring watches known metrics against thresholds and alerts when one crosses a line, like error rate above one percent. Observability is broader: whether you can ask brand-new questions of your data after the fact, without shipping new code. A well-instrumented app gives you all three, so you catch expected failures and investigate the ones you didn't.

4. Log groups, log streams, and retention

CloudWatch Logs organizes everything into two levels. A log group is a named container for related logs, usually one per application or function. Inside it, a log stream is a sequence of events from a single source, like one instance of a Lambda function, code that runs without managing servers. Picture a filing cabinet: the log group is the drawer for one application, and each log stream is a single folder inside it. By default logs are kept forever, which quietly grows your bill, so you set a retention period that matches how long the data is useful, maybe thirty days for an app, longer for audit logs. Storing everything forever is rarely worth the cost.

5. Metrics and custom metrics

Metrics are numeric measurements tracked over time, like invocation count or duration. AWS publishes a large set automatically for Lambda and API Gateway, so you get basic monitoring free. Custom metrics are numbers your application emits, like orders processed. Dimensions slice a metric by attribute, each one a label you attach to filter the metric, like error count by region or customer tier. Automatic and custom metrics, sliced by useful dimensions, turn raw activity into something you can chart and alarm on.

6. The embedded metric format

There's a clever way to publish custom metrics without a separate API call: the embedded metric format, or EMF. You write a structured JSON log line that includes both your metric values and a small metadata block telling CloudWatch which fields are metrics. CloudWatch reads that line and automatically extracts the metrics, so a single write gives you both a detailed log entry and a queryable metric. This is especially powerful for high-cardinality data, values with many distinct possibilities, like a per-customer ID, where you want the rich log for investigation and the metric for alarming.

7. Dashboards and alarms

Once metrics exist, you make them actionable. A dashboard gathers the charts your team cares about, so on-call has one place to look. An alarm fires when a metric crosses a threshold for a set period, like error rate over one percent for five minutes, and routes through SNS to email, SMS, or automation. The discipline: alarm on symptoms users actually feel, like latency or errors, not on every metric, or you'll drown in noise.

8. A structured logging strategy

How you log matters as much as that you log. Free-text lines are hard to search and impossible to aggregate. Structured logging means emitting JSON, where each field has a name: a timestamp, a level like ERROR, a request ID, and a user ID. Because the fields are named, you can filter and query them precisely, like "every ERROR for this user ID in the last hour." When every service logs the same core fields, you can correlate a single request as it moves across them. Structure is what makes logs genuinely useful.

9. Let's practice!

Let's now go make this application observable.

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.