Distributed tracing with AWS X-Ray and structured logging

1. Distributed tracing with AWS X-Ray and structured logging

Welcome back. Logs and metrics tell you that something is slow; tracing tells you where. In this video, you'll learn how AWS X-Ray follows a single request across services. We'll break down segments and the service map, separate annotations from metadata, correlate structured logs with trace IDs, and read a service map to find the slowest call. Let's get started.

2. The slow request nobody could find

A request takes four seconds, and it shouldn't. It crosses API Gateway, a Lambda function, code that runs without managing servers, a database, and a third-party call. Each team checks its own logs and says everything looks normal. Nobody sees the whole journey, so the slow hop hides. Distributed tracing stitches it back together.

3. What distributed tracing solves

One request might pass through API Gateway, a Lambda, a database, and a third-party call. When it's slow, each service's logs in isolation can't tell you where the time went, because no single log sees the whole journey. Distributed tracing stitches the request into one trace across every service it touches. Instead of guessing, you look at one trace and see which hop consumed the time, the difference between hours of correlation and a glance.

4. Segments, subsegments, and the service map

X-Ray builds a trace from segments. A segment is the work one service does for the request, like the Lambda execution. Inside it, subsegments capture finer units, like a database query or HTTP call. X-Ray assembles segments into a service map. On the right, each circle is one service and the arrows are the calls between them; the node glowing red is the one carrying the latency, so your eye goes straight to the suspect. That's usually where you start hunting a performance problem.

5. Annotations vs metadata

You can attach your own context to a trace, and the choice between two types matters. Annotations are indexed key-value pairs, meaning X-Ray keeps a lookup table so you can filter and search traces by them. You annotate with things you'll want to slice by later, like customer tier or feature flag. Metadata is extra detail you attach for context but cannot search on, because it isn't indexed, like a full request payload. The rule of thumb: if you'll ever want to find traces by it, make it an annotation; otherwise make it metadata.

6. Correlating logs with trace IDs

Traces and logs are far more powerful together than apart. Every X-Ray trace has a unique trace ID, and the trick is to include that ID as a field in your structured JSON logs. Now when a trace shows a slow or failing request, you can take its trace ID and pull up exactly the log lines for that request, across every service. In reverse, a suspicious log entry points you straight to its full trace. This correlation turns two separate tools into a single investigation.

7. Reading the service map to find the bottleneck

Here's the workflow when something is slow. You start at the service map, not the logs, because it shows at a glance which node carries the latency. You find the node with the highest latency or error rate, then drill into a representative trace to see its timeline, where each segment is a bar sized by duration. The longest bar is your suspect, often a downstream call you don't control. This top-down approach finds bottlenecks in minutes instead of guessing across scattered logs.

8. Tracing across the full request

Putting it together, X-Ray traces a request from API Gateway, through the Lambda, and into each database query or external call. Every service contributes a segment to the same trace, sharing one trace ID. Because your structured logs carry that ID too, one identifier ties the whole picture together: the path, the timing of each hop, and the logs. That end-to-end visibility is what lets you operate a distributed application with confidence.

9. Let's practice!

Time to follow a request end to end. Let's practice!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.