Foundational concepts of observability
1. Foundational concepts of observability
It's one thing to be able to observe a failure, but it's another to be able to pinpoint the exact root cause behind that failure. This is why understanding the foundational concepts of observability is critical. Observability is often made up of three core pillars, logs, traces, and metrics. These are important to understand before diving into implementing observability. Let's understand each one in a little more detail. Logs are immutable, time-stamped records of events that have occurred. They capture information about processes that have started, completed, or failed. You can think of them as receipts for individual events that occur in a system. Traces, on the other hand, are much more detailed. They represent the series of related events that occurred for actions taken within a system. They're often chained together in a time-series fashion. For example, a trace could represent the several events that occurred when moving data from, say, source to destination. It might include things like connection requests, connection acknowledgments, transfer initiations, and more. And each of these events would be connected to a timestamp. Traces don't necessarily represent a summarized final outcome, but rather represent the itemized journey of an activity. Because of this, traces are absolutely critical when debugging and pinpointing a root cause. And finally, metrics. Metrics are a little unique. They represent numerical data about a system's performance and health. Sometimes they might represent a single, raw measurement. Other times they might represent an aggregation of measurements. They can be used to represent things like total number of requests, CPU, GPU, and memory usage, latency distribution, and more. The combination of these three aspects is what helps data engineers use observability to troubleshoot and fix issues in a pipeline. This is because metrics provide great information for identifying system-wide issues. Traces provide support in identifying which step in a chain is causing the issue. And logs provide support for detailed debugging and root cause analyses. So if something fails, you probably don't just want a failure output message. For effective observability, you want a detailed trail of what's occurred so that you can understand exactly what failed to execute, along with the conditions that were present in the system when that thing failed. Logs, traces, and metrics help you achieve this for data pipelines. And when used effectively, you'll be able to easily tell whether a failure is related to transformation logic, data quality, compute resources, or anything else you're keeping track of. And you'd be able to do this without diving into a line-by-line audit of your code. So join me in the next video to understand how Snowflake supports observability.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.