Metrics Explorer
1. Metrics Explorer
person: We've explored the DataFlow UI and its monitoring capabilities. Let's look at how Cloud Monitoring Metrics Explorer can be used with DataFlow. Cloud Monitoring Metrics Explorer is a sink into which all the DataFlow metrics we've seen are exported. You can explore DataFlow metrics using the Metrics Explorer and build custom dashboards to view them. The available metrics range from plotting the job status to custom metrics that you create. Here is an example of a custom dashboard using Cloud Monitoring. This dashboard shows the data watermark leg across all pipelines that start with a specific name. While other metrics can be used, it depends on what you want to measure. Some metrics that can be used are shown on this page. For example, if you want to see if your job failed, set is_failed to greater than 0 and filter by job name. If you want to see if a dependency is failing, set up a counter to measure the number of times the dependency is called and plot the results using the user counter metric. You may have noticed that most of the graphs we looked at had a Create Alerting Policy button. Cloud Monitoring gives you the ability to create alerts and be notified when a certain metric crosses a specific threshold. Where can this be useful? In streaming pipelines, if an element fails to get processed, it is retried indefinitely. Streaming pipelines have no concept of failure unless you specifically cancel or drain a job. You can catch indefinite retries by setting an alert if system latency increases over a predefined value. Every time an alert is triggered, an incident and a corresponding event are created. If you specified a notification mechanism in the alert, such as an email or SMS, you will also receive a notification. The alerting policies provided are on a per-pipeline basis, but you can build your own custom alerting policy grouping more than one pipeline using Cloud Monitoring. This is the end of this module. You should now be able to: navigate the DataFlow job details UI, interpret job metric graphs to diagnose pipeline regressions, and set up alerts on DataFlow jobs using Cloud Monitoring.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.