Logging and metrics
1. Logging and metrics
MIKE DUNKER: Cloud Logging is one of the tools that can be most valuable for app developers. Logs and metrics will help you understand how your application is working. A robust system of logging is crucial for developer productivity and to help you understand the state of your application. Cloud Logging is a real-time log management system with storage, search, analysis, and monitoring support. Cloud Logging automatically collects logs from Google Cloud resources. You can also collect logs from your applications. Logging is an important tool for developers. Logs can help you troubleshoot your application. Using the Logs Explorer, you can view individual log entries and find related log entries. Logging details about the flow of calls within your application helps you understand how your application is working or why it's not working as intended. Logs can also be used to analyze application performance. For example, you can use the Log Analytics interface to perform aggregate operations on your logs. With this interface, you use SQL to query your log data and help you understand performance. You can configure Cloud Logging to notify you when certain kinds of events occur in your logs. You can create a log-based alert to send a notification when a particular pattern appears in a log entry. Alternatively, you might want to monitor trends or the occurrence of events over time. For these situations, you can create a log-based metric. Log-based metrics are suitable whenever you want to count the occurrences of a message and be notified when the number of occurrences crosses a threshold. Metrics can also be used to observe trends in your data and receive a notification if the values change in an unacceptable way. You can install the Ops Agent on Compute Engine instances to stream logs and metrics from third-party applications into Cloud Logging. The Ops Agent uses Fluent Bit for high-throughput logging and the OpenTelemetry Collector for metrics. Ops Agent automatically collects logs from standard system log locations, like /var, /log, /syslog. File-based logs can also be configured with customizable paths and refresh interval. Ops Agent also supports flexible processing of log data. You can configure Ops Agent to parse text logs into structured logs, modify log entries by removing renaming or setting fields, or exclude logs based on labels and regular expressions. Ops Agent also supports collection of standard system metrics without any configuration. Collected system metrics include CPU, disk, memory, network, and processes. Curated third-party application metrics can also be collected. Apache Tomcat, Apache web server, and NGINX are examples of third-party applications that are supported by Ops Agent. Cloud Logging is preconfigured in other compute environments. Cloud Run services and functions have built in support for logging. Logs written by applications are automatically sent to Cloud Logging. You can also enable logging on Google Kubernetes Engine by enabling the observability for GKE integration for your cluster. Kubernetes logs are not stored permanently within GKE. For example, GKE container logs are removed when their host pod is removed. System logs are periodically removed to free up space for new logs. And cluster events are removed after one hour. Sending logs to Cloud Logging ensures that log entries are kept as long as you deem necessary. For Cloud Run services and functions, you can simply write to standard out and standard error, and the logs will be automatically delivered to Cloud Logging. Text strings are put into the text payload field of the log entry. Other fields, like the time the log was received, the resource that produced the log entry, and the log name, will also be logged with the text message. It's recommended to use structured logs instead of text logs. Text logs do not have a log level, so it can be hard to find the really important content inside the text logs Fields within a structured log data can be searched using queries, which makes log analysis much easier. For a structured log entry, you log a single line of serialized JSON, which is placed in the JSON payload field of the log entry. When you use structured JSON data, some special fields are stripped from the JSON payload and written to the corresponding field in the generated log entry. For example, you can set the log level by specifying a severity field. The message property will be used as the main display text of the log when it's provided. In this example, the first line would create a text log entry. The rest of the code shows an example of how to create a structured log entry. The structured entry uses special fields to create labels, the log severity, and a component field within the log entry. Another tool you might want to consider for logging and alerting is Prometheus. Prometheus is an open source systems monitoring and alerting toolkit. It can monitor services running on VMs and Kubernetes. It's a very popular solution for monitoring Kubernetes workloads and clusters and alerting when the workloads and clusters are not healthy. Prometheus collects and stores metrics as time series data. This time series data can help you see trends in your application metrics. Prometheus also provides a powerful query language, PromQL, which can be used to create dashboards and alerts from your metrics. Prometheus provides many benefits, but it can be difficult to manage and scale Prometheus infrastructure. One solution to this problem is to use Google Cloud Managed Service for Prometheus. Google Cloud Managed Service for Prometheus is a fully-managed Prometheus solution that removes the burden of manually managing and operating Prometheus at scale. The solution is also multi-cloud and cross-project. All of your applications, whether running in Google Cloud, other clouds, or on premises, can be managed in a single pane of glass using Cloud Monitoring. There are many data collectors available. Managed collectors are recommended for all Kubernetes environments, including GKE. For managed collectors, operation of Prometheus is fully handled by the Kubernetes operator. Self-deployed collectors are drop-in replacements for the normal Prometheus binary. OpenTelemetry collectors and Ops Agent can also both collect metrics that can be used with Prometheus. Managed Service for Prometheus supports any query UI that can call the PromQL query API, including Cloud Monitoring and Grafana. Over 1,500 free metrics available in Cloud Monitoring can also be queried, even without sending data to Managed Service for Prometheus. Managed Service for Prometheus also provides a fully cloud-based alerting pipeline for your Cloud Monitoring and Prometheus metrics.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.