Inspecting logs with the kubectl command
1. Inspecting logs with the kubectl command
Let’s explore how to inspect Kubernetes logs with the kubectl command. Regarding Kubernetes-native logging, logs from Kubernetes system components, such as kubelet and kube-proxy, are stored in the /var/log directory of each node’s file system. Messages written by each container to its standard output and standard error are also logged in the /var/log directory. This allows log messages to be captured and centrally managed. Let’s explore some common kubectl commands that can be used to produce GKE logs. To view the logs of a specific Pod, run the kubectl logs command, followed by the name of the Pod. If you don't know the exact name of the Pod, or if the Pod is part of a Deployment, you can run the kubectl get pods command to get a list of all the available Pods. When troubleshooting an issue, you might only need to inspect the most recent log from a specific Pod or container. To limit the number of logs returned, add the --tail option to the kubectl logs command. By adding –tail=20, the output would be limited to the 20 most recent lines. But there might be instances when you don’t actually know how many log messages to retrieve, but you do know the problem occurred in the last few hours. To restrict the output based on time, you can add the --since option, followed by the time period. For example, --since=3h would return all logs for a specific Pod or container from the last 3 hours. Now let’s say you need to view a previous instantiation of a container before it crashed. This can be accomplished by adding the --previous option. If your Pod has multiple containers, you’ll need to specify which container’s log the kubectl command should return. These native Kubernetes logs, and the compressed archives aren’t persistent. If, for any reason, a Pod is evicted or restarted, logs and the archives associated with that Pod are lost. As a result, there’s a need to store logs outside a container, Pod, or node. This is called cluster-level logging. As mentioned earlier, Kubernetes itself doesn’t offer any log storage solution, but it does support various implementations. In GKE this is handled for you by the integration with Cloud Logging. When a container starts on a node, a log file is created. Node-level logging implements log archiving using a rotation mechanism. As the container runs, events happen and the log file grows. Either once per day, or when the log file reaches 100 MB (whichever comes first) the logrotate utility creates a new log and compresses the old log file, saving it into an archive. It then deletes all but the five most recent compressed log archives. This ensures that logs don’t consume all of the available storage on the node. If a container restarts, the default behavior of kubelet is that it keeps one terminated container with its logs. However, all log events are streamed to Cloud Logging, which retains all log event data in the _default bucket for 30 days by default. If your organization requires you to retain log data for longer than 30 days, configure Cloud Logging to export the data to long-term storage such as BigQuery or Cloud Storage.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.