Logging
1. Logging
Next, let’s talk about Logging. Now, you’ve already been exposed to Cloud Logging throughout this course. In this module, we’ll focus on VPC Flow Logs and exporting your logs to BigQuery and Data Studio so that you can analyze and visualize your logs. VPC Flow Logs records a sample of network flows sent from and received by VM instances, as you can see in this animation. These logs can be used for network monitoring, forensics, real-time security analysis, and expense optimization. Google Cloud is unique for its near-real-time visibility, providing log updates every 5 seconds. Also, there is no extra delay and no performance penalty in routing the logged IP packets to their destination. DNS provides a lookup for sites on the internet. You can think of it as a phone book, but instead of using the name of an organization to look up its phone number, you use the name of an organization to find an IP address. A DNS service is provided by your ISP (internet service provider). For example, suppose a request comes from a client computer to access cymbal.com. To direct the client computer to the cymbal.com site, the internet service provider needs the IP address of cymbal.com. The ISP connects to get this information from its DNS service. The DNS service recursive resolver issues a request to look up the IP address of cymbal.com from one of its name servers. The name server responds with the ISP. You can enable or disable VPC Flow Logs per VPC subnet. Once enabled for a subnet, VPC Flow Logs collect data from all VM instances in that subnet. Each log entry contains a record of different fields. For example, this table illustrates the IP connection information that is recorded. This consists of the source IP address and port, the destination IP address and port, and the protocol number. This set is commonly referred to as 5-tuple. Other fields include the start and end time of the first and last observed packet, the bytes and packets sent, instance details, VPC details, and geographic details. For more information on all data recorded by VPC Flow Logs, please refer to the documentation. VPC Flow Logs capture traffic from both ends of a VM-to-VM conversation within the same VPC network. To use this feature, ensure both the communicating VMs reside in subnets with VPC Flow Logs enabled. In this example, VM 10.10.0.2 sends a request with 1,224 bytes to VM 10.50.0.2, which is also in a subnet that has logging enabled. In turn, 10.50.0.2 responds to the request with a reply containing 5,342 bytes. Both the request and reply are recorded from both the requesting and responding VMs. Flow Analyzer lets you quickly and efficiently understand your Virtual Private Cloud traffic flows without the need to write complex SQL queries for analyzing VPC Flow Logs. Flow Analyzer lets you perform opinionated network traffic analysis with 5-tuple granularity (source IP, destination IP, source port, destination port, and protocol). Developed using Log Analytics and powered by BigQuery, Flow Analyzer enables in-depth analysis of inbound and outbound traffic of your VM instances. It lets you monitor, troubleshoot, and optimize your networking deployment for better performance and enhanced security which helps ensure compliance, and save on costs. Packet Mirroring clones the traffic of specific instances in your Virtual Private Cloud network and forwards it for examination. It also captures all ingress and egress traffic and packet data, such as payloads and headers. The mirroring happens on the virtual machines, not on the network. Consequently, Packet Mirroring consumes additional bandwidth on the hosts. Packet Mirroring is useful when you need to monitor and analyze your security status. It exports all traffic, not only the traffic between sampling periods. For example, you can use security software that analyzes mirrored traffic to detect all threats or anomalies. Additionally, you can inspect the full traffic flow to detect application performance issues. Cloud NAT logging allows you to log NAT connections and errors. When Cloud NAT logging is enabled, one log entry can be generated for each of the following scenarios: When a network connection using NAT is created. And when a packet is dropped because no port was available for NAT. You can choose to log both kinds of events, or only one. Created logs are sent to Cloud Logging. Cloud NAT logging handles TCP and UDP traffic only. Cloud NAT logging only logs dropped packets if they are egress (outbound) TCP and UDP packets. It does not log dropped incoming packets, for example, if an inbound response to an outbound request is dropped for any reason, no error is logged. Although Tal can explore each log entry within Google Cloud Logging, we recommend exporting logs to BigQuery. BigQuery runs blazing-fast SQL queries on gigabytes to petabytes of data. This allows Tal to analyze network traffic to better understand traffic growth to forecast capacity, analyze network usage to optimize network traffic expenses, or analyze network forensics to examine incidents. For example, in this screenshot we queried my logs to identify the top IP addresses that have exchanged traffic with my web server. Depending on where they are and who they belong to, we could relocate part of my infrastructure to save on networking costs or deny some of these IP addresses if we don’t want them to access my web server. If Tal wants to visualize logs, we recommend connecting BigQuery tables to Data Studio. Data Studio transforms raw data into the metrics and dimensions that Tal can use to create easy-to-understand reports and dashboards.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.