Internal TCP/UDP Load Balancing logging and monitoring

This document shows you how to configure and use Cloud Logging and Cloud Monitoring for Internal TCP/UDP Load Balancing.

Logging

Preview

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA products and features might have limited support, and changes to pre-GA products and features might not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

The logs provide useful information for troubleshooting and monitoring the pass-through Google Cloud load balancer. Logs are aggregated per-connection and exported in near real time. Logs are generated for each load balanced instances' TCP and UDP flows for both ingress and egress traffic. For more information about the fields provided in the log entry, see Log fields.

There are no additional charges for using logs. Based on how you ingest logs, standard pricing for Cloud Logging, BigQuery, or Pub/Sub apply. Enabling logs has no effect on the performance of the load balancer.

Benefits

Here are the benefits of using logging:

Internal TCP/UDP load balancer traffic monitoring

Per-connection logging gives you insight into how each connection is routed to serving backends.

Network troubleshooting

You can utilize internal TCP/UDP load balancer logs for troubleshooting. For more information, see Troubleshoot internal TCP/UDP load balancer.

Sample log format for VM to VM flows

The following diagram shows the inbound and outbound traffic for an internal client (192.168.1.2), internal TCP/UDP load balancer (10.240.0.200), and backend instance (10.240.0.3).

Inbound and outbound flows for VM to VM flows.

Internal TCP/UDP load balancer logs for connections from the client to the backend instance are formatted as follows:

• connection.clientIp: 192.168.1.2
• connection.serverIp: 10.240.0.200
• bytesSent: 1256
• bytesReceived: 4521

Logs sampling and collection

Google Cloud samples packets that leave and enter load balancer backend VMs. Those sampled packets are processed to generate logs.

Not every packet is sampled. Google Cloud samples a variable subset of packets depending on the amount of traffic on the physical host. The lowest possible sampling rate is one out of 1,024 packets. The sampling rate is dynamically controlled by Google Cloud. You cannot adjust the sampling rate.

The packet sampling interacts with firewall rules in the following ways:

• Packets are sampled before egress firewall rules are applied.
• Packets are sampled after ingress firewall rules are applied.

After packet sampling, Google Cloud processes the sampled packets according to the following procedure:

1. Aggregation: Sampled packets are aggregated over a five second interval to produce a single flow entry.

2. Configurable (secondary) log sampling: This is a second sampling process, sampling the flows. You control the fraction of the flow entries that are emitted as log entries according to the logConfig.sampleRate parameter. When logConfig.sampleRate is 1.0 (100%), this means all of the sampled packets are processed.

3. Write to logging: The log entries are written to Cloud Logging.

Response packet source IP address requirements

Internal TCP/UDP Load Balancing logging samples only response packets from backend VMs if the source IP address for those packets matches the load balancer's forwarding rule IP address to which a corresponding request packet was sent. For TCP connections, response packets must always have sources that match the request packet's destination; however, for other protocols, it's possible for response packets to use a different source IP address. For additional details, see IP addresses for request and return packets.

The packet sampling process used by Internal TCP/UDP Load Balancing logging omits any response packets from backend VMs if those response packets have sources that do not match an IP address of a forwarding rule for an internal TCP/UDP load balancer.

Note: After logging is enabled, you can no longer edit the load balancer settings by using the Google Cloud console. The Google Cloud console continues to remain disabled even if logging is disabled. Use either the gcloud CLI or the REST API to update the load balancer.

Enable logging for a new backend service

gcloud

Create a backend service and enable logging with the gcloud beta compute backend-services create command.

gcloud beta compute backend-services create BACKEND_SERVICE \
--region=REGION \
--enable-logging \
--logging-sample-rate=SAMPLE_RATE

Replace the following:

• BACKEND_SERVICE: the name of the backend service.
• REGION: Region of the backend service to create.
• SAMPLE_RATE: This field can only be specified if logging is enabled for this backend service. The value of the field must be from 0.0 to 1.0, where 0.0 means no logs are reported and 1.0 means all logged requests are reported. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.

api

Make a POST request to the regionBackendServices.insert method.

POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/regions/REGION/backendServices

 {
 "name": "BACKEND_SERVICE",
  "loadBalancingScheme": "INTERNAL",
  "logConfig": {
   "enable": true,
   "sampleRate": SAMPLE_RATE
  }
 }
 

Enable logging on an existing backend service

gcloud

Enable logging on an existing backend service with the gcloud beta compute backend-services update command.

gcloud beta compute backend-services update BACKEND_SERVICE \
--region=REGION \
--enable-logging \
--logging-sample-rate=SAMPLE_RATE

Replace the following:

• BACKEND_SERVICE: the name of the backend service.
• REGION: Region of the backend service to create.
• SAMPLE_RATE: This field can only be specified if logging is enabled for this backend service. The value of the field must be from 0.0 to 1.0, where 0.0 means no logs are reported and 1.0 means all logged requests are reported. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.

api

Make a PATCH request to the regionBackendServices/patch method.

PATCH https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/regions/REGION/backendServices/BACKEND_SERVICE

 {
 "logConfig": {
   "enable": true,
   "sampleRate": SAMPLE_RATE
  }
 }
 

Disable logging on an existing backend service

gcloud

Disable logging on a backend service with the gcloud beta compute backend-services update command.

gcloud beta compute backend-services update BACKEND_SERVICE \
--region=REGION \
--no-enable-logging

Replace the following:

• BACKEND_SERVICE: the name of the backend service.
• REGION: Region of the backend service to create.

api

Make a PATCH request to the regionBackendServices/patch method.

PATCH https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/regions/REGION/backendServices/BACKEND_SERVICE

 {
 "logConfig": {
   "enable": false
  }
 }
 

View logs

When logs are ingested into Cloud Logging and not excluded through a Log router sink , you can read logs using the Cloud Logging API and the Google Cloud CLI.

To view all the internal TCP/UDP load balancer logs:

Console

1. In the Google Cloud console, go to the Logs Explorer page.
   Go to Logs Explorer
2. Select the Google Cloud Internal Network Load Balancer Rule resource type.
3. Select the loadbalancing.googleapis.com/flows log name.

Console query

1. In the Google Cloud console, go to the Logs Explorer page.
   Go to Logs Explorer
2. Click the Show query toggle.
3. Paste the following into the query field. Ensure that you replace PROJECT_ID with your project ID.

   resource.type="loadbalancing.googleapis.com/InternalNetworkLoadBalancerRule"
   logName="projects/PROJECT_ID/logs/loadbalancing.googleapis.com%2Fflows"
   

4. Click Run query.

View logs for a specific backend service

To view the internal TCP/UDP load balancer logs for a specific backend service:

Console

1. In the Google Cloud console, go to the Logs Explorer page.
   Go to Logs Explorer
2. Enable Log fields.
3. Select the Google Cloud Internal Network Load Balancer Rule resource type.
4. Select the loadbalancing.googleapis.com/flows log name.

Console query

1. In the Google Cloud console, go to the Logs Explorer page.
   Go to Logs Explorer
2. Click the Show query toggle.
3. Paste the following into the query field. Ensure that you replace PROJECT_ID with your project ID and the BACKEND_SERVICE_NAME with the name of your backend service.

   resource.type="loadbalancing.googleapis.com/InternalNetworkLoadBalancerRule"
   logName="projects/PROJECT_ID/logs/loadbalancing.googleapis.com%2Fflows"
   resource.labels.backend_service_name="BACKEND_SERVICE_NAME"
   

4. Click Run query.

View logs for a specific backend

To view the internal TCP/UDP load balancer logs for a specific backend instance group or network endpoint group (NEG) with GCE_VM_IP endpoints, follow these steps:

Console

1. In the Google Cloud console, go to the Logs Explorer page.
   Go to Logs Explorer
2. Enable Log fields.
3. Select Google Cloud Internal Network Load Balancer Rule resource type.
4. Select loadbalancing.googleapis.com/flows log name.

Console query

1. In the Google Cloud console, go to the Logs Explorer page.
   Go to Logs Explorer
2. Click the Show query toggle.

3. Paste the following into the query field. Replace PROJECT_ID with your project ID and the BACKEND_GROUP_NAME with the name of the instance group or NEG.

   resource.type="loadbalancing.googleapis.com/InternalNetworkLoadBalancerRule"
   logName="projects/PROJECT_ID/logs/loadbalancing.googleapis.com%2Fflows"
   resource.labels.backend_group_name="BACKEND_GROUP_NAME"
   

4. Click Run query.

Log fields

Some log fields are in a structured format, with more than one piece of data in a given field. For example, the connection field is of type IpConnection, which contains the source and destination IP address, port, and protocol in a single field. These multi-field fields are described in the following tables.

The monitored resource is loadbalancing.googleapis.com/InternalNetworkLoadBalancerRule.

Field	Type	Description
connection	IpConnection	5-Tuple describing this connection.
startTime	string	Timestamp (RFC 3339 date string format) of the first observed packet during the aggregated time interval.
endTime	string	Timestamp (RFC 3339 date string format) of the last observed packet during the aggregated time interval.
bytesSent	int64	Number of bytes sent from the server to the client.
bytesReceived	Int64	Number of bytes received by the server from the client.
packetsSent	int64	Number of packets sent from the server to the client.
packetsReceived	int64	Number of packets received by the server from the destination.
rtt	string	Latency is measured only for TCP connections. Latency is the sum of estimated network round-trip time (RTT) plus time consumed processing the packet within the client's VM operating system. For sampled packets, the RTT is calculated from the perspective of a load balanced backend by measuring time differences between the backend sending a TCP segment and the backend receiving a TCP acknowledgement for the sequence number of the sent segment. The latency is formatted as a string that begins with number of seconds and ends with "s" to indicate seconds. Nanoseconds is expressed as fractional seconds. For example, latency of 250 milliseconds is formatted as "0.250000000s".

IpConnection

Field	Type	Description
clientIp	string	Client IP address
clientPort	int32	Client port. Set for TCP and UDP connections only.
serverIp	string	Server IP address(forwarding rule IP)
serverPort	int32	Server port. Set for TCP and UDP connections only.
protocol	int32	The IANA protocol number

Monitoring

Internal TCP/UDP Load Balancing exports monitoring data to Cloud Monitoring.

Monitoring metrics can be used for the following purposes:

• Evaluate an internal TCP/UDP load balancer's configuration, usage, and performance
• Troubleshoot problems
• Improve resource utilization and user experience

In addition to the predefined dashboards in Monitoring, you can create custom dashboards, set up alerts, and query the metrics through the Monitoring API.

View Monitoring dashboards

1. In the Google Cloud console, go to Monitoring.
   Go to Monitoring
2. If Resources appears in the navigation pane, select Resources and then select Google Cloud Load Balancers. Otherwise, select Dashboards and then select the dashboard named Google Cloud Load Balancers.
3. Click the name of your load balancer.

In the left pane, you can see various details for the selected load balancer. In the right pane, you can see time series graphs. Click the Breakdowns link to see specific breakdowns. The left pane presents currently configured data, while the right pane can present data served by historical configurations not currently reflected in the left pane.

Note: Monitoring collects internal TCP/UDP load balancer data every 15 minutes. However, it might take up to two hours after you create your first internal TCP/UDP load balancer for data to be available in Cloud Logging and Monitoring.

Define Monitoring custom dashboards

You can create custom Monitoring dashboards over Internal TCP/UDP Load Balancing metrics:

1. In the Google Cloud console, go to Monitoring.
   Go to Monitoring
2. Select Dashboards > Create Dashboard.
3. Click on Add Chart.
4. Give the chart a title.
5. Select metrics and filters. For metrics, the resource type is Google Cloud TCP Load Balancer (Internal) Rule (internal_tcp_lb_rule) or Google Cloud UDP Load Balancer (Internal) Rule (internal_udp_lb_rule).
6. Click Save.

Define alerting policies

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.

1. In the Google Cloud console, go to the Monitoring page.

   Go to Monitoring

2. In the Monitoring navigation pane, select Alerting.
3. If you haven't created your notification channels and if you want to be notified, then click Edit Notification Channels and add your notification channels. Return to the Alerting page after you add your channels.
4. From the Alerting page, select Create policy.
5. To select the metric, expand the Select a metric menu and then do the following:
   1. To limit the menu to relevant entries, enter Google Cloud TCP Load Balancer or Google Cloud UDP Load Balancer into the filter bar. If there are no results after you filter the menu, then disable the Show only active resources & metrics toggle.
   2. For the Resource type, select Google Cloud TCP Load Balancer or Google Cloud UDP Load Balancer.
   3. Select a Metric category and a Metric, and then select Apply.
6. Click Next.
7. The settings in the Configure alert trigger page determine when the alert is triggered. Select a condition type and, if necessary, specify a threshold. For more information, see Condition trigger.
8. Click Next.
9. Optional: To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu, and then click OK.
10. Optional: Update the Incident autoclose duration. This field determines when Monitoring closes incidents in the absence of metric data.
11. Optional: Click Documentation, and then add any information that you want included in a notification message.
12. Click Alert name and enter a name for the alerting policy.
13. Click Create Policy.

For more information, see Alerting policies.

Metrics for internal TCP/UDP load balancers

The following metrics for internal TCP/UDP load balancers are reported into Monitoring.

Metric	Description
Inbound Throughput	The number of bytes sent towards internal TCP/UDP load balancer forwarding rules, as received by the backends.
Inbound Packets	The number of packets sent towards internal TCP/UDP load balancer forwarding rules, as received by the backends.
Outbound Throughput	The number of bytes sent by internal load balanced backends on connections bound to forwarding rule IPs.
Outbound Packets	The number of packets sent by internal load balanced backends on connections bound to forwarding rule IPs.
Latency(*)	The distribution of the RTT measured for bundles of packets over each internal load balanced connection. Typically reduced to 95th-percentile in Monitoring views.

(*) Available only for TCP traffic.

Filter dimensions for internal TCP/UDP load balancer metrics

Metrics are aggregated for each internal TCP/UDP load balancer. Metrics can be further broken down by the following dimensions:

Property	Description
BACKEND NAME	The name of the instance group or network endpoint group (NEG) with `GCE_VM_IP` endpoints.
BACKEND SCOPE	The scope (region or zone) of the backend that received the connection.
BACKEND ZONE	For zonal instance groups and network endpoint groups, zone of the backend that served the connection.
CLIENT NETWORK	The network of the client instance that connected to the internal TCP/UDP load balancer.
CLIENT SUBNETWORK	The subnetwork of the client instance that connected to the internal TCP/UDP load balancer.
CLIENT ZONE	The Google Cloud zone of the instance that connected to the forwarding rule.
FORWARDING RULE	The name of the internal TCP/UDP load balancer's forwarding rule.

Metric reporting frequency and retention

Metrics for the internal TCP/UDP load balancers are exported to Monitoring in one-minute granularity batches. Monitoring data is retained for six weeks. Metrics are based on sampled traffic (sampling rate is dynamic and cannot be adjusted). The dashboard provides data analysis in default intervals of one hour (1H), six hours (6H), one day (1D), one week (1W), and six weeks (6W). You can manually request analysis in any interval from six weeks to one minute.

What's next

• See Internal TCP/UDP Load Balancing Concepts for important fundamentals.
• See Failover concepts for Internal TCP/UDP Load Balancing for important information about failover.
• See Internal Load Balancing and DNS Names for available DNS name options your load balancer can use.
• See Setting Up Internal TCP/UDP Load Balancing for an example internal TCP/UDP load balancer configuration.
• See Configuring failover for Internal TCP/UDP Load Balancing for configuration steps and an example internal TCP/UDP load balancer failover configuration.
• See Internal TCP/UDP Load Balancing and connected networks for information about accessing internal load balancers from peer networks connected to your VPC network.
• See Troubleshooting Internal TCP/UDP Load Balancing for information about how to troubleshoot issues with your internal TCP/UDP load balancer.
• See Troubleshoot logging issues for information about how to troubleshoot issues with logs.