This document shows you how to configure and use Cloud Logging and Cloud Monitoring for Network Load Balancing.
Logging
The logs provide useful information for troubleshooting and monitoring the pass-through Google Cloud load balancer. Logs are aggregated per-connection and exported in near real time. Logs are generated for each load balanced instances' TCP, UDP, ESP, GRE, ICMP, and ICMPv6 flows for both ingress and egress traffic. For more information about the fields provided in the log entry, see Log fields.
There are no additional charges for using logs. Based on how you ingest logs, standard pricing for Cloud Logging, BigQuery, or Pub/Sub apply. Enabling logs has no effect on the performance of the load balancer.
Benefits
Here are the benefits of using logging:
Network load balancer traffic monitoring
Per-connection logging gives you insight into how each connection is routed to serving backends.
Network troubleshooting
You can utilize network load balancer logs for troubleshooting. For more information, see Troubleshoot network load balancer.
Sample log format for an external client to VM flows
The following diagram shows the inbound and outbound traffic for an external client (203.0.113.7), network load balancer (198.51.100.99), and backend instance (10.240.0.2).
Network load balancer logs for connections from the client to the backend instance are formatted as follows:
connection.clientIp
: 203.0.113.7connection.serverIp
: 198.51.100.99bytesSent
: 1256bytesReceived
: 4521
Logs sampling and collection
Google Cloud samples packets that leave and enter load balancer backend VMs. Those sampled packets are processed to generate logs.
Not every packet is sampled. Google Cloud samples a variable subset of packets depending on the amount of traffic on the physical host. The lowest possible sampling rate is one out of 1,024 packets. The sampling rate is dynamically controlled by Google Cloud. You cannot adjust the sampling rate.
The packet sampling interacts with firewall rules in the following ways:
- Packets are sampled before egress firewall rules are applied.
- Packets are sampled after ingress firewall rules are applied.
After packet sampling, Google Cloud processes the sampled packets according to the following procedure:
Aggregation: Sampled packets are aggregated over a five second interval to produce a single flow entry.
Configurable (secondary) log sampling: This is a second sampling process, sampling the flows. You control the fraction of the flow entries that are emitted as log entries according to the logConfig.sampleRate parameter. When
logConfig.sampleRate
is1.0
(100%), this means all of the sampled packets are processed.Write to logging: The log entries are written to Cloud Logging.
Response packet source IP address requirements
Network Load Balancing logging only samples response packets from backend VMs if the source IP address for those packets matches the load balancer's forwarding rule IP address to which a corresponding request packet was sent. For TCP connections, response packets must always have sources that match the request packet's destination; however, for other protocols, it's possible for response packets to use a different source IP address. For more information, see IP addresses for request and return packets.
The packet sampling process used by Network Load Balancing logging omits any response packets from backend VMs if those response packets have sources that do not match an IP address of a forwarding rule for an network load balancer.
Enable logging on a new backend service
Create the backend service to enable logging with the gcloud beta compute backend-services create command.
gcloud
gcloud beta compute backend-services create BACKEND_SERVICE \ --region=REGION \ --enable-logging \ --logging-sample-rate=SAMPLE_RATE
Replace the following:
BACKEND_SERVICE
: the name of the backend service.REGION
: Region of the backend service to create.SAMPLE_RATE
: This field can only be specified if logging is enabled for this backend service. The value of the field must be from 0.0 to 1.0, where 0.0 means no logs are reported and 1.0 means all logged requests are reported. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.
api
Make a POST
request to the
regionBackendServices.insert
method.
POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID
/regions/REGION
/backendServices
{ "name": "BACKEND_SERVICE", "loadBalancingScheme": "EXTERNAL", "logConfig": { "enable": true, "sampleRate": SAMPLE_RATE } }
Enable logging on an existing backend service
Enable logging on an existing backend service with the gcloud beta compute backend-services update command.
gcloud
gcloud beta compute backend-services update BACKEND_SERVICE \ --region=REGION \ --enable-logging \ --logging-sample-rate=SAMPLE_RATE
Replace the following:
BACKEND_SERVICE
: the name of the backend service.- `
REGION
: Region of the backend service to create. SAMPLE_RATE
: This field can only be specified if logging is enabled for this backend service. The value of the field must be from 0.0 to 1.0, where 0.0 means no logs are reported and 1.0 means all logged requests are reported. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.
api
Make a PATCH
request to the
regionBackendServices/patch
method.
PATCH https://compute.googleapis.com/compute/beta/projects/PROJECT_ID
/regions/REGION
/backendServices/BACKEND_SERVICE
{ "logConfig": { "enable": true, "sampleRate": SAMPLE_RATE } }
Disable logging on an existing backend service
Disable logging on the backend service with the
gcloud beta compute backend-services update
command.
gcloud
gcloud beta compute backend-services update BACKEND_SERVICE \ --region=REGION \ --no-enable-logging
API
Make a PATCH
request to the
regionBackendServices/patch
method.
PATCH https://compute.googleapis.com/compute/beta/projects/PROJECT_ID
/regions/REGION
/backendServices/BACKEND_SERVICE
{ "logConfig": { "enable": false } }
View logs
When logs are ingested into Cloud Logging and not excluded through a Log router sink , you can read logs using the Cloud Logging API and the Google Cloud CLI.
To view all the network load balancer logs:
Console
- In the Google Cloud console, go to the Logs Explorer page.
Go to Logs Explorer - Select the Google Cloud External Network Load Balancer Rule resource type.
- Select the loadbalancing.googleapis.com/flows log name.
Console query
- In the Google Cloud console, go to the Logs Explorer page.
Go to Logs Explorer - Click the Show query toggle.
- Paste the following into the query field. Ensure that you replace
PROJECT_ID
with your project ID.resource.type="loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule" logName="projects/PROJECT_ID/logs/loadbalancing.googleapis.com%2Fflows"
- Click Run query.
View logs for a specific backend
To view the network load balancer logs for a specific backend service:
Console
- In the Google Cloud console, go to the Logs Explorer page.
Go to Logs Explorer - Enable Log fields.
- Select Google Cloud External Network Load Balancer Rule resource type.
- Select loadbalancing.googleapis.com/flows log name.
Console query
- In the Google Cloud console, go to the Logs Explorer page.
Go to Logs Explorer - Click the Show query toggle.
- Paste the following into the query field. Ensure that you replace
PROJECT_ID
with your project ID and theBACKEND_SERVICE_NAME
with the name of your backend service.resource.type="loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule" logName="projects/PROJECT_ID/logs/loadbalancing.googleapis.com%2Fflows" resource.labels.backend_service_name="BACKEND_SERVICE_NAME"
- Click Run query.
View logs for a backend
To view the network load balancer logs for a specific backend instance group:
Console
- In the Google Cloud console, go to the Logs Explorer page.
Go to Logs Explorer - Enable Log fields.
- Select Google Cloud External Network Load Balancer Rule resource type.
- Select loadbalancing.googleapis.com/flows log name.
Console query
- In the Google Cloud console, go to the Logs Explorer page.
Go to Logs Explorer - Click the Show query toggle.
Paste the following into the query field. Replace
PROJECT_ID
with your project ID and theBACKEND_GROUP_NAME
with the name of the instance group.resource.type="loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule" logName="projects/PROJECT_ID/logs/loadbalancing.googleapis.com%2Fflows" resource.labels.backend_group_name="BACKEND_GROUP_NAME"
Click Run query.
Log fields
Some log fields are in a structured format, with more than one piece of data in
a given field. For example, the connection
field is of type IpConnection
,
which contains the source and destination IP address, port, and
protocol in a single field. These multi-field fields are described in the
following tables:
The monitored resource is loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule.
General information
Field | Type | Description |
---|---|---|
connection | IpConnection | 5-Tuple describing this connection. |
startTime | string | Timestamp (RFC 3339 date string format) of the first observed packet during the aggregated time interval. |
endTime | string | Timestamp (RFC 3339 date string format) of the last observed packet during the aggregated time interval. |
bytesSent | int64 | Number of bytes sent from the server to the client. |
bytesReceived | Int64 | Number of bytes received by the server from the client. |
packetsSent | int64 | Number of packets sent from the server to the client. |
packetsReceived | int64 | Number of packets received by the server from the destination. |
rtt | string | Latency is measured only for TCP connections. Latency is the sum of estimated network round-trip time (RTT) plus time consumed processing the packet within the client's operating system. For sampled packets, the RTT is calculated from the perspective of a load balanced backend by measuring time differences between the backend sending a TCP segment and the backend receiving a TCP acknowledgement for the sequence number of the sent segment. The latency is formatted as a string that begins with number of seconds and ends with "s" to indicate seconds. Nanoseconds is expressed as fractional seconds. For example, latency of 250 milliseconds is formatted as "0.250000000s". |
IpConnection
Field | Type | Description |
---|---|---|
clientIp | string | Client IP address |
clientPort | int32 | Client port. Set for TCP and UDP connections only. |
serverIp | string | Server IP address(forwarding rule IP) |
serverPort | int32 | Server port. Set for TCP and UDP connections only. |
protocol | int32 | The IANA protocol number |
Monitoring
Network Load Balancing exports key metrics to Monitoring.
Monitoring metrics can be used for the following purposes:
- Evaluate a load balancer's configuration, usage, and performance
- Troubleshoot problems
- Improve resource utilization and user experience
In addition to the predefined dashboards in Monitoring, you can create custom dashboards, set up alerts, and query the metrics through the Monitoring API.
View Monitoring dashboards
- In the Cloud Console, go to the Monitoring page.
Go to Monitoring - In the navigation pane, select Dashboards.
- Select the Google Cloud Load Balancers dashboard. All the load balancers are displayed.
- Select the load balancer name from the list of load balancers.
In the Network Load Balancing details pane, you can see various details for the selected load balancer. This pane displays your current configurations.
In the Key metrics pane, you can see charts for each key metrics. Click Breakdowns to see specific breakdowns. This pane presents data informed by historical configurations, while the Network Load Balancing details pane only displays the current configurations. For more information, see Dashboards and charts.
Define Monitoring custom dashboards
You can create custom Monitoring dashboards over Network Load Balancing metrics.
Refer to Metrics and resource types for a list of collected metrics. Refer to Filters for lists of attributes you can use to filter your results.
- In the Cloud Console, go to the Monitoring page.
Go to Monitoring - Select Dashboards > Create Dashboard.
- Click Add chart.
- Give the chart a title.
- Select metrics and filters.
You can find Network Load Balancing metrics by searching for theloadbalancing.googleapis.com/l3/external
prefix. - Click Save.
Define Monitoring alerts
You can define Monitoring alerts over various Network Load Balancing metrics:
- In the Cloud Console, go to the Monitoring page.
Go to Monitoring - Select Alerting > Create a Policy.
- Select resource types and metrics.
You can find Network Load Balancing resource types and metrics by searching forloadbalancing.googleapis.com/l3/external
in the Find resource type and metric field. - Specify a Configuration to set when an alert should be triggered.
- Add Filters if desired.
- Click Save.
Metric reporting frequency and retention
Metrics for the network load balancers are exported to Monitoring in one-minute granularity batches. Monitoring data is retained for six weeks. Metrics are based on sampled traffic (sampling rate is dynamic and cannot be adjusted). The dashboard provides data analysis in default intervals of one hour (1H), six hours (6H), one day (1D), one week (1W), and six weeks (6W). You can manually request analysis in any interval from six weeks to sixty seconds.
Metrics and resource types
Metrics
The following metrics for network load balancers are reported into Monitoring. You can use these metric names when making API requests.
metric_name | Type | Description |
---|---|---|
loadbalancing.googleapis.com/l3/external/ingress_bytes_count |
counter | The number of bytes sent from a client to a network load balancer backend. For TCP flows, only bytes in the application stream are counted. |
loadbalancing.googleapis.com/l3/external/ingress_packets_count |
counter | The number of packets sent from a client to a network load balancer backend. |
loadbalancing.googleapis.com/l3/external/egress_bytes_count |
counter | The number of bytes sent from a network load balancer backend to a client. For TCP flows, only bytes in on the application stream are counted. |
loadbalancing.googleapis.com/l3/external/egress_packets_count |
counter | The number of packets sent from a network load balancer backend to a client. |
loadbalancing.googleapis.com/l3/external/rtt_latencies
|
distribution | A distribution of rtt measured over TCP connections for network load balancer
flows.
Available only for TCP traffic. |
Resource types
Network Load Balancing uses the following resource types:
tcp_lb_rule
udp_lb_rule
loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule
A network load balancer can support TCP, UDP, ESP, GRE, ICMP, and ICMPv6 traffic.
You can specify the resource type as
tcp_lb_rule
or
udp_lb_rule
to limit
monitored data to either TCP or UDP protocols.
You can specify the resource type as
loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule
to monitor all the supported protocols including
TCP, UDP, ESP, GRE, ICMP, and ICMPv6
protocols.
Filters
Resource labels for tcp_lb_rule
or udp_lb_rule
You can restrict your query by resource. You can also group results by these values.
resource.label.<var>LABEL_KEY</var>:
label_key | Type | Description |
---|---|---|
project |
string | The identifier of the Google Cloud project associated with this resource. |
load_balancer_name |
string | The name of the load balancer. |
region |
string | The region
where the load balancer's backend instance is located. For example,
us-central1 , europe-west1 ,
asia-east1 . |
network_name |
string | The VPC network in which the load balancer backend resides. |
backend_target_type |
string | Indicates whether the network load balancer is target pool-based or backend
service-based. Valid values are BACKEND_SERVICE or
TARGET_POOL . |
backend_target_name |
string | For target pool-based network load balancers, the name of the target pool. For backend service-based network load balancers, the name of the backend service." |
forwarding_rule_name |
string | The name of the forwarding rule." |
forwarding_rule_network_tier |
string | The network tier of the forwarding rule. |
backend_name |
string | The name of the backend instance group that handled the connection.
For a backend service-based network load balancer, the value is the name of the backend instance group that handled the connection. For a target pool-based network load balancer, the value is the name of the target pool. |
backend_type |
string | The type of the backend that handled the connection. The valid values are
INSTANCE_GROUP or TARGET_POOL .
For a backend service-based network load balancer, the value is INSTANCE_GROUP .
For a target pool-based network load balancer, the value is TARGET_POOL . |
backend_scope |
string | For a backend service-based network load balancer, the zone or region of the
backend instance group. For a target pool-based network load balancer, the
value is UNSUPPORTED_FOR_TARGET_POOL . |
backend_scope_type |
string | The type of the scope of the backend group that handled the
connection. Valid values are ZONE or REGION .
For a target pool-based network load balancer, the value is
UNSUPPORTED_FOR_TARGET_POOL . |
backend_failover_configuration |
string | Failover configuration of the backend group that
handled the connection. Valid values are PRIMARY ,
BACKUP , or UNKNOWN . |
backend_subnetwork_name |
string | The name of the subnetwork of the instance that received the connection. |
endpoint_zone |
string | Zone of the backend VM that handled the connection. |
Resource labels for loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule
You can restrict your query by resource. You can also group results by these values.
resource.label.<var>LABEL_KEY</var>:
label_key | Type | Description |
---|---|---|
project |
string | The identifier of the Google Cloud project associated with this resource. |
region |
string | The region
where the load balancer's backend instance is located. For example,
us-central1 , europe-west1 ,
asia-east1 . |
backend_network_name |
string | The VPC network in which the load balancer backend resides. |
backend_target_type |
string | The type of the backend target that handled the connection. Valid values
are BACKEND_SERVICE or TARGET_POOL . |
backend_service_name |
string | The name of the backend service that handled the connection. If the
backend_target_type is TARGET_POOL , then the value is
UNSUPPORTED_FOR_TARGET_POOL . |
primary_target_pool |
string | The name of the primary target pool. If the backend_target_type
is BACKEND_SERVICE , then the value is
UNSUPPORTED_FOR_BACKEND_SERVICE . |
target_pool |
string | The name of the target pool. If the backend_target_type is
BACKEND_SERVICE , then the value is UNSUPPORTED_FOR_BACKEND_SERVICE .
|
forwarding_rule_name |
string | The name of the network load balancer's forwarding rule. |
backend_group_name |
string | The name of the backend group that handled the connection.
If the backend_target_type is TARGET_POOL , then the
value is UNSUPPORTED_FOR_TARGET_POOL . |
backend_group_type |
string | The type of the backend group that handled the connection.
If the backend_target_type is BACKEND_SERVICE ,
then the value is INSTANCE_GROUP . If the backend_target_type is TARGET_POOL , then the value
is UNSUPPORTED_FOR_TARGET_POOL . |
backend_group_scope |
string | The scope of the backend group (name of the zone or region) that handled the
connection. If the backend_target_type is TARGET_POOL ,
then the value is UNSUPPORTED_FOR_TARGET_POOL . |
backend_subnetwork_name |
string | The name of the subnetwork of the instance that received the connection. |
backend_zone |
string | Zone of the backend VM that handled the connection. |
Metric labels
You can restrict your query by metric label. You can also group results by these values.
metric.label.<var>LABEL_KEY</var>:
label_key | Type | Description |
---|---|---|
client_country |
string | Country of the client which initiated the connection to the network load balancer. |
client_continent |
string | Continent of the client which initiated the connection to the network load balancer. |
protocol |
string | The protocol in the connection for the
loadbalancing.googleapis.com/ExternalNetworkLoadBalancerRule resource.
This field is left blank for the
tcp_lb_rule and udp_lb_rule resources.
|
Monitoring API requests
You can construct arbitrary queries over the Network Load Balancing metrics using the Monitoring API`s v3 projects.timeSeries.list request. The exact structure of the various definitions follow the generic monitoring filters semantics.
Example API requests
- Get all bytes sent from all network load balancers in the project, during 12:00-12:02 PM UTC on 2019-07-01, with 1m aggregates:
timeSeries.list parameters:
- name: projects/
PROJECT_ID
- Filter: resource.type = "tcp_lb_rule" AND metric.name="loadbalancing.googleapis.com/l3/external/egress_bytes_count"
- interval.start_time: 2019-07-01T12:00:00Z
- interval.end_time: 2019-07-01T12:02:00Z
- aggregation.alignmentPeriod: 60s
- aggregation.crossSeriesReducer: REDUCE_SUM
- aggregation.perSeriesAligner: ALIGN_SUM
- Get median RTT measurement over all network load balancers in the project, during 12:19-12:20 PM UTC on 2019-07-01, broken down by client country:
timeSeries.list parameters:
- name: projects/
PROJECT_ID
- Filter: resource.type = "tcp_lb_rule" AND metric.name="loadbalancing.googleapis.com/l3/external/rtt_latencies"
- interval.start_time: 2019-07-01T12:19:00Z
- interval.end_time: 2019-07-01T12:20:00Z
- aggregation.alignmentPeriod: 60s
- aggregation.crossSeriesReducer: REDUCE_SUM
- aggregation.groupByFields: metric.label.client_country
- aggregation.perSeriesAligner: ALIGN_PERCENTILE_50
- Get total bytes from a specific network load balancer during 12:19-12:20 PM UTC on 2019-07-01, broken down by endpoint_zone and client continent:
timeSeries.list parameters:
- name: projects/
PROJECT_ID
- Filter: resource.type = "tcp_lb_rule" AND resource.label.load_balancer_name = "netlb-bs-1" AND metric.name="loadbalancing.googleapis.com/l3/external/ingress_bytes_count"
- interval.start_time: 2019-07-01T12:19:00Z
- interval.end_time: 2017-07-01T12:20:00Z
- aggregation.alignmentPeriod: 60s
- aggregation.crossSeriesReducer: REDUCE_SUM
- aggregation.groupByFields: metric.label.client_continent
- aggregation.groupByFields: resource.label.endpoint_zone
- aggregation.perSeriesAligner: ALIGN_SUM
What's next
- Read conceptual information about Network Load Balancing
- Set up a network load balancer
- Read about Monitoring Metrics, time series, and resources
- Read about forwarding rules
- See troubleshoot logging issues