Proxy Network Load Balancer logging and monitoring

This page shows you how to configure and use Cloud Logging and Cloud Monitoring for proxy Network Load Balancers.

Monitoring resources

The following table specifies the resource names for the load balancers.

Regional external proxy Network Load Balancer

Regional internal proxy Network Load Balancer

Cross-region internal proxy Network Load Balancer (Preview)

Global external proxy Network Load Balancer

Classic proxy Network Load Balancer
Logging monitored-resource type "Proxy Network Load Balancer Rule"
l4_proxy_rule
"Global External Proxy Network Load Balancer Rule"
tcp_ssl_proxy_rule
Monitoring monitored-resource type "Proxy Network Load Balancer Rule"
l4_proxy_rule
"Global External Proxy Network Load Balancer Rule"
tcp_ssl_proxy_rule

Logging for Proxy Network Load Balancers

Logs provide useful information for troubleshooting and monitoring load balancers. Logs are aggregated for each connection and give you insight into how each connection is routed to the serving backends.

There are no additional charges for using logs. However, based on how you import logs, standard pricing for Cloud Logging, BigQuery, or Pub/Sub applies. Also, enabling logs does not affect the performance of the load balancer.

Logs sampling and collection

The connections that leave and enter load balancer backend virtual machine (VM) instances are sampled. These sampled connections are then processed to generate logs. You control the fraction of the connections that are emitted as log entries according to the logConfig.sampleRate parameter. When logConfig.sampleRate is 1.0 (100%), this means that logs are generated for all of the connections and written to Cloud Logging.

Enable logging on a new backend service

gcloud

Use the gcloud compute backend-services create command.

For regional external proxy Network Load Balancers and regional internal proxy Network Load Balancers:

    gcloud compute backend-services create BACKEND_SERVICE \
        --region=REGION \
        --enable-logging \
        --logging-sample-rate=SAMPLE_RATE
    

For global external proxy Network Load Balancers, classic proxy Network Load Balancers, or cross-region internal proxy Network Load Balancers:

    gcloud compute backend-services create BACKEND_SERVICE \
        --global \
        --enable-logging \
        --logging-sample-rate=SAMPLE_RATE
    

Replace the following:

  • BACKEND_SERVICE: the name of the backend service.
  • REGION: the region of the backend service to create.
  • SAMPLE_RATE: this field can only be specified if logging is enabled for this backend service.

    The value of the field must be from 0.0 to 1.0, where 0.0 means that no logs are reported and 1.0 means that all connections are logged. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.

API

Make a POST request to the regionBackendServices.insert method:

For regional internal proxy Network Load Balancers:

    {
    "name": "BACKEND_SERVICE",
    "loadBalancingScheme": "INTERNAL_MANAGED",
    "logConfig": {
       "enable": true,
       "sampleRate": SAMPLE_RATE
      }
    }
    

For regional external proxy Network Load Balancers:

    {
    "name": "BACKEND_SERVICE",
    "loadBalancingScheme": "EXTERNAL_MANAGED",
    "logConfig": {
       "enable": true,
       "sampleRate": SAMPLE_RATE
      }
    }
    

For global external proxy Network Load Balancers:

Make a POST request to the backendServices.insert method:

    {
    "name": "BACKEND_SERVICE",
    "loadBalancingScheme": "EXTERNAL_MANAGED",
    "logConfig": {
       "enable": true,
       "sampleRate": SAMPLE_RATE
      }
    }
    

For classic proxy Network Load Balancers:

Make a POST request to the backendServices.insert method:

    {
    "name": "BACKEND_SERVICE",
    "loadBalancingScheme": "EXTERNAL",
    "logConfig": {
       "enable": true,
       "sampleRate": SAMPLE_RATE
      }
    }
    

For cross-region internal proxy Network Load Balancers:

Make a POST request to the backendServices.insert method:

    {
    "name": "BACKEND_SERVICE",
    "loadBalancingScheme": "INTERNAL_MANAGED",
    "logConfig": {
       "enable": true,
       "sampleRate": SAMPLE_RATE
      }
    }
    

Replace the following:

  • BACKEND_SERVICE: the name of the backend service.
  • SAMPLE_RATE: this field can only be specified if logging is enabled for this backend service.

    The value of the field must be from 0.0 to 1.0, where 0.0 means that no logs are reported and 1.0 means that all connections are logged. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.

Enable logging on an existing backend service

gcloud

Use the gcloud compute backend-services update command.

For regional external proxy Network Load Balancers and regional internal proxy Network Load Balancers:

    gcloud compute backend-services update BACKEND_SERVICE \
        --region=REGION \
        --enable-logging \
        --logging-sample-rate=SAMPLE_RATE
    

For global external proxy Network Load Balancers, classic proxy Network Load Balancers, or cross-region internal proxy Network Load Balancers:

    gcloud compute backend-services update BACKEND_SERVICE \
        --global \
        --enable-logging \
        --logging-sample-rate=SAMPLE_RATE
    

Replace the following:

  • BACKEND_SERVICE: the name of the backend service.
  • REGION: the region of the backend service to create.
  • SAMPLE_RATE: this field can only be specified if logging is enabled for this backend service.

    The value of the field must be from 0.0 to 1.0, where 0.0 means that no logs are reported and 1.0 means that all connections are logged. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.

API

Make a PATCH request to the regionBackendServices/patch method:

      PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/backendServices/BACKEND_SERVICE
     

For regional internal proxy Network Load Balancers:

    {
    "name": "BACKEND_SERVICE",
    "loadBalancingScheme": "INTERNAL_MANAGED",
    "logConfig": {
       "enable": true,
       "sampleRate": SAMPLE_RATE
      }
    }
    

For regional external proxy Network Load Balancers:

    {
    "name": "BACKEND_SERVICE",
    "loadBalancingScheme": "EXTERNAL_MANAGED",
    "logConfig": {
       "enable": true,
       "sampleRate": SAMPLE_RATE
      }
    }
    

For global external proxy Network Load Balancers:

Make a PATCH request to the backendServices/patch method:

      PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/backendServices/BACKEND_SERVICE
    {
    "name": "BACKEND_SERVICE",
    "loadBalancingScheme": "EXTERNAL_MANAGED",
    "logConfig": {
       "enable": true,
       "sampleRate": SAMPLE_RATE
      }
    }
    

For classic proxy Network Load Balancers:

Make a PATCH request to the backendServices/patch method:

      PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/backendServices/BACKEND_SERVICE
    {
    "name": "BACKEND_SERVICE",
    "loadBalancingScheme": "EXTERNAL",
    "logConfig": {
       "enable": true,
       "sampleRate": SAMPLE_RATE
      }
    }
    

For cross-region internal proxy Network Load Balancers:

Make a PATCH request to the backendServices/patch method:

      PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/backendServices/BACKEND_SERVICE
    {
    "name": "BACKEND_SERVICE",
    "loadBalancingScheme": "INTERNAL_MANAGED",
    "logConfig": {
       "enable": true,
       "sampleRate": SAMPLE_RATE
      }
    }
    

Replace the following:

  • PROJECT_ID: the name of your project.
  • BACKEND_SERVICE: the name of the backend service.
  • SAMPLE_RATE: this field can only be specified if logging is enabled for this backend service.

    The value of the field must be from 0.0 to 1.0, where 0.0 means that no logs are reported and 1.0 means that all connections are logged. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.

Disable logging on an existing backend service

gcloud

Use the gcloud compute backend-services update command.

For regional external proxy Network Load Balancers and regional internal proxy Network Load Balancers:

gcloud compute backend-services update BACKEND_SERVICE \
   --region=REGION \
   --no-enable-logging

For global external proxy Network Load Balancers, classic proxy Network Load Balancers, or cross-region internal proxy Network Load Balancers:

gcloud compute backend-services update BACKEND_SERVICE \
   --global \
   --no-enable-logging

Replace the following:

  • BACKEND_SERVICE: the name of the backend service.
  • REGION: the region of the backend service.

API

For regional external proxy Network Load Balancers and regional internal proxy Network Load Balancers:

Make a PATCH request to the regionBackendServices/patch method:

 PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/backendServices/BACKEND_SERVICE
  {
  "logConfig": {
    "enable": false
   }
  }
 

For global external proxy Network Load Balancers, classic proxy Network Load Balancers, or cross-region internal proxy Network Load Balancers:

Make a PATCH request to the backendServices/patch method:

 PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/backendServices/BACKEND_SERVICE
  {
  "logConfig": {
    "enable": false
   }
  }
 

Replace the following:

  • PROJECT_ID: the name of your project.
  • REGION: the region of the backend service.
  • BACKEND_SERVICE: the name of the backend service.

View logs

When logs are ingested into Cloud Logging and not excluded through a Log Router sink, you can read logs by using the Cloud Logging API and the Google Cloud CLI.

To view all the logs, complete the following steps.

Console

  1. In the Google Cloud console, go to the Logs Explorer page.

    Go to Logs Explorer

  2. Select the Proxy Network Load Balancer Rule resource type.

  3. Select the loadbalancing.googleapis.com/connections log name.

Console query

  1. In the Google Cloud console, go to the Logs Explorer page.

    Go to Logs Explorer

  2. Click the Show query toggle.

  3. Paste the following into the query field.

    resource.type="LOG_RESOURCE_TYPE"
    logName="projects/PROJECT_ID/logs/loadbalancing.googleapis.com/connections"
    
  4. Click Run query.

Replace the following:

  • LOG_RESOURCE_TYPE: the logging monitored-resource type set to either l4_proxy_rule or tcp_ssl_proxy_rule.
  • PROJECT_ID: the name of your project.

View logs for a specific backend service

To view the logs for a specific backend service, complete the following steps.

Console query

  1. In the Google Cloud console, go to the Logs Explorer page.

    Go to Logs Explorer

  2. Click the Show query toggle.

  3. Paste the following into the query field.

    resource.type="LOG_RESOURCE_TYPE"
    logName="projects/PROJECT_ID/logs/loadbalancing.googleapis.com/connections"
    resource.labels.backend_service_name="BACKEND_SERVICE_NAME"
    
  4. Click Run query.

Replace the following:

  • LOG_RESOURCE_TYPE: the logging monitored-resource type set to either l4_proxy_rule or tcp_ssl_proxy_rule.
  • PROJECT_ID: the name of your project.
  • BACKEND_SERVICE_NAME: the name of the backend service.

View logs for a backend instance group

To view the logs for a specific backend instance group, complete the following steps.

Console query

  1. In the Google Cloud console, go to the Logs Explorer page.

    Go to Logs Explorer

  2. Click the Show query toggle.

  3. Paste the following into the query field.

    resource.type="LOG_RESOURCE_TYPE"
    logName="projects/PROJECT_ID/logs/loadbalancing.googleapis.com/connections"
    resource.labels.backend_group_name="BACKEND_GROUP_NAME"
    
  4. Click Run query.

Replace the following:

  • LOG_RESOURCE_TYPE: the logging monitored-resource type set to either l4_proxy_rule or tcp_ssl_proxy_rule.
  • PROJECT_ID: the name of your project.
  • BACKEND_GROUP_NAME: the name of the instance group.

What is logged

Log entries contain information useful for monitoring and debugging your traffic. Log records contain required fields, which are the default fields of every log record.

Field Field format Field type: Required or Optional Description
severity
timestamp
receiveTimestamp
insertID
logName
LogEntry Required The general fields as described in a log entry.
resource MonitoredResource Required

The MonitoredResource is the resource type associated with a log entry.

The MonitoredResourceDescriptor describes the schema of a MonitoredResource object by using a type name and a set of labels. For more information, see Resource labels.

jsonPayload object (Struct format) Required The log entry payload that is expressed as a JSON object. The JSON object contains the following fields:

Log fields

Log records contain required fields, which are the default fields of every log record.

Some log fields contain more than one piece of data in a given field—these log fields are in a multi-field format. For example, the connection field is of the IpConnection format, which contains the source and destination IP address and port, plus the protocol, in a single field. These multi-field log fields are described in the following record format table.

The following table lists all the required log fields for the resource l4_proxy_rule.

Field Field format Description
connection IpConnection 5-Tuple describing this connection.
startTime string Timestamp (RFC 3339 date string format) when the connection from the client was accepted by the load balancer.
endTime string Timestamp (RFC 3339 date string format) when the client or the backend terminated the connection.
bytesSent int64 Number of bytes sent from the server to the client.
bytesReceived int64 Number of bytes received by the server from the client.

IpConnection field format

Field Type Description
clientIp string Client IP address
clientPort int32 Client port. Set for TCP and UDP connections only.
serverIp string Server IP address (forwarding rule IP)
serverPort int32 Server port. Set for TCP and UDP connections only.
protocol int32 IANA protocol number

Resource labels

The following table lists the resource labels for resource type l4_proxy_rule.

Field Type Description
network_name string The name of the load balancer's VPC network.
project_id string The identifier of the Google Cloud project associated with this resource.
region string The region where the load balancer is defined.
target_proxy_name string The name of the target proxy object referenced by the forwarding rule.
forwarding_rule_name string The name of the forwarding rule object.
loadbalancing_scheme_name string An attribute on the forwarding rule and the backend service of a load balancer that indicates whether the load balancer can be used for internal or external traffic.
backend_target_name string The name of the backend selected to handle the request.
backend_target_type string The type of backend target (BACKEND_SERVICE / UNKNOWN).
backend_name string The name of the backend instance group or network endpoint group (NEG).
backend_type string

The type of backend, either an instance group or a NEG, or unknown.

Cloud Logging logs requests when the backend_type is UNKNOWN even if logging is disabled. For example, if a client closes the connection to the load balancer before the load balancer can pick a backend, the backend_type is set to UNKNOWN and the request is logged. These logs provide useful debugging information about client requests that were closed because the load balancer couldn't select a backend.

backend_scope string The scope of the backend, either a zone name or a region name. Might be UNKNOWN whenever backend_name is unknown.
backend_scope_type string The scope of the backend (REGION/ZONE). Might be UNKNOWN whenever backend_name is unknown.

Monitoring

The proxy Network Load Balancers export monitoring data to Cloud Monitoring.

Monitoring metrics can be used to do the following:

  • Evaluate a load balancer's configuration, usage, and performance.
  • Troubleshoot problems.
  • Improve resource utilization and user experience.

In addition to the predefined dashboards in Monitoring, you can create custom dashboards, set up alerts, and query metrics by using the Cloud Monitoring API.

View Monitoring dashboards

  1. In the Google Cloud console, go to the Monitoring page.

    Go to Monitoring

  2. If Resources appears in the navigation pane, select Resources, and then select Google Cloud Load Balancers. Otherwise, select Dashboards, and then select the dashboard named Google Cloud Load Balancers.

  3. Click the name of your load balancer.

In the left pane, you can see various details for this load balancer. In the right pane, you can see timeseries graphs. To see specific breakdowns, click Breakdowns.

Metric reporting frequency and retention

Metrics for the load balancers are exported to Monitoring in one-minute granularity batches. Monitoring data is retained for six (6) weeks. Metrics are based on sampled traffic (sampling rate is dynamic and cannot be adjusted).

The dashboard provides data analysis in default intervals of 1H (one hour), 6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You can manually request analysis in any interval from six weeks to one minute.

Metrics for classic proxy Network Load Balancers

The following metrics for classic proxy Network Load Balancers are reported into Monitoring.

Metric Name Description
Inbound traffic tcp_ssl_proxy/ingress_bytes_count The number of bytes sent from external endpoints to configured backends through the Google Front End (GFE)—in bytes per second.
Outbound traffic tcp_ssl_proxy/egress_bytes_count The number of bytes sent from configured backends to external endpoints through the GFE—in bytes per second.
Open connections tcp_ssl_proxy/open_connections The number of connections open at the given sample moment. Samples are taken one minute apart.
New connections per second tcp_ssl_proxy/new_connections The number of connections that were created (client successfully connected to backend). The counting granularity is per minute, but graphs are adjusted to show per second values. For more information, see the Monitoring documentation.
Closed connections per second tcp_ssl_proxy/closed_connections The number of connections that were closed. The counting granularity is per minute, but graphs are adjusted to show per second values. For more information, see the Monitoring documentation.
Frontend RTT tcp_ssl_proxy/frontend_tcp_rtt A distribution of the smoothed round-trip time (RTT) measured for each connection between the client and the GFE (measured by the GFE's TCP stack, each time application layer bytes pass from the GFE to the client). Smoothed RTT is an algorithm that deals with variations and anomalies that might occur in RTT measurements.

Metrics for other load balancers

The following metrics for regional internal proxy Network Load Balancers, regional external proxy Network Load Balancers, cross-region internal proxy Network Load Balancer, and global external proxy Network Load Balancers are reported into Monitoring.

Metric Name Description
Inbound traffic l4_proxy/ingress_bytes_count The number of bytes sent from the client to the backend VM by using the proxy. Sampled every 60 seconds. After sampling, data is not visible for up to 210 seconds.
Outbound traffic l4_proxy/egress_bytes_count The number of bytes sent from the backend VM to the client by using the proxy. Sampled every 60 seconds. After sampling, data is not visible for up to 210 seconds.
Closed connections per second l4_proxy/tcp/closed_connections_count The number of connections that were terminated by using a TCP RST or TCP FIN message. Sampled every 60 seconds. After sampling, data is not visible for up to 210 seconds.

Filtering dimensions for metrics

Metrics are aggregated for each load balancer. Metrics can be further broken down by the following dimensions.

Property Description
BACKEND SCOPE The scope (region or zone) of the instance group that served the connection.
BACKEND ZONE If the instance group was a zonal instance group, the zone of the instance group that served the connection.
BACKEND REGION If the instance group was a regional instance group, the region of the instance group that served the connection.
PROXY CONTINENT The continent of the GFE that terminated the user TCP/SSL connection—for example, America, Europe, Asia.
INSTANCE GROUP The name of the instance group that received the user connection.
FORWARDING RULE The name of the forwarding rule used to connect to the GFE.
CLIENT COUNTRY The name of the country of the user.

What's next