Regional external Application Load Balancer logging and monitoring

This document shows you how to configure and use Cloud Logging and Cloud Monitoring with regional external Application Load Balancers.

Logging

You can enable, disable, and view logs for an external Application Load Balancer backend service.

You enable or disable logging for each backend service. You can configure whether to log all requests or a randomly sampled fraction.

You must ensure that you don't have a logs exclusion that applies to external Application Load Balancers. For instructions about how to verify that Cloud HTTP Load Balancer logs are allowed, see Viewing resource-type exclusions.

Note: If the backend service is created after the General Availability release of Logging for external Application Load Balancers, logging is disabled for a backend service. If you enabled logging during the alpha release or beta release, logging continues to be enabled for the backend service.

Optional fields

Log records contain required fields and optional fields. The What is logged section lists which fields are optional and which are required. All required fields are always included. You can customize which optional fields you keep.

• If you select include all optional, all optional fields in the log record format are included in the logs. When new optional fields are added to the record format, the logs automatically include the new fields.

• If you select exclude all optional, all optional fields are omitted.

• If you select custom, you can specify the optional fields that you want to include, such as tls.protocol,tls.cipher.

For instructions about customizing optional fields, see Enable logging on a new backend service.

Enabling logging on a new backend service

Console

1. In the Google Cloud console, go to the Load Balancing page.

   Go to Load balancing

2. Click the name of your load balancer.

3. Click Edit.

4. Click Backend Configuration.

5. Select Create a backend service.

6. Complete the required backend service fields.

7. Click Enable logging.

8. In the Sample rate field, set the sampling probability. You can set a number from 0.0 through 1.0, where 0.0 means that no requests are logged and 1.0 means that 100% of the requests are logged. The default value is 1.0.

9. Optional: To include all the optional fields in the logs, in the Optional fields section, click Include all optional fields.

   Pro tip: To specify the CUSTOM option, use the gcloud CLI and the REST API.

10. To finish editing the backend service, click Update.

11. To finish editing the load balancer, click Update.

gcloud: Regional mode

Create a backend service and enable logging by using the gcloud compute backend-services create command.

gcloud compute backend-services create BACKEND_SERVICE \
    --region=REGION \
    --enable-logging \
    --logging-sample-rate=VALUE \
    --load-balancing-scheme=EXTERNAL_MANAGED \
    --logging-optional=LOGGING_OPTIONAL_MODE \
    --logging-optional-fields=OPTIONAL_FIELDS

where

• --region indicates that the backend service is regional. Use this field for backend services used with regional external Application Load Balancers.
• --enable-logging enables logging for that backend service.
• --logging-sample-rate lets you specify a value from 0.0 through 1.0, where 0.0 means that no requests are logged and 1.0 means that 100% of the requests are logged. Only meaningful with the --enable-logging parameter. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.

• --logging-optional lets you specify the optional fields that you want to include in the logs:

  • INCLUDE_ALL_OPTIONAL to include all optional fields.

  • EXCLUDE_ALL_OPTIONAL (default) to exclude all optional fields.

  • CUSTOM to include a custom list of optional fields that you specify in OPTIONAL_FIELDS.

• --logging-optional-fields lets you specify a comma-separated list of optional fields that you want to include in the logs.

  For example, tls.protocol,tls.cipher can only be set if LOGGING_OPTIONAL_MODE is set to CUSTOM.

Enabling logging on an existing backend service

Console

1. In the Google Cloud console, go to the Load Balancing page.

   Go to Load balancing

2. Click the name of your load balancer.

3. Click Edit.

4. Click Backend Configuration.

5. Click Edit next to your backend service.

6. Click Enable logging.

7. In the Sample rate field, set the sampling probability. You can set a number from 0.0 through 1.0, where 0.0 means that no requests are logged and 1.0 means that 100% of the requests are logged. The default value is 1.0.

8. Optional: To include all the optional fields in the logs, in the Optional fields section, click Include all optional fields.

   Pro tip: To specify the CUSTOM option, use the gcloud CLI and the REST API.

9. To finish editing the backend service, click Update.

10. To finish editing the load balancer, click Update.

gcloud: Regional mode

Enable logging on an existing backend service with the gcloud compute backend-services update command.

gcloud compute backend-services update BACKEND_SERVICE \
    --region=REGION \
    --enable-logging \
    --logging-sample-rate=VALUE \
    --logging-optional=LOGGING_OPTIONAL_MODE \
    --logging-optional-fields=OPTIONAL_FIELDS

where

• --region indicates that the backend service is regional. Use this field for backend services used with regional external Application Load Balancers.
• --enable-logging enables logging for that backend service.
• --logging-sample-rate lets you specify a value from 0.0 through 1.0, where 0.0 means that no requests are logged and 1.0 means that 100% of the requests are logged. Only meaningful with the --enable-logging parameter. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.

• --logging-optional lets you specify the optional fields that you want to include in the logs.

  • INCLUDE_ALL_OPTIONAL to include all optional fields.

  • EXCLUDE_ALL_OPTIONAL (default) to exclude all optional fields.

  • CUSTOM to include a custom list of optional fields that you specify in OPTIONAL_FIELDS.

• --logging-optional-fields lets you specify a comma-separated list of optional fields that you want to include in the logs.

  For example, tls.protocol,tls.cipher. Can only be set if LOGGING_OPTIONAL_MODE is set to CUSTOM.

Disabling or modifying logging on an existing backend service

Console

1. In the Google Cloud console, go to the Load Balancing page.

   Go to Load balancing

2. Click the name of your load balancer.

3. Click Edit.

4. Click Backend Configuration.

5. Click Edit next to your backend service.

6. Clear Enable logging to disable logging entirely.

7. If you leave logging enabled, you can set a different sampling probability in the Sample rate field. You can set a number from 0.0 through 1.0, where 0.0 means that no requests are logged and 1.0 means that 100% of the requests are logged. The default value is 1.0. To reduce the number of stored logs to 20%, set the value to 0.2.

8. To finish editing the backend service, click Update.

9. To finish editing the load balancer, click Update.

gcloud: Regional mode

Disable logging on a backend service with the gcloud compute backend-services update command.

Disabling logging entirely

gcloud compute backend-services update BACKEND_SERVICE \
    --region=REGION \
    --no-enable-logging

where

• --region indicates that the backend service is regional. Use this field for backend services used with regional external Application Load Balancers.
• --no-enable-logging disables logging for that backend service.

Modifying the logging sample rate

gcloud compute backend-services update BACKEND_SERVICE \
 --global | --region=REGION \
 --logging-sample-rate=VALUE

Viewing logs

HTTP(S) logs are indexed first by a forwarding rule, then by a URL map.

To view logs, go to the Logs Explorer page:

Go to Logs Explorer

• To view all logs, in the Resource filter menu, select Cloud HTTP Load Balancer > All forwarding rules.

• To view logs for one forwarding rule, select a single forwarding rule name.

• To view logs for one URL map, select a forwarding rule, and then select a URL map.

Log fields of type boolean typically only appear if they have a value of true. If a boolean field has a value of false, that field is omitted from the log.

UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks. For regional external Application Load Balancers, you can export logs-based metrics using resource logs (resource.type="http_external_regional_lb_rule").

What is logged

External Application Load Balancer log entries contain information useful for monitoring and debugging your HTTP(S) traffic. Log records contain required fields, which are the default fields of every log record. Log records contain optional fields that add additional information about your HTTP(S) traffic. Optional fields can be omitted to save storage costs.

Some log fields are in a multi-field format, with more than one piece of data in a given field. For example, the tls field is of the TlsDetails format, which contains the TLS protocol and TLS cipher in a single field. These multi-field fields are described in the following record format table.

Field	Field format	Field type: Required or Optional	Description
				severity timestamp insertID logName	LogEntry	Required	The general fields as described in a log entry.
httpRequest	HttpRequest	Required	A common protocol for logging HTTP requests.
resource	MonitoredResource	Required	The MonitoredResource is the resource type associated with a log entry. The MonitoredResourceDescriptor describes the schema of a MonitoredResource object by using a type name and a set of labels. For more information, see Resource labels.
jsonPayload	object (Struct format)	Required	The log entry payload that is expressed as a JSON object. The JSON object contains the following fields: proxyStatus tls
	string	Required	The proxyStatus field holds a string that specifies why the regional external Application Load Balancer returned the HttpRequest.status. This field is only populated when the proxy returns an error code with 4XX or 5XX. Possible values can be connection_timeout, destination_unavailable, or http_request_error. The field is not logged if the value is an empty string. For more information, see proxyStatus message.
	TlsDetails	Optional	The tls field holds the TlsDetails that specifies the TLS metadata for the connection between the client and the regional external Application Load Balancer. This field is only available if the client is using TLS/SSL encryption.

Note: Redirects (HTTP response status code 302 Found) issued from the load balancer are not logged. Redirects issued from the backend instances are logged.

TlsDetails field format

Field	Field format	Field type: Required or Optional	Description
protocol	string	Optional	TLS protocol that clients can use to establish a connection with the load balancer. Possible values can be TLS 1.0, 1.1, 1.2, 1.3, or QUIC. This value is set to NULL if the client is not using TLS/SSL encryption.
cipher	string	Optional	TLS cipher that clients can use to establish a connection with the load balancer. This value is set to NULL if the client is not using HTTP(S) or the client is not using TLS/SSL encryption.

Resource labels

The following table lists the resource labels for resource.type="http_external_regional_lb_rule".

Field	Type	Description
backend_name	string	The name of the backend instance group or NEG.
backend_scope	string	The scope of the backend (either a zone name or a region name). Might be UNKNOWN whenever backend_name is unknown.
backend_scope_type	string	The scope of the backend (REGION/ZONE). Might be UNKNOWN whenever backend_name is unknown.
backend_target_name	string	The name of the backend selected to handle the request, based on the URL map path rule or route rule that matches the request.
backend_target_type	string	The type of backend target. Can be BACKEND_SERVICE, or UNKNOWN if the backend wasn't assigned.
backend_type	string	The type of the backend group. Can be INSTANCE_GROUP, NETWORK_ENDPOINT_GROUP, or UNKNOWN if the backend wasn't assigned.
forwarding_rule_name	string	The name of the forwarding rule object.
matched_url_path_rule	string	The URL map path rule or route rule configured as part of the URL map key. Can be UNMATCHED or UNKNOWN as fallbacks. UNMATCHED refers to a request that matches no URL path rules, so it uses the default path rule. UNKNOWN indicates an internal error.
network_name	string	The name of the load balancer's VPC network.
project_id	string	The identifier of the Google Cloud project associated with this resource.
region	string	The region in which the load balancer is defined.
target_proxy_name	string	The name of the target proxy object referenced by the forwarding rule.
url_map_name	string	The name of the URL map object configured to select a backend service.

proxyStatus message

Filter this table:

proxyStatus	Meaning	Common accompanying response codes
destination_unavailable	The load balancer considers the backend to be unavailable. For example, recent attempts to communicate with the backend have failed, or a health check might indicate failure.	500, 503
connection_timeout	The load balancer's attempt to open a connection to the backend has timed out.	504
connection_terminated	The load balancer's connection to the backend was closed before a complete response was received.	502, 503
connection_refused	The load balancer's connection to the backend was refused.	502, 503
connection_limit_reached	The load balancer is configured to limit the number of connections it has to the backend, and that limit has been exceeded.	502, 503
destination_not_found	The load balancer cannot determine the appropriate backend to use for this request—for example, it might not be configured.	500, 404
http_response_timeout	The load balancer reached a configured time limit waiting for the complete response from the backend.	504, 408
http_request_error	The load balancer is generating a client (4xx) response on the client's behalf.	400, 403, 405, 406, 408, 411, 413, 414, 415, 416, 417, or 429
proxy_configuration_error	The load balancer encountered an error regarding its configuration.	500
http_protocol_error	The load balancer encountered an HTTP protocol error when communicating with the backend.	502

Interacting with the logs

You can interact with the external Application Load Balancer logs by using the Cloud Logging API. The Logging API provides ways to interactively filter logs that have specific fields set. It exports matching logs to Cloud Logging, Cloud Storage, BigQuery, or Pub/Sub. For more information about the Logging API, see Cloud Logging API overview.

Monitoring

The load balancer exports monitoring data to Cloud Monitoring.

You can use monitoring metrics to do the following:

• Evaluate a load balancer's configuration, usage, and performance
• Troubleshoot problems
• Improve resource utilization and user experience

In addition to the predefined dashboards in Cloud Monitoring, you can create custom dashboards, set up alerts, and query the metrics through the Cloud Monitoring API.

Defining alerting policies

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.

1. In the Google Cloud console, select Monitoring, and then select  Alerting, or click the following button:

   Go to Alerting

2. If you haven't created your notification channels and if you want to be notified, then click Edit Notification Channels and add your notification channels. Return to the Alerting page after you add your channels.
3. From the Alerting page, select Create policy.
4. To select the metric, expand the Select a metric menu and then do the following:
   1. To limit the menu to relevant entries, enter Regional External Application Load Balancer Rule into the filter bar. If there are no results after you filter the menu, then disable the Show only active resources & metrics toggle.
   2. For the Resource type, select Regional External Application Load Balancer Rule.
   3. Select a Metric category and a Metric, and then select Apply.
5. Click Next.
6. The settings in the Configure alert trigger page determine when the alert is triggered. Select a condition type and, if necessary, specify a threshold. For more information, see Create metric-threshold alerting policies.
7. Click Next.
8. Optional: To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu, and then click OK.
9. Optional: Update the Incident autoclose duration. This field determines when Monitoring closes incidents in the absence of metric data.
10. Optional: Click Documentation, and then add any information that you want included in a notification message.
11. Click Alert name and enter a name for the alerting policy.
12. Click Create Policy.

For more information, see Alerting policies.

Defining Cloud Monitoring custom dashboards

You can create custom Cloud Monitoring dashboards for the load balancer's metrics:

1. In the Google Cloud console, go to the Monitoring page.

   Go to Monitoring

2. Select Dashboards > Create Dashboard.

3. Click Add Chart, and then give the chart a title.

4. To identify the time series to be displayed, choose a resource type and metric type:

   1. In the Resource & Metric section, click the chart, and then in the Select a metric section, select from the available options:
   2. For a regional external Application Load Balancer, select the resource type Regional External Application Load Balancer Rule.
   3. Click Apply.

5. To specify monitoring filters, click Filters > Add filter.

6. Click Save.

Metric reporting frequency and retention

Metrics for the external Application Load Balancers are exported to Cloud Monitoring in 1-minute granularity batches. Monitoring data is retained for six (6) weeks. Metrics are based on sampled traffic (sampling rate is dynamic and cannot be adjusted). The dashboard provides data analysis in default intervals of 1H (one hour), 6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You can manually request analysis in any interval from 6W to 1 minute.

Monitoring metrics

You can monitor the following metrics for external Application Load Balancers.

The following metrics for regional external Application Load Balancers are reported into Cloud Monitoring. These metrics are prepended with loadbalancing.googleapis.com/.

Filter this table:

Metric	Name	Description
Request count	https/external/regional/request_count	The number of requests served by the regional external Application Load Balancer.
Request bytes count	https/external/regional/request_bytes	The number of bytes sent as requests from clients to the regional external Application Load Balancer.
Response bytes count	https/external/regional/response_bytes	The number of bytes sent as responses from the regional external Application Load Balancer to the client.
Total latencies	https/external/regional/total_latencies	A distribution of the latency, in milliseconds. Latency is measured from the time when the proxy receives the first byte of the request to the time when the proxy sends the last byte of the response.
Backend latencies	https/external/regional/backend_latencies	A distribution of the latency, in milliseconds. Latency is measured from the time when the proxy sends the first byte of the request to the backend to the time when the proxy receives the last byte of the response from the backend.

Filtering dimensions for metrics

You can apply filters for metrics for external Application Load Balancers.

Metrics are aggregated for each regional external Application Load Balancer. You can filter aggregated metrics by using the following dimensions for resource.type="http_external_regional_lb_rule".

Filter this table:

Property	Description
backend_name	The name of the backend instance group or NEG.
backend_scope	The scope of the backend (either a zone name or a region name). Might be UNKNOWN whenever backend_name is unknown.
backend_scope_type	The scope of the backend (REGION/ZONE). Might be UNKNOWN whenever backend_name is unknown.
backend_target_name	The name of the backend selected to handle the request, based on the URL map path rule or route rule that matches the request.
backend_target_type	The type of backend target. Can be BACKEND_SERVICE, or UNKNOWN if the backend wasn't assigned.
backend_type	The type of the backend group. Can be INSTANCE_GROUP, NETWORK_ENDPOINT_GROUP, or UNKNOWN if the backend wasn't assigned.
forwarding_rule_name	The name of the forwarding rule object.
matched_url_path_rule	The URL map path rule or route rule configured as part of the URL map key. Can be UNMATCHED or UNKNOWN as fallbacks. UNMATCHED refers to a request that matches no URL path rules, so it uses the default path rule. UNKNOWN indicates an internal error.
network_name	The name of the load balancer's VPC network.
project_id	The identifier of the Google Cloud project associated with this resource.
region	The region in which the load balancer is defined.
target_proxy_name	The name of the target proxy object referenced by the forwarding rule.
url_map_name	The name of the URL map object configured to select a backend service.

What's next

• Read conceptual information about external Application Load Balancers.