Internal Application Load Balancer logging and monitoring

This document provides you with the information that you need to understand logging and monitoring metrics for internal Application Load Balancers. The logging and monitoring metrics for both regional internal Application Load Balancers and cross-region internal Application Load Balancers are the same.

Logging

You can enable logging on a per-backend service basis. A single internal Application Load Balancer's URL map can reference more than one backend service. You might need to enable logging for multiple backend services depending on your configuration.

Logs sampling and collection

Google Cloud samples the packets that leave and enter load balancer backend virtual machine (VM) instances. Those sampled packets are processed to generate logs.

Not every packet is sampled. Google Cloud samples a variable subset of packets depending on the amount of traffic on the physical host. The lowest possible sampling rate is one out of 1,024 packets. The sampling rate is dynamically controlled by Google Cloud. You cannot adjust the sampling rate.

The packet sampling interacts with firewall rules in the following ways:

  • Packets are sampled before egress firewall rules are applied.
  • Packets are sampled after ingress firewall rules are applied.

After packet sampling, Google Cloud processes the sampled packets according to the following procedure:

  1. Aggregation: Sampled packets are aggregated over a five-second interval to produce a single flow entry.

  2. Configurable (secondary) log sampling: This is a second sampling process, sampling the flows. You control the fraction of the flow entries that are emitted as log entries according to the logConfig.sampleRate parameter. When logConfig.sampleRate is 1.0 (100%), this means that all of the sampled packets are processed.

  3. Write to logging: The log entries are written to Cloud Logging.

Optional fields

Log records contain required fields and optional fields. The What is logged section lists which fields are optional and which are required. All required fields are always included. You can customize which optional fields you keep.

  • If you select include all optional, all optional fields in the log record format are included in the flow logs. When new optional fields are added to the record format, the flow logs automatically include the new fields.

  • If you select exclude all optional, all optional fields are omitted.

  • If you select custom, you can specify the optional fields that you want to include, such as tls.protocol,tls.cipher.

For instructions about customizing optional fields, see Enable logging on an existing backend service.

Enabling logging on an existing backend service

For regional internal Application Load Balancers, use the following steps:

Console

  1. In the Google Cloud console, go to the Load balancing page.

    Go to Load balancing

  2. Click the name of your load balancer.

  3. Click Edit.

  4. Click Backend Configuration.

  5. Click Edit next to your backend service.

  6. Click Advanced configurations (Session affinity, connection draining timeout).

  7. Click Enable logging.

  8. Set a Sample rate fraction. You can set a number from 0.0 through 1.0, where 0.0 means that no requests are logged and 1.0 means that 100% of the requests are logged. The default value is 1.0.

  9. Optional: To include all the optional fields in the logs, in the Optional fields section, click Include all optional fields.

  10. To finish editing the backend service, click Update.

  11. To finish editing the load balancer, click Update.

gcloud

To update the backend service to enable logging, use the gcloud compute backend-services update command.

gcloud compute backend-services update BACKEND_SERVICE \
    --enable-logging \
    --logging-sample-rate=RATE \
    --region=REGION \
    --logging-optional=LOGGING_OPTIONAL_MODE \
    --logging-optional-fields=OPTIONAL_FIELDS

where

  • --enable-logging enables logging for that backend service.
  • --logging-sample-rate lets you specify a value from 0.0 through 1.0, where 0.0 means no requests are logged and 1.0 means 100% of requests are logged. Only meaningful with the --enable-logging parameter. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.
  • --logging-optional lets you specify the optional fields that you want to include in the logs:

    • INCLUDE_ALL_OPTIONAL to include all optional fields.

    • EXCLUDE_ALL_OPTIONAL (default) to exclude all optional fields.

    • CUSTOM to include a custom list of optional fields that you specify in OPTIONAL_FIELDS.

  • --logging-optional-fields lets you specify a comma-separated list of optional fields that you want to include in the logs.

    For example, tls.protocol,tls.cipher can only be set if LOGGING_OPTIONAL_MODE is set to CUSTOM.

For cross-region internal Application Load Balancers, use the following steps:

Console

  1. In the Google Cloud console, go to the Load balancing page.

    Go to Load balancing

  2. Click the name of your load balancer.

  3. Click Edit.

  4. Click Backend Configuration.

  5. Click Edit next to your backend service.

  6. Click Advanced configurations (Session affinity, connection draining timeout).

  7. Click Enable logging.

  8. Set a Sample rate fraction. You can set a number from 0.0 through 1.0, where 0.0 means that no requests are logged and 1.0 means that 100% of the requests are logged. The default value is 1.0.

  9. Optional: To include all the optional fields in the logs, in the Optional fields section, click Include all optional fields.

  10. To finish editing the backend service, click Update.

  11. To finish editing the load balancer, click Update.

gcloud

To update the backend service to enable logging, use the gcloud compute backend-services update command.

gcloud compute backend-services update BACKEND_SERVICE \
    --enable-logging \
    --logging-sample-rate=RATE \
    --global \
    --logging-optional=LOGGING_OPTIONAL_MODE \
    --logging-optional-fields=OPTIONAL_FIELDS

where

  • --enable-logging enables logging for that backend service.
  • --logging-sample-rate lets you specify a value from 0.0 through 1.0, where 0.0 means no requests are logged and 1.0 means 100% of requests are logged. Only meaningful with the --enable-logging parameter. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging. The default value is 1.0.
  • --logging-optional lets you specify the optional fields that you want to include in the logs:

    • INCLUDE_ALL_OPTIONAL to include all optional fields.

    • EXCLUDE_ALL_OPTIONAL (default) to exclude all optional fields.

    • CUSTOM to include a custom list of optional fields that you specify in OPTIONAL_FIELDS.

  • --logging-optional-fields lets you specify a comma-separated list of optional fields that you want to include in the logs.

    For example, tls.protocol,tls.cipher can only be set if LOGGING_OPTIONAL_MODE is set to CUSTOM.

After you enable logging on the backend service, each HTTP(S) request is logged by using Cloud Logging.

How to view logs

To view logs, in the Google Cloud console, go to the Logs Explorer page.

Internal Application Load Balancer logs are indexed first by network and then by region.

  • To see logs for all internal Application Load Balancers, in the first pull-down menu, select Internal Application Load Balancer Rule.
  • To see logs for only one network, select Internal Application Load Balancer Rule, and then select the name of a network.
  • To see logs for just one region of the network, select Internal Application Load Balancer Rule > NETWORK > REGION.

Log fields of type boolean typically only appear if they have a value of true. If a boolean field has a value of false, that field is omitted from the log.

UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks.

You can configure export of logs-based metrics for resource logs (resource.type="internal_http_lb_rule"). The metrics created are based on the "Internal Application Load Balancer Rule" resource, which is available under Cloud Monitoring dashboards:

Go to Monitoring

What is logged

Internal Application Load Balancer log entries contain information useful for monitoring and debugging your HTTP(S) traffic. Log records contain required fields, which are the default fields of every log record, and optional fields that add additional information about your HTTP(S) traffic. Optional fields can be omitted to save storage costs. Log entries contain the following types of information:

  • General information shown in most Google Cloud logs, such as severity, project ID, project number, and timestamp as described in the LogEntry.
  • HttpRequest log fields.

Some log fields are in a multi-field format, with more than one piece of data in a given field. For example, the tls field is of the TlsDetails format, which contains the TLS protocol and TLS cipher in a single field. These multi-field fields are described in the following record format table.

Field Type Field type: Required or Optional Description
logName string Required The resource name of the log to which this log entry belongs.
In the form "projects/PROJECT_ID/logs/requests".
timestamp string Required The time at which the request began.
severity LogSeverity format Required The severity of the log entry. Defaults to LogSeverity.DEFAULT.
httpRequest HttpRequest object Required An HttpRequest proto that describes the HTTP(S) request being logged.
trace string Required The resource name of the trace associated with the log entry, if any. If it contains a relative resource name, the name is assumed to be relative to https://tracing.googleapis.com. Example: projects/PROJECT_ID/traces/06796866738c859f2f19b7cfb3214824.

Internal Application Load Balancers don't support this field.

spanId string Required The span ID within the trace associated with the log entry. For Trace spans, this string has the same format that the Trace API v2 uses: a 16-character hexadecimal encoding of an 8-byte array, such as 000000000000004a.

Internal Application Load Balancers don't support this field.

resource MonitoredResource object Required

The monitored resource that produced this log entry.

The MonitoredResourceDescriptor object describes the schema of a MonitoredResource object by using a type name and a set of labels.

For example, monitored resource descriptors for internal Application Load Balancers have a resource type of internal_http_lb_rule and use resource labels to identify the actual resource and its attributes. For a list of resource labels, see the Resource labels for resource.type="internal_http_lb_rule".

jsonPayload object (Struct format) Required The log entry payload that is expressed as a JSON object. The JSON object contains the following fields:
  • tls
  • proxyStatus
  • backendTargetProjectNumber
  • serviceDirectoryService
  • cloudFitExperiment
  • cloudFitFault
  • serviceExtensionInfo
  • mtls
string Required

The proxyStatus field holds a string that specifies why the internal Application Load Balancer returned the HttpRequest.status. This field is only populated when the proxy returns an error code with 4xx or 5xx.

The field is not logged if the value is an empty string. For more information, see proxyStatus message.

string Required The backendTargetProjectNumber field holds the project number that identifies the owner of the backend service or backend bucket.
string Required The serviceDirectoryService field holds the name of the Service Directory service on which the Cloud FIT fault was configured.
string Required The cloudFitExperiment field holds the name of the Cloud FIT experiment.
string Required The cloudFitFault field holds the name of the fault injected by a Cloud FIT fault experiment in this request path.
ServiceExtensionInfo Required The serviceExtensionInfo field stores information about the gRPC streams from the load balancer to Service Extensions. For more information, see what is logged for callout extensions.
TlsDetails Optional The tls field holds the TlsDetails that specifies the TLS metadata for the connection between the client and the internal Application Load Balancer. This field is only available if the client is using TLS/SSL encryption.
MtlsDetails Optional The mtls field holds the MtlsDetails value that specifies the mTLS metadata for the connection between the client and the internal Application Load Balancer. This field is only available if the load balancer uses frontend mutual TLS (mTLS).

TlsDetails field format

Field Field format Field type: Required or Optional Description
protocol string Optional TLS protocol that clients use to establish a connection with the load balancer. Possible values can be TLS 1.0, 1.1, 1.2, 1.3, or QUIC. This value is set to NULL if the client is not using TLS/SSL encryption.
cipher string Optional TLS cipher that clients use to establish a connection with the load balancer. This value is set to NULL if the client is not using HTTP(S) or the client is not using TLS/SSL encryption.

MtlsDetails field format

Field Field format Field type: Required or Optional Description
clientCertPresent bool Optional

true if the client has provided a certificate during the TLS handshake; otherwise, false.

clientCertChainVerified bool Optional

true if the client certificate chain is verified against a configured TrustStore; otherwise, false.

clientCertError string Optional

Predefined strings representing the error conditions. For more information about the error strings, see mTLS client validation modes.

clientCertSha256Fingerprint string Optional

Base64-encoded SHA-256 fingerprint of the client certificate.

clientCertSerialNumber string Optional

The serial number of the client certificate. If the serial number is longer than 50 bytes, the string client_cert_serial_number_exceeded_size_limit is added to client_cert_error, and the serial number is set to an empty string.

clientCertValidStartTime string Optional

Timestamp (RFC 3339 date string format) before which the client certificate is not valid. For example, 2022-07-01T18:05:09+00:00.

clientCertValidEndTime string Optional

Timestamp (RFC 3339 date string format) after which the client certificate is not valid. For example, 2022-07-01T18:05:09+00:00.

clientCertSpiffeId string Optional

The SPIFFE ID from the subject alternative name (SAN) field. If the value is not valid or exceeds 2048 bytes, the SPIFFE ID is set to an empty string.

If the SPIFFE ID is longer than 2048 bytes, the string client_cert_spiffe_id_exceeded_size_limit is added to client_cert_error.

clientCertUriSans string Optional

Comma-separated Base64-encoded list of the SAN extensions of type URI. The SAN extensions are extracted from the client certificate. The SPIFFE ID is not included in the client_cert_uri_sans field.

If the client_cert_uri_sans field is longer than 512 bytes, the string client_cert_uri_sans_exceeded_size_limit is added to client_cert_error, and the comma-separated list is set to an empty string.

clientCertDnsnameSans string Optional

Comma-separated Base64-encoded list of the SAN extensions of type DNSName. The SAN extensions are extracted from the client certificate.

If the client_cert_dnsname_sans field is longer than 512 bytes, the string client_cert_dnsname_sans_exceeded_size_limit is added to client_cert_error, and the comma-separated list is set to an empty string.

clientCertIssuerDn string Optional

Base64-encoded full Issuer field from the certificate.

If the client_cert_issuer_dn field is longer than 512 bytes, the string client_cert_issuer_dn_exceeded_size_limit is added to client_cert_error, and client_cert_issuer_dn is set to an empty string.

clientCertSubjectDn string Optional

Base64-encoded full Subject field from the certificate.

If the client_cert_subject_dn field is longer than 512 bytes, the string client_cert_subject_dn_exceeded_size_limit is added to client_cert_error, and client_cert_subject_dn is set to an empty string.

clientCertLeaf string Optional

The client leaf certificate for an established mTLS connection where the certificate passed validation. Certificate encoding is compliant with RFC 9440: the binary DER certificate is encoded using Base64 (without line breaks, spaces, or other characters outside the Base64 alphabet) and delimited with colons on either side.

If client_cert_leaf exceeds 16 KB unencoded, the string client_cert_validated_leaf_exceeded_size_limit is added to client_cert_error, and client_cert_leaf is set to an empty string.

clientCertChain string Optional

The comma-delimited list of certificates, in standard TLS order, of the client certificate chain for an established mTLS connection where the client certificate passed validation, not including the leaf certificate. Certificate encoding is compliant with RFC 9440.

If the combined size of client_cert_leaf and client_cert_chain before Base64 encoding exceeds 16 KB, the string client_cert_validated_chain_exceeded_size_limit is added to client_cert_error, and client_cert_chain is set to an empty string.

Resource labels

The following table lists the resource labels for resource.type="internal_http_lb_rule".

Field Type Description
network_name string The name of the load balancer's VPC network.
project_id string The identifier of the Google Cloud project associated with this resource.
region string The region in which the load balancer is defined.
url_map_name string The name of the URL map object configured to select a backend service.
forwarding_rule_name string The name of the forwarding rule object.
target_proxy_name string The name of the target proxy object referenced by the forwarding rule.
matched_url_path_rule string The URL map path rule or route rule configured as part of the URL map key. Can be UNMATCHED or UNKNOWN as fallbacks.
  • UNMATCHED refers to a request that matches no URL path rules, so it uses the default path rule.
  • UNKNOWN indicates an internal error.
backend_target_name string The name of the backend selected to handle the request, based on the URL map path rule or route rule that matches the request.
backend_target_type string The type of backend target (BACKEND_SERVICE / UNKNOWN).
backend_name string The name of the backend instance group or NEG.
backend_type string

The type of backend, either an instance group or a NEG, or unknown.

Cloud Logging logs requests when the backend_type is UNKNOWN even if logging is disabled. For example, if a client closes the connection to the load balancer before the load balancer can pick a backend, the backend_type is set to UNKNOWN and the request is logged. These logs provide useful debugging information about client requests that were closed because the load balancer could not select a backend.

backend_scope string The scope of the backend, either a zone name or a region name. Might be UNKNOWN whenever backend_name is unknown.
backend_scope_type string The scope of the backend (REGION/ZONE). Might be UNKNOWN whenever backend_name is unknown.

proxyStatus messages

The strings logged in the proxyStatus field are:

proxyStatus Meaning Common accompanying response codes
destination_unavailable The load balancer considers the backend to be unavailable; For example, recent attempts to communicate with the backend has failed, or a health check may indicate failure. 500, 503
connection_timeout The load balancer's attempt to open a connection to the backend has timed out. 504
connection_terminated The load balancer's connection to the backend was closed before a complete response was received. 502, 503
connection_refused The load balancer's connection to the backend was refused. 502, 503
connection_limit_reached The load balancer is configured to limit the number of connections it has to the backend, and that limit has been exceeded. 502, 503
destination_not_found The load balancer cannot determine the appropriate backend to use for this request; for example, it may not be configured. 500, 404
dns_error The load balancer encountered a DNS error when trying to find an IP address for the backend hostname. 502, 503
http_response_timeout The load balancer reached a configured time limit waiting for the complete response from the backend. 504, 408
http_request_error The load balancer is generating a client (4xx) response on the client's behalf. 400, 403, 405, 406, 408, 411, 413, 414, 415, 416, 417, or 429
proxy_configuration_error The load balancer encountered an error regarding its configuration. 500
http_protocol_error The load balancer encountered an HTTP protocol error when communicating with the backend. 502
proxy_internal_error The load balancer encountered an internal error. 500, 502
proxy_internal_response The load balancer generated the response without attempting to connect to the backend. You might see any HTTP status code depending on the type of problem. For example, the HTTP `410` status code means that the backend is unavailable due to payment delinquency.

View logs for mTLS client certificate validation

To view the logged errors for closed connections during mutual TLS client certificate validation, complete the following steps.

Console

  1. In the Google Cloud console, go to the Logs Explorer page.

    Go to Logs Explorer

  2. Click the Show query toggle to enable the query editor.

  3. Paste the following into the Query field. Replace FORWARDING_RULE_NAME with the name of your forwarding rule.

    jsonPayload.statusDetails=~"client_cert"
    jsonPayload.@type="type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
    resource.labels.forwarding_rule_name=FORWARDING_RULE_NAME
    
  4. Click Run query.

Monitoring

Internal Application Load Balancers export monitoring data to Monitoring.

Monitoring metrics can be used for the following purposes:

  • Evaluating a load balancer's configuration, usage, and performance
  • Troubleshooting problems
  • Improving resource utilization and user experience

In addition to the predefined dashboards in Monitoring, you can create custom dashboards, set up alerts, and query the metrics through the Monitoring API.

Viewing Cloud Monitoring metrics

Console

To view the metrics for a monitored resource by using the Metrics Explorer, do the following:

  1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Metrics explorer:

    Go to Metrics explorer

  2. In the Metric element, expand the Select a metric menu, enter Internal Application Load Balancer Rule in the filter bar, and then use the submenus to select a specific resource type and metric:
    1. In the Active resources menu, select Internal Application Load Balancer Rule.
    2. To select a metric, use the Active metric categories and Active metrics menus.
    3. Click Apply.
  3. To remove time series from the display, use the Filter element.

  4. To combine time series, use the menus on the Aggregation element. For example, to display the CPU utilization for your VMs, based on their zone, set the first menu to Mean and the second menu to zone.

    All time series are displayed when the first menu of the Aggregation element is set to Unaggregated. The default settings for the Aggregation element are determined by the metric type you selected.

  5. For quota and other metrics that report one sample per day, do the following:
    1. In the Display pane, set the Widget type to Stacked bar chart.
    2. Set the time period to at least one week.

Defining alerting policies

Console

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.

  1. In the navigation panel of the Google Cloud console, select Monitoring, and then select  Alerting:

    Go to Alerting

  2. If you haven't created your notification channels and if you want to be notified, then click Edit Notification Channels and add your notification channels. Return to the Alerting page after you add your channels.
  3. From the Alerting page, select Create policy.
  4. To select the metric, expand the Select a metric menu and then do the following:
    1. To limit the menu to relevant entries, enter Internal Application Load Balancer Rule into the filter bar. If there are no results after you filter the menu, then disable the Show only active resources & metrics toggle.
    2. For the Resource type, select Internal Application Load Balancer Rule.
    3. Select a Metric category and a Metric, and then select Apply.
  5. Click Next.
  6. The settings in the Configure alert trigger page determine when the alert is triggered. Select a condition type and, if necessary, specify a threshold. For more information, see Create metric-threshold alerting policies.
  7. Click Next.
  8. Optional: To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu, and then click OK.
  9. Optional: Update the Incident autoclose duration. This field determines when Monitoring closes incidents in the absence of metric data.
  10. Optional: Click Documentation, and then add any information that you want included in a notification message.
  11. Click Alert name and enter a name for the alerting policy.
  12. Click Create Policy.
For more information, see Alerting policies.

Defining Monitoring custom dashboards

Console

You can create custom Monitoring dashboards over internal Application Load Balancer metrics:

  1. In the Google Cloud console, go to the Monitoring page.

    Go to Monitoring

  2. Select Dashboards > Create Dashboard.

  3. Click Add Chart.

  4. Give the chart a title.

  5. Select metrics and filters. For metrics, the resource type is Internal HTTP/S Load Balancer.

  6. Click Save.

Metric reporting frequency and retention

Metrics for the load balancers are exported to Monitoring in 1-minute granularity batches. Monitoring data is retained for six (6) weeks. The dashboard provides data analysis in default intervals of 1H (one hour), 6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You can manually request analysis in any interval from 6W to 1 minute.

Monitoring metrics for internal Application Load Balancers

The following metrics for internal Application Load Balancers are reported into Monitoring:

Metric FQDN Description
Request count loadbalancing.googleapis.com/https/internal/request_count The number of requests served by the internal Application Load Balancer.
Request bytes count loadbalancing.googleapis.com/https/internal/request_bytes The number of bytes sent as requests from clients to the internal Application Load Balancer.
Response bytes count loadbalancing.googleapis.com/https/internal/response_bytes The number of bytes sent as responses from the internal HTTP(S) load balancer to the client.
Total latencies loadbalancing.googleapis.com/https/internal/total_latencies A distribution of the latency, in milliseconds. Latency is measured from the time when the proxy receives the first byte of the request, to the time when the proxy sends the last byte of the response.
Backend latencies loadbalancing.googleapis.com/https/internal/backend_latencies A distribution of the latency, in milliseconds. Latency is measured from the time when the proxy sends the first byte of the request to the backend, to the time when the proxy receives the last byte of the response from the backend.

Filtering dimensions for internal Application Load Balancer metrics

Metrics are aggregated for each internal Application Load Balancer. You can filter aggregated metrics by the following dimensions.

Property Description
BACKEND_SCOPE The Google Cloud zone or region of the backend group that served the client request, or a special string for cases in which the backend group wasn't assigned. Examples: us-central1-a, europe-west1-b, asia-east1, UNKNOWN.
PROXY_REGION Region of the internal Application Load Balancer, client, and backend. Examples: us-central1, europe-west1 or asia-east1.
BACKEND The name of the backend instance group or NEG that served the client request.
BACKEND_TARGET The name of the backend service that served the client request.
MATCHED_URL_RULE The URL map path rule or route rule that matched the prefix of the client HTTP(S) request (up to 50 characters).

The Response code class fraction metric is supported for the entire load balancer. No further granularity is supported.

What's next