HTTP(S) Load Balancing Logging and Monitoring

This document provides you with the information you need to understand Cloud Logging and Cloud Monitoring for HTTP(S) Load Balancing.

Logging

You can enable, disable, and view logs for an HTTP(S) Load Balancing backend service.

You enable or disable logging on a per-backend service basis. If you have enabled logging, you can configure whether all requests or just a randomly sampled fraction are logged.

In addition, you must ensure that you don't have a logs exclusion that applies to external HTTP(S) load balancer. For instructions about how to verify that Cloud HTTP Load Balancer logs are allowed, see Viewing resource-type exclusions.

Enabling logging on a new backend service

Console

  1. Go to the Load balancing page in the Google Cloud Console.
    Go to the Load balancing page
  2. Click the name of your load balancer.
  3. Click Edit .
  4. Click Backend Configuration.
  5. Click Edit next to your backend service.
  6. Uncheck Enable logging to disable logging entirely.
  7. If you leave logging enabled, you can set a different Sample rate fraction. You can set it to anything from 0.0 through 1.0 (default). To cut the number of stored logs to 20%, set the value to 0.2.
  8. Click Update.

gcloud

gcloud compute backend-services create BACKEND_SERVICE \
 --global \
 --enable-logging \
 --logging-sample-rate=VALUE \
 ... other values

where

  • --global indicates that this is a global backend service.
  • --enable-logging enables logging for that backend service.
  • --logging-sample-rate=<var>VALUE</var> lets you specify a value from 0.0 through 1.0, where 0.0 means no packets are logged and 1.0 means 100% of packets are logged. Only meaningful with the --enable-logging parameter. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging.

Enabling logging on an existing backend service

Console

  1. Go to the Load balancing page in the Google Cloud Console.
    Go to the Load balancing page
  2. Click the name of your load balancer.
  3. Click Edit .
  4. Click Backend Configuration.
  5. Click Edit next to your backend service.
  6. Click Enable logging.
  7. Set a Sample rate fraction. You can set a rate to 0.0 through 1.0 (default).
  8. Click Update.

gcloud

gcloud compute backend-services update BACKEND_SERVICE \
    --global \
    --enable-logging \
    --logging-sample-rate=VALUE

where

  • --global indicates that this is a global backend service.
  • --enable-logging enables logging for that backend service.
  • --logging-sample-rate=<var>VALUE</var> lets you specify a value from 0.0 through 1.0, where 0.0 means no packets are logged and 1.0 means 100% of packets are logged. Only meaningful with the --enable-logging parameter. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging.

Disabling or modifying logging on an existing backend service

Console

  1. Go to the Load balancing page in the Google Cloud Console.
    Go to the Load balancing page
  2. Click the name of your load balancer.
  3. Click Edit .
  4. Click Backend Configuration.
  5. Click next to your backend service.
  6. Uncheck Enable logging to disable logging entirely.
  7. If you leave logging enabled, you can set a different Sample rate fraction. You can set the rate to 0.0 through 1.0 (default). To cut the number of stored logs to 20%, set the value to 0.2.
  8. Click Update.

gcloud

Disabling logging entirely

gcloud compute backend-services update BACKEND_SERVICE \
    --global \
    --no-enable-logging

where

  • --global indicates that this is a global backend service.
  • --no-enable-logging disables logging for that backend service.

Modifying the logging sample rate

gcloud compute backend-services update BACKEND_SERVICE \
 --global \
 --logging-sample-rate=VALUE

where

  • --global indicates that this is a global backend service.
  • --logging-sample-rate=<var>VALUE</var> lets you specify a value from 0.0 through 1.0, where 0.0 means no packets are logged and 1.0 means 100% of packets are logged. Only meaningful with the --enable-logging parameter. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging.

Viewing logs

To view logs, go to the Logs Viewer.

HTTP(S) logs are indexed first by forwarding rule, then by URL map.

  • To see all logs, in the first pull-down menu select Cloud HTTP Load Balancer > All forwarding rules.
  • To see logs for just one forwarding rule, select a single forwarding rule name from the list.
  • To see logs for just one URL map used by a forwarding rule, highlight a forwarding rule and choose the URL map of interest.

Log fields of type boolean typically only appear if they have a value of true. If a boolean field has a value of false, that field is omitted from the log.

UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks.

You can configure export of logs-based metrics for external HTTP(S) load balancer resource logs (resource.type=http_load_balancer). The metrics created are based on the "Google Cloud HTTP Load Balancing Rule (Logs-based Metrics)" resource (l7_lb_rule), which is available under Cloud Monitoring dashboards instead of under the https_lb_rule resource:

Go to Monitoring

What is logged

HTTP(S) Load Balancing log entries contain information useful for monitoring and debugging your HTTP(S) traffic. Log entries contain the following types of information:

  • General information shown in most logs, such as severity, project ID, project number, timestamp, and so on.
  • HttpRequest log fields. However, HttpRequest.protocol is not populated for HTTP(S) Load Balancing Cloud Logging logs.
  • A statusDetails field inside the structPayload. This field holds a string that explains why the load balancer returned the HTTP status that it did. The tables below contain further explanations of these log strings.
  • Redirects (HTTP response status code 302 Found) issued from the load balancer are not logged. Redirects issued from the backend instances are logged.

statusDetail HTTP success messages

statusDetails (successful) Meaning Common accompanying response codes
byte_range_caching The HTTP request was served using Cloud CDN byte range caching. Any cacheable response code is possible.
response_from_cache The HTTP request was served from a Cloud CDN cache. Any cachable response code is possible.
response_from_cache_validated The return code was set from a Cloud CDN cached entry that was validated by a backend. Any cachable response code is possible.
response_sent_by_backend The HTTP request was proxied successfully to the backend. Returned from backend - the HTTP response code is set by the software running on the backend.

statusDetail HTTP failure messages

statusDetails (failure) Meaning Common accompanying response codes
aborted_request_due_to_backend_early_response A request with body was aborted due to the backend sending an early response with an error code. The response was forwarded to the client. The request was terminated. 4XX or 5XX
backend_connection_closed_after_partial_response_sent The backend connection closed unexpectedly after a partial response had been sent to the client. The HTTP response code is set by the software running on the backend. HTTP response code 0 (zero) means that the backend sent incomplete HTTP headers.
backend_connection_closed_before_data_sent_to_client The backend unexpectedly closed its connection to the load balancer before the response was proxied to the client. This can happen if the load balancer is sending traffic to another entity, such as a third party load balancer running on a VM instance, that has a TCP timeout that is shorter than the external HTTP(S) load balancer's 10 minute (600 second) timeout. Manually setting the TCP timeout (keepalive) on the target service to greater than 600 seconds may fix this problem. 502
backend_early_response_with_non_error_status The backend sent a non-error response (1XX or 2XX) to a request before receiving the whole request body. 502
backend_interim_response_not_supported The backend sent an interim 1XX response to the request in a context where interim responses are not supported. 502
backend_response_corrupted The HTTP response body sent by the backend has invalid chunked transfer-encoding or is otherwise corrupted. Any response code possible depending on the nature of the corruption. Often 502.
backend_response_headers_too_long The HTTP response headers sent by the backend exceeded the allowed limit. See Header size for HTTP(S) Load Balancing limits for more information. 502
backend_timeout The backend timed out while generating a response. 502
body_not_allowed The client sent a HTTP request with a body, but the HTTP method used does not allow a body. 400
byte_range_caching_aborted The load balancer previously received a response that indicated the resource was cacheable and supported byte ranges, but Cloud CDN received an inconsistent response (for example, one with a response code other than the expected 206 Partial Content) when attempting to perform cache fill using a byte range request. The load balancer aborted the response to the client as a result. 2XX
byte_range_caching_forwarded_backend_response The load balancer previously received a response that indicated the resource was cacheable and supported byte ranges, but Cloud CDN received an inconsistent response (for example, one with a response code other than the expected 206 Partial Content) when attempting to perform cache fill using a byte range request. The load balancer then forwarded the inconsistent response to the client. Returned from backend - any response code is possible.
byte_range_caching_retrieval_abandoned The user canceled a byte range request or validation request initiated by Cloud CDN. Returned from backend - any response code is possible.
byte_range_caching_retrieval_from_backend_failed_after_partial_response A byte range request or validation request initiated by Cloud CDN encountered an error. Refer to the corresponding Cloud Logging log entry for the request initiated by Cloud CDN for the detailed backend status. 2XX
cache_lookup_failed_after_partial_response The load balancer failed to serve a full response from Cloud CDN cache due to an internal error. 2XX
cache_lookup_timeout_after_partial_response The Cloud CDN cache lookup stream timed out because the client did not retrieve the content in a timely manner. 2XX
client_disconnected_after_partial_response The connection to the client was broken after the load balancer sent a partial response. Returned from the backend - any response code is possible.
client_disconnected_before_any_response The connection to the client was broken before the load balancer sent any response. 0
client_timed_out The Google Front End (GFE) idled out the client connection due to lack of progress while it was proxying either the request or the response. 0 or 408
denied_by_security_policy The load balancer denied this request because of a Cloud Armor Security Policy. Configured in the Security Policy.
error_uncompressing_gzipped_body There was an error uncompressing a gzipped HTTP response. 503
failed_to_connect_to_backend The load balancer failed to connect to the backend. This includes timeouts during the connection phase. 502
failed_to_pick_backend The load balancer failed to pick a healthy backend to handle the request. 502
failed_to_negotiate_alpn The load balancer and the backend failed to negotiate an application layer protocol (such as HTTP/2) to use to communicate with each other over TLS. 502
headers_too_long The request headers were larger than the maximum allowed. 413
http_version_not_supported HTTP version not supported. Currently only HTTP 0.9, 1.0, 1.1, and 2.0 are supported. 400
internal_error Internal error at the load balancer. Normally represents a transient error in the load balancer infrastructure. Please try your query again. 4XX
invalid_external_origin_endpoint The configuration for the custom origin is invalid. Review the internet NEG configuration and ensure that it specifies a valid FQDN/IP address and port. 4XX
invalid_http2_client_header_format The HTTP/2 headers from a client are invalid. 400
multiple_iap_policies Multiple Identity-Aware Proxy (IAP) policies cannot be combined. If you have an IAP policy attached to a backend service and another policy attached to a serverless object (App Engine, Cloud Run (fully managed), or Cloud Functions), remove one of the policies and try again. 500
malformed_chunked_body The request body was improperly chunk encoded. 411
request_loop_detected The load balancer detected a request loop. This may have been caused by misconfiguration where the backend forwarded the request back to the load balancer. 502
required_body_but_no_content_length The HTTP request requires a body but the request headers did not include a content length or transfer-encoding chunked header. 400 or 403
secure_url_rejected A request with a https:// URL was received over a plaintext HTTP/1.1 connection. 400
ssl_san_verification_failed The load balancer could not find a SAN (Subject Alternative Name) in the SSL certificate presented by backend that matches configured hostname. 502
ssl_certificate_chain_verification_failed The SSL certificate presented by the backend failed SSL certificate verification. 502
unsupported_method The client supplied an unsupported HTTP request method. 400
upgrade_header_rejected The client HTTP request contained the Upgrade header and was refused. 400
uri_too_long The HTTP request URI was longer than the maximum allowed length. 414
websocket_closed The WebSocket connection was closed. 101
websocket_handshake_failed The WebSocket handshake failed. Any response code possible depending on the nature of the handshake failure.
request_body_too_large The HTTP request body exceeded the maximum supported by the backend. Not applicable for VM backends. 413
handled_by_identity_aware_proxy This response was generated by Identity-Aware Proxy during identity verification of the client before allowing access. 200, 302, 400, 401, 403, 500, 502
serverless_neg_routing_failed The serverless NEG request cannot be dispatched. This can happen when the region specified in the NEG cannot be reached, or when the resource name (for example, the Cloud Functions name) cannot be found. 404, 502

Logging for IP deny list/allow list

The following log entries in the Logs Viewer are for HTTP(S) IP denylist/allowlist logging and include the following structure in jsonPayload. HTTP request details appear in the httpRequest message.

  • status_details (string) - a textual description of the response code
  • enforced_security_policy - the security policy rule that was actually enforced
    • outcome (string) - ACCEPT, DENY or UNKNOWN_OUTCOME
    • configured_action (string) - same as outcome
    • name (string) - name of the security policy
    • priority (number) - matching rule priority
  • preview_security_policy - populated if request matches a rule configured for preview (present only when a preview rule would have taken priority over the enforced rule)
    • outcome (string) - ACCEPT, DENY or UNKNOWN_OUTCOME
    • configured_action (string) - same as outcome
    • name (string) - name of the security policy
    • priority (number) - matching rule priority

You can interact with the external HTTP(S) load balancer logs using the Cloud Logging API. The logging API provides ways to interactively filter logs that have specific fields set, and export matching logs to Cloud Logging, Cloud Storage, BigQuery, or Pub/Sub. For more information on the Cloud Logging API, see Viewing Logs.

Monitoring

HTTP(S) Load Balancing exports monitoring data to Cloud Monitoring.

Monitoring metrics can be used for the following purposes:

  • Evaluate a load balancer's configuration, usage, and performance
  • Troubleshoot problems
  • Improve resource utilization and user experience

In addition to the predefined dashboards in Cloud Monitoring, you can create custom dashboards, set up alerts, and query the metrics through the Cloud Monitoring API.

Viewing Cloud Monitoring dashboards

  1. Go to Monitoring in the Google Cloud Console.

    Go to Monitoring

  2. If the Monitoring navigation pane displays Services, then select Services > Cloud load balancers. To view the dashboard for a specific load balancer, locate the load balancer in the list and then click its name.

  3. Otherwise, select Dashboards:

    • To view a list of dashboards for your Google Cloud load balancers, select the dashboard named Google Cloud Load Balancers. To view a specific load balancer's dashboard, locate the load balancer in the list and click its name.

    • To view a list of dashboards for your AWS load balancers, select the dashboard named Amazon Web Services. To view a specific load balancer's dashboard, locate the load balancer in the list and click its name.

In the left pane, you can see various details for this external HTTP(S) load balancer. In the right pane you can see timeseries graphs. Click the Breakdowns link to see specific breakdowns.

Defining alerting policies

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition. The general steps for creating an alerting policy that monitors one or more Google Cloud HTTP/S Load Balancing Rule resources are as follows:

  1. In the Google Cloud Console, go to Monitoring.

    Go to Monitoring

  2. In the Monitoring navigation pane, select Alerting, and then select Create policy.
  3. Click Add condition:
    1. The settings in the Target pane specify the resource and metric to be monitored. Click the text box to enable a menu, and then select the resource Google Cloud HTTP/S Load Balancing Rule. Next, select a metric from the metrics list.
    2. The settings in the Configuration pane of the alerting policy determine when the alert is triggered. Most fields in this pane are populated with default values. For more information about the fields in the pane, see Configuration in the Alerting policies documentation.
    3. Click Add.
  4. To advance to the notifications section, click Next.
  5. Optional: To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu, and then click OK.

    If a notification channel that you want to add isn't listed, then click Manage notification channels. You are taken to the Notification channels page in a new browser tab. From this page, you can update the configured notification channels. After you have completed your updates, return to the original tab, click Refresh , and then select the notification channels to add to the alerting policy.

  6. To advance to the documentation section, click Next.
  7. Click Name and enter a name for the alerting policy.
  8. Optional: Click Documentation, and then add any information that you want included in a notification message.
  9. Click Save.
For more information, see Alerting policies.

Defining Cloud Monitoring custom dashboards

You can create custom Cloud Monitoring dashboards over HTTP(S) Load Balancing metrics:

  1. Go to Monitoring in the Google Cloud Console.
    Go to Monitoring
  2. Select Dashboards > Create Dashboard.
  3. Click on Add Chart.
  4. Give the chart a title.
  5. Select metrics and filters. For metrics, the resource type is Cloud HTTP Load Balancer.
  6. Click Save.

Metric reporting frequency and retention

Metrics for the external HTTP(S) load balancers are exported to Cloud Monitoring in 1-minute granularity batches. Monitoring data is retained for six (6) weeks. The dashboard provides data analysis in default intervals of 1H (one hour), 6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You can manually request analysis in any interval from 6W to 1 minute.

Monitoring metrics for external HTTP(S) load balancers

The following metrics for external HTTP(S) load balancers are reported into Cloud Monitoring.

Metric Name Description
Request count https/request_count The number of requests served by the external HTTP(S) load balancer
Request bytes count https/request_bytes_count The number of bytes sent as requests from users to the external HTTP(S) load balancer
Response bytes count https/response_bytes_count The number of bytes sent as responses from the external HTTP(S) load balancer to users
Total latencies https/total_latencies A distribution of the latency measured from the time the first byte of the request was received by the GFE until the GFE receives an ACK from the requesting client on the last response byte. Total latencies are measured by request/response, so pauses between requests on the same connection using Connection: keep-alive do not affect the measurement. This measurement is typically reduced to 95th percentile in Cloud Monitoring views.

Example: a load balancer has 1 request per second from the UK, all with 100ms latency, and 9 requests per second from the US, all with 50ms latency. Over a certain minute there were 60 requests from the UK and 540 requests from the US. Monitoring metrics preserves the distribution over all dimensions. You can request the following:
  • median overall latency (300/600) - 50ms
  • median UK latency (30/60) - 100ms
  • 95th percentile overall latency (570/600) - 100ms
  • and so on...
Frontend RTT(*) https/frontend_tcp_rtt A distribution of the smoothed RTT measured for each connection between the client and the GFE (measured by the GFE's TCP stack).
Backend latencies(*) https/backend_latencies A distribution of the latency measured from when the first request byte was sent by the GFE to the backend, until the GFE received from the backend the last byte of the response.
Response code class fraction Fraction of total external HTTP(S) load balancer responses that are in each response code class (2XX, 4XX, ...). In Cloud Monitoring, this value is only available on default dashboards. It is not available for custom dashboards. You can set alerts for it via the API.
Backend request count https/backend_request_count The number of requests sent from the external HTTP(S) load balancer to the backends.
Backend request bytes count https/backend_request_bytes_count The number of bytes sent as requests from the external HTTP(S) load balancer to the backends.
Backend response bytes count https/backend_response_bytes_count The number of bytes sent as responses from the backends to the HTTP(S) load balancer.

(*) The sum of Frontend RTT and Backend latencies is not guaranteed to be less than or equal to Total latencies. This is because although we poll RTT over the socket from the GFE to client at the time the HTTP response is ACKed, we rely on kernel reporting for some of these measurements, and we cannot guarantee that the kernel will have an RTT measurement for the given HTTP response. The end result is a smoothed RTT value that is also affected by previous HTTP responses, SYN/ACKs, and SSL handshakes that aren't affecting current HTTP request actual timings.

Filtering dimensions for external HTTP(S) load balancer metrics

Metrics are aggregated for each external HTTP(S) load balancer. You can filter aggregated metrics by the following dimensions(*).

Property Description
BACKEND SCOPE The Google Cloud scope (region or zone) of the backend service instance group that served the connection.

If no instance group was available or if the request was served by another entity, you will see one of the following values instead of the region or zone of the backend service instance group.
  • FRONTEND_5XX - An internal error occurred before the GFE could select a backend. The GFE returned 5XX to the client.
  • INVALID_BACKEND - The GFE could not find a healthy backend to assign the request to, so it returned a 5XX response to the requestor.
  • NO_BACKEND_SELECTED - An error or other interruption occurred before a backend could be selected.
  • SERVED_FROM_BACKEND_BUCKET - The request was handled by a backend bucket, not a backend service instance group.
  • SERVED_FROM_CACHE - The request was handled by a GFE cache, so no backend was assigned.
  • MULTIPLE_BACKENDS - The request was served by potentially multiple backends. The cache has sent one or more byte range requests to a different backend. Use the BACKEND_SCOPE breakdown to visualize each load balancer-to-backend request.

When this breakdown is chosen, the charts show backend metrics (load balancer-to-backends), not frontend metrics (client-to-load balancer).
BACKEND ZONE If the instance group was a zonal instance group, the Google Cloud zone of the instance group that served the user's request. (Examples: us-central1-a, europe-west1-b, asia-east1-c)

When this breakdown is chosen, the charts show backend metrics (load balancer-to-backends), not frontend metrics (client-to-load balancer).
BACKEND REGION If the instance group was a regional instance group, the Google Cloud region of the instance group that served the user's request. (Examples: us-central1, europe-west1, asia-east1)

When this breakdown is chosen, the charts show backend metrics (load balancer-to-backends), not frontend metrics (client-to-load balancer).
PROXY CONTINENT Continent of the HTTP(S) GFE that terminated the user HTTP(S) connection. (Examples: America, Europe, Asia)
INSTANCE GROUP The name of the instance group that served the user's request.

If no instance group was available or if the request was served by another entity, you will see one of the following values instead of an instance group.
  • FRONTEND_5XX - An internal error occurred before the GFE could select a backend. The GFE returned 5XX to the client.
  • INVALID_BACKEND - The GFE could not find a healthy backend to assign the request to, so returned a 5XX response to the requestor.
  • NO_BACKEND_SELECTED - An error or other interruption occurred before a backend could be selected.
  • INVALID_BACKEND - The GFE couldn't find a valid healthy backend, so it returned 5xx to the requestor.
  • SERVED_FROM_BACKEND_BUCKET - The request was handled by a backend bucket, not a backend service instance group.
  • SERVED_FROM_CACHE - The request was handled by Cloud CDN, so no backend was assigned.
  • MULTIPLE_BACKENDS - The request was served by multiple backends.

When this breakdown is chosen, the charts show backend metrics (load balancer-to-backends), not frontend metrics (client-to-load balancer).
BACKEND SERVICE The name of the backend service that served the user's request.
MATCHED URL RULE The URL map path rule that matched the prefix of the user HTTP(S) request (up to 50 characters).
FORWARDING RULE The name of the forwarding rule used by the client to send the request.

(*) Currently the "Response code class fraction" metric is available per entire load balancer only, with no further breakdowns available.

What's next