HTTP(S) Load Balancing Logging and Monitoring

This document provides you with the information you need to understand Stackdriver logging and monitoring for HTTP(S) Load Balancing.

Logging

Alpha

This is an Alpha release of GCP HTTP(S) Load Balancing Logging. This feature might be changed in backward-incompatible ways and is not recommended for production use. It is not subject to any SLA or deprecation policy.

Each HTTP(S) request is logged temporarily via Stackdriver Logging. During the Alpha testing phase, logging is automatic and does not need to be enabled.

How to view logs

To view logs, go to the Logs Viewer.

HTTP(S) logs are indexed first by forwarding rule, then by URL map.

• To see all logs, in the first pull-down menu select Cloud HTTP Load Balancer > All forwarding_rule_name.
• To see logs for just one forwarding rule, select a single forwarding rule name from the list.
• To see logs for just one URL map used by a forwarding rule, highlight a forwarding rule and choose the URL map of interest.

Log fields of type boolean typically only appear if they have a value of true. If a boolean field has a value of false, that field is omitted from the log.

UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks.

You can configure export of Stackdriver logs based metrics for Cloud HTTP(S) Load Balancer resource logs (resource.type=http_load_balancer). The metrics created are based on the "Google Cloud HTTP Load Balancing Rule (Logs-based Metrics)" resource (l7_lb_rule), which is available under Stackdriver monitoring dashboards instead of under the https_lb_rule resource.

What is logged

HTTP(S) Load Balancing log entries contain information useful for monitoring and debugging your HTTP(S) traffic. Log entries contain the following types of information:

• General information shown in most GCP logs, such as severity, project ID, project number, timestamp, and so on.
• HttpRequest log fields. However, HttpRequest.protocol and HttpRequest.latency are not populated for HTTP(S) Load Balancing Stackdriver logs.
• a statusDetails field inside the structPayload. This field holds a string that explains why the load balancer returned the HTTP status that it did. The tables below contain further explanations of these log strings.

statusDetail HTTP success messages

statusDetails (successful)	Meaning	Common Accompanying Response Codes
byte_range_caching	The HTTP request was served using byte range caching.	Any cachable response code is possible.
response_from_cache	The HTTP request was served from cache.	Any cachable response code is possible.
response_from_cache_validated	The return code was set from a cached entry that was validated by a backend.	Any cachable response code is possible.
response_sent_by_backend	The HTTP request was proxied successfully to the backend.	Returned from VM backend - any response code is possible.

statusDetail HTTP failure messages

statusDetails (failure)	Meaning	Common Accompanying Response Codes
aborted_request_due_to_backend_early_response	A request with body was aborted due to backend sending an early response with an error code. The response was forwarded to the client. The request was terminated.	4XX or 5XX
backend_connection_closed_after_partial_response_sent	The backend connection closed unexpectedly after a partial response had been sent to the client.	Returned from VM backend - any response code is possible. A 0 indicates the backend did not send full response headers.
backend_connection_closed_before_data_sent_to_client	The backend unexpectedly closed its connection to the load balancer before the response was proxied to the client. This can happen if the load balancer is sending traffic to another entity, such as a third party load balancer running on a VM instance, that has a TCP timeout that is shorter than the HTTP(S) load balancer's 10 minute (600 second) timeout. Manually setting the TCP timeout (keepalive) on the target service to greater than 600 seconds may fix this problem.	502
backend_early_response_with_non_error_status	The backend sent a non-error response (1XX or 2XX) to a request before receiving the whole request body.	502
backend_interim_response_not_supported	The backend sent an interim 1XX response to the request in a context where interim responses are not supported.	502
backend_response_corrupted	The HTTP response body sent by the backend has invalid chunked transfer-encoding or is otherwise corrupted.	Any response code possible depending on the nature of the corruption. Often 502.
backend_timeout	The backend timed out while generating a response.	502
body_not_allowed	The client sent a HTTP request with a body, but the HTTP method used does not allow a body.	400
byte_range_caching_aborted	The load balancer aborted a response due to inconsistent byte ranges.	2XX
byte_range_caching_forwarded_backend_response	The load balancer failed to serve a response using byte range caching, but was able to serve a response from the backend.	Returned from backend - any response code is possible.
byte_range_caching_retrieval_from_backend_failed_after_partial_response	A byte range request or validation request initiated by Cloud CDN encountered an error. Refer to the corresponding Stackdriver Logging log entry for the request initiated by Cloud CDN for the detailed backend status.	2XX
cache_lookup_failed_after_partial_response	The load balancer failed to serve a full response from cache due to an internal error.	2XX
client_disconnected_after_partial_response	The connection to the client was broken after the load balancer sent a partial response.	Returned from the VM backend - any response code is possible.
client_disconnected_before_any_response	The connection to the client was broken before the load balancer sent any response.	0
client_timed_out	The load balancer idled out the client connection due to lack of progress while proxying either the request or response.	0 or 408
denied_by_security_policy	The load balancer denied this request because of a Cloud Armor Security Policy.	Configured in the Security Policy.
error_uncompressing_gzipped_body	There was an error uncompressing a gzipped HTTP response.	503
failed_to_connect_to_backend	The load balancer failed to connect to the backend.	502
failed_to_pick_backend	The load balancer failed to pick a healthy backend to handle the request.	502
failed_to_negotiate_alpn	The load balancer and the backend failed to negotiate an application layer protocol (such as HTTP/2) to use to communicate with each other over TLS.	502
headers_too_long	The request headers were larger than the maximum allowed.	413
http_version_not_supported	HTTP version not supported. Currently only HTTP 0.9, 1.0, 1.1, and 2.0 are supported.	400
internal_error	Internal error at the load balancer.	4XX
invalid_http2_client_header_format	The HTTP/2 headers from client are invalid.	400
malformed_chunked_body	The request body was improperly chunk encoded.	411
required_body_but_no_content_length	The HTTP request requires a body but the request headers did not include a content length or transfer-encoding chunked header.	400 or 403
secure_url_rejected	A request with a https:// URL was received over a plaintext HTTP/1.1 connection.	400
unsupported_method	The client supplied an unsupported HTTP request method.	400
upgrade_header_rejected	The client HTTP request contained the Upgrade header and was refused.	400
uri_too_long	The HTTP request URI was longer than the maximum allowed length.	414
websocket_closed	The websocket connection was closed.
websocket_handshake_failed	The websocket handshake failed.	501

Logging for IP deny list/allow list

The following log entries in the Logs Viewer are for HTTP(S) IP deny list/ allow list logging and include the following structure in jsonPayload. HTTP request details appear in the httpRequest message.

• status_details (string) - a textual description of the response code
• enforced_security_policy - the security policy rule that was actually enforced
  • outcome (string) - ACCEPT, DENY or UNKNOWN_OUTCOME
  • configured_action (string) - same as outcome
  • name (string) - name of the security policy
  • priority (number) - matching rule priority
• preview_security_policy - populated if request hit a rule configured for preview (present only when a preview rule would have taken priority over the enforced rule)
  • outcome (string) - ACCEPT, DENY or UNKNOWN_OUTCOME
  • configured_action (string) - same as outcome
  • name (string) - name of the security policy
  • priority (number) - matching rule priority

You can interact with the HTTP(S) Load Balancer logs using the Stackdriver logging API. The logging API provides ways to interactively filter logs that have specific fields set, and export matching logs to Stackdriver Console, Google Cloud Storage, BigQuery, or Cloud Pub/Sub. For more information on the Stackdriver logging API, see Viewing Logs.

Monitoring

HTTP(S) Load Balancing exports monitoring data to Stackdriver.

Monitoring metrics can be used for the following purposes:

• Evaluate a load balancer's configuration, usage, and performance
• Troubleshoot problems
• Improve resource utilization and user experience

In addition to the predefined dashboards in Stackdriver, you can create custom dashboards, set up alerts, and query the metrics through the Stackdriver monitoring API.

Viewing Stackdriver monitoring dashboards

1. Go to Stackdriver in the Google Cloud Platform Console.
   Go to Stackdriver
2. Select Resources > Google Cloud Load Balancers.
3. Click the name of your load balancer.

In the left pane, you can see various details for this HTTP(S) load balancer. In the right pane you can see timeseries graphs. Click the Breakdowns link to see specific breakdowns.

Defining Stackdriver alerts

You can define Stackdriver alerts over various HTTP(S) Load Balancing metrics:

1. Go to Stackdriver in the Google Cloud Platform Console.
   Go to Stackdriver
2. Select Alerting > Create a Policy.
3. Click on Add Condition and select condition type.
4. Select metrics and filters. For metrics, the resource type is HTTP(S) Load Balancer.
5. Click Save Condition.
6. Enter policy name and click Save Policy.

Defining Stackdriver custom dashboards

You can create custom Stackdriver dashboards over HTTP(S) Load Balancing metrics:

1. Go to Stackdriver in the Google Cloud Platform Console.
   Go to Stackdriver
2. Select Dashboards > Create Dashboard.
3. Click on Add Chart.
4. Give the chart a title.
5. Select metrics and filters. For metrics, the resource type is HTTP(S) Load Balancer.
6. Click Save.

Metric reporting frequency and retention

Metrics for the HTTP(S) load balancers are exported to Stackdriver in 1-minute granularity batches. Monitoring data is retained for six (6) weeks. The dashboard provides data analysis in default intervals of 1H (one hour), 6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You can manually request analysis in any interval from 6W to 1 minute.

Monitoring metrics for HTTP(S) load balancers

The following metrics for HTTP(S) load balancers are reported into Stackdriver:

Metric	Description
Request count	The number of requests served by the HTTP(S) load balancer
Request bytes count	The number of bytes sent as requests from users to the HTTP(S) load balancer
Response bytes count	The number of bytes sent as responses from the HTTP(S) load balancer to users
Total latencies	A distribution of the latency measured from the time the first byte of the request was received by the load balancer proxy until the proxy receives an ACK from the requesting client on the last response byte. Total latencies are measured by request/response, so pauses between requests on the same connection using `Connection: keep-alive` do not affect the measurement. This measurement is typically reduced to 95th percentile in Stackdriver views. Example: a load balancer has 1 request per second from the UK, all with 100ms latency, and 9 requests per second from the US, all with 50ms latency. Over a certain minute there were 60 requests from the UK and 540 requests from the US. Monitoring metrics preserves the distribution over all dimensions. You can request the following: median overall latency (300/600) - 50ms median UK latency (30/60) - 100ms 95th percentile overall latency (570/600) - 100ms and so on...
Frontend RTT(*)	A distribution of the smoothed RTT measured for each connection between client and proxy (measured by the proxy's TCP stack). Typically reduced to 95th percentile in Stackdriver views.
Backend latencies(*)	A distribution of the latency measured from when the first request byte was sent by the proxy to the backend, until the proxy received from the backend the last byte of the response. Typically reduced to 95th percentile in Stackdriver views.
Response code class fraction	Percentage of total HTTP(S) load balancer responses that are in each response code class (2XX, 4XX, ...). In Stackdriver, this value is only available on default dashboards. It is not available for custom dashboards. You can set alerts for it via the API.
Backend request count	The number of requests sent from the HTTP(S) load balancer to the backends.
Backend request bytes count	The number of bytes sent as requests from the HTTP(S) load balancer to the backends.
Backend response bytes count	The number of bytes sent as responses from the backends to the HTTP(S) load balancer.

(*) The sum of Frontend RTT and Backend latencies is not guaranteed to be less than or equal to Total latencies. This is because although we poll RTT over the socket from proxy to client at the time the HTTP response is ACKed, we rely on kernel reporting for some of these measurements, and we cannot guarantee that the kernel will have an RTT measurement for the given HTTP response. The end result is a smoothed RTT value that is also affected by previous HTTP responses, SYN/ACKs, and SSL handshakes that aren't affecting current HTTP request actual timings.

Filtering dimensions for HTTP(S) load balancer metrics

Metrics are aggregated for each HTTP(S) load balancer. You can filter aggregated metrics by the following dimensions(*):

Property	Description
BACKEND SCOPE	The GCP scope (region or zone) of the backend service instance group that served the connection. If no instance group was available or if the request was served by another entity, you will see one of the following values instead of the region or zone of the backend service instance group. FRONTEND_5XX - An internal error occurred before the proxy could select a backend. Proxy returned 5XX to the client. INVALID_BACKEND - Proxy could not find a healthy backend to assign the request to, so returned a 5XX response to the requestor. NO_BACKEND_SELECTED - An error or other interruption occurred before a backend could be selected. SERVED_FROM_BACKEND_BUCKET - The request was handled by a backend bucket, not a backend service instance group. SERVED_FROM_CACHE - The request was handled by a proxy cache, so no backend was assigned. MULTIPLE_BACKENDS - The request was served by multiple backends. When this breakdown is chosen, the charts show backend metrics (load balancer-to-backends), not frontend metrics (client-to-load balancer).
BACKEND ZONE	If the instance group was a zonal instance group, the GCP zone of the instance group that served the user request. (Examples: us-central1-a, europe-west1-b, asia-east1-c) When this breakdown is chosen, the charts show backend metrics (load balancer-to-backends), not frontend metrics (client-to-load balancer).
BACKEND REGION	If the instance group was a regional instance group, the GCP region of the instance group that served the user request. (Examples: us-central1, europe-west1, asia-east1) When this breakdown is chosen, the charts show backend metrics (load balancer-to-backends), not frontend metrics (client-to-load balancer).
PROXY CONTINENT	Continent of the HTTP(S) proxy that terminated the user HTTP(S) connection. (Examples: America, Europe, Asia)
INSTANCE GROUP	The name of the instance group that served the user request. If no instance group was available or if the request was served by another entity, you will see one of the following values instead of an instance group. FRONTEND_5XX - An internal error occurred before the proxy could select a backend. Proxy returned 5XX to the client. INVALID_BACKEND - Proxy could not find a healthy backend to assign the request to, so returned a 5XX response to the requestor. NO_BACKEND_SELECTED - An error or other interruption occurred before a backend could be selected. SERVED_FROM_BACKEND_BUCKET - The request was handled by a backend bucket, not a backend service instance group. SERVED_FROM_CACHE - The request was handled by a proxy cache, so no backend was assigned. MULTIPLE_BACKENDS - The request was served by multiple backends. When this breakdown is chosen, the charts show backend metrics (load balancer-to-backends), not frontend metrics (client-to-load balancer).
BACKEND SERVICE	The name of the backend service that served the user request.
MATCHED URL RULE	The URL map path rule that matched the prefix of the user HTTP(S) request (up to 50 characters).
FORWARDING RULE	The name of the forwarding rule used by the client to send the request.

(*) Currently the "Response code class fraction" metric is available per entire load balancer only, with no further breakdowns available.

What's next

• Read conceptual information about HTTP(S) Load Balancing
• Create a cross-region HTTP(S) load balancer
• Create a content-based HTTP(S) load balancer
• Add a backend bucket to a content-based load balancer
• Read about forwarding rules
• Read about setting up URL maps or read conceptual information about URL maps
• Configure SSL certificates
• Create SSL policies