Logging
You can enable, disable, and view logs for an external Application Load Balancer backend service. For external Application Load Balancers with backend buckets, logging is automatically enabled and cannot be disabled.
You enable or disable logging for each backend service. You can configure whether to log all requests or a randomly sampled fraction.
You must ensure that you don't have a logs exclusion that applies to
external Application Load Balancers. For instructions about how to verify that Cloud HTTP Load
Balancer logs are allowed, see Viewing resource-type
exclusions.
Logs sampling and collection
The requests (and corresponding responses) handled by load balancer backend
virtual machine (VM) instances are sampled. These sampled requests are then
processed to generate logs. You control the fraction of the requests that are
emitted as log entries according to the
logConfig.sampleRate
parameter. When logConfig.sampleRate is 1.0 (100%), this means that logs are
generated for all of the requests and written to Cloud Logging.
Enabling logging on a new backend service
Console
In the Google Cloud console, go to the Load Balancing page.
Click the name of your load balancer.
Click Edit.
Click Backend Configuration.
Select Create a backend service.
Complete the required backend service fields.
In the Logging section, select the Enable logging checkbox.
Set a Sample rate fraction. You can set a number from
0.0through1.0, where0.0means that no requests are logged and1.0means that 100% of the requests are logged. The default value is1.0.To finish editing the backend service, click Update.
To finish editing the load balancer, click Update.
gcloud: Global mode
Create a backend service and enable logging by using the
gcloud compute backend-services create command.
gcloud compute backend-services create BACKEND_SERVICE \
--global \
--enable-logging \
--logging-sample-rate=VALUE \
--load-balancing-scheme=EXTERNAL_MANAGED
where
--globalindicates that the backend service is global. Use this field for backend services used with global external Application Load Balancers.--enable-loggingenables logging for that backend service.--logging-sample-ratelets you specify a value from0.0through1.0, where0.0means that no requests are logged and1.0means that 100% of the requests are logged. Only meaningful with the--enable-loggingparameter. Enabling logging but setting the sampling rate to0.0is equivalent to disabling logging. The default value is1.0.
gcloud: Classic mode
Create a backend service and enable logging by using the
gcloud compute backend-services create command.
gcloud compute backend-services create BACKEND_SERVICE \ --global \ --enable-logging \ --logging-sample-rate=VALUE \ --load-balancing-scheme=EXTERNAL
where
--globalindicates that the backend service is global. Use this field for backend services used with a classic Application Load Balancer.--enable-loggingenables logging for that backend service.--logging-sample-ratelets you specify a value from0.0through1.0, where0.0means that no requests are logged and1.0means that 100% of the requests are logged. Only meaningful with the--enable-loggingparameter. Enabling logging but setting the sampling rate to0.0is equivalent to disabling logging. The default value is1.0.
Enabling logging on an existing backend service
Console
In the Google Cloud console, go to the Load Balancing page.
Click the name of your load balancer.
Click Edit.
Click Backend Configuration.
Click Edit next to your backend service.
In the Logging section, select the Enable logging checkbox.
In the Sample rate field, set the sampling probability. You can set a number from
0.0through1.0, where0.0means that no requests are logged and1.0means that 100% of the requests are logged. The default value is1.0.To finish editing the backend service, click Update.
To finish editing the load balancer, click Update.
gcloud: Global mode
Enable logging on an existing backend service with the
gcloud compute backend-services update command.
gcloud compute backend-services update BACKEND_SERVICE \
--global \
--enable-logging \
--logging-sample-rate=VALUE
where
--globalindicates that the backend service is global. Use this field for backend services used with global external Application Load Balancers.--enable-loggingenables logging for that backend service.--logging-sample-ratelets you specify a value from0.0through1.0, where0.0means that no requests are logged and1.0means that 100% of the requests are logged. Only meaningful with the--enable-loggingparameter. Enabling logging but setting the sampling rate to0.0is equivalent to disabling logging. The default value is1.0.
gcloud: Classic mode
Enable logging on an existing backend service with the
gcloud compute backend-services update command.
gcloud compute backend-services update BACKEND_SERVICE \
--global \
--enable-logging \
--logging-sample-rate=VALUE
where
--globalindicates that the backend service is global. Use this field for backend services used with a classic Application Load Balancer.--enable-loggingenables logging for that backend service.--logging-sample-ratelets you specify a value from0.0through1.0, where0.0means that no requests are logged and1.0means that 100% of the requests are logged. Only meaningful with the--enable-loggingparameter. Enabling logging but setting the sampling rate to0.0is equivalent to disabling logging. The default value is1.0.
Disabling or modifying logging on an existing backend service
Console
In the Google Cloud console, go to the Load Balancing page.
Click the name of your load balancer.
Click Edit.
Click Backend Configuration.
Click Edit next to your backend service.
To disable logging entirely, in the Logging section, clear the Enable logging checkbox.
If you leave logging enabled, you can set a different Sample rate fraction. You can set a number from
0.0through1.0, where0.0means that no requests are logged and1.0means that 100% of the requests are logged. The default value is1.0. For example,0.2means 20% of the sampled requests generate logs.To finish editing the backend service, click Update.
To finish editing the load balancer, click Update.
gcloud: Global mode
Disable logging on a backend service with the
gcloud compute backend-services update command.
Disabling logging entirely
gcloud compute backend-services update BACKEND_SERVICE \
--global \
--no-enable-logging
where
--globalindicates that the backend service is global. Use this field for backend services used with global external Application Load Balancers.--regionindicates that the backend service is regional. Use this field for backend services used with regional external Application Load Balancers.--no-enable-loggingdisables logging for that backend service.
Modifying the logging sample rate
gcloud compute backend-services update BACKEND_SERVICE \ --global \ --logging-sample-rate=VALUE
gcloud: Classic mode
Disable logging on a backend service with the
gcloud compute backend-services update command.
Disabling logging entirely
gcloud compute backend-services update BACKEND_SERVICE \
--global \
--no-enable-logging
where
--globalindicates that the backend service is global. Use this field for backend services used with a classic Application Load Balancer.--no-enable-loggingdisables logging for that backend service.
Modifying the logging sample rate
gcloud compute backend-services update BACKEND_SERVICE \ --global \ --logging-sample-rate=VALUE
where
--globalindicates that the backend service is global. Use this field for backend services used with a classic Application Load Balancer.--logging-sample-ratelets you specify a value from0.0through1.0, where0.0means that no requests are logged and1.0means that 100% of the requests are logged. Only meaningful with the--enable-loggingparameter. Enabling logging but setting the sampling rate to0.0is equivalent to disabling logging.
View logs
To follow step-by-step guidance for this task directly in the Google Cloud console, click Guide me:
HTTP(S) logs are indexed first by a forwarding rule, then by a URL map.
To view logs, go to the Logs Explorer page:
To view all logs, in the Resource filter menu, select Cloud HTTP Load Balancer > All forwarding rules.
To view logs for one forwarding rule, select a single forwarding rule name.
To view logs for one URL map, select a forwarding rule, and then select a URL map.
Log fields of type boolean typically only appear if they have a value of true.
If a boolean field has a value of false, that field is omitted from the log.
UTF-8 encoding
is enforced for log fields. Characters that are not UTF-8
characters are replaced with question marks.
For classic Application Load Balancers and global external Application Load Balancers, you can export
logs-based metrics using
resource logs (resource.type="http_load_balancer"). The metrics
created are based on the "Application Load Balancer Rule
(Logs-based Metrics)" resource (l7_lb_rule), which is available under
Cloud Monitoring dashboards instead of under the https_lb_rule resource.
What is logged
External Application Load Balancer log entries contain information useful for monitoring and debugging your HTTP(S) traffic. Log records contain required fields, which are the default fields of every log record.
| Field | Field format | Field type: Required or Optional | Description |
|---|---|---|---|
| severity insertID logName |
LogEntry | Required | The general fields as described in a log entry. |
| timestamp | string (
|
Optional | The time when the first layer GFE receives the request. |
| httpRequest | HttpRequest | Required | A common protocol for logging HTTP requests.
|
| resource | MonitoredResource | Required | The MonitoredResource is the resource type associated with a log entry. The MonitoredResourceDescriptor
describes the schema of a |
| jsonPayload | object (Struct format) | Required | The log entry payload that is expressed as a JSON object. The JSON
object contains the following fields:
|
| string | Required | The statusDetails field
holds a string that explains why the load balancer returned
the HTTP status that it did. For more information about these log strings,
see statusDetails HTTP success messages and
statusDetails HTTP failure messages. |
|
| string | Required | The backendTargetProjectNumber field holds the project
number where the backend target—backend service or backend
bucket—has been created. This field is in the format:
"projects/PROJECT_NUMBER". This information is
only available for global external Application Load Balancers using custom error
responses. |
|
| integer | Required | The overrideResponseCode holds the override response code
applied to the response sent to the client. This information is
only available for global external Application Load Balancers using custom error
responses. |
|
| string | Required | The errorService field holds the backend service that
provided the custom error response. This information is
only available for global external Application Load Balancers using custom error
responses. |
|
| string | Required | The errorBackendStatusDetails field holds the
statusDetails of the final response served to the client.
This information is only available for global external Application Load Balancers using custom error
responses. |
|
| string | Optional | The loadBalancingScheme field is only populated for
customers using the classic Application Load Balancer migration
feature. This field holds a string that describes which load balancing
scheme was used to route the request. The possible values are either
EXTERNAL or EXTERNAL_MANAGED. |
Resource labels
The following table lists the resource labels for
resource.type="http_load_balancer".
| Field | Type | Description |
|---|---|---|
backend_service_name |
string | The name of the backend service. |
forwarding_rule_name |
string | The name of the forwarding rule object. |
project_id |
string | The identifier of the Google Cloud project associated with this resource. |
target_proxy_name |
string | The name of the target proxy object referenced by the forwarding rule. |
url_map_name |
string | The name of the URL map object configured to select a backend service. |
zone |
string | The zone in which the load
balancer is running. The zone is global. |
statusDetails HTTP success messages
| statusDetails (successful) | Meaning | Common accompanying response codes |
|---|---|---|
byte_range_caching
|
The HTTP request was served using Cloud CDN byte range caching. | Any cacheable response code is possible. |
response_from_cache
|
The HTTP request was served from a Cloud CDN cache. | Any cacheable response code is possible. |
response_from_cache_validated
|
The return code was set from a Cloud CDN cached entry that was validated by a backend. | Any cacheable response code is possible. |
response_sent_by_backend
|
The HTTP request was proxied successfully to the backend, and the response was returned by the backend. | The HTTP response code is set by the software running on the backend. |
statusDetails HTTP failure messages
| statusDetails (failure) | Meaning | Common accompanying response codes |
|---|---|---|
aborted_request_due_to_backend_early_response
|
A request with body was aborted due to the backend sending an early response with an error code. The response was forwarded to the client. The request was terminated. | 4XX or 5XX |
backend_connection_closed_after_partial_response_sent
|
The backend connection closed unexpectedly after a partial response had been sent to the client. | The HTTP response code is set by the software running on the backend. HTTP response code 0 (zero) means that the backend sent incomplete HTTP headers. The HTTP response code is 101 if the HTTP(S) connection was upgraded to a websocket connection. |
backend_connection_closed_before_data_sent_to_client
|
The backend unexpectedly closed its connection to the load balancer before the response was proxied to the client. | 502, 503 The HTTP response code is 101 if the HTTP(S) connection was upgraded to a websocket connection. |
backend_early_response_with_non_error_status
|
The backend sent a non-error response (1XX or 2XX) to a request before receiving the whole request body. | 502, 503 |
backend_interim_response_not_supported
|
The backend sent an interim 1XX response to the request in a context where interim responses are not supported. | 502, 503 |
backend_response_corrupted
|
The HTTP response body sent by the backend has invalid chunked transfer-encoding or is otherwise corrupted. | Any response code possible depending on the nature of the corruption. Often 502, 503. |
backend_response_headers_too_long
|
The HTTP response headers sent by the backend exceeded the allowed limit. See the Header size for external Application Load Balancers section for more information. | 502, 503 |
backend_timeout
|
The backend timed out while generating a response. For a websocket connection:
|
502, 503 The HTTP response code is 101 if the HTTP(S) connection was upgraded to a websocket connection. |
banned_by_security_policy
|
The request was banned by a Google Cloud Armor rate-based ban rule. | 429 |
body_not_allowed
|
The client sent an HTTP request with a body, but the HTTP method used does not allow a body. | 400 |
byte_range_caching_aborted
|
The load balancer previously received a response indicating that the resource was cacheable and supported byte ranges. Cloud CDN received an inconsistent response (for example, one with a response code other than the expected 206 Partial Content). This happened when attempting to perform cache fill using a byte range request. As a result, the load balancer aborted the response to the client. | 2XX |
byte_range_caching_forwarded_backend_response
|
The load balancer previously received a response indicating that the resource was cacheable and supported byte ranges. Cloud CDN received an inconsistent response (for example, one with a response code other than the expected 206 Partial Content). This happened when attempting to perform cache fill using a byte range request. The load balancer then forwarded the inconsistent response to the client. | Returned from the backend—any response code is possible. |
byte_range_caching_retrieval_abandoned
|
The client canceled a byte range request or validation request initiated by Cloud CDN. | Returned from the backend—any response code is possible. |
byte_range_caching_retrieval_from_backend_failed_after_partial_response
|
A byte range request or validation request initiated by Cloud CDN encountered an error. Refer to the corresponding Cloud Logging log entry for the request initiated by Cloud CDN for the detailed backend status. | 2XX |
cache_lookup_failed_after_partial_response
|
The load balancer failed to serve a full response from Cloud CDN cache due to an internal error. | 2XX |
cache_lookup_timeout_after_partial_response
|
The Cloud CDN cache lookup stream timed out because the client did not retrieve the content in a timely manner. | 2XX |
client_disconnected_after_partial_response
|
The connection to the client was broken after the load balancer sent a partial response. | Returned from the backend—any response code is possible. The HTTP response code is 101 if the HTTP(S) connection was upgraded to a websocket connection. |
client_disconnected_before_any_response
|
The connection to the client was broken before the load balancer sent any response. | 0 The HTTP response code is 101 if the HTTP(S) connection was upgraded to a websocket connection. |
client_timed_out
|
The Google Front End (GFE) idled out the client connection due to lack of progress while it was proxying either the request or the response. | 0 or 408 |
client_cert_invalid_rsa_key_size
|
A client leaf or intermediate certificate had an invalid RSA key size. For more information, see Logged errors for closed connections. | 0 |
client_cert_unsupported_elliptic_curve_key
|
A client or intermediate certificate is using an unsupported elliptic curve. For more information, see Logged errors for closed connections. | 0 |
client_cert_unsupported_key_algorithm
|
A client or intermediate certificate is using a non-RSA or non-ECDSA algorithm. For more information, see Logged errors for closed connections. | 0 |
client_cert_pki_too_large
|
The PKI to be used for validation has more than three intermediate certificates that share the same Subject and Subject Public Key Info. For more information, see Logged errors for closed connections. | 0 |
client_cert_chain_max_name_constraints_exceeded
|
An intermediate certificate provided for validation had more than ten name constraints. For more information, see Logged errors for closed connections. | 0 |
client_cert_chain_invalid_eku |
Either the client certificate or its issuer does not have
Extended Key Usage (EKU) that includes clientAuth. For more information, see
Logged errors for closed connections.
|
0 |
client_cert_validation_timed_out
|
Time limit exceeded while validating the certificate chain. For more information, see Logged errors for closed connections. | 0 |
client_cert_validation_search_limit_exceeded
|
The depth or iteration limit is reached while attempting to validate the certificate chain. For more information, see Logged errors for closed connections. | 0 |
client_cert_validation_not_performed
|
You have configured mTLS without setting up a TrustConfig.
For more information, see
Logged errors for closed connections.
|
0 |
client_cert_not_provided
|
The client did not provide the requested certificate during the handshake. For more information, see Logged errors for closed connections. | 0 |
client_cert_validation_failed
|
Client certificate fails validation with the TrustConfig when hashing algorithms such as MD4, MD5, and SHA-1 are used.
For more information, see
Logged errors for closed connections.
|
0 |
config_not_found
|
The load balancer is missing project configuration. This might occur intermittently, especially after you've made configuration changes that add a new resource. As there are two layers of proxies, sometimes, the first-layer GFE might fail to reach the second-layer GFE. This could be due to an internal error, such as an in-progress rollout, a load balancer overload, or intermittent configuration issues. |
404, 502, 503 |
direct_response
|
The load balancer overrode this request and returned a fixed response. | You might see any HTTP response code depending on the nature of the issue. For example, the HTTP 410 response code means that the backend is unavailable due to payment delinquency. |
denied_by_security_policy
|
The load balancer denied this request because of a Google Cloud Armor security policy. | Configured in the security policy. |
error_uncompressing_gzipped_body
|
There was an error uncompressing a gzipped HTTP response. | 502, 503 |
failed_to_connect_to_backend
|
The load balancer failed to connect to the backend. This includes timeouts during the connection phase. | 502, 503 |
failed_to_pick_backend
|
The load balancer failed to pick a healthy backend to handle the request. | 502, 503 |
failed_to_negotiate_alpn
|
The load balancer and the backend failed to negotiate an application layer protocol (such as HTTP/2) to use to communicate with each other over TLS. | 502, 503 |
headers_too_long
|
The request headers were larger than the maximum allowed. | 413 |
http_version_not_supported
|
HTTP version not supported. Currently only HTTP 0.9, 1.0, 1.1, and 2.0 are supported. | 400 |
internal_error
|
Internal error at the load balancer. Normally represents a transient error in the load balancer infrastructure. Retry your query. | 4XX |
invalid_external_origin_endpoint
|
The configuration for the external backend is invalid. Review the internet NEG configuration and ensure that it specifies a valid FQDN/IP address and port. | 4XX |
invalid_request_headers
|
The HTTP request headers received from a client contain at least one character that isn't allowed under an applicable HTTP specification. For example, header field names that include a double quotation mark
( For more information, see: |
400 |
invalid_http2_client_header_format
|
The HTTP/2 headers from a client are invalid. For more information, see
invalid_request_headers.
|
400 |
invalid_http2_client_request_path
|
The HTTP/2 request path from a client contains at least one character that isn't allowed under the URI specification. For more information, see the "3.3. Path" section of RFC 3986. |
400 |
multiple_iap_policies
|
Multiple Identity-Aware Proxy (IAP) policies cannot be combined. If you have an IAP policy attached to a backend service and another policy attached to a serverless object, remove one of the policies and try again. Serverless objects include App Engine, Cloud Run, and Cloud Run functions. | 500 |
malformed_chunked_body
|
The request body was improperly chunk encoded. | 411 |
request_loop_detected
|
The load balancer detected a request loop. This loop might be caused by a misconfiguration where the backend forwarded the request back to the load balancer. | 502, 503 |
required_body_but_no_content_length
|
The HTTP request requires a body but the request headers did not include a content length or transfer-encoding chunked header. | 400 or 403 |
secure_url_rejected
|
A request with an https:// URL was received over a plaintext HTTP/1.1 connection. | 400 |
ssl_certificate_san_verification_failed
|
The load balancer couldn't find a SAN (Subject Alternative Name) in the SSL certificate presented by backend that matches the configured hostname. | 502, 503 |
ssl_certificate_chain_verification_failed
|
The SSL certificate presented by the backend failed SSL certificate verification. | 502, 503 |
throttled_by_security_policy
|
The request was blocked by a Google Cloud Armor throttle rule. | 429 |
unsupported_method
|
The client supplied an unsupported HTTP request method. | 400 |
unsupported_100_continue
|
The client request included 'Expect: 100-continue' header on a protocol that doesn't support it. | 400 |
upgrade_header_rejected
|
The client HTTP request contained the Upgrade header and was refused. | 400 |
websocket_closed
|
The websocket connection was closed. | 101 |
websocket_handshake_failed
|
The websocket handshake failed. | Any response code possible depending on the nature of the handshake failure. |
request_body_too_large
|
The HTTP request body exceeded the maximum supported by the backend. Not applicable for VM backends. | 413 |
handled_by_identity_aware_proxy
|
This response was generated by Identity-Aware Proxy during identity verification of the client before allowing access. | 200, 302, 400, 401, 403, 500, 502, 503 |
serverless_neg_routing_failed
|
The serverless NEG request cannot be dispatched. This error can happen when the region specified in the NEG cannot be reached, or when the resource name (for example, the Cloud Run functions name) cannot be found. | 404, 502, 503 |
fault_filter_abort
|
This error can happen if the customer has configured a fault filter and the fault filter was triggered for the given request. | The value must be from 200 to 599. |
View logs for mTLS client certificate validation
To view the logged errors for closed connections during mutual TLS client certificate validation, complete the following steps.
Console query
In the Google Cloud console, go to the Logs Explorer page.
Click the Show query toggle.
Paste the following into the query field. Replace
FORWARDING_RULE_NAMEwith the name of your forwarding rule.jsonPayload.statusDetails=~"client_cert" jsonPayload.@type="type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry" resource.labels.forwarding_rule_name=FORWARDING_RULE_NAME
Click Run query.
Authorization policy request logs
The authz_info object in the Load Balancer Log Entry JSON payload contains
information about authorization policies. You can configure log-based metrics
for traffic allowed or denied by these policies.
| Field | Type | Description |
|---|---|---|
authz_info.policies[] |
object | The list of policies that match the request. |
authz_info.policies[].name |
string | The name of the authorization policy that matches the request. The name is empty for the following reasons:
|
authz_info.policies[].result |
enum | The result can be ALLOWED or DENIED. |
authz_info.policies[].details |
string | The details include the following:
|
authz_info.overall_result |
enum | The result can be ALLOWED or DENIED. |
Logging for backend buckets
Logging is automatically enabled for load balancers with backend buckets. You cannot modify or disable logging for backend buckets.
Logging for Google Cloud Armor
The table for statusDetail HTTP failure messages contains some messages that
apply to Google Cloud Armor. For more information about what
Google Cloud Armor logs, see Using request logging.
Logging for Shared VPC deployments
Application Load Balancer logs and metrics are typically exported to the project that has the forwarding rule. Therefore, service admins—owners or users of projects where the backend service is created—won't have access to the load balancer's logs and metrics by default. You can use IAM roles to grant these permissions to service admins. To learn more about the IAM roles that are available, and the steps to provide access, see Grant access to Monitoring.
Interacting with the logs
You can interact with the external Application Load Balancer logs by using the Cloud Logging API. The Logging API provides ways to interactively filter logs that have specific fields set. It exports matching logs to Cloud Logging, Cloud Storage, BigQuery, or Pub/Sub. For more information about the Logging API, see Cloud Logging API overview.
Monitoring
The load balancer exports monitoring data to Cloud Monitoring.
You can use monitoring metrics to do the following:
- Evaluate a load balancer's configuration, usage, and performance
- Troubleshoot problems
- Improve resource utilization and user experience
In addition to the predefined dashboards in Cloud Monitoring, you can create custom dashboards, set up alerts, and query the metrics through the Cloud Monitoring API.
Viewing predefined Cloud Monitoring dashboards
Cloud Monitoring provides predefined dashboards to monitor your load balancers. These dashboards are automatically populated by Cloud Monitoring.
Perform the following steps to access the predefined dashboards:
Go to Monitoring in the Google Cloud console.
In the Monitoring navigation panel, click Dashboards.
Under Categories, click GCP.
To view a list of dashboards for all your Google Cloud load balancers, select the dashboard named Google Cloud Load Balancers. To view a specific load balancer's dashboard, locate the load balancer in the list and click its name.
To view the predefined dashboards for only your external Application Load Balancers, select the dashboard named External HTTP(S) Load Balancers. This page displays a dashboard that shows the 5XX response ratios and backend latency for all external Application Load Balancers in your project. It also provides a list of dashboards for all the external Application Load Balancers in your project.
You can click through to each load balancer's dashboard. Each dashboard includes the following:- Pre-populated graphs that display breakdowns for responses by response code classes (5XX, 4XX, 3XX, 2XX)
- Total latency
- Backend latency
- Frontend RTT
- Request count
- A link to the logs for the load balancer
To view dashboards for third-party services, go back to the Dashboards page. Under Categories, click Other.
- To view a specific third-party service dashboard, locate it in the list and click its name.
Defining alerting policies
To follow step-by-step guidance for this task directly in the Google Cloud console, click Guide me:
You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.
-
In the Google Cloud console, go to the notifications Alerting page:
If you use the search bar to find this page, then select the result whose subheading is Monitoring.
- If you haven't created your notification channels and if you want to be notified, then click Edit Notification Channels and add your notification channels. Return to the Alerting page after you add your channels.
- From the Alerting page, select Create policy.
- To select the metric, expand the Select a metric menu and then do the following:
- To limit the menu to relevant entries, enter
Global External Application Load Balancer Ruleinto the filter bar. If there are no results after you filter the menu, then disable the Show only active resources & metrics toggle. - For the Resource type, select Global External Application Load Balancer Rule.
- Select a Metric category and a Metric, and then select Apply.
- To limit the menu to relevant entries, enter
- Click Next.
- The settings in the Configure alert trigger page determine when the alert is triggered. Select a condition type and, if necessary, specify a threshold. For more information, see Create metric-threshold alerting policies.
- Click Next.
- Optional: To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu, and then click OK.
- Optional: Update the Incident autoclose duration. This field determines when Monitoring closes incidents in the absence of metric data.
- Optional: Click Documentation, and then add any information that you want included in a notification message.
- Click Alert name and enter a name for the alerting policy.
- Click Create Policy.
Defining Cloud Monitoring custom dashboards
You can create custom Cloud Monitoring dashboards for the load balancer's metrics:
In the Google Cloud console, go to the Monitoring page.
Select Dashboards > Create Dashboard.
Click Add Chart, and then give the chart a title.
To identify the time series to be displayed, choose a resource type and metric type:
- In the Resource & Metric section, click the chart, and then
in the Select a metric section, select from the available options:
- For a global external Application Load Balancer, select the resource type Global External Application Load Balancer Rule.
- Click Apply.
- In the Resource & Metric section, click the chart, and then
in the Select a metric section, select from the available options:
To specify monitoring filters, click Filters > Add filter.
Click Save.
Metric reporting frequency and retention
Metrics for the external Application Load Balancers are exported to Cloud Monitoring in 1-minute granularity batches. Monitoring data is retained for six (6) weeks.
The dashboard provides data analysis in default intervals of 1H (one hour), 6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You can manually request analysis in any interval from 6W to 1 minute.
Monitoring metrics
You can monitor the following metrics for external Application Load Balancers.
The following metrics for global external Application Load Balancers are
reported into Cloud Monitoring.
These metrics are prepended with loadbalancing.googleapis.com/.
| Metric | Name | Description |
|---|---|---|
| Request count | https/request_count |
The number of requests served by the external Application Load Balancer |
| Request bytes count | https/request_bytes_count |
The number of bytes sent as requests from clients to the external Application Load Balancer |
| Response bytes count | https/response_bytes_count |
The number of bytes sent as responses from the external Application Load Balancer to clients |
| Total latencies | https/total_latencies |
A distribution of the latency. Latency is the time between the first byte of the request received to the last byte of the response sent by the GFE. Total latencies are measured by request/response. Pauses between requests
on the same connection that use For websocket connections, this field refers to the entire time duration of the connection.† Example: A load balancer has 1 request per second from the UK, all with 100 ms latency, and 9 requests per second from the US, all with 50 ms latency. Over a certain minute there were 60 requests from the UK and 540 requests from the US. Monitoring metrics preserves the distribution over all dimensions. You can request information such as the following:
|
| Frontend RTT* | https/frontend_tcp_rtt |
A distribution of the smoothed round trip time (RTT) measured for each connection between the client and the GFE (measured by the GFE's TCP stack). Smoothed RTT is an algorithm that deals with variations and anomalies that may occur in RTT measurements. |
| Backend latencies* | https/backend_latencies |
A distribution of the latency measured from when the first request byte was sent by the GFE to the backend, until the GFE received from the backend the last byte of the response. For websocket connections, backend latencies apply to the entire duration of the websocket session.† |
| Response code class fraction | Fraction of total external Application Load Balancer responses that are in each response code class (2XX, 4XX, ...). In Cloud Monitoring, this value is only available on default dashboards. It is not available for custom dashboards. You can use the API to set alerts for it. | |
| Backend request count | https/backend_request_count |
The number of requests sent from the external Application Load Balancer to the backends. |
| Backend request bytes count | https/backend_request_bytes_count |
The number of bytes sent as requests from the external Application Load Balancer to the backends. |
| Backend response bytes count | https/backend_response_bytes_count |
The number of bytes sent as responses from the backends (including cache) to the external Application Load Balancer. |
* The sum of Frontend RTT and Backend latencies is not guaranteed to be less than or equal to Total latencies. This is because although we poll RTT over the socket from the GFE to the client at the time the HTTP response is ACKed, we rely on kernel reporting for some of these measurements, and we cannot guarantee that the kernel will have an RTT measurement for the given HTTP response. The end result is a smoothed RTT value that is also affected by previous HTTP responses, SYN/ACKs, and SSL handshakes that aren't affecting current HTTP request actual timings.
† For monitoring websocket connections, create a backend service specifically for websockets.
Filtering dimensions for metrics
You can apply filters for metrics for external Application Load Balancers.
Metrics are aggregated for each classic Application Load Balancer and global external Application Load Balancer. You can filter
aggregated metrics by the following dimensions for
resource.type="http_load_balancer" or resource.type="https_lb_rule". Note
that not all dimensions are available on all metrics.
| Property | Description |
|---|---|
backend_scope |
The Google Cloud scope
(region or zone)
of the backend service instance group that served the connection. If no instance group was available or if the request was served by another entity, you will see one of the following values instead of the region or zone of the backend service instance group.
When this breakdown is chosen, the charts show backend metrics (load balancer-to-backends), not frontend metrics (client-to-load balancer). |
backend_type |
The name of the backend group that served the client's request.
Can be
|
backend_target_type |
The name of the backend service that served the request. Can be
BACKEND_SERVICE, BACKEND_BUCKET,
UNKNOWN if the backend wasn't assigned, or
NO_BACKEND_SELECTED if an error or other interruption occurred before a
backend could be selected or a URL redirect happened. |
matched_url_path_rule |
The URL map path rule that matched the prefix of the HTTP(S) request (up to 50 characters). |
forwarding_rule_name |
The name of the forwarding rule used by the client to send the request. |
url_map_name |
The URL map path rule or route rule configured as part of the URL map
key. Can be
|
target_proxy_name |
The name of the target HTTP(S) proxy object referenced by the forwarding rule. |
backend_target_name |
The name of the backend target. The target can be either a backend
service or backend bucket. UNKNOWN is returned if a backend
wasn't assigned.
|
backend_name |
The name of the backend instance group, bucket, or NEG.
UNKNOWN is returned if the backend wasn't assigned, or
NO_BACKEND_SELECTED if an error or other interruption occurred
before a backend could be selected or a URL redirect happened. |
backend_scope_type |
The type of the scope of the backend group. Can be
|
proxy_continent |
Continent of the HTTP(S) GFE that terminated the HTTP(S) connection.
(Examples: America, Europe, Asia) |
protocol |
Protocol used by the client, one of HTTP/1.0,
HTTP/1.1, HTTP/2.0, QUIC/HTTP/2.0,
UNKNOWN. |
response_code |
The HTTP response code of the request. |
response_code_class |
The HTTP response code class of the request: 200, 300, 400, 500 or 0 for none. |
cache_result |
Cache result for serving HTTP request by proxy: HIT,
MISS, DISABLED, PARTIAL_HIT (for a
request served partially from cache and partially from backend) or
UNKNOWN. |
client_country |
Country of the client that issued the HTTP request (for example,
United States or Germany) |
load_balancing_scheme |
The load balancing scheme used. If classic Application Load Balancer is used, the
value is EXTERNAL. If global external Application Load Balancer is used, the value
is EXTERNAL_MANAGED. |