Internal HTTP(S) Load Balancing logging and monitoring

This document provides you with the information you need to understand logging and monitoring metrics for Internal HTTP(S) Load Balancing.

Logging

You can enable logging on a per-backend service basis. A single internal HTTP(S) load balancer's URL map can reference more than one backend service, so you might need to enable logging for more than one backend service, depending on your configuration.

Enabling logging on an existing backend service

Console

  1. Go to the Load balancing page in the Google Cloud Console.
    Go to the Load balancing page
  2. Click the name of your load balancer.
  3. Click Edit .
  4. Click Backend Configuration.
  5. Click Edit next to your backend service.
  6. Click Advanced configurations (Session affinity, connection draining timeout).
  7. Click Enable logging.
  8. Set a Sample rate fraction. You can set it to anything from 0.0 through 1.0 (default).
  9. Click Update.

gcloud

Update the backend service to enable logging with the gcloud compute instance-templates create command.

gcloud compute backend-services update BACKEND_SERVICE \
    --enable-logging \
    --logging-sample-rate=RATE \
    --region=REGION

where

  • --enable-logging enables logging for that backend service.
  • --logging-sample-rate lets you specify a value from 0.0 through 1.0, where 0.0 means no requests are logged and 1.0 means 100% of requests are logged. Only meaningful with the --enable-logging parameter. Enabling logging but setting the sampling rate to 0.0 is equivalent to disabling logging.

After you enable logging on the backend service, each HTTP(S) request is logged by using Cloud Logging.

How to view logs

To view logs, go to the Logs Viewer.

internal HTTP(S) load balancer logs are indexed first by network and then by region.

  • To see logs for all internal HTTP(S) load balancers, in the first pull-down menu select Internal HTTP/S Load Balancing Rule.
  • To see logs for only one network, select Internal HTTP/S Load Balancing Rule, and then select the name of a network.
  • To see logs for just one region of the network, select Internal HTTP/S Load Balancing Rule > NETWORK > REGION.

Log fields of type boolean typically only appear if they have a value of true. If a boolean field has a value of false, that field is omitted from the log.

UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks.

You can configure export of logs-based metrics for resource logs (resource.type="internal_http_lb_rule). The metrics created are based on the "Internal HTTP/S Load Balancing Rule (Logs-based Metrics)" resource, which is available under Cloud Monitoring dashboards:

Go to Monitoring

What is logged

Internal HTTP(S) Load Balancing log entries contain information useful for monitoring and debugging your HTTP(S) traffic. Log entries contain the following types of information:

  • General information shown in most Google Cloud logs, such as severity, project ID, project number, timestamp, and so on.
  • HttpRequest log fields.

Log fields

Logs are LogEntry protos.

Field Type Description
logName string The resource name of the log to which this log entry belongs.
In the form "projects/PROJECT_ID/logs/requests".
timestamp Timestamp format The time at which the request began.
severity LogSeverity format The severity of the log entry. Defaults to LogSeverity.DEFAULT.
httpRequest HttpRequest object An HttpRequest proto that describes the HTTP(S) request being logged.
trace string The resource name of the trace associated with the log entry, if any. If it contains a relative resource name, the name is assumed to be relative to https://tracing.googleapis.com. Example: projects/PROJECT_ID/traces/06796866738c859f2f19b7cfb3214824.
spanId string The span ID within the trace associated with the log entry. For Trace spans, this is the same format that the Trace API v2 uses: a 16-character hexadecimal encoding of an 8-byte array, such as 000000000000004a.
resource MonitoredResource object The monitored resource associated with this log entry. See MonitoredResource object.

MonitoredResource

Field Type Description
networkName string The name of the load balancer's VPC network.
region string The name of the load balancer's Google Cloud region.
urlMapName string The name of the URL map object configured to select a backend service.
forwardingRuleName string The name of the forwarding rule object.
targetProxyName string The name of the target proxy object referenced by the forwarding rule.
matchedUrlPathRule string The URL map path rule or route rule configured as part of the URL map key. Can be UNMATCHED or UNKNOWN as fallbacks.
  • UNMATCHED refers to a request that matches no URL path rules, and so uses the default path rule.
  • UNKNOWN indicates an internal error.
backendTargetName string The name of the backend selected to handle the request, based on the URL map path rule or route rule that matches the request.
backendTargetType string The type of the backend target (BACKEND_SERVICE / UNKNOWN).
backendName string The name of the backend instance group or NEG.
backendType string The type of the backend, either instance group or NEG.
backendScope string The scope of the backend (either zone name or region name). Might be UNKNOWN whenever backendName is unknown.
backendScopeType string The scope of the backend (REGION/ZONE). Might be UNKNOWN whenever backendName is unknown.

Monitoring

Internal HTTP(S) Load Balancing exports monitoring data to Monitoring.

Monitoring metrics can be used for the following purposes:

  • Evaluating a load balancer's configuration, usage, and performance
  • Troubleshooting problems
  • Improving resource utilization and user experience

In addition to the predefined dashboards in Monitoring, you can create custom dashboards, set up alerts, and query the metrics through the Monitoring API.

Viewing Cloud Monitoring metrics

To view the metrics for a monitored resource using Metrics Explorer, do the following:

  1. In the Google Cloud Console navigation pane, select Monitoring:
    Go to Google Cloud Console

    If this is the first access of Cloud Monitoring for this Google Cloud project, then Cloud Monitoring creates a Workspace. Typically, this process is automatic and completes within a few minutes. If prompted to either select a Workspace or to create a Workspace, select create.

  2. In the Monitoring navigation pane, click Metrics Explorer.
  3. Ensure Metric is the selected tab.
  4. Click in the box labeled Find resource type and metric, and then select from the menu or enter the name for the resource and metric. Use the following information to complete the fields for this text box:
    1. For the Resource, select or enter internal_http_lb_rule. This selects the resource Internal HTTP/S Load Balancing Rule.
    2. For the Metric, select from the menu.
  5. Use the Filter, Group By, and Aggregator menus to modify how the data is displayed. For example, you can group by resource or metric labels. For more information, see Selecting metrics.

Defining alerting policies

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition. The general steps for creating an alerting policy that monitors one or more Internal HTTP/S Load Balancer resources are as follows:

  1. In the Google Cloud Console, go to Monitoring.

    Go to Monitoring

  2. In the Monitoring navigation pane, select Alerting, and then select Create policy.
  3. Click Add condition:
    1. The settings in the Target pane specify the resource and metric to be monitored. Click the text box to enable a menu, and then select the resource Internal HTTP/S Load Balancer. Next, select a metric from the metrics list.
    2. The settings in the Configuration pane of the alerting policy determine when the alert is triggered. Most fields in this pane are populated with default values. For more information about the fields in the pane, see Configuration in the Alerting policies documentation.
    3. Click Add.
  4. To advance to the notifications section, click Next.
  5. Optional: To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu, and then click OK.

    If a notification channel that you want to add isn't listed, then click Manage notification channels. You are taken to the Notification channels page in a new browser tab. From this page, you can update the configured notification channels. After you have completed your updates, return to the original tab, click Refresh , and then select the notification channels to add to the alerting policy.

  6. To advance to the documentation section, click Next.
  7. Click Name and enter a name for the alerting policy.
  8. Optional: Click Documentation, and then add any information that you want included in a notification message.
  9. Click Save.
For more information, see Alerting policies.

Defining Monitoring custom dashboards

You can create custom Monitoring dashboards over Internal HTTP(S) Load Balancing metrics:

  1. Go to Monitoring in the Google Cloud Console.
    Go to Monitoring
  2. Select Dashboards > Create Dashboard.
  3. Click on Add Chart.
  4. Give the chart a title.
  5. Select metrics and filters. For metrics, the resource type is Internal HTTP/S Load Balancer.
  6. Click Save.

Metric reporting frequency and retention

Metrics for the load balancers are exported to Monitoring in 1-minute granularity batches. Monitoring data is retained for six (6) weeks. The dashboard provides data analysis in default intervals of 1H (one hour), 6H (six hours), 1D (one day), 1W (one week), and 6W (six weeks). You can manually request analysis in any interval from 6W to 1 minute.

Monitoring metrics for internal HTTP(S) load balancers

The following metrics for internal HTTP(S) load balancers are reported into Monitoring:

Metric FQDN Description
Request count loadbalancing.googleapis.com/https/internal/request_count The number of requests served by the internal HTTP(S) load balancer.
Request bytes count loadbalancing.googleapis.com/https/internal/request_bytes The number of bytes sent as requests from clients to the internal HTTP(S) load balancer.
Response bytes count loadbalancing.googleapis.com/https/internal/response_bytes The number of bytes sent as responses from the internal HTTP(S) load balancer to the client.
Total latencies loadbalancing.googleapis.com/https/internal/total_latencies A distribution of the latency, in milliseconds, measured from the time the first byte of the request is received by the proxy to the time when the last byte of the response is sent by the proxy.
Backend latencies loadbalancing.googleapis.com/https/internal/backend_latencies A distribution of the latency, in milliseconds, measured from when the proxy sends the first byte of the request to the backend to when the proxy receives the last byte of the response from the backend.

Filtering dimensions for internal HTTP(S) load balancer metrics

Metrics are aggregated for each internal HTTP(S) load balancer. You can filter aggregated metrics by the following dimensions.

Property Description
BACKEND_SCOPE The Google Cloud zone or region of the backend group that served the client request, or a special string for cases in which the backend group wasn't assigned. Examples: us-central1-a, europe-west1-b, asia-east1, UNKNOWN.
PROXY_REGION Region of the internal HTTP(S) load balancer, client, and backend. Examples: us-central1, europe-west1 or asia-east1.
BACKEND The name of the backend instance group or NEG that served the client request.
BACKEND_TARGET The name of the backend service that served the client request.
MATCHED_URL_RULE The URL map path rule or route rule that matched the prefix of the client HTTP(S) request (up to 50 characters).

The Response code class fraction metric is supported for the entire load balancer. No further granularity is supported.

What's next