Health check logging

You can get logs of load balancing health checks when an endpoint changes in health status. You can use health check logs for the following:

  • Live debugging and troubleshooting your endpoint health status
  • Getting visibility into your endpoint's health status
  • Auditing and compliance purposes

Health checks log health transition information to Logging. You enable or disable logging on a per-health check basis.

To see health check logs in Logging, you must ensure that you don't have a logs exclusion that applies to health checks. For instructions about how to verify that GCE Instance Group and Network Endpoint Group logs are allowed, see Viewing resource-type exclusions.

Limitations

  • Health check logging on interfaces other than nic0 of VMs connected to internal TCP/UDP load balancers is not supported.
  • Logs are generated for endpoint health transition only.
  • Legacy health checks are not supported.
  • Target pools are not supported.

Enabling and disabling logging

Enabling logging on a new health check

Console

  1. Go to the Health check page in the Google Cloud Console.
    Go to the Health check page
  2. Click Create a health check.
  3. Set the Logs radio button to On.
  4. Continue setting up your health check.

gcloud 

gcloud compute health-checks create PROTOCOL HEALTH_CHECK_NAME \
   --enable-logging

where

  • --enable-logging enables logging for that health check.

Enabling logging on an existing health check

Console

  1. Go to the Health check page in the Google Cloud Console.
    Go to the Health check page
  2. Click the name of your health check.
  3. Click Edit .
  4. Set the Logs radio button to On.
  5. Click Save.

gcloud 

gcloud compute health-checks update PROTOCOL HEALTH_CHECK_NAME \
    --enable-logging

where

  • --enable-logging enables logging for that health check.

Disabling or modifying logging on an existing health check

Console

  1. Go to the Health check page in the Google Cloud Console.
    Go to the Health check page
  2. Click the name of your health check.
  3. Click Edit .
  4. Set the Logs radio button to Off.
  5. Click Save.

gcloud

Disabling logging entirely

gcloud compute health-checks update PROTOCOL HEALTH_CHECK_NAME \
    --no-enable-logging

where

  • --no-enable-logging disables logging for that health check.

Viewing logs

To view logs, go to the Logs viewer.

Health check logs are indexed by instance group or network endpoint group.

To see all logs, in the first pull-down menu select GCE Instance Group or Network Endpoint Group, depending on your type of backend.

Alternatively, go to the Logs viewer and paste the following into the Filter by label or text search field. Replace PROJECT_ID with your project's ID.

logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks"

You can also get logs based on more specific searches. For example, the following filter shows all logs for a specified backend instance IP address:

  logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks"
jsonPayload.healthCheckProbeResult.ipAddress="IP_ADDRESS"

Log fields of type boolean typically only appear if they have a value of true. If a boolean field has a value of false, that field is omitted from the log.

UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks.

You can configure export of logs-based metrics for load balancer health checks.

What is logged

Health check log entries contain information useful for monitoring and debugging the state of your endpoints. Log entries contain the following types of information:

  • General information shown in most logs, such as severity, project ID, project number, timestamp, and so on.
  • Fields specific to health checks, described in the following tables.

Health check states

An endpoint is considered either HEALTHY or UNHEALTHY. These are the basic states. Within each of these basic states, there are several more detailed states. The following table shows the mapping between basic and detailed health states.

Basic health state Detailed health state
HEALTHY HEALTHY
DRAINING
UNHEALTHY UNKNOWN
UNHEALTHY
TIMEOUT

State changes don't always alter the behavior of the load balancer. Consider the following case:

  1. The server is giving the wrong response, so the endpoint is considered UNHEALTHY.
  2. Then the server stops responding, and the new state is TIMEOUT.
  3. The load balancer still considers the endpoint to be UNHEALTHY because the detailed TIMEOUT state maps to the basic UNHEALTHY state.

The following table provides a definition of each health state.

Detailed health check state Meaning Basic state
HEALTHY The endpoint is reachable. The endpoint conforms to the requirements defined by the health check. HEALTHY
UNHEALTHY The endpoint is reachable, but does not conform to the requirements defined by the health check. UNHEALTHY
DRAINING The endpoint is being drained. The existing connections to the endpoint are allowed to complete, but the new ones are being refused. Endpoint is considered HEALTHY. HEALTHY
TIMEOUT The endpoint is unreachable. Depending on the health check type, either a connection to the endpoint cannot be established or the server did not respond within the specified timeout. Endpoint is considered UNHEALTHY. UNHEALTHY
UNKNOWN The health check system is aware of the endpoint, but its health is not known. Endpoint is considered UNHEALTHY. UNHEALTHY

There are multiple health checkers probing each endpoint; Google Cloud de-duplicates the log entries before logging so that only unique logs are generated.

You can interact with the logs using the Cloud Logging API. The logging API provides ways to interactively filter logs that have specific fields set, and export matching logs to Cloud Logging, Cloud Storage, BigQuery, or Pub/Sub. For more information on the Cloud Logging API, see Viewing Logs.

Health Check Log Entry

The LogEntry jsonPayload is populated with a field healthCheckProbeResult containing the following information:

Fields
ipAddress string The primary internal IP address associated with the primary network interface of each backend VM. This is a human-readable string.
healthCheckProtocol HealthCheckProtocol The Health check protocol used for health checking the endpoint. Examples: TCP, HTTP, HTTPS
healthState HealthState Current health status of the endpoint: HEALTHY or UNHEALTHY.
previousHealthState HealthState The previous health status of the endpoint: HEALTHY or UNHEALTHY.
detailedHealthState DetailedHealthState Current detailed health status of the endpoint. See Health check states for a list of possibilities.
previousDetailedHealthState DetailedHealthState The previous detailed health status of the endpoint. See Health check states for a list of possibilities.
probeRequest string For HTTP, HTTPS, and HTTP/2, this is the URL request path (requestPath field in the resource config). For TCP/SSL this is the configured optional string that is sent once the health check connection is established (request field in the resource config).
probeCompletionTimestamp Timestamp Probe completion timestamp.
connectLatency Duration Time spent on setting up the connection for connection-oriented health check protocols: TCP, SSL, HTTP, HTTPS, HTTP/2.
responseLatency Duration Latency between request and response, as measured by the prober.
probeResultText string Descriptive text related to the probe result. It might say something like "Connection timed out" or "HTTP response: Bad Gateway" or it could be empty.
probeSourceIp string The IP address from which the health check probe was sent.
targetIp string IP address that is the target of the probe. This can be different than ipAddress. The destination IP address of the probe depends on the type of load balancer:
  • For internal TCP/UDP load balancers and network load balancers, the destination of health check packets is the IP address of the load balancer's forwarding rule.
  • For external HTTP(S) load balancers, TCP proxy load balancers, and SSL proxy load balancers that use instance groups as backends, the destination of health check packets is the primary internal IP address associated with the primary network interface of each backend VM.
  • For external HTTP(S) load balancers, TCP proxy load balancers, and SSL proxy load balancers that use network endpoint groups as backends, the destination of health check packets is the IP address of the endpoint, which can be either a primary or secondary (alias IP) address.
targetPort int The port that was the target of the probe. This may be the default port of the probe or the port you specified when you created the health check.

Example filters

Find all health check results for a particular instance group

logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks"  AND
resource.type="gce_instance_group" AND
resource.labels.instance_group_name="INSTANCE_GROUP_NAME"

Find all health check results for a particular NEG

logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks"  AND
resource.type="gce_network_endpoint_group" AND
resource.labels.network_endpoint_group_id="ENDPOINT_GROUP_ID"

Find all health check transitions for backend instance IP address 10.128.15.201

logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks"
jsonPayload.healthCheckProbeResult.ipAddress="10.128.15.201"

Find all endpoints that were previously HEALTHY but now TIMEOUT

logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks"
jsonPayload.healthCheckProbeResult.previousDetailedHealthState="HEALTHY"
jsonPayload.healthCheckProbeResult.detailedHealthState="TIMEOUT"

Find health logs from a specific time range.

logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks"
timestamp>"2019-02-14T02:20:00.0Z"
timestamp<"2019-02-14T03:30:00.0Z"

What's next