Health check logging

You can get logs of load balancing health checks when an endpoint changes in health status. You can use health check logs for the following:

  • Live debugging and troubleshooting your endpoint health status
  • Getting visibility into your endpoint's health status
  • Auditing and compliance purposes

Health checks log health transition information to Logging. You enable or disable logging on a per-health check basis.

To see health check logs in Logging, you must ensure that you don't have a logs exclusion that applies to health checks. See Viewing resource-type exclusions for instructions on how to verify that GCE instance group and Network endpoint group logs are allowed.

Limitations

  • Logs are generated for endpoint health transition only.
  • Legacy health checks are not supported
  • Target pools are not supported.

Enabling and disabling logging

Enabling logging on a new health check

Console

  1. Go to the Health check page in the Google Cloud Console.
    Go to the Health check page
  2. Click Create a health check.
  3. Set the Logs radio button to On.
  4. Continue setting up your health check.

gcloud

gcloud beta compute health-checks create PROTOCOL HEALTH_CHECK_NAME \
   --enable-logging \
 ... other values

where

  • --enable-logging enables logging for that health check.

Enabling logging on an existing health check

Console

  1. Go to the Health check page in the Google Cloud Console.
    Go to the Health check page
  2. Click the name of your health check.
  3. Click Edit edit .
  4. Set the Logs radio button to On.
  5. Click Save.

gcloud

gcloud beta compute health-checks update PROTOCOL HEALTH_CHECK_NAME \
    --enable-logging

where

  • --enable-logging enables logging for that health check.

Disabling or modifying logging on an existing health check

Console

  1. Go to the Health check page in the Google Cloud Console.
    Go to the Health check page
  2. Click the name of your health check.
  3. Click Edit edit .
  4. Set the Logs radio button to Off.
  5. Click Save.

gcloud

Disabling logging entirely

gcloud beta compute health-checks update PROTOCOL HEALTH_CHECK_NAME \
    --no-enable-logging

where

  • --no-enable-logging disables logging for that health check.

Viewing logs

To view logs, go to the Logs viewer.

Health check logs are indexed by instance group or network endpoint group.

To see all logs, in the first pull-down menu select GCE Instance Group or Network Endpoint Group, depending on your type of backend.

Alternatively, go to the Logs viewer and paste the following into the Filter by label or text search field. Replace project-id with your project's ID.

logName="projects/project-id/logs/compute.googleapis.com%2Fhealthchecks"

You can also get logs based on more specific searches. For example, the following filter shows all logs for a specified backend instance IP address:

  logName="projects/project-id/logs/compute.googleapis.com%2Fhealthchecks"
jsonPayload.healthCheckProbeResult.ipAddress="ip-address"

Log fields of type boolean typically only appear if they have a value of true. If a boolean field has a value of false, that field is omitted from the log.

UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks.

You can configure export of logs-based metrics for load balancer health checks.

What is logged

Health check log entries contain information useful for monitoring and debugging your the state of endpoints. Log entries contain the following types of information:

  • General information shown in most logs, such as severity, project ID, project number, timestamp, and so on.
  • Fields specific to health checks, described in the following tables.

Health check states

Health checks generate a log whenever an endpoint changes state from one state to another. However, the behaviour of the load balancer may not change because a state changed. If an endpoint is considered UNHEALTHY because the server is giving the wrong response, but then stops responding entirely, the health check sends a log with a new state of TIMEOUT, but the load balancer still considers the endpoint to be unhealthy.

Detailed health check state Meaning Basic state
HEALTHY The endpoint is reachable. The endpoint conforms to the requirements defined by the health check. HEALTHY
UNHEALTHY The endpoint is reachable, but does not conform to the requirements defined by the health check. UNHEALTHY
DRAINING The endpoint is being drained. The existing connections to the endpoint are allowed to complete, but the new ones are being refused. Endpoint is considered HEALTHY. HEALTHY
TIMEOUT The endpoint is unreachable. Depending on the health check type, either a connection to the endpoint cannot be established or the server did not respond within the specified timeout. Endpoint is considered UNHEALTHY. UNHEALTHY
UNKNOWN The health check system is aware of the endpoint, but its health is not known. Endpoint is considered UNHEALTHY. UNHEALTHY

You can interact with the logs using the Cloud Logging API. The logging API provides ways to interactively filter logs that have specific fields set, and export matching logs to Cloud Logging, Cloud Storage, BigQuery, or Pub/Sub. For more information on the Cloud Logging API, see Viewing Logs.

Health Check Log Entry

The LogEntry json-payload is populated with a field healthCheckProbeResult containing the following information:

Fields
ipAddress string The primary internal IP address associated with the primary network interface of each backend VM. This is a human-readable string.
healthCheckProtocol HealthCheckProtocol The Health check protocol used for health checking the endpoint. e.g. TCP, HTTP, HTTPS etc.
healthState HealthState Current health status of the endpoint: HEALTHY or UNHEALTHY.
previousHealthState HealthState The previous health status of the endpoint: HEALTHY or UNHEALTHY.
detailedHealthState DetailedHealthState Current detailed health status of the endpoint. See Health check states for a list of possibilities.
previousDetailedHealthState DetailedHealthState The previous detailed health status of the endpoint. See Health check states for a list of possibilities.
probeRequest string For HTTP, HTTPS, and HTTP2, this is the URL request path (requestPath field in the resource config. For TCP/SSL this is the configured optional string that is sent once the health check connection is established (request field in the resource config).
probeCompletionTimestamp Timestamp Probe completion timestamp.
connectLatency Duration Time spent on setting up the connection for connection-oriented health check protocols: TCP, SSL, HTTP, HTTPS, HTTP/2.
responseLatency Duration Latency between request and response, as measured by the prober.
probeResultText string Descriptive text related to the probe result. It might say something like "Connection timed out" or "HTTP response: Bad Gateway" or it could be empty.
probeSourceIp string The IP from which the health check probe was sent.
targetIp string IP address that is the target of the probe. This can be different than ipAddress. The destination IP address of the probe depends on the type of load balancer:
  • For internal TCP/UDP load balancers and network load balancers, the destination of health check packets is the IP address of the load balancer's forwarding rule.
  • For external HTTP(S) load balancers, TCP proxy load balancers, and SSL proxy load balancers that use instance groups as backends, the destination of health check packets is the primary internal IP address associated with the primary network interface of each backend VM.
  • For external HTTP(S) load balancers, TCP proxy load balancers, and SSL proxy load balancers that use network endpoint groups as backends, the destination of health check packets is the IP address of the endpoint, which can be either a primary or secondary (alias IP) address.
targetPort int The port that was the target of the probe. This may be the default port of the probe or the port you specified when you created the health check.

Example filters

Find all health check results for a particular instance group

logName="projects/project-id/logs/compute.googleapis.com%2Fhealthchecks"  AND
resource.type="gce_instance_group" AND
resource.labels.instance_group_name="instance-group-name"

Find all health check results for a particular NEG

logName="projects/project-id/logs/compute.googleapis.com%2Fhealthchecks"  AND
resource.type="gce_network_endpoint_group" AND
resource.labels.network_endpoint_group_id="endpoint-group-id"

Find all health check transitions for backend instance IP address 10.128.15.201

logName="projects/project-id/logs/compute.googleapis.com%2Fhealthchecks"
jsonPayload.healthCheckProbeResult.ipAddress="10.128.15.201"

Find all endpoints that were previously HEALTHY but now TIMEOUT

logName="projects/project-id/logs/compute.googleapis.com%2Fhealthchecks"
jsonPayload.healthCheckProbeResult.previousDetailedHealthState="HEALTHY"
jsonPayload.healthCheckProbeResult.detailedHealthState="TIMEOUT"

Find health logs from a specific time range.

logName="projects/project-id/logs/compute.googleapis.com%2Fhealthchecks"
timestamp>"2019-02-14T02:20:00.0Z"
timestamp<"2019-02-14T02:30:00.0Z"

What's next