You can get logs of load balancing health checks when an endpoint changes in health status. You can use health check logs for the following:
- Live debugging and troubleshooting your endpoint health status
- Getting visibility into your endpoint's health status
- Auditing and compliance purposes
Health checks log health transition information to Logging. You enable or disable logging on a per-health check basis.
To see health check logs in Logging, you must ensure that you don't have a logs exclusion that applies to health checks. For instructions about how to verify that GCE Instance Group and Network Endpoint Group logs are allowed, see Viewing resource-type exclusions.
Limitations
- Health check logging on interfaces other than nic0 of VMs connected to internal TCP/UDP load balancers is not supported.
- Logs are generated for endpoint health transition only.
- Legacy health checks are not supported.
- Target pools are not supported.
Enabling and disabling logging
Enabling logging on a new health check
Console
- Go to the Health check page in the Google Cloud Console.
Go to the Health check page - Click Create a health check.
- Set the Logs radio button to On.
- Continue setting up your health check.
gcloud
gcloud compute health-checks create PROTOCOL HEALTH_CHECK_NAME \ --enable-logging
where
--enable-loggingenables logging for that health check.
Enabling logging on an existing health check
Console
- Go to the Health check page in the Google Cloud Console.
Go to the Health check page - Click the name of your health check.
- Click Edit .
- Set the Logs radio button to On.
- Click Save.
gcloud
gcloud compute health-checks update PROTOCOL HEALTH_CHECK_NAME \
--enable-logging
where
--enable-loggingenables logging for that health check.
Disabling or modifying logging on an existing health check
Console
- Go to the Health check page in the Google Cloud Console.
Go to the Health check page - Click the name of your health check.
- Click Edit .
- Set the Logs radio button to Off.
- Click Save.
gcloud
Disabling logging entirely
gcloud compute health-checks update PROTOCOL HEALTH_CHECK_NAME \
--no-enable-logging
where
--no-enable-loggingdisables logging for that health check.
Viewing logs
To view logs, go to the Logs viewer.
Health check logs are indexed by instance group or network endpoint group.
To see all logs, in the first pull-down menu select GCE Instance Group or Network Endpoint Group, depending on your type of backend.
Alternatively, go to the Logs viewer and paste the
following into the Filter by label or text search field. Replace
PROJECT_ID with your project's ID.
logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks"
You can also get logs based on more specific searches. For example, the following filter shows all logs for a specified backend instance IP address:
logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks" jsonPayload.healthCheckProbeResult.ipAddress="IP_ADDRESS"
Log fields of type boolean typically only appear if they have a value of true.
If a boolean field has a value of false, that field is omitted from the log.
UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks.
You can configure export of logs-based metrics for load balancer health checks.
What is logged
Health check log entries contain information useful for monitoring and debugging the state of your endpoints. Log entries contain the following types of information:
- General information shown in most logs, such as severity, project ID, project number, timestamp, and so on.
- Fields specific to health checks, described in the following tables.
Health check states
Health checks generate a log whenever an endpoint changes state from one state
to another. However, the behaviour of the load balancer may not change because
a state changed. If an endpoint is considered UNHEALTHY because
the server is giving the wrong response, but then stops responding entirely,
the health check sends a log with a new state of TIMEOUT, but
the load balancer still considers the endpoint to be unhealthy.
| Detailed health check state | Meaning | Basic state |
|---|---|---|
HEALTHY
|
The endpoint is reachable. The endpoint conforms to the requirements defined by the health check. | HEALTHY
|
UNHEALTHY
|
The endpoint is reachable, but does not conform to the requirements defined by the health check. | UNHEALTHY
|
DRAINING
|
The endpoint is being drained. The existing connections to the endpoint
are allowed to complete, but the new ones are being refused. Endpoint
is considered HEALTHY.
|
HEALTHY
|
TIMEOUT
|
The endpoint is unreachable. Depending on the health check type, either
a connection to the endpoint cannot be established or the server did not
respond within the specified timeout. Endpoint is considered
UNHEALTHY.
|
UNHEALTHY
|
UNKNOWN
|
The health check system is aware of the endpoint, but its health is not
known. Endpoint is considered UNHEALTHY.
|
UNHEALTHY
|
You can interact with the logs using the Cloud Logging API. The logging API provides ways to interactively filter logs that have specific fields set, and export matching logs to Cloud Logging, Cloud Storage, BigQuery, or Pub/Sub. For more information on the Cloud Logging API, see Viewing Logs.
Health Check Log Entry
The
LogEntry
jsonPayload is populated with a field healthCheckProbeResult containing the
following information:
| Fields | |
|---|---|
ipAddress
|
string
The primary internal IP address associated with the primary network interface of each backend VM. This is a human-readable string.
|
healthCheckProtocol
|
HealthCheckProtocol
The Health check protocol used for health checking the endpoint.
Examples: TCP, HTTP, HTTPS
|
healthState
|
HealthState
Current health status of the endpoint: HEALTHY or
UNHEALTHY.
|
previousHealthState
|
HealthState
The previous health status of the endpoint: HEALTHY or
UNHEALTHY.
|
detailedHealthState
|
DetailedHealthState
Current detailed health status of the endpoint. See
Health check states for a list of
possibilities.
|
previousDetailedHealthState
|
DetailedHealthState
The previous detailed health status of the endpoint. See
Health check states for a list of
possibilities.
|
probeRequest
|
string
For HTTP, HTTPS, and HTTP/2, this is the URL request path
(requestPath field in the resource config). For TCP/SSL
this is the configured optional string that is sent once the health
check connection is established (request field in the
resource config).
|
probeCompletionTimestamp
|
Timestamp
Probe completion timestamp.
|
connectLatency
|
Duration
Time spent on setting up the connection for connection-oriented
health check protocols: TCP, SSL, HTTP, HTTPS, HTTP/2.
|
responseLatency
|
Duration
Latency between request and response, as measured by the prober.
|
probeResultText
|
string
Descriptive text related to the probe result. It might say something
like "Connection timed out" or "HTTP response: Bad Gateway" or it
could be empty.
|
probeSourceIp
|
string
The IP address from which the health check probe was sent.
|
targetIp
|
string
IP address that is the target of the probe. This can be different
than ipAddress.
The destination IP address of the probe depends on the type of
load balancer:
|
targetPort
|
int
The port that was the target of the probe. This may be the default
port of the probe or the port you specified when you created the
health check.
|
Example filters
Find all health check results for a particular instance group
logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks" AND resource.type="gce_instance_group" AND resource.labels.instance_group_name="INSTANCE_GROUP_NAME"
Find all health check results for a particular NEG
logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks" AND resource.type="gce_network_endpoint_group" AND resource.labels.network_endpoint_group_id="ENDPOINT_GROUP_ID"
Find all health check transitions for backend instance IP address 10.128.15.201
logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks" jsonPayload.healthCheckProbeResult.ipAddress="10.128.15.201"
Find all endpoints that were previously HEALTHY but now TIMEOUT
logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks" jsonPayload.healthCheckProbeResult.previousDetailedHealthState="HEALTHY" jsonPayload.healthCheckProbeResult.detailedHealthState="TIMEOUT"
Find health logs from a specific time range.
logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fhealthchecks" timestamp>"2019-02-14T02:20:00.0Z" timestamp<"2019-02-14T03:30:00.0Z"
What's next
- Read conceptual information about health checks.
- Create a health check.
- Read about Logging.