Problem
An External HTTP/HTTPS Load Balancer has a backend configured with a Google Remote Procedure Call (gRPC) health check. The health check returns healthy even though the gRPC response has the status field set to NOT_SERVING.
The following is what the health check logs look like:
detailedHealthState: "HEALTHY" healthCheckProtocol: "GRPC" healthState: "HEALTHY" probeRequest: "ranking_service_check.Health.Check" probeResultText: "The backend has no health checking service. Treated as healthy."
Environment
- gRPC Health Check
- External HTTP(S) Load Balancer.
Solution
The solution here is to fix where the Check method is located. In the example above, it points to package_name.Service.Check where package_name is ranking_service_check.
According to the gRPC Health Check Protocol's service definition, the package_name must be grpc.health.v1. So the Check method would be located in grpc.health.v1.Health.Check.
Receiving a probeRequest with anything other than grpc.health.v1.Health.Check will trigger the error described in this article.
Cause
The Health Check prober from Google Cloud Platform will be checking the path established in the gRPC Health Check Protocol. If nothing is there, the health check will respond with a gRPC status code.