Troubleshooting Cloud Run for Anthos on Google Cloud

This page provides troubleshooting strategies as well as solutions for some common errors.

When troubleshooting Cloud Run for Anthos on Google Cloud, first confirm that you can run your container image locally.

If your application is not running locally, you will need to diagnose and fix it. You should use Cloud Logging to help debug a deployed project.

When troubleshooting Cloud Run for Anthos on Google Cloud, consult the following sections for possible solutions to the problem.

Checking command line output

If you use the gcloud command line, check your command output to see if it succeeded or not. For example if your deployment terminated unsuccessfully, there should be an error message describing the reason for the failure.

Deployment failures are most likely due to either a misconfigured manifest or an incorrect command. For example, the following output says that you must configure route traffic percent to sum to 100

Error from server (InternalError): error when applying patch:</p><pre>{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"serving.knative.dev/v11\",\"kind\":\"Route\",\"metadata\":{\"annotations\":{},\"name\":\"route-example\",\"namespace\":\"default\"},\"spec\":{\"traffic\":[{\"configurationName\":\"configuration-example\",\"percent\":50}]}}\n"}},"spec":{"traffic":[{"configurationName":"configuration-example","percent":50}]}}
to:
&{0xc421d98240 0xc421e77490 default route-example STDIN 0xc421db0488 264682 false}
for: "STDIN": Internal error occurred: admission webhook "webhook.knative.dev" denied the request: mutation failed: The route must have traffic percent sum equal to 100.
ERROR: Non-zero return code '1' from command: Process exited with status 1

Checking logs for your service

You can use Cloud Logging or the Cloud Run page in the Cloud Console to check request logs and container logs. For complete details, read Logging and viewing logs.

If you use Cloud Logging, the resource you need to filter on is Kubernetes Container.

Checking Route status

Run the following command to get the status of the Route object you used to deploy your application:

kubectl get route ROUTE

The conditions in status provide the reason for the failure.

Checking Istio routing

Compare your Istio Route object's configuration, obtained by checking Route status, to the Istio RouteRule object's configuration.

Enter the following, replacing ROUTERULE-NAME with the appropriate value:

kubectl get routerule ROUTERULE-NAME -o yaml

If you don't know the name of your route rule, use kubectl get routerule to find it.

The command returns the configuration of your route rule. Compare the domains between your route and route rule; they should match.

Checking Revision status

If you configure your Route with Configuration, run the following command to get the name of the Revision created for you deployment, and look up the configuration name in the Route's .yaml file:

kubectl get configuration CONFIGURATION-NAME -o jsonpath="{.status.latestCreatedRevisionName}"

If you configure your Route with Revision directly, look up the revision name in the Route yaml file.

Then run:

kubectl get revision REVISION-NAME -o yaml

A ready Revision should has the following condition in status:

conditions:
  - reason: ServiceReady
    status: "True"
    type: Ready

If you see this condition, check the following to continue debugging:

  • Check Pod status
  • Check application logs
  • Check Istio routing

If you see other conditions, to debug further:

Checking Pod status

To get the Pods for all your deployments:

kubectl get pods

This should list all Pods with brief status. For example:

NAME                                                      READY     STATUS             RESTARTS   AGE
configuration-example-00001-deployment-659747ff99-9bvr4   2/2       Running            0          3h
configuration-example-00002-deployment-5f475b7849-gxcht   1/2       CrashLoopBackOff   2          36s

Choose one and use the following command to see detailed information for its status. Some useful fields are conditions and containerStatuses:

kubectl get pod POD-NAME -o yaml

EXTERNAL-IP is <pending> for a long time

Sometimes, you may not get an external IP address immediately after you create a cluster, but instead see the external IP as pending. For example you could see this by invoking the command:

To get the external IP for the Istio ingress gateway:

kubectl get svc ISTIO-GATEWAY -n NAMESPACE 
Replace ISTIO-GATEWAY and NAMESPACE as follows:
Cluster version ISTIO-GATEWAY NAMESPACE
1.15.3-gke.19 and greater
1.14.3-gke.12 and greater
1.13.10-gke.8 and greater
istio-ingress gke-system
All other versions istio-ingressgateway istio-system

where the resulting output looks something like this:

NAME            TYPE           CLUSTER-IP     EXTERNAL-IP  PORT(S)
ISTIO-GATEWAY    LoadBalancer   XX.XX.XXX.XX   pending     80:32380/TCP,443:32390/TCP,32400:32400/TCP

The EXTERNAL-IP for the Load Balancer is the IP address you must use.

This may mean that you have run out of external IP address quota in Google Cloud. You can check the possible cause by invoking:

kubectl describe svc ISTIO-GATEWAY -n NAMESPACE

replacing ISTIO-GATEWAY and NAMESPACE with the values from the table above. This yields output similar to the following:

stem
Name:                     ISTIO-GATEWAY
Namespace:                NAMESPACE
Labels:                   addonmanager.kubernetes.io/mode=Reconcile
                          app=ISTIO-GATEWAY
                          chart=gateways-1.0.3
                          heritage=Tiller
                          istio=ingressgateway
                          k8s-app=istio
                          kubernetes.io/cluster-service=true
                          release=istio
Annotations:              kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","app":"istio-ingressgateway","...
Selector:                 app=ISTIO-GATEWAY,istio=ingressgateway
Type:                     LoadBalancer
IP:                       10.XX.XXX.XXX
LoadBalancer Ingress:     35.XXX.XXX.188
Port:                     http2  80/TCP
TargetPort:               80/TCP
NodePort:                 http2  31380/TCP
Endpoints:                XX.XX.1.6:80
Port:                     https  443/TCP
TargetPort:               443/TCP
NodePort:                 https  3XXX0/TCP
Endpoints:                XX.XX.1.6:XXX
Port:                     tcp  31400/TCP
TargetPort:               3XX00/TCP
NodePort:                 tcp  3XX00/TCP
Endpoints:                XX.XX.1.6:XXXXX
Port:                     tcp-pilot-grpc-tls  15011/TCP
TargetPort:               15011/TCP
NodePort:                 tcp-pilot-grpc-tls  32201/TCP
Endpoints:                XX.XX.1.6:XXXXX
Port:                     tcp-citadel-grpc-tls  8060/TCP
TargetPort:               8060/TCP
NodePort:                 tcp-citadel-grpc-tls  31187/TCP
Endpoints:                XX.XX.1.6:XXXX
Port:                     tcp-dns-tls  853/TCP
TargetPort:               XXX/TCP
NodePort:                 tcp-dns-tls  31219/TCP
Endpoints:                10.52.1.6:853
Port:                     http2-prometheus  15030/TCP
TargetPort:               XXXXX/TCP
NodePort:                 http2-prometheus  30944/TCP
Endpoints:                10.52.1.6:15030
Port:                     http2-grafana  15031/TCP
TargetPort:               XXXXX/TCP
NodePort:                 http2-grafana  31497/TCP
Endpoints:                XX.XX.1.6:XXXXX
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age                  From                Message
  ----    ------                ----                 ----                -------
  Normal  EnsuringLoadBalancer  7s (x4318 over 15d)  service-controller  Ensuring load balancer

If your output contains an indication that the IN_USE_ADDRESSES quota was exceeded, you can request additional quota by navigating to the IAM & Admin page in the Google Cloud Console to request additional quota.

The gateway will continue to retry until an external IP address is assigned. This may take a few minutes

Troubleshooting automatic TLS issues

Use the troubleshooting steps listed below to resolve general issues for the automatic TLS certificates feature.

Check status of a specific domain mapping

To check the status of a specific domain mapping:

  1. Run the command:

    kubectl get domainMapping DOMAIN -n NAMESPACE -oyaml

    Replace

    • DOMAIN with the name of the domain you are using.
    • NAMESPACE with the namespace you use for the domain mapping.
  2. In the yaml results from this command, examine the condition of the CertificateProvisioned field to determine the nature of the error.

Check Order status

The Order status records the process of interacting with LetsEncrypt, and therefore can be used to debug the issues related to LetsEncrypt. If it is necessary, check the status of Order by running this command:

kubectl get order DOMAIN -n NAMESPACE -oyaml

Replace

  • DOMAIN with the name of the domain you are using.
  • NAMESPACE with the namespace you use for the domain mapping.

The results will contain the certificates issued and other information if the order was successful.

Exceeding LetsEncrypt quota

Check the DomainMapping status. If you exceed your LetsEncrypt quota, you will see an error message in the DomainMapping such as this:

acme: urn:ietf:params:acme:error:rateLimited: Error creating new order
:: too many certificates already issued for exact set of domains:
test.example.com:
see https://letsencrypt.org/docs/rate-limits/'

Refer to the LetsEncrypt documentation on rate limits to increase the certificate quota.

Order Timeout

An Order object will be timed out after 20 minutes if it still cannot get certificates.

  1. Check the domain mapping status. For a timeout, look for an error message such as this in the status output:

    order (test.example.com) timed out (20.0 minutes)
  2. A common cause of the timeout issue is that your DNS record is not configured properly to map the domain you are using to the IP address of the istio-ingress service under gke-system. Run the following command to check the DNS record:

    host DOMAIN
  3. Run the following command to check the external IP address of istio-ingress service under gke-system:

    kubectl get svc istio-ingress -n gke-system

    If the external IP address of your domain does not match the ingress IP address, then reconfigure your DNS record to map to the correct IP address.

  4. After the (updated) DNS record becomes effective, run the following command to delete the Order object in order to re-trigger the process of requesting a TLS certificate:

    kubectl delete order DOMAIN -n NAMESPACE

    Replace

    • DOMAIN with the name of the domain you are using.
    • NAMESPACE with the namespace you use.

Authorization Failures

Authorization failures can occur when a DNS record is not propagated globally in time. As a result, LetsEncrypt fails to verify the ownership of the domain.

  1. Check Order status. Find out the authz link under the acmeAuthorizations field of status. The URL should look like this:

    https://acme-v02.api.letsencrypt.org/acme/authz-v3/1717011827
  2. Open the link. If you see a message similar to:

    urn:ietf:params:acme:error:dns

    then the issue is due to incomplete DNS propagation.

  3. To resolve the DNS propagation error:

    1. Get the external IP of istio-ingress service under gke-system by running the following command:
      kubectl get svc istio-ingress -n gke-system
    2. Check your DNS record for the domain by running the following command:

      host DOMAIN

      If the IP address of the DNS record does not match the external IP of the istio-ingress service under gke-system, configure your DNS record to map the user's domain to the external IP.

    3. After the (updated) DNS record becomes effective, run the following command to delete the Order object to re-trigger the process of requesting a TLS certificate:

      kubectl delete order DOMAIN -n NAMESPACE

    Replace

    • DOMAIN with the name of the domain you are using.
    • NAMESPACE with the namespace you use for the domain mapping.

Deployment to private cluster failure: Failed calling webhook error

Your firewall may not be set up properly if your deployment to a private cluster fails with the message:

ERROR: (gcloud.run.deploy) Error: failed calling webhook "webhook.serving.knative.dev": Post
https://webhook.knative-serving.svc:443/?timeout=30s: context deadline exceeded (Client.Timeout
exceeded while awaiting headers)

For information on firewall changes required to support deployment to a private cluster, see enabling deployments on a private cluster.

Services report status of IngressNotConfigured

If IngressNotConfigured shows up in your service status, you may need to restart the istio-pilot deployment in the gke-system namespace. This error, which has been observed more frequently on kubernetes 1.14, can occur if the services are created before istio_pilot is ready to begin its work of reconciling VirtualServices and pushing envoy configuration to the ingress gateways.

To fix this issue, scale the deployment down and then back up again using commands similar to the following:

kubectl scale deployment istio-pilot -n gke-system --replicas=0
kubectl scale deployment istio-pilot -n gke-system --replicas=1