Troubleshoot DNS in GKE

Stay organized with collections Save and categorize content based on your preferences.

This page shows you how to resolve issues related to DNS providers in Google Kubernetes Engine (GKE) clusters.

See Debugging DNS Resolution for general information about diagnosing Kubernetes DNS issues and Troubleshooting to learn more about diagnosing issues with Cloud DNS.

You can also enable Cloud DNS Logging.

Cloud DNS for GKE events

GKE records events that occur in Cloud DNS.

Cloud DNS disabled

The following event occurs when the Cloud DNS API is disabled:

Warning   FailedPrecondition        service/default-http-backend
Failed to send requests to Cloud DNS: Cloud DNS API Disabled. Please enable the Cloud DNS API in your project PROJECT_NAME: Cloud DNS API has not been used in project PROJECT_NUMBER before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/dns.googleapis.com/overview?project=PROJECT_NUMBER then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

This error occurs because the Cloud DNS API is not enabled by default, you must enable it manually.

To resolve the issue, enable the Cloud DNS API.

Failed to send requests to Cloud DNS: API rate limit exceeded.

The following event occurs when a project has exceeded a Cloud DNS quota or limit:

kube-system   27s         Warning   InsufficientQuota
managedzone/gke-cluster-quota-ee1bd2ca-dns     Failed to send requests to Cloud DNS: API rate limit exceeded. Contact Google Cloud support team to request a quota increase for your project PROJECT_NAME: Quota exceeded for quota metric 'Write requests' and limit 'Write limit for a minute for a region' of service 'dns.googleapis.com' for consumer 'project_number:PROJECT_NUMBER.

To resolve this issue, review the Cloud DNS quotas and Compute Engine quotas and limits. You can increase quota using the Google Cloud console.

Failed to send to requests to Cloud DNS due to a previous error

The following event occurs when errors cause cascading failures:

kube-system   27s         Warning   InsufficientQuota
managedzone/gke-cluster-quota-ee1bd2ca-dns     Failed to send requests to Cloud DNS: API rate limit exceeded. Contact Google Cloud support team to request a quota increase for your project PROJECT_NAME: Quota exceeded for quota metric 'Write requests' and limit 'Write limit for a minute for a region' of service 'dns.googleapis.com' for consumer 'project_number:PROJECT_NUMBER.
kube-system   27s         Warning   FailedPrecondition               service/default-http-backend                         Failed to send requests to Cloud DNS due to a previous error. Please check the cluster events.

To resolve this issue, check the cluster events to find the error, and follow the instructions to resolve the issue.

In the preceding example, the error InsufficientQuota for the managed zone triggered cascading failures, and the failure is recorded as an event that indicates a previous error. In this case, you would follow the instructions for the Cloud DNS quota error to resolve the issue.

Failed to bind response policy

The following event occurs when a response policy is bound to the network of the cluster and Cloud DNS for GKE attempts to bind a response policy to the network:

kube-system   9s          Warning   FailedPrecondition               responsepolicy/gke-2949673445-rp
Failed to bind response policy gke-2949673445-rp to test. Please verify that another Response Policy is not already associated with the network: Network 'https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/global/networks/NETWORK_NAME' cannot be bound to this response policy because it is already bound to another response policy.
kube-system   9s          Warning   FailedPrecondition               service/kube-dns
Failed to send requests to Cloud DNS due to a previous error. Please check the cluster events.

To resolve the issue, do the following:

  1. Get the response policy bound to the network:

    gcloud dns response-policies list --filter='networks.networkUrl: NETWORK_URL'
    

    Replace NETWORK_URL with the network URL from the error. For example, https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/NETWORK_NAME.

    If the output is empty, the response policy might not be in the same project. Proceed to the next step to search for the response policy.

    If the output is similar to the following, skip to step 4 to delete the response policy.

    [
       {
          "description": "Response Policy for GKE cluster \"CLUSTER_NAME\" with cluster suffix \"cluster.local.\" in project \"PROJECT_ID\" with scope \"CLUSTER_SCOPE\".",
          ...
          "kind": "dns#responsePolicy",
          "responsePolicyName": "gke-CLUSTER_NAME-POLICY_ID-rp"
       }
    ]
    
  2. Get a list of projects with the permission dns.networks.bindDNSResponsePolicy using the IAM Policy Analyzer.

  3. Check if each project has the response policy that is bound to the network:

    gcloud dns response-policies list --filter='networks.networkUrl:NETWORK_URL' \
        --project=PROJECT_NAME
    
  4. Delete the response policy.

Invalid configuration specified in kube-dns

The following event occurs when you apply a custom kube-dns ConfigMap that is not valid for Cloud DNS for GKE:

kube-system   49s         Warning   FailedValidation                 configmap/kube-dns
Invalid configuration specified in kube-dns: error parsing stubDomains for ConfigMap kube-dns: dnsServer [8.8.8.256] validation: IP address "8.8.8.256" invalid

To resolve this issue, follow the details in the error to resolve the invalid part of the ConfigMap. In the preceding example, 8.8.8.256 is not a valid IP address.