Troubleshoot DNS in GKE

This page shows you how to resolve issues related to DNS providers in Google Kubernetes Engine (GKE) clusters.

If you need additional assistance, reach out to Cloud Customer Care.

Cloud DNS for GKE events

This section details common Cloud DNS problems in GKE.

Cloud DNS disabled

The following event occurs when the Cloud DNS API is disabled:

Warning   FailedPrecondition        service/default-http-backend
Failed to send requests to Cloud DNS: Cloud DNS API Disabled. Please enable the Cloud DNS API in your project PROJECT_NAME: Cloud DNS API has not been used in project PROJECT_NUMBER before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/dns.googleapis.com/overview?project=PROJECT_NUMBER then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

This error occurs because the Cloud DNS API is not enabled by default. You must enable the Cloud DNS API manually.

To resolve the issue, enable the Cloud DNS API.

Failed to send requests to Cloud DNS: API rate limit exceeded.

The following event occurs when a project has exceeded a Cloud DNS quota or limit:

kube-system   27s         Warning   InsufficientQuota
managedzone/gke-cluster-quota-ee1bd2ca-dns     Failed to send requests to Cloud DNS: API rate limit exceeded. Contact Google Cloud support team to request a quota increase for your project PROJECT_NAME: Quota exceeded for quota metric 'Write requests' and limit 'Write limit for a minute for a region' of service 'dns.googleapis.com' for consumer 'project_number:PROJECT_NUMBER.

To resolve this issue, review the Cloud DNS quotas and Compute Engine quotas and limits. You can increase quota using the Google Cloud console.

Failed to send to requests to Cloud DNS due to a previous error

The following event occurs when errors cause cascading failures:

kube-system   27s         Warning   InsufficientQuota
managedzone/gke-cluster-quota-ee1bd2ca-dns     Failed to send requests to Cloud DNS: API rate limit exceeded. Contact Google Cloud support team to request a quota increase for your project PROJECT_NAME: Quota exceeded for quota metric 'Write requests' and limit 'Write limit for a minute for a region' of service 'dns.googleapis.com' for consumer 'project_number:PROJECT_NUMBER.
kube-system   27s         Warning   FailedPrecondition               service/default-http-backend                         Failed to send requests to Cloud DNS due to a previous error. Please check the cluster events.

To resolve this issue, check the cluster events to find the source of the original error, and follow the instructions to resolve that root issue.

In the preceding example, the InsufficientQuota error for the managed zone triggered cascading failures. The second error for FailedPrecondition indicates that a previous error occurred, which was that initial insufficient quota problem. To resolve this example issue, you would follow the guidance for the Cloud DNS quota error.

Failed to bind response policy

The following event occurs when a response policy is bound to the network of the cluster and Cloud DNS for GKE attempts to bind a response policy to the network:

kube-system   9s          Warning   FailedPrecondition               responsepolicy/gke-2949673445-rp
Failed to bind response policy gke-2949673445-rp to test. Please verify that another Response Policy is not already associated with the network: Network 'https://www.googleapis.com/compute/v1/projects/PROJECT_NAME/global/networks/NETWORK_NAME' cannot be bound to this response policy because it is already bound to another response policy.
kube-system   9s          Warning   FailedPrecondition               service/kube-dns
Failed to send requests to Cloud DNS due to a previous error. Please check the cluster events.

To resolve the issue, complete the following steps:

  1. Get the response policy bound to the network:

    gcloud dns response-policies list --filter='networks.networkUrl: NETWORK_URL'

    Replace NETWORK_URL with the network URL from the error, such as https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/NETWORK_NAME.

    If the output is empty, the response policy might not be in the same project. Proceed to the next step to search for the response policy.

    If the output is similar to the following, skip to step 4 to delete the response policy.

    [
   {
      "description": "Response Policy for GKE cluster \"CLUSTER_NAME\" with cluster suffix \"cluster.local.\" in project \"PROJECT_ID\" with scope \"CLUSTER_SCOPE\".",
      ...
      "kind": "dns#responsePolicy",
      "responsePolicyName": "gke-CLUSTER_NAME-POLICY_ID-rp"
   }
]

  2. Get a list of projects with the dns.networks.bindDNSResponsePolicy permission using the IAM Policy Analyzer.

  3. Check if each project has the response policy that is bound to the network:

    gcloud dns response-policies list --filter='networks.networkUrl:NETWORK_URL' \
    --project=PROJECT_NAME

  4. Delete the response policy.

Invalid configuration specified in kube-dns

The following event occurs when you apply a custom kube-dns ConfigMap that is not valid for Cloud DNS for GKE:

kube-system   49s         Warning   FailedValidation                 configmap/kube-dns
Invalid configuration specified in kube-dns: error parsing stubDomains for ConfigMap kube-dns: dnsServer [8.8.8.256] validation: IP address "8.8.8.256" invalid

To resolve this issue, review the details in the error for the invalid part of the ConfigMap. In the preceding example, 8.8.8.256 is not a valid IP address.

What's next