Using kube-dns

Standard

---

This page describes how Google Kubernetes Engine (GKE) implements service discovery using kube-dns, the default DNS provider for GKE clusters.

For Autopilot clusters, you cannot modify the default kube-dns configuration.

Architecture

When you create a cluster, GKE automatically deploys kube-dns Pods in the kube-system namespace. Pods access the kube-dns deployment through a corresponding Service that groups the kube-dns Pods and gives them a single IP address (ClusterIP). By default, all Pods in a cluster use this Service to resolve DNS queries. The following diagram shows the relationship between Pods and the kube-dns Service.

kube-dns scales to meet the DNS demands of the cluster. This scaling is controlled by the kube-dns-autoscaler, a Pod that is deployed by default in all GKE clusters. The kube-dns-autoscaler adjusts the number of replicas in the kube-dns Deployment based on the number of nodes and cores in the cluster.

kube-dns supports up to 1000 endpoints per headless service.

How Pod DNS is configured

The kubelet running on each Node configures the Pod's etc/resolv.conf to use the kube-dns service's ClusterIP. The following example configuration shows that the IP address of the kube-dns service is 10.0.0.10. This IP address is different in other clusters.

nameserver 10.0.0.10
search default.svc.cluster.local svc.cluster.local cluster.local c.my-project-id.internal google.internal
options ndots:5

kube-dns is the authoritative name server for the cluster domain (cluster.local) and it resolves external names recursively. Short names that are not fully qualified, such as myservice, are completed first with local search paths.

Note: In GKE Autopilot clusters, in version 1.27 and later, the default Service IP range is a Google-managed range: 34.118.224.0/20 and the kube-dns service will be addressed 34.118.x.10 by default (x being in in the range 224-239).

Adding custom resolvers for stub domains

You can modify the ConfigMap for kube-dns to set stub domains as part of DNS infrastructure within your clusters.

Stub domains let you configure custom per-domain resolvers so that kube-dns forwards DNS requests to specific upstream DNS servers when resolving these domains.

Note: When you set a custom resolver for a stub domain, such as example.com, kube-dns forwards all name resolution requests to the defined server(s) including **example.com and *.example.com.

The following example ConfigMap manifest for kube-dns includes a stubDomains configuration that sets custom resolvers for the domain example.com.

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {
      "example.com": [
        "8.8.8.8",
        "8.8.4.4",
        "1.1.1.1",
        "1.0.0.1"
      ]
    }

Run the following command to open a text editor:

kubectl edit configmap kube-dns -n kube-system

Replace the contents of the file with the manifest and then exit the text editor to apply the manifest to the cluster.

Upstream nameservers

If you modify the ConfigMap for kube-dns to include upstreamNameservers, kube-dns forwards all DNS requests except *.cluster.local to those servers. This includes metadata.internal and *.google.internal, which are not resolvable by the upstream server.

If you enable Workload Identity Federation for GKE or any workloads that rely on metadata.internal resolution, to retain *.internal name resolution, add a stubDomain to the ConfigMap.

data:
  stubDomains: |
    {
      "internal": [
        "169.254.169.254"
      ]
    }
  upstreamNameservers: |
    ["8.8.8.8"]

Troubleshooting

For information about troubleshooting kube-dns, see the following pages:

• For advice about kube-dns in GKE, see Troubleshoot kube-dns in GKE.
• For general advice about diagnosing Kubernetes DNS issues, see Debugging DNS Resolution.

Limitations

Read the following sections for information about kube-dns limitations.

Search domain limit

There is a limit of six DNS search domains for /etc/resolv.conf. If you define more than six search domains, the following warning appears when you run the command kubectl describe pod:

Search Line limits were exceeded, some search paths have been omitted, the applied search line is: default.svc.cluster.local svc.cluster.local cluster.local c.<project ID>.internal google.internal

This warning is logged in Cloud Logging in the container logs section.

To resolve this issue, remove the extra search paths from the configuration.

Consider the upstreamNameservers limit

Kubernetes imposes a limit of up to three upstreamNameservers values. If you define more than three upstreamNameservers, you see the following error in Cloud Logging in the kube-dns deployment logs:

Invalid configuration: upstreamNameserver cannot have more than three entries (value was &TypeMeta{Kind:,APIVersion:,}), ignoring update

When this happens, kube-dns behaves as if it has no upstreamNameservers configured. To resolve this issue, remove the extra upstreamNameservers from the configuration.

Performance limitations with kube-dns

If you are experiencing high latency with DNS lookups or DNS resolution failures with the default kube-dns provider, this might be caused by:

• Performing frequent DNS lookups within your workload
• Deploying a higher Pod density per node.
• Exceeding the query per second (QPS) limit of 20 for each kube-dns Pod.
• Running kube-dns on Spot or preemptible VMs, which can lead to unexpected node deletions and subsequent DNS resolution issues.

To improve DNS lookup times, you can choose one of the following options:

• Avoid running critical system components like kube-dns on Spot or preemptible VMs. Using Spot or preemptible VMs for DNS can cause failures and disrupt your cluster.
• As best practices, create at least one node pool comprised of standard (non-Spot or preemptible) VMs to host critical system components like kube-dns. To ensure that critical workloads are only scheduled on the reliable node pool preventing them from running on Spot or preemptible VMs, you can use taints and tolerations for Spot VMs.
• Enable NodeLocal DNSCache.
• Scale up kube-dns.
• Ensure that your application uses dns.resolve* based functions rather than dns.lookup based function as dns.lookup is synchronous. dns.resolve* functions always perform an asynchronous DNS query on the network.

Note: During GKE cluster upgrades, DNS resolution failures may occur due to concurrent upgrades of the control plane components, including kube-dns. This can happen regardless of whether NodeLocal DNSCache is enabled or not. Typically, only a small percentage of nodes (around 10%) will be affected at any given time. To minimize disruptions, we strongly recommended that you thoroughly test cluster upgrades in a non-production environment before applying them to your production clusters.

Service DNS records

kube-dns only creates DNS records for Services that have Endpoints{track-name="k8sLink" track-type="concept"}.

Large TTL from DNS upstream servers

If kube-dns receives a DNS response from an upstream DNS resolver with a large or "infinite" TTL, it keeps this TTL value for the DNS entry in the cache. The entry never expires and could create a discrepancy between the entry and the actual IP address resolved for the TTL name.

GKE resolves this issue in the following control plane versions by setting a max TTL value to 30 seconds for any DNS response that has a TTL higher than 30 seconds:

• 1.21.14-gke.9100
• 1.22.15-gke.2100
• 1.23.13-gke.500
• 1.24.7-gke.500
• 1.25.2-gke.500 or later

This behavior is similar to NodeLocal DNSCache.

What's next

• Read an overview of cluster DNS in GKE.
• Read DNS for Services and Pods for a general overview of how DNS is used in Kubernetes clusters.