Setting up NodeLocal DNSCache

This page explains how to configure NodeLocal DNSCache on a Google Kubernetes Engine (GKE) cluster. NodeLocal DNSCache improves DNS lookup latency, makes DNS lookup times more consistent, and reduces the number of DNS queries to kube-dns by running a DNS cache on each cluster node.

For an overview of how service discovery and managed DNS works on GKE, see Service discovery and DNS.

Overview

NodeLocal DNSCache is an optional GKE add-on that you can run in addition to kube-dns. NodeLocal DNSCache is implemented as a DaemonSet that runs a DNS cache on each node in your cluster. When a Pod makes a DNS request, the request goes to the DNS cache running on the same node as the Pod. If the cache can't resolve the DNS request, the cache forwards the request to:

• Cloud DNS for external hostname queries. These queries are forwarded to Cloud DNS by the local MetaData Server running on the same node as the Pod the query originated from.
• kube-dns for all other DNS queries. The kube-dns-upstream service is used by node-local-dns Pods to reach out to kube-dns Pods.

Pods do not need to be modified to use NodeLocal DNSCache. NodeLocal DNSCache consumes compute resources on each node of your cluster.

Benefits of NodeLocal DNSCache

• Reduced average DNS lookup time
• Connections from Pods to their local cache don't create conntrack table entries. This prevents dropped and rejected connections caused by conntrack table exhaustion and race conditions.

Details

• NodeLocal DNSCache requires GKE version 1.15 or higher.
• Connections between the local DNS cache and kube-dns use TCP instead of UDP for improved reliability.
• DNS queries for external URLs (URLs that don't refer to cluster resources) are forwarded directly to the local Cloud DNS metadata server, bypassing kube-dns.

• The local DNS caches automatically pick up stub domains and upstream nameservers that are specified in the kube-dns ConfigMap.

  Note: NodeLocal DNSCache currently reaches out to these upstreamServers via TCP, so the upstream servers need to support DNS resolution via TCP and UDP. This will be changed to use UDP in GKE 1.15.11-gke.14, 1.17.5-gke.6, and future versions. This change is only for custom configurations provided via kube-dns ConfigMap. The recommended best practice is to use a DNS server that is reachable via TCP and UDP.

• DNS records are cached for:

  • The record's TTL, or 30 seconds if the TTL is more than 30 seconds.
  • 5 seconds if the DNS response is NXDOMAIN.

• NodeLocal DNSCache Pods listen on port 53, 9253 and 8080 on the nodes. Running any other hostNetwork Pod using the above ports or configuring hostPorts with

• the above ports causes NodeLocal DNSCache to fail and result in DNS errors.

Enabling NodeLocal DNSCache

You can enable NodeLocal DNSCache in an existing cluster or when creating a new cluster. Enabling NodeLocal DNSCache in an existing cluster is a disruptive process, all cluster nodes running GKE 1.15 and higher are recreated. Nodes are recreated per the GKE node upgrade process.

Note: The local DNS cache only runs on node pools running GKE version 1.15 and higher. If you enable NodeLocal DNSCache in a cluster with nodes running lower versions, Pods on those nodes use kube-dns.

Warning: If you are using maintenance windows, your nodes will not be recreated until a maintenance window occurs. If you prefer not to wait, you can manually "upgrade" the node pool to the same version it is already using, by setting the --cluster-version flag to the same GKE version the master is already running. You must use the gcloud command if you use this workaround. See the caveats section of the maintenance window documentation for more information.

gcloud

Enabling NodeLocal DNSCache in a new cluster

To enable enable NodeLocal DNSCache in a new cluster, use the --addons NodeLocalDNS flag:

gcloud container clusters create cluster-name \
  --zone compute-zone \
  --cluster-version cluster-version \
  --addons NodeLocalDNS

Replace the following:

• cluster-name: the name of your new cluster.
• compute-zone: the zone for your cluster.
• cluster-version: the version for your cluster (1.15 or higher).

Enabling NodeLocal DNSCache in an existing cluster

Warning: Enabling NodeLocal DNSCache in an existing cluster is a disruptive operation. All cluster nodes running GKE 1.15 and higher are recreated. Nodes are recreated per the GKE node upgrade process.

To enable enable NodeLocal DNSCache on a new cluster, use the --update-addons=NodeLocalDNS=ENABLED flag:

gcloud container clusters update cluster-name \
  --update-addons=NodeLocalDNS=ENABLED

Console

You can use Google Cloud Console to enable NodeLocal DNSCache when creating a new cluster.

1. Visit the Create a Kubernetes cluster page in Cloud Console.

   Visit Kubernetes clusters page

2. For Name, enter cluster-name.

3. For Zone, select us-central1-a.

4. For Number of nodes, enter 1.

5. At the left side the page, click Networking.

6. Select Enable NodeLocal DNSCache checkbox under Advanced Networking options.

7. Click Create.

Verifying that NodeLocal DNSCache is enabled

You can verify that NodeLocal DNSCache is running by listing the node-local-dns Pods. There should be a node-local-dns Pod running on each node running GKE version 1.15 or higher.

kubectl get pods -n kube-system -o wide | grep node-local-dns

Disabling NodeLocal DNSCache

Warning: Disabling NodeLocal DNSCache is a disruptive operation. All cluster nodes running GKE 1.15 and higher are recreated. Nodes are recreated per the GKE node upgrade process.

NodeLocal DNSCache can be disabled using gcloud:

gcloud container clusters update cluster-name \
  --update-addons=NodeLocalDNS=DISABLED

Troubleshooting NodeLocal DNSCache

See Debugging DNS Resolution for general information about diagnosing Kubernetes DNS issues.

Validating Pod configuration

To verify that a Pod is using NodeLocal DNSCache, check /etc/resolv.conf on the Pod to see if the Pod is configured to use the correct nameserver:

kubectl exec -it pod-name -- cat /etc/resolv.conf | grep nameserver

The nameserver IP should match the IP address output by:

kubectl get svc -n kube-system kube-dns -o jsonpath="{.spec.clusterIP}"

If the nameserver IP address configured in /etc/resolv.conf doesn't match, you need to modify the configuration to use the correct nameserver IP address.

What's next

• Read an overview of how GKE provides managed DNS.
• Read DNS for Services and Pods for a general overview of how DNS is used in Kubernetes clusters.