This page explains how to improve DNS lookup latency in a Google Kubernetes Engine (GKE) cluster by using NodeLocal DNSCache.
For GKE Autopilot clusters, NodeLocal DNSCache is enabled by default and cannot be overridden. Skip to Verify that NodeLocal DNSCache is enabled.
Architecture
NodeLocal DNSCache is a GKE add-on that you can run in addition to kube-dns.
GKE implements NodeLocal DNSCache as a DaemonSet that runs a DNS cache on each node in your cluster.
When a Pod makes a DNS request, the request goes to the DNS cache running on the same node as the Pod. If the cache can't resolve the DNS request, the cache forwards the request to one of the following places based on the query destination:
- kube-dns: all queries for the cluster DNS domain (
cluster.local
) are forwarded to kube-dns. The node-local-dns Pods use the kube-dns-upstream Service to access kube-dns Pods. In the following diagram, the IP address of the kube-dns Service is10.0.0.10:53
. - Custom stub domains or upstream nameservers: queries are forwarded directly from NodeLocal DNSCache Pods.
- Cloud DNS: all other queries are forwarded to the local metadata server that runs on the same node as the Pod the query originated from. The local metadata server accesses Cloud DNS.
When you enable NodeLocal DNSCache on an existing cluster, GKE recreates all cluster nodes running GKE version 1.15 and later according to the node upgrade process.
After GKE recreates the nodes, GKE automatically
adds the label addon.gke.io/node-local-dns-ds-ready=true
to the nodes. You
must not add this label to the cluster nodes manually.
Benefits of NodeLocal DNSCache
NodeLocal DNSCache provides the following benefits:
- Reduced average DNS lookup time
- Connections from Pods to their local cache don't create conntrack table entries. This prevents dropped and rejected connections caused by conntrack table exhaustion and race conditions.
- You can use NodeLocal DNSCache with Cloud DNS for GKE.
- DNS queries for external URLs (URLs that don't refer to cluster resources) are forwarded directly to the local Cloud DNS metadata server, bypassing kube-dns.
- The local DNS caches automatically pick up stub domains and upstream nameservers that are specified in the kube-dns ConfigMap.
Requirements and limitations
- NodeLocal DNSCache consumes compute resources on each node of your cluster.
- NodeLocal DNSCache is not supported with Windows Server node pools.
- NodeLocal DNSCache requires GKE version 1.15 or later.
- NodeLocal DNSCache accesses kube-dns pods via TCP.
- NodeLocal DNSCache accesses
upstreamServers
andstubDomains
using TCP and UDP on GKE versions 1.18 or later. The DNS server must be reachable using TCP and UDP. - DNS records are cached for the following periods:
- The record's time to live (TTL), or 30 seconds if the TTL is more than 30 seconds.
- 5 seconds if the DNS response is
NXDOMAIN
.
- NodeLocal DNSCache Pods listen on port 53, 9253, 9353, and 8080 on the nodes. If
you run any other
hostNetwork
Pod or configure ahostPorts
with those ports, NodeLocal DNSCache fails and DNS errors occur. NodeLocal DNSCache Pods do not usehostNetwork
mode when using GKE Dataplane V2. If your cluster has PodSecurityPolicy enabled, your cluster must be on one of the following versions to use NodeLocal DNSCache:
- 1.15.12-gke.9 and later
- 1.16.11-gke.5 and later
- 1.17.7-gke.15 and later
The local DNS cache only runs on node pools running GKE versions 1.15 and later. If you enable NodeLocal DNSCache in a cluster with nodes running earlier versions, Pods on those nodes use kube-dns.
Enable NodeLocal DNSCache
You can enable NodeLocal DNSCache on new or existing clusters using the Google Cloud CLI. You can enable NodeLocal DNSCache in new clusters using the Google Cloud console.
gcloud
Enable NodeLocal DNSCache in a new cluster
To enable NodeLocal DNSCache in a new cluster, use the --addons
flag with the
argument NodeLocalDNS
:
gcloud container clusters create CLUSTER_NAME \
--region=COMPUTE_REGION \
--cluster-version=CLUSTER_VERSION \
--addons=NodeLocalDNS
Replace the following:
CLUSTER_NAME
: the name of your new cluster.COMPUTE_REGION
: the Compute Engine region for your cluster.CLUSTER_VERSION
: the version for your cluster (1.15 or later).
Enable NodeLocal DNSCache in an existing cluster
To enable NodeLocal DNSCache in an existing cluster, use the
--update-addons
flag with the argument NodeLocalDNS=ENABLED
:
gcloud container clusters update CLUSTER_NAME \
--update-addons=NodeLocalDNS=ENABLED
Replace the following:
CLUSTER_NAME
: the name of your cluster.
Console
To enable NodeLocal DNSCache on a new cluster, use the following steps:
Go to the Google Kubernetes Engine page in the Google Cloud console.
Next to Standard, click Configure.
Configure your cluster how you want.
From the navigation pane, click Networking.
In the Advanced networking options section, select the Enable NodeLocal DNSCache checkbox.
Click Create.
Verify that NodeLocal DNSCache is enabled
You can verify that NodeLocal DNSCache is running by listing the node-local-dns
Pods:
kubectl get pods -n kube-system -o wide | grep node-local-dns
The output is similar to the following:
node-local-dns-869mt 1/1 Running 0 6m24s 10.128.0.35 gke-test-pool-69efb6b8-5d7m <none> <none>
node-local-dns-htx4w 1/1 Running 0 6m24s 10.128.0.36 gke-test-pool-69efb6b8-wssk <none> <none>
node-local-dns-v5njk 1/1 Running 0 6m24s 10.128.0.33 gke-test-pool-69efb6b8-bhz3 <none> <none>
The output shows a node-local-dns
Pod for each node that is running
GKE version 1.15 or later.
Disable NodeLocal DNSCache
You can disable NodeLocal DNSCache using the following command:
gcloud container clusters update CLUSTER_NAME \
--update-addons=NodeLocalDNS=DISABLED
Replace the following:
CLUSTER_NAME
: the name of the cluster to disable.
Troubleshoot NodeLocal DNSCache
For general information about diagnosing Kubernetes DNS issues, see Debugging DNS Resolution.
NodeLocal DNSCache is not enabled immediately
When you enable NodeLocal DNSCache on an existing cluster, GKE might not update the nodes immediately if the cluster has a configured maintenance window or exclusion. For more information, see caveats for node re-creation and maintenance windows.
If you prefer not to wait, you can manually apply the changes to the nodes by
calling the gcloud container clusters upgrade
command and passing the --cluster-version
flag with the same
GKE version that the node pool is already running. You must use
the Google Cloud CLI for this workaround.
NodeLocal DNSCache with Cloud DNS
If you use NodeLocal DNSCache with
Cloud DNS, the cluster
uses the nameserver IP address 169.254.20.10
, as shown in the following
diagram:
You can view the IP address of the Cluster IP by using the following command:
kubectl get svc -n kube-system kube-dns -o jsonpath="{.spec.clusterIP}"
The output is similar to the following:
169.254.20.10
Network policy with NodeLocal DNSCache
If you use network policy
with NodeLocal DNSCache and you are not using
Cloud DNS or
GKE Dataplane V2,
you must configure rules to permit your workloads and the node-local-dns
Pods to send DNS queries.
Use an ipBlock
rule in your manifest to allow communication between
your Pods and kube-dns.
The following manifest describes a network policy that uses an ipBlock
rule:
spec:
egress:
- ports:
- port: 53
protocol: TCP
- port: 53
protocol: UDP
to:
- ipBlock:
cidr: KUBE_DNS_SVC_CLUSTER_IP/32
podSelector: {}
policyTypes:
- Egress
Replace KUBE_DNS_SVC_CLUSTER_IP
with the IP address of the
kube-dns service. You can get the IP address of the kube-dns service using the
following command:
kubectl get svc -n kube-system kube-dns -o jsonpath="{.spec.clusterIP}"
Known issues
DNS timeout in ClusterFirstWithHostNet dnsPolicy when using NodeLocal DNSCache and GKE Dataplane V2
On clusters using GKE Dataplane V2 and NodeLocal DNSCache, pods with hostNetwork
set to true
and dnsPolicy
set to ClusterFirstWithHostNet
cannot reach
cluster DNS backends. DNS logs might contain entries similar to the following:
nslookup: write to 'a.b.c.d': Operation not permitted
;; connection timed out; no servers could be reached
The output indicates that the DNS requests cannot reach the backend servers.
A workaround is to set the dnsPolicy
and dnsConfig
for hostNetwork
pods:
spec:
dnsPolicy: "None"
dnsConfig:
nameservers:
- KUBE_DNS_UPSTREAM
searches:
- cluster.local
- svc.cluster.local
- NAMESPACE.svc.cluster.local
- c.PROJECT_ID.internal
- google.internal
options:
- name: ndots
value: "5"
Replace the following:
NAMESPACE
: the namespace of thehostNetwork
pod.PROJECT_ID
: the ID of your Google Cloud project.KUBE_DNS_UPSTREAM
: the ClusterIP of the upstream kube-dns service. You can get this value using the following command:kubectl get svc -n kube-system kube-dns-upstream -o jsonpath="{.spec.clusterIP}"
DNS requests from the Pod can now reach kube-dns and bypass NodeLocal DNSCache.
NodeLocal DNSCache timeout errors
On clusters with NodeLocal DNSCache enabled, the logs might contain entries similar to the following:
[ERROR] plugin/errors: 2 <hostname> A: read tcp <node IP: port>-><kubedns IP>:53: i/o timeout
The output includes the IP address of the kube-dns-upstream
Cluster IP
Service. In this example, the response to a DNS request was not received from
kube-dns in 2 seconds. This could be due to one of the following reasons:
- An underlying network connectivity problem.
- A known issue with
dnsmasq
handling TCP connections.
node-local-dns
Pods access kube-dns using TCP for improved reliability.
When handling connections from multiple source IPs, dnsmasq
prioritizes
connections from existing connections over new connections. As a result, on a
cluster with high DNS queries per second (QPS), node-local-dns
Pods on newly
created nodes can experience higher DNS latency. This can also occur on clusters
with cluster autoscaler enabled because cluster autoscaler dynamically changes
the number of nodes.
This issue is resolved in the following GKE versions:
- 1.19.7-gke.1500 and later
- 1.18.16-gke.1200 and later
- 1.17.17-gke.5400 and later
One workaround is to increase the number of kube-dns replicas by tuning the autoscaling parameters.
Scaling up kube-dns
You can use a lower value for nodesPerReplica
to ensure that more kube-dns
Pods are created as cluster nodes scale up. We highly recommend setting an
explicit max
value to ensure that the GKE control plane virtual
machine (VM) is not overwhelmed due to large number of kube-dns pods watching
the Kubernetes API.
You can set max
to the number of nodes in the cluster. If the cluster has more
than 500 nodes, set max
to 500.
For Standard clusters, you can modify the number of kube-dns replicas by editing the
kube-dns-autoscaler
ConfigMap. This configuration is not supported in
Autopilot clusters.
kubectl edit configmap kube-dns-autoscaler --namespace=kube-system
The output is similar to the following:
linear: '{"coresPerReplica":256, "nodesPerReplica":16,"preventSinglePointFailure":true}'
The number of kube-dns replicas is calculated using the following formula:
replicas = max( ceil( cores × 1/coresPerReplica ) , ceil( nodes × 1/nodesPerReplica ), maxValue )
To scale up, change nodesPerReplica
to a smaller value and include a
max
value.
linear: '{"coresPerReplica":256, "nodesPerReplica":8,"max": 15,"preventSinglePointFailure":true}'
The config creates 1 kube-dns pod for every 8 nodes in the cluster. A 24-node
cluster will have 3 replicas and a 40-node cluster will have 5 replicas. If the
cluster grows beyond 120 nodes, the number of kube-dns replicas does not grow
beyond 15, the max
value.
What's next
- Read an overview of how GKE provides managed DNS.
- Read DNS for Services and Pods for a general overview of how DNS is used in Kubernetes clusters.
- Learn how to use Cloud DNS for GKE.