Using Cloud DNS for GKE


This page explains how to use Cloud DNS for Google Kubernetes Engine (GKE).

For more information about using kube-dns as a DNS provider, see Service discovery and DNS. To learn how to use a custom version of kube-dns or a custom DNS provider, see Setting up a custom kube-dns Deployment.

Overview

Cloud DNS can be used as the DNS provider for GKE, providing Pod and Service DNS resolution with managed DNS that does not require a cluster-hosted DNS provider. DNS records for Pods and Services are automatically provisioned in Cloud DNS for cluster IP, headless and external name Services.

Cloud DNS supports the full Kubernetes DNS specification and provides resolution for A, AAAA, SRV and PTR records for Services within a GKE cluster. PTR records are implemented using response policy rules.

Using Cloud DNS as the DNS provider for GKE offers many benefits over cluster-hosted DNS:

  • Removes overhead of managing the cluster-hosted DNS server. Cloud DNS requires no scaling, monitoring, or managing of DNS instances because it is a hosted Google service.
  • Local resolution of DNS queries on each GKE node. Similar to NodeLocal DNSCache, Cloud DNS caches DNS responses locally, providing low latency and high scalability DNS resolution.
  • Integration with Google Cloud's operations suite for DNS monitoring and logging.
  • VPC scope DNS for multi-cluster, multi-environment, and VPC-wide Kubernetes Service resolution.

Architecture

When Cloud DNS is the DNS provider for GKE, a controller runs as a GKE-managed Pod. This Pod syncs the cluster DNS records into a private, managed DNS zone.

The following diagram shows how the Cloud DNS control plane and data plane resolve cluster names:

A Pod requests the IP address of a service using Cloud DNS
Diagram: Resolving cluster names

In the diagram, the Service backend selects the running backend Pods. The clouddns-controller creates a DNS record for Service backend.

The Pod frontend sends a DNS request for the IP address of the Service backend to the Compute Engine local metadata server at 169.254.169.254. The metadata server runs locally on the node, sending cache misses to Cloud DNS.

The Cloud DNS data plane runs locally within each Compute Engine virtual machine (VM) instance. Depending on the type of Kubernetes Service, Cloud DNS resolves the Service name to its virtual IP address or the list of endpoint IP addresses.

After the Pod frontend resolves the IP address, the Pod can send traffic to the Service backend and any Pods behind the Service.

DNS scopes

Cloud DNS has two kinds of DNS scopes, GKE cluster scope and Virtual Private Cloud (VPC) scope. A cluster cannot operate in both modes simultaneously.

  • GKE cluster scope: DNS records are only resolvable within the cluster, which is the same behavior as kube-dns. Only nodes running in the GKE cluster can resolve Service names. By default, clusters have DNS names that end in *.cluster.local. These DNS names are only visible within the cluster and do not overlap or conflict with *.cluster.local DNS names for other GKE clusters in the same project. This is the default mode.
  • VPC scope: DNS records are resolvable within the entire VPC. Compute Engine VMs and on-premise clients can connect using Cloud Interconnect or Cloud VPN and directly resolve GKE Service names. You must set a unique custom domain for each cluster, which means that all Service and Pod DNS records are unique within the VPC. This mode reduces communication friction between GKE and non-GKE resources.

Pricing

When Cloud DNS is the DNS provider for GKE, DNS queries from Pods inside the GKE cluster are billed according to Cloud DNS pricing. DNS queries to a cluster scope DNS zone managed by GKE are not charged during preview, but will start billing in general availability (GA).

Queries to a VPC scoped DNS zone managed by GKE are billed using the standard Cloud DNS pricing.

Requirements

Cloud DNS for GKE has the following requirements:

  • GKE version 1.18 or later for new clusters or GKE version 1.19 or later for existing clusters.
  • Cloud SDK version 341.0.0 or later.

Restrictions

Cloud DNS for GKE has the following restrictions:

  • Only available for GKE clusters on Google Cloud.
  • After you enable Cloud DNS for GKE in a cluster, you cannot disable it in the cluster and kube-dns continues to run in the cluster. You can disable kube-dns by scaling the kube-dns Deployment and autoscaler to zero.
  • You cannot change the DNS scope in a cluster after you have set the scope with the --cluster-dns-scope flag.
  • Cloud DNS is a global infrastructure with global dependencies, which is different from DNS providers like kube-dns, where the entire DNS infrastructure exists locally within the cluster. This restriction only applies during preview.
  • Custom stub domains and upstream DNS server configurations apply to the DNS configurations of Pods and nodes. Pods using host networking or processes that run directly on the host also use the stub domain and upstream nameserver configurations.
  • Custom stub domains and upstream nameservers configured through the kube-dns Configmap are automatically applied to Cloud DNS for cluster scope DNS. VPC scope DNS ignores the kube-dns ConfigMap and you must apply these configurations directly on Cloud DNS.
  • For VPC scope, the services secondary range must not be shared with any other clusters in that subnetwork.

Before you begin

Before you start, make sure you have performed the following tasks:

Set up default gcloud settings using one of the following methods:

  • Using gcloud init, if you want to be walked through setting defaults.
  • Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

  1. Run gcloud init and follow the directions:

    gcloud init

    If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

    gcloud init --console-only
  2. Follow the instructions to authorize gcloud to use your Google Cloud account.
  3. Create a new configuration or select an existing one.
  4. Choose a Google Cloud project.
  5. Choose a default Compute Engine zone for zonal clusters or a region for regional or Autopilot clusters.

Using gcloud config

  • Set your default project ID:
    gcloud config set project PROJECT_ID
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone COMPUTE_ZONE
  • If you are working with Autopilot or regional clusters, set your default compute region:
    gcloud config set compute/region COMPUTE_REGION
  • Update gcloud to the latest version:
    gcloud components update

Cluster scope DNS

In cluster scope DNS, only nodes running in the GKE cluster can resolve service names, and service names do not conflict between clusters. This behavior is the same as kube-dns in GKE clusters, which means that you can migrate clusters from kube-dns to Cloud DNS cluster scope without downtime or changes to your applications.

The following diagram shows how Cloud DNS creates a private DNS zone for a GKE cluster. Only processes and Pods running on the nodes in the cluster can resolve the cluster's DNS records, because only the nodes are in the DNS scope.

Pods on different nodes resolving Services within the GKE cluster
Diagram: Cluster scope DNS

Enabling cluster scope DNS in a new cluster

You can create a cluster with cluster scope Cloud DNS enabled using the following command:

gcloud beta container clusters create CLUSTER_NAME \
  --cluster-dns clouddns --cluster-dns-scope cluster \
  --cluster-version VERSION \
  [--zone ZONE_NAME | --region REGION_NAME]

Replace the following:

  • CLUSTER_NAME: the name of the cluster.
  • VERSION: the GKE version, which must be 1.18 or later. You can also use the --release-channel flag to select a release channel. The release channel must have a default version 1.18 or later.
  • ZONE_NAME or REGION_NAME: the location for the cluster. These arguments are mutually exclusive. For more information, see Types of clusters.

The --cluster-dns-scope cluster flag is optional in the command because cluster is the default value.

Enabling cluster scope DNS in an existing cluster

You can migrate an existing GKE cluster from kube-dns to Cloud DNS cluster scope.

When you migrate an existing cluster, the nodes in the cluster do not use Cloud DNS as a DNS provider until you recreate the nodes.

After you enable Cloud DNS for a cluster, the settings only apply to new GKE nodes that are added to the cluster. When you upgrade a node pool, the nodes are recreated.

You can also migrate clusters that have running applications without interrupting cluster communication by enabling Cloud DNS as a DNS provider in each node pool separately. A subset of the nodes are operational at all times because some node pools use kube-dns and some node pools use Cloud DNS.

In the following steps, you enable Cloud DNS for a cluster and then upgrade your node pools. Upgrading your node pools recreates the nodes. The nodes then use Cloud DNS for DNS resolution instead of kube-dns.

  1. Update the existing cluster:

    gcloud beta container clusters update CLUSTER_NAME \
      --cluster-dns clouddns --cluster-dns-scope cluster
    

    Replace the following:

    • CLUSTER_NAME: the name of the cluster.

    The response is similar to the following:

    Enabling CloudDNS is a one-way operation. Once enabled, this
    configuration cannot be disabled. All the node pools in the cluster
    need to be re-created by the user to start using CloudDNS for DNS
    lookups. It is highly recommended to complete this step shortly after
    enabling CloudDNS.
    Do you want to continue (Y/n)?
    

    After you confirm, the Cloud DNS controller runs on the GKE control plane, but your Pods do not use Cloud DNS for DNS resolution until you recreate your nodes or you add new nodes to the cluster.

  2. Upgrade the node pools in the cluster to use Cloud DNS:

    gcloud beta container clusters upgrade CLUSTER_NAME \
      --node-pool POOL_NAME \
      --cluster-version VERSION
    

    Replace the following:

    • CLUSTER_NAME: the name of the cluster.
    • POOL_NAME: the name of the node pool to upgrade.
    • VERSION: the GKE version, which must be 1.18 or later. You can also use the --release-channel flag to select a release channel. The release channel must have a default version 1.18 or later.

    If the node pool and control plane are running the same version, upgrade the control plane first, as described in Manually upgrading the control plane and then perform the node pool upgrade.

    Confirm the response and repeat this command for each node pool in the cluster. If your cluster has one node pool, omit the --node-pool flag.

  3. Verify the your nodes are using Cloud DNS by connecting to a Pod on a node and running the command cat /etc/resolv.conf:

    kubectl exec -it POD_NAME -- cat /etc/resolv.conf | grep nameserver
    

    Replace POD_NAME with the name of the Pod.

    The output is similar to the following:

    nameserver 169.254.169.254
    

    If the output is an IP address similar to 10.x.y.10, then the Pod is using kube-dns. If the output is 169.254.20.10, then the Pod is using NodeLocalDNS. 169.254.169.254 is the IP address of the metadata server where the Cloud DNS data plane listens for requests on port 53. The nodes no longer use the kube-dns Service address for DNS resolution and all DNS resolution occurs on the local node.

Verifying cluster scope DNS

To verify that Cloud DNS for GKE is working correctly for your cluster, perform the following steps:

  1. Deploy a sample application to your cluster using the following command:

    kubectl run dns-test --image gcr.io/google-samples/hello-app:2.0
    
  2. Expose the sample application with a service using the following command:

    kubectl expose pod dns-test --name dns-test-svc --port 8080
    
  3. Verify that the service deployed successfully:

    kubectl get svc dns-test-svc
    

    The output is similar to the following:

    NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
    dns-test-svc   ClusterIP   10.47.255.11    <none>        8080/TCP   6m10s
    

    The value of CLUSTER-IP is the virtual IP address for your cluster. In this example, the virtual IP address is 10.47.255.11.

  4. Connect to a Pod using kubectl exec and resolve the service name using the nslookup command to confirm that Cloud DNS is serving DNS records within your cluster:

    kubectl exec -it dns-test -- nslookup dns-test-svc
    

    The output is similar to the following:

    Server:   169.254.169.254
    Address:  169.254.169.254#53
    
    Non-authoritative answer:
    Name: dns-test-svc.default.svc.cluster.local
    Address: 10.47.255.11
    

    The response shows that Cloud DNS resolved the service name dns-test-svc to the virtual IP of the cluster, 10.47.255.11.

Cleaning up

After completing the exercise, follow these steps to remove resources and prevent unwanted charges incurring on your account:

  1. Delete the service:

    kubectl delete service dns-test-svc
    
  2. Delete the Pod:

    kubectl delete Pod dns-test
    
  3. You can also delete the cluster.

VPC scope DNS

In VPC scope DNS, a cluster's DNS names are resolvable within the entire VPC. Any client in the VPC can resolve cluster DNS records.

VPC scope DNS enables the following use cases:

  • Headless service discovery for non-GKE clients within the same VPC.
  • GKE Service resolution from on-premise or 3rd party cloud clients.
  • Service resolution where a client can decide which cluster they want to communicate with using the custom cluster DNS domain.

In the following diagram, two GKE clusters use VPC scope DNS in the same VPC. Both clusters have a custom DNS domain, .cluster1 and .cluster2, instead of the default .cluster.local domain. A VM communicates with the headless backend Service by resolving backend.default.svc.cluster1. Cloud DNS resolves the headless Service to the individual Pod IPs in the Service and the VM communicates directly with the Pod IPs.

Clients resolving to headless Services from outside the GKE cluster
Diagram: VPC scope DNS

You can also perform this type of resolution from other networks when connected to the VPC through Cloud Interconnect or Cloud VPN. DNS server policies enable clients from connected networks to resolve names in Cloud DNS, which includes GKE services if the cluster is using VPC scope DNS.

Enabling VPC scope DNS in a new cluster

You can create a new cluster with VPC scope Cloud DNS enabled using the following command:

gcloud beta container clusters create CLUSTER_NAME \
  --cluster-dns clouddns --cluster-dns-scope vpc \
  --cluster-version VERSION \
  --cluster-dns-domain CUSTOM_DOMAIN \
  [--zone ZONE_NAME | --region REGION_NAME]

Replace the following:

  • CLUSTER_NAME: the name of the cluster.
  • VERSION: the GKE version, which must be 1.18 or later. You can also use the --release-channel flag to select a release channel. The release channel must have a default version 1.18 or later.
  • ZONE_NAME or REGION_NAME: the location for the cluster. These arguments are mutually exclusive. For more information, see Types of clusters.
  • CUSTOM_DOMAIN: the name of a domain. You must ensure this name is unique within the VPC because GKE does not confirm this value. You cannot change this value after it is set. You must not use a domain that ends in ".local", or you might experience DNS resolution failures.

Using Cloud DNS with Shared VPC

Cloud DNS for GKE supports Shared VPC for both VPC and cluster scope.

The GKE controller creates a managed private zone in the same project as the GKE cluster.

The GKE service account for the GKE cluster does not require Identity and Access Management (IAM) for DNS outside of its own project because the managed zone and GKE cluster reside within the same project.

Supporting custom stub domains and upstream nameservers

Cloud DNS for GKE supports custom stub domains and upstream nameservers configured using kube-dns ConfigMap. This support is only available for GKE cluster scope.

Cloud DNS translates stubDomains and upstreamNameservers values into Cloud DNS forwarding zones.

Troubleshooting

See Debugging DNS Resolution for general information about diagnosing Kubernetes DNS issues and Troubleshooting to learn more about diagnosing issues with Cloud DNS.

You can also enable Cloud DNS Logging.

Unable to update existing cluster or create cluster with Cloud DNS enabled

Ensure you are using the correct version. Cloud DNS for GKE requires GKE version 1.18 or later to create new clusters, or GKE version 1.19 or later for existing clusters.

Pods use kube-dns even after Cloud DNS is enabled on an existing cluster

Ensure you have upgraded or recreated your node pools after you enable Cloud DNS on the cluster. Until this step is complete, Pods continue to use kube-dns.

Pod is unable to resolve DNS lookups

  1. Verify the Pod is using Cloud DNS by connecting to a Pod and running the command cat /etc/resolv.conf:

    kubectl exec -it POD_NAME -- cat /etc/resolv.conf | grep nameserver
    

    Replace POD_NAME with the name of the Pod.

    The output is similar to the following:

    nameserver 169.254.169.254
    

    If the output is an IP address similar to 10.x.y.10, then the Pod is using kube-dns. If the output is 169.254.20.10, then the Pod is using NodeLocalDNS.

  2. Confirm the managed zone exists and contains the necessary DNS entry:

    gcloud dns managed-zones list --format list
    

    The output is similar to the following:

     - creationTime: 2021-02-12T19:24:37.045Z
       description: Private zone for GKE cluster "cluster1" with cluster suffix "cluster.local" in project "project-243723"
       dnsName: cluster.local.
       id: 5887499284756055830
       kind: dns#managedZone
       name: gke-cluster1-aa94c1f9-dns
       nameServers: ['ns-gcp-private.googledomains.com.']
       privateVisibilityConfig: {'kind': 'dns#managedZonePrivateVisibilityConfig'}
       visibility: private
    

    The value of name in the response shows that Google Cloud created a private zone named gke-cluster1-aa94c1f9-dns.

  3. Verify that Cloud DNS contains entries for your cluster:

    gcloud dns record-sets list --zone ZONE_NAME | grep SERVICE_NAME
    

    Replace the following:

    • ZONE_NAME: the name of the private zone.
    • SERVICE_NAME: the name of the service.

    The output shows that Cloud DNS contains an A record for the domain dns-test.default.svc.cluster.local. and the IP address of your cluster, 10.47.255.11:

    dns-test.default.svc.cluster.local.                A     30     10.47.255.11
    
  4. Enable Cloud DNS logging to track queries. Every log entry contains information about the query, including DNS latency.

DNS lookups on nodes fail after enabling Cloud DNS on a cluster

If you enable cluster scope Cloud DNS in a GKE cluster that has custom stub domains or upstream nameservers, the custom config applies to both nodes and Pods in the cluster because Cloud DNS cannot distinguish between Pod and node DNS requests. DNS lookups on nodes might fail if the custom upstream server cannot resolve the queries.

What's next