Enable DNS queries log in Google Kubernetes Engine

Problem

When you enable logging of DNS queries on Google Kubernetes Engine temporarily for troubleshooting, any changes made to the Google Kubernetes Engine kube-dns deployment are removed shortly after, preventing further diagnostic.

Environment

• Google Kubernetes Engine version 1.17.15-gke.800

Solution

In order to log DNS queries, a new kube-dns debug pod with log-queries enabled needs to be created.

Please note this can be very resource intensive and is not recommended to be done for long periods of time in production environments, the jq Open Source tool is needed to handle JSON. 

1. Step 1 Create the kube-dns-debug based on an existing kube-dns pod:

   $ POD=$(kubectl -n kube-system get pods --selector=k8s-app=kube-dns -o jsonpath="{.items[0].metadata.name}")
   
   $ kubectl apply -f <(kubectl get pod -n kube-system ${POD} -o json | jq -e '
   
   (
   
   (.spec.containers[] | select(.name == "dnsmasq") | .args) += ["--log-queries"]
   
   )
   
   | (.metadata.name = "kube-dns-debug")
   
   | (del(.metadata.labels."pod-template-hash"))
   
   ')
   
   
   pod "kube-dns-debug" created
   

2. Step 2 You should then see logs start to show in the Cloud Logging, as with other workloads. You can also inspect logs using kubectl:

   $ kubectl logs -f --tail 100 -c dnsmasq -n kube-system kube-dns-debug

3. Step 3 It is important to remember to clear this configuration when done inspecting DNS queries, by deleting the kube-dns-debug pod:

   $ kubectl -n kube-system delete pod kube-dns-debug
   
   pod "kube-dns-debug" deleted
   

Cause

The kube-dns is a managed workload and its settings are not currently easy to adjust, as the kube-dns deployment is set to reconcile. Any changes to it are reverted shortly after they are made.