Troubleshooting

The troubleshooting page describes common issues you might encounter using Cloud DNS when creating private zones, forwarding zones, and reverse lookup zones, and tells you how to handle them:

Private zones

This section covers issues with private zones.

Private zones problems in Shared VPC service projects

Refer to Private zones and Shared VPC on the overview page for important information about using private zones with Shared VPC networks.

Cannot create a private zone, cannot list or create policies

You must have the DNS Admin role to perform various private zone actions, such as listing DNS server policies or creating a private zone. This role can be granted through the Cloud IAM tool.

Private zones are not resolving in the same VPC network

Make sure that you are performing the test from a VM instance whose primary interface is on the same network as the private zone that you have created

A VM sends all traffic out of its primary interface unless the traffic is destined for a subnet attached to one of the other interfaces or if the VM has policy-based routing configured. Make sure your test VM has its primary interface on the same network as the private zone that you're testing.

Determine that your VM is using:

gcloud compute instances describe ${vm_name} \
     --zone=${gce_zone} \
     --format="csv[no-heading](networkInterfaces['network'])"

Ensure that network is in the list of networks authorized to search your private zone:

gcloud dns managed-zones describe ${private-zone-name} \
     --format="csv(privateVisibilityConfig['networks'])"

Verify that the DNS name in the query matches your zone

Google Cloud uses longest-suffix matching to decide which zone to query for a given DNS name. Make sure the suffix for the record you're querying matches at least one private zone accessible in your VPC network. For example, Google Cloud first looks for myapp.dev.gcp.example.lan in a zone that serves dev.gcp.example.lan, if accessible, before it looks for it in a zone that serves gcp.example.lan, if accessible.

The output of the following command shows the DNS suffix for a given private zone:

gcloud dns managed-zones describe ${private-zone-name} \
--format="csv[no-heading](dnsName)"

Query for the DNS name using the metadata server

Use dig to submit the DNS name query directly to the Google Cloud metadata server, 169.254.169.254:

dig ${dns-name} @169.254.169.254

Use dig to query the VM's default name server:

dig ${dns-name}

If the output of the two dig commands produces different answers, check the ;; SERVER: section of the second command. The responding server must be the metadata server, 169.254.169.254. If it is not, then you've configured the guest operating system to use a custom DNS name server instead of the default Google Cloud metadata server. Cloud DNS private zones require that you use the metadata server for name resolution. Both the Linux Guest Environment and the Windows Guest Environment do this for you. If you've imported the image you're using for a VM, verify that the appropriate guest environment has been installed.

Private zones not resolving through Cloud VPN or Cloud Interconnect

First make sure you can successfully query and resolve the DNS name from within an authorized VPC network.

Verify connectivity through Cloud VPN or Cloud Interconnect

Ensure that connectivity from an on-premises system to your VPC network is operational. Specifically, TCP and UDP traffic on port 53 must be able to leave your on-premises network and be delivered to your VPC network. Verify the configuration of on-premises firewalls to make sure that such egress traffic is allowed.

You can use any protocol, such as ping (ICMP), to test connectivity to a sample VM in your VPC network from on-premises. While Cloud DNS requests are not sent to Google Cloud VMs, testing connectivity to a sample VM allows you to verify connectivity through a Cloud VPN tunnel or Cloud Interconnect. Make sure you configure an appropriate ingress allow firewall rule for the sample Google Cloud VM because the implied deny ingress rule blocks all incoming traffic otherwise.

Ensure inbound forwarding is enabled for the authorized VPC network

Inbound forwarding must be enabled for each VPC network that is authorized to query your Cloud DNS private zone. Use the following command to list all policies:

gcloud dns policies list

Identify the network in the output table, and check for the Forwarding field to see if forwarding is enabled.

Make sure the authorized network is a VPC network

DNS Forwarding requires subnets, which are only available to VPC networks, not legacy networks.

gcloud compute networks list \
     --format="csv[no-heading](name, SUBNET_MODE)"

Legacy networks are identified in the output as LEGACY.

Make sure there is a valid DNS forwarding address reserved in that network

The following command shows all the reserved DNS forwarder IP addresses in your project.

gcloud compute addresses list \
     --filter="purpose=DNS_RESOLVER" \
     --format='csv[no-heading](address, subnetwork)'

You can add an AND clause to the filter to limit the output to only the subnetwork that you care about.

Example: --filter="name ~ ^dns-forwarding AND subnetwork ${subnetwork-name}"

If you do not see an IP address in the network/region you expected, please file a ticket with Cloud Support.

Perform dig command pointing the query at the address that you found above. Do so from a VM in the same network. This test verifies that inbound forwarder is working, and the problem lies elsewhere.

dig ${dns-name} @10.150.0.1 # - address returned by command above.

Repeat the same dig command but from a VM across the VPN.

CNAME record defined in a private zone is not working

CNAME records that refer to other records in the same private zone are supported. CNAME records that point to domain names defined in other private zones, .internal zones, or the Internet are not supported.

Reverse Lookup Zones

This section provides troubleshooting tips for reverse lookup zones. Some private zone issues also apply to reverse lookup zones. Please consult the Private zones troubleshooting section for additional tips.

VM with non-RFC 1918 address not resolving

Reconciliate your VM with a non-RFC 1918 address

If you created a VM during non-RFC 1918 alpha before Cloud DNS support was launched, these VMs might not resolve correctly. To resolve this issue, pause and restart your VMs.

Forwarding zones

This section provides troubleshooting tips for forwarding zones. Some private zone issues also apply to forwarding zones. Please consult the Private zones troubleshooting section for additional tips.

Forwarding zones (Outbound forwarding) not working

First make sure you can successfully query and resolve the DNS name from within an authorized VPC network.

Forwarding queries from VMs in a consumer VPC network to a producer VPC network not working

If you are using DNS peering and you want to forward queries from VMs in a consumer VPC network to a producer VPC network, and then to one or more on-premises name servers, you must ensure that the producer VPC network has a VM or a VPN tunnel in the same region as the subnetwork used by the client VMs. For example, suppose that VPC1 uses a peering zone that sends queries for example.com. to VPC2. Suppose also that VPC2 has a private forwarding zone for example.com. that forwards the queries to an on-premises name server using a Cloud VPN tunnel or Cloud Interconnect. For a VM located in us-east1 in VPC1 to be able to query example.com., VPC2 must also have a VM or VPN tunnel located in us-east1.

Verify connectivity through Cloud VPN or Cloud Interconnect

You can use any protocol, such as ping (ICMP), to test connectivity to a sample VM in your VPC network from on-premises. You must also attempt to query your on-premises name server directly from a sample Google Cloud VM using a tool like dig:

dig ${dns-name} @192.168.x.x # - address of the onprem DNS server.

Check firewall rules in your VPC network

The implied allow egress firewall rule allows outbound connections and established responses from VMs in your VPC network unless you have created custom rules to deny egress. If you've created custom deny egress rules, you'll need to create higher priority allow rules that permit egress to at least your on-premises name servers.

Check on-premises firewall

Make sure that your on-premises firewall is configured to allow incoming traffic from and outgoing traffic to 35.199.192.0/19.

Check logs in your on-premises firewall, looking for DNS requests from 35.199.192.0/19. To search using a regex expression, use "35\.199\.(19[2-9]|20[0-9]|21[0-9]|22[0-3]).*"

Check on-premises name server

Verify that you do not have any ACL configured in your on-premises name server that would block queries from 35.199.192.0/19.

Check your on-premises name server to see if it is receiving packets from 35.199.192.0/19. If you have shell access, you can use a tool such as tcpdump to look for both incoming packets from and outgoing packets to 35.199.192.0/19:

sudo tcpdump port 53 and tcp -vv

Verify return routes

Your on-premises network must have a route to the 35.199.192.0/19 destination with a next hop being a VPN tunnel for the same VPC network that sent the DNS request as described in creating forwarding zones.

If you use multiple VPN tunnels to connect the same VPC network to your on-premises network, the response does not have to be delivered using the same tunnel that sent it. However, the response must be delivered using a VPN tunnel with a next hop in the same VPC network from which the request originated.

If you have connected more than one VPC network to your on-premises network, you must ensure that DNS replies are not sent to the wrong one. Google Cloud discards DNS responses sent to the wrong VPC network.

CNAME record defined in a private zone is not working

Cloud DNS does not support recursively resolving an IP address via a CNAME that points to an A record in another managed private zone (CNAME chasing). For example, suppose you have two managed private zones, zone-a.private and zone-b.private, and you create records like this:

  • CNAME cname.zone-a.private pointing to target.zone-b.private
  • A record target.zone-b.private pointing to 192.168.1.9

Cloud DNS does not resolve cname.zone-a.private as 192.168.1.9 because the two records are not in the same zone (zone-a.private and zone-b.private are different).

Cloud DNS does chase CNAMEs within a managed private zone.

Outbound forwarding to a name server that uses a non-RFC 1918 IP address fails

By default, Cloud DNS uses standard routing. That is, Cloud DNS routes queries through the public internet when the target name server has a non-RFC 1918 IP address. Using standard routing, name servers with non-RFC 1918 addresses that are not reachable from the public internet cannot be used as forwarding targets.

In order to forward queries to a name server that uses a non-RFC 1918 IP address through your VPC network, configure the Cloud DNS forwarding zone or server policy to use private routing for the target name server.

For an explanation of the differences between standard and private routing, see forwarding targets and routing methods.

This limitation applies even if there is a VPC route for the target name server; routes for non-RFC 1918 addresses have no effect on Cloud DNS's outbound forwarding behavior when standard routing is configured.

Outbound forwarding to a secondary NIC fails

Cloud DNS does not currently support forwarding to secondary NICs of Compute Engine VMs.

Outbound forwarded queries receive SERVFAIL errors

The Cloud DNS control plane uses an algorithm to determine the responsiveness of a forwarding target. The responsiveness score is based on the number of SERVFAIL errors and the latency of responses received. SERVFAIL errors have a greater impact on the score than latency. The scores reset every hour.

If all of the targets for a zone have low scores, Cloud DNS does not send queries to those targets. You may see one or more of these symptoms:

  • Using Cloud DNS forwarding results in a quick (~2 ms) SERVFAIL error
  • In the Cloud Logging logs for the queries, the fields destinationIP, egressIP, and egressError are not populated
  • You don't see the query attempts on your DNS servers
  • Queries that you send directly to the forwarding targets (for example, dig @<TARGET> <QNAME>) may succeed

To resolve this issue, verify that your on-premises name servers are properly configured. Then, verify that your on-premises name servers respond to queries within 4 seconds. If your on-premises name servers consult upstream name servers (for example, due to being configured as caching resolvers), check that there are no issues with upstream name servers.

Inbound forwarding on a VPC using non-RFC 1918 address ranges fails

Cloud DNS does not support inbound forwarding if the VPC is using a non-RFC 1918 address range.

Next steps

  • The Overview provides more context on features.
  • See the Support area for pointers to additional help.