Troubleshooting

This page provides solutions for common issues that you might encounter when using Cloud DNS to create private zones, reverse lookup zones, forwarding zones, and resource records.

Private zones

This section covers issues with private zones.

Private zone problems in Shared VPC service projects

For important information about using private zones with Shared VPC networks, see Private zones and Shared VPC.

Cannot create a private zone, cannot list or create policies

You must have the DNS Administrator role to perform various private zone actions, such as listing Domain Name System (DNS) server policies or creating a private zone. This role can be granted through the Identity and Access Management (IAM) tool.

Private zones not resolving in the same VPC network

Make sure that you are performing the test from a VM instance whose primary interface is on the same network as the private zone that you have created

A virtual machine (VM) instance sends all traffic out of its primary interface unless the traffic is destined for a subnet attached to one of the other interfaces or if the VM has policy routing configured. Make sure that your test VM has its primary interface on the same network as the private zone that you're testing.

Determine that your VM is using:

gcloud compute instances describe ${VM_NAME} \
     --zone=${GCE_ZONE} \
     --format="csv[no-heading](networkInterfaces['network'])"

Ensure that the network is in the list of networks authorized to query your private zone:

gcloud dns managed-zones describe ${PRIVATE_ZONE_NAME} \
     --format="csv(privateVisibilityConfig['networks'])"

Verify that the DNS name in the query matches your zone

Google Cloud uses longest-suffix matching to decide which zone to query for a given DNS name. Make sure that the suffix for the record that you're querying matches at least one private zone accessible in your VPC network. For example, Google Cloud first looks for myapp.dev.gcp.example.lan in a zone that serves dev.gcp.example.lan, if accessible, before it looks for it in a zone that serves gcp.example.lan, if accessible.

The output of the following command shows the DNS suffix for a given private zone:

gcloud dns managed-zones describe ${PRIVATE_ZONE_NAME} \
--format="csv[no-heading](dnsName)"

Query for the DNS name using the metadata server

Use dig to submit the DNS name query directly to the Google Cloud metadata server, 169.254.169.254:

dig ${DNS_NAME} @169.254.169.254

Use dig to query the VM's default name server:

dig ${DNS_NAME}

If the output of the two dig commands produces different answers, check the ;; SERVER: section of the second command. The responding server must be the metadata server, 169.254.169.254. If it is not, then you've configured the guest operating system to use a custom DNS name server instead of the default Google Cloud metadata server. Cloud DNS private zones require that you use the metadata server for name resolution. Both the Linux guest environment and the Windows guest environment do this for you. If you've imported the image that you're using for a VM, verify that the appropriate guest environment has been installed.

Private zones not resolving through Cloud VPN or Cloud Interconnect

First make sure that you can successfully query and resolve the DNS name from within an authorized VPC network.

Verify connectivity through Cloud VPN or Cloud Interconnect

Ensure that connectivity from an on-premises system to your VPC network is operational. Specifically, TCP and UDP traffic on port 53 must be able to leave your on-premises network and be delivered to your VPC network. Verify the configuration of on-premises firewalls to make sure that such egress traffic is allowed.

You can use any protocol, such as ping (ICMP), to test connectivity to a sample VM in your VPC network from on-premises. While Cloud DNS requests are not sent to Google Cloud VMs, testing connectivity to a sample VM allows you to verify connectivity through a Cloud VPN tunnel or Cloud Interconnect connection. Make sure that you configure an appropriate ingress allow firewall rule for the sample Google Cloud VM because the implied deny ingress rule blocks all incoming traffic otherwise.

Ensure inbound forwarding is enabled for the authorized VPC network

Inbound forwarding must be enabled for each VPC network that is authorized to query your Cloud DNS private zone. Use the following command to list all policies:

gcloud dns policies list

Identify the network in the output table, and check for the Forwarding field to see if forwarding is enabled.

Make sure the authorized network is a VPC network

DNS forwarding requires subnets, which are only available to VPC networks, not legacy networks.

gcloud compute networks list \
     --format="csv[no-heading](name, SUBNET_MODE)"

Legacy networks are identified in the output as LEGACY.

Make sure there is a valid DNS forwarding address reserved in that network

The following command shows all the reserved DNS forwarding IP addresses in your project.

gcloud compute addresses list \
     --filter="purpose=DNS_RESOLVER" \
     --format='csv[no-heading](address, subnetwork)'

You can add an AND clause to the filter to limit the output to only the subnetwork that you care about.

Example:

--filter="name ~ ^dns-forwarding AND subnetwork ${subnetwork-name}"

If you do not see an IP address in the network/region that you expected, file a ticket with Google Cloud Support.

Run the dig command pointing the query at the address that you found in the preceding command. Do so from a VM in the same network. This test verifies that the inbound forwarder is working, and that the problem lies elsewhere.

dig ${DNS_NAME} @10.150.0.1 # address returned by previous command

Repeat the same dig command but from an on-premises host across the VPN.

CNAME record defined in a private zone not working

CNAME records in private Cloud DNS zones are supported by the Compute Engine DNS resolver when the target is one of the following:

CNAME records cannot target any other resource. For more information, see CNAME chasing.

Reverse lookup zones

This section provides troubleshooting tips for reverse lookup zones. Some private zone issues also apply to reverse lookup zones. For additional tips, see the Private zones troubleshooting section.

VM with non-RFC 1918 address not resolving

Reconciliate your VM with a non-RFC 1918 address

If you created a VM during the non-RFC 1918 alpha before Cloud DNS support was launched, these VMs might not resolve correctly. To resolve this issue, pause and restart your VMs.

Forwarding zones

This section provides troubleshooting tips for forwarding zones. Some private zone issues also apply to forwarding zones. For additional tips, see the Private zones troubleshooting section.

Forwarding zones (outbound forwarding) not working

First make sure that you can successfully query and resolve the DNS name from within an authorized VPC network.

Forwarding queries from VMs in a consumer VPC network to a producer VPC network not working

If you are using DNS peering and you want to forward queries from VMs in a consumer VPC network to a producer VPC network, and then to one or more on-premises name servers, you must ensure that the producer VPC network has a VM or a VPN tunnel in the same region as the subnetwork used by the client VMs.

For example, suppose that VPC1 uses a peering zone that sends queries for example.com. to VPC2. Suppose also that VPC2 has a private forwarding zone for example.com. that forwards the queries to an on-premises name server using a Cloud VPN tunnel or Cloud Interconnect. For a VM located in us-east1 in VPC1 to be able to query example.com., VPC2 must also have a VM or VPN tunnel located in us-east1.

Verify connectivity through Cloud VPN or Cloud Interconnect

You can use any protocol, such as ping (ICMP), to test connectivity to a sample VM in your VPC network from on-premises. You must also attempt to query your on-premises name server directly from a sample Google Cloud VM by using a tool like dig:

dig ${DNS_NAME} @192.168.x.x # address of the onprem DNS server

Check firewall rules in your VPC network

The implied allow egress firewall rule allows outbound connections and established responses from VMs in your VPC network unless you have created custom rules to deny egress. If you've created custom deny egress rules, you'll need to create higher priority allow rules that permit egress to at least your on-premises name servers.

Check on-premises firewall

Make sure that your on-premises firewall is configured to allow incoming traffic from and outgoing traffic to 35.199.192.0/19.

Check logs in your on-premises firewall, looking for DNS requests from 35.199.192.0/19. To use a regex expression to search, use the following:

"35\.199\.(19[2-9]|20[0-9]|21[0-9]|22[0-3]).*"

Check on-premises name server

Verify that you do not have any ACL configured in your on-premises name server that would block queries from 35.199.192.0/19.

Check your on-premises name server to see if it is receiving packets from 35.199.192.0/19. If you have shell access, you can use a tool such as tcpdump to look for both incoming packets from and outgoing packets to 35.199.192.0/19:

sudo tcpdump port 53 and tcp -vv

Verify return routes

Your on-premises network must have a route to the 35.199.192.0/19 destination with the next hop being a VPN tunnel or Interconnect connection for the same VPC network that sent the DNS request. This behavior is described in Creating forwarding zones.

If you use multiple VPN tunnels to connect the same VPC network to your on-premises network, the response does not have to be delivered using the same tunnel that sent it. However, the response must be delivered by using a VPN tunnel with a next hop in the same VPC network from which the request originated.

If you have connected more than one VPC network to your on-premises network, you must ensure that DNS replies are not sent to the wrong one. Google Cloud discards DNS responses sent to the wrong VPC network. For a recommended solution, see our Best practices guide.

Outbound forwarding to a name server that uses a non-RFC 1918 IP address fails

By default, Cloud DNS uses standard routing, which routes queries through the public internet when the target name server has a non-RFC 1918 IP address. Standard routing doesn't support forwarding queries to name servers with non-RFC 1918 addresses that are not reachable from the public internet.

To forward queries to a name server that uses a non-RFC 1918 IP address through your VPC network, configure the Cloud DNS forwarding zone or server policy to use private routing for the target name server.

For an explanation of the differences between standard and private routing, see forwarding targets and routing methods.

This limitation applies even if there is a VPC route for the target name server; routes for non-RFC 1918 addresses have no effect on Cloud DNS's outbound forwarding behavior when standard routing is configured.

Outbound forwarding to a secondary NIC fails

Cloud DNS does not currently support forwarding to secondary NICs of Compute Engine VMs.

Outbound forwarded queries receive SERVFAIL errors

The Cloud DNS control plane uses an algorithm to determine the responsiveness of a forwarding target. The responsiveness score is based on the number of SERVFAIL errors and the latency of responses received. SERVFAIL errors have a greater impact on the score than latency. The scores reset every hour.

If all of the targets for a zone have low scores, Cloud DNS does not send queries to those targets. You might see one or more of these symptoms:

  • Using Cloud DNS forwarding results in a quick (~2 ms) SERVFAIL error.
  • In the Cloud Logging logs for the queries, the fields destinationIP, egressIP, and egressError are not populated.
  • You don't see the query attempts on your DNS servers.
  • Queries that you send directly to the forwarding targets (for example, dig @<TARGET> <QNAME>) might succeed.

To resolve this issue, verify that your on-premises name servers are properly configured. Then, verify that your on-premises name servers respond to queries within 4 seconds. If your on-premises name servers consult upstream name servers (for example, due to being configured as caching resolvers), check that there are no issues with upstream name servers.

Additionally, if the forwarding target is an on-premises system, be aware that the routes configured for that path can be custom dynamic routes or custom static routes, with the important exception that custom static routes with network tags are not valid for forwarding DNS queries. Ensure that the route used to reach the on-premises name server does not specify a network tag.

Inbound forwarding on a VPC network that uses non-RFC 1918 IP address ranges fails

Cloud DNS does not support inbound forwarding if the VPC network is using a non-RFC 1918 IP address range.

Resource records

This section provides troubleshooting tips for resource records.

Unexpected data when querying for resource record sets present in the zone

There are several reasons why you might receive unexpected data when querying for resource record sets present in a Cloud DNS managed zone:

  1. Resource record sets created by using the @ syntax specified in RFC 1035 are not supported. Cloud DNS interprets the @ symbol in such resource record sets literally; therefore, in the example.com. zone, a record set created for the QNAME @ is interpreted as @.example.com. instead of example.com.. To resolve this issue, ensure that you create record sets without the @ symbol; all names are relative to the apex of the zone.

  2. Like all records, wildcard CNAME records are subject to the existence rules described in RFC 4592. For example, suppose that you define the following record sets in the example.com. zone:

    *.images.example.com. IN CNAME _static.example.com.

    srv1.images.example.com. IN A 192.0.2.91

    _static.example.com. IN A 192.0.2.92

    A query for public.srv1.images.example.com. returns NOERROR with an empty answer section. The existence of a record between the CNAME and the QNAME prevents the CNAME from being returned, but there's no record that matches the QNAME exactly, so Cloud DNS returns an empty answer. This is standard DNS behavior.

Cloud DNS resource record sets returned in random order

Consistent with DNS industry practices, Cloud DNS name servers now randomize the order of the resource record sets and ignore the ordering given by the authoritative server. This is normal DNS behavior and applies to both public and private Cloud DNS zones.

What's next