Service producer unable to access consumers Private DNS zones

Problem

Issue connecting to On-Premise service using the proxy deployed in APIGEE X.

Private Service Access with servicenetworking.googleapis.com and required routes are established, but when an API request is triggered, APIGEE is experiencing the following error.
{"fault":{"faultstring":"Unable to resolve host abc.example.com","detail":{"errorcode":"protocol.http.NoResolvedHost","reason":"TARGET_CONNECT_HOST_NOT_REACHABLE"}}}
In addition, Service producer is unable to resolve DNS records in consumer managed Private DNS Zones through Private Service Access and when you run the following command in cloud shell, nothing is returned:
gcloud services peered-dns-domains list --network=VPC_NETWORK

Environment

Project A:
  • Transit project with VPN connecting to on-premise.
  • Has a Cloud DNS forwarding zone for the Domain "example.com" which forwards the request to three on-premise DNS servers.
  • All required routes are in places.
Project B:
  • Host projects with a Cloud DNS peering zone for the domain "example.com" and peers with Project A.
  • VPC has Private Service Access created for Apigee, which in-turn created a Connection Named: "servicenetworking-googleapis-com" and a VPC network peering "servicenetworking-googleapis-com" with Google managed tenant project.
  • VM's in host project can resolve abc.example.com

Solution

Cloud DNS private zones are private to your VPC network. If you want to let a service producer network resolve names from your private zone, you need to configure DNS peering between the two networks.

From the host project, run the commands (listed here).
gcloud services peered-dns-domains create <PEERING_NAME> \
    --network=<VPC_NETWORK> \
    --dns-suffix=<DNS_SUFFIX>

Replace the following placeholders with relevant values:
  • PEERING_NAME: a name for this DNS peering configuration.
  • VPC_NETWORK: the name of your VPC network that is connected to the service producer using private services access.
  • DNS_SUFFIX: the DNS suffix you want to peer with the service producer. You must provide a complete DNS domain name, including the dot. For example, example.com. is a valid DNS suffix.

Cause

The existing DNS peering zone in the host project is the peering from Host to Transit project. This will not provide the Service Producer Project's (Tenant Project) VPC access to the consumer's private zones. For this, a Cloud DNS peering zone for the domain 'example.com' (or any Private domain name) needs to be created within the Tenant Project.  The solution does that for you as you do not have access to the Tenant Project.

The 'gcloud services peered-dns-domains list' command returned nothing initially because there was not a DNS peering zone in the Tenant Project for the service 'servicenetworking-googleapis-com'.