DNS resolution is available in private instances running in Cloud Data Fusion version 6.7.0 or later. It lets you use hostnames or domain names for sources and sinks when you design your pipeline in a private Cloud Data Fusion instance so that you don't have to change the pipeline definition during failover.
Without it, you use hard-coded IP addresses, which are problematic when the addresses change in the event of a failover in highly-available deployments. DNS resolution is recommended when you design a pipeline in a private instance and want to retrieve schema, run previews, wrangle, and test connections.
For more information, see DNS resolution in Cloud Data Fusion.
Before you begin
Create a private instance
Create a private instance in Cloud Data Fusion and set up a VPC network or shared VPC network in the instance with the following steps.
Create a private Cloud Data Fusion instance. When you create the instance, choose the following options:
- Choose version 6.7.0 or later.
- In the Network field, choose the appropriate associated network for the private connection.
- Enter any other instance details and click Create. For information about all fields, see Create a private Cloud Data Fusion instance.
Cloud Data Fusion uses VPC Network Peering to establish network connectivity to your VPC or shared VPC network. This lets Cloud Data Fusion access resources on your network through internal IP addresses.
When you set it up, do the following:
- Enter a Name for your peering connection.
- For Your VPC network, choose the network where you created your Cloud Data Fusion instance.
- For Peered VPC network, choose In another project.
- For Project ID, enter the tenant project ID.
For VPC network name, enter
Replace the following:
INSTANCE_REGION: the region in which you created your Cloud Data Fusion instance.
INSTANCE_ID: the ID of the Cloud Data Fusion instance.
For Exchange custom routes, click Export custom routes. This allows for exchanging any custom routes defined in your VPC network with the tenant VPC network.
If the VPC Network Peering is successfully set up, the Status on the VPC network details page is Active.
Create a managed private DNS zone
To resolve domain names in Cloud Data Fusion, create a new managed private DNS zone in the same VPC network or shared VPC network that was used for VPC network peering. For more information, see Create a private zone.
To create a private managed DNS zone, do the following:
- In the Google Cloud console, go to the Create a DNS zone page.
- For Zone name, enter a name.
- For Zone type, click Private.
- For DNS name, enter a domain name.
- For Options, click Default (private).
- For Networks, select the VPC network or shared VPC network that was used for VPC Network Peering.
- Click Create.
Create a record set in the DNS zone
Create the record set in the DNS zone that has the internal IP addresses to resolve when you design your pipeline. For more information, see Add a record.
To create the record set, do the following:
In the Google Cloud console, go to the Cloud DNS page.
Click the zone for which you want to create a resource record set.
On the Zone details page, click Add record set.
For DNS name, enter a domain name that must be resolved during design time.
Enter the record type details. For the IPv4 address, provide the internal IP address for the sink or source to be used in the pipeline. For more information about record types, see Add a record.
Add DNS Peering in an instance
To create DNS peering in the Google Cloud console, do the following:
Open your instance.
In the Google Cloud console, go to the Cloud Data Fusion page.
Click Instances, and then click the instance's name to go to the Instance details page.
Click Add DNS peering.
The Add DNS peering dialog opens.
For DNS peering ID, enter a name.
For Domain, enter the domain name that you used for creating the private DNS zone.
For the Target project, select the project name where the private DNS zone was created.
For the Network, select the VPC network name where the private DNS zone was created.
Click Save to create the DNS peering.
To resolve domain and hostnames with the Cloud Data Fusion DNS Peering
API, call its
For more information and methods, see the Cloud Data Fusion DNS Peering API reference.
DNS resolution in shared VPC networks
To create the Cloud DNS private zones in a shared VPC network, create a private zone in the host project and authorize access to the zone for the shared VPC network or set up the zone in a service project using cross-project binding. For more information, see Best practices for Cloud DNS private zones.
Required roles for shared VPC networks
To ensure that the Cloud Data Fusion Service Account has the necessary
permissions to create a DNS peering in a shared VPC network,
ask your administrator to grant the Cloud Data Fusion Service Account the
DNS Peer (
roles/dns.peer) IAM role on the shared VPC host project.
For more information about granting roles, see
Your administrator might also be able to give the Cloud Data Fusion Service Account the required permissions through custom roles or other predefined roles.
For more information, see Granting access to the required service accounts.
- Learn more about Networking in Cloud Data Fusion.
- Refer to the Cloud Data Fusion API.