This page provides information about networking with Dataproc Metastore.
Dataproc Metastore services use private IPs, which provide many benefits. Private IPs provide lower network latency than public IPs. You can connect through private IP from any region. You can also connect through Shared VPCs between projects.
Private IP connectivity between hosts in your VPC network and a Dataproc Metastore service is facilitated by peering your VPC network with Dataproc Metastore's service producer VPC network. Google allocates IP ranges in your VPC network that are used in the service producer project.
In the following example, the customer VPC network allocated the
10.100.0.0/17 address range for Google services and established a private
connection that uses the allocated range.
- On the Google services side of the VPC peering, Google creates a project for the customer. The project is isolated, meaning no other customers share it and the customer is billed for only the resources the customer provisions.
- When creating the first Dataproc Metastore service
in a region, Dataproc Metastore allocates a
/17range and a
/20range in the customer's network for all future Dataproc Metastore services usage in that region and network. Dataproc Metastore further subdivides these ranges to create subnetworks and address ranges in the service producer project.
- VM services in the customer's network can access Dataproc Metastore service resources in any region if the Google Cloud service supports it. Some Google Cloud services might not support cross-region communication.
- Egress costs for cross-regional traffic, where a VM instance communicates with resources in a different region, still apply.
- Google assigns the Dataproc Metastore service the IP address
10.100.0.100. In the customer VPC network, requests with a destination of
10.100.0.100are routed through the VPC peering to the service producer's network. After reaching the service network, the service network contains routes that direct the request to the correct resource.
- Traffic between VPC networks travels internally within Google's network, not through the public internet.
Dataproc Metastore allocates a
/17 range and a
/20 range from
the address space for each region. For example, placing Dataproc Metastore
services in two regions requires that the allocated IP address range contains at
least two unused address blocks of size
/17 and two unused address blocks of
Connections to a Dataproc Metastore service using a private IP address use RFC 1918 address ranges. If RFC 1918 address blocks aren't found, then Dataproc Metastore will find suitable non-RFC 1918 address blocks instead. Note that the allocation of non-RFC 1918 blocks doesn't take into account whether or not those addresses are in use in your VPC network or on-premises.
Traffic over VPC network peering is provided with a certain level of encryption. For more information, see Google Cloud's virtual network encryption and authentication.
Creating one VPC network for each service with a private IP address provides better network isolation than putting all services in the "default" VPC network.
Quick reference for networking topics
When you manage Dataproc Metastore services, some of the following topics could be of interest:
|Shared VPC networks||You can create Dataproc Metastore services in a Shared VPC network. To set up shared VPC when creating a service, see Set up shared VPC.|
|Legacy networks||You can't connect to a Dataproc Metastore service from a legacy network. Legacy networks don't support VPC Network Peering.|
|Existing Dataproc Metastore services||You cannot change the network that a Dataproc Metastore service is connected to.|
|Static IP addresses||The private IP address of the Dataproc Metastore service doesn't change.|
|VPC Network Peering||You don't create the VPC Network Peering explicitly, because
the peering is internal to Google Cloud. After you create a
Dataproc Metastore service, you can see its underlying VPC
Network Peering on the VPC Network Peering page in the Cloud Console.
The peering is shared by services in the same region and network. Don't delete
Learn more about VPC Network Peering.
|VPC Service Controls||VPC Service Controls improve your ability to mitigate the risk of data exfiltration. With VPC Service Controls, you create perimeters around the Dataproc Metastore service. VPC Service Controls restrict access to resources within the perimeter from the outside. Only clients and resources within the perimeter can interact with one another. For more information, see the Overview of VPC Service Controls. Also review the Dataproc Metastore limitations when using VPC Service Controls. To use VPC Service Controls with Dataproc Metastore, see VPC Service Controls with Dataproc Metastore.|
|Transitive peering||Only directly peered networks can communicate. Transitive peering is not
supported. In other words, if VPC network N1 is peered with N2 and N3, but
N2 and N3 are not directly connected, VPC network N2 can't communicate with
VPC network N3 over
VPC Network Peering. In addition, VMs in networks peered with your
Dataproc Metastore project network can't reach Dataproc Metastore.
Only hosts on the VPC network can reach Dataproc Metastore.
A Dataproc Metastore service in one project can be connected to by clients in multiple different projects using Shared VPC networks.
- Learn more about accessing a service.
- Learn more about VPC Service Controls.
- Learn more about Dataproc Metastore IAM and access control.