With Private Service Connect, you can create a Dataproc Metastore service that doesn't use VPC peering. This page explains what Private Service Connect is and how to use it as an alternative networking option for Dataproc Metastore.
How Dataproc Metastore works with VPC
Dataproc Metastore protects its metadata access by only exposing private IP endpoints. It also restricts connectivity to VMs in your VPC network by using VPC peering.
Dataproc Metastore requires the following configurations per region for each VPC network:
- 1 peering quota
/17
and/20
CIDR blocks
As a result, setting up VPC peering and IP address reservations can be difficult on crowded VPC networks. Similarly, a VPC network may not have enough peering quota to accommodate additional peering requests. Both of these limitations can prevent you from creating new Dataproc Metastore service.
To circumvent these issues, you can use Dataproc Metastore with Private Service Connect.
How Dataproc Metastore works with Private Service Connect
Private Service Connect provides a private connection to Dataproc Metastore metadata across VPC networks.
To use Private Service Connect with Dataproc Metastore, the following configurations are required:
- A single address reservation in the subnetwork.
- A forwarding rule targeting the service attachment that exposes the Dataproc Metastore endpoint. The address reservation and forwarding rule are created as a part of the Dataproc Metastore service create call.
Considerations
Dataproc Metastore services that use Private Service Connect only support access from VPC networks of the subnetworks specified during the service creation.
Dataproc Metastore reserves addresses and creates forwarding rules in each of the specified subnetworks. Each subnetwork has a Thrift endpoint URI that you can use to access the Dataproc Metastore metadata endpoint from.
Limitations
- Dataproc Metastore services that use the gRPC endpoints protocol don't support Private Service Connect.
- Reverse connectivity is not supported with Private Service Connect. This means you can't use a Kerberos configuration with Private Service Connect.
- You can't dynamically add or remove subnets from a Dataproc Metastore service that is configured with Private Service Connect. Instead, you must recreate a service if you'd like to add or remove subnets.
- You can't update a Dataproc Metastore service that uses Private Service Connect to use VPC or the other way around.
Create a Dataproc Metastore service with Private Service Connect
The following instructions demonstrate how to configure Private Service Connect during service creation.
Console
In the Google Cloud console, open the Dataproc Metastore page:
At the top of the Dataproc Metastore page, click Create.
The Create service page opens.
Configure your service as needed.
Under Network configuration, click Make services accessible in multiple VPC subnetworks.
Select the Subnetworks. You can specify up to 5 subnetworks.
Click Done.
Click Submit.
Verify the service's network configuration:
In the Google Cloud console, open the Dataproc Metastore page:
On the Dataproc Metastore page, click the service name of the service want like to view.
The Service detail page for that service opens.
Under the Configuration tab, verify that the details show multiple VPC subnetwork URIs.
gcloud
Run the following
gcloud metastore services create
command to create a service with Private Service Connect:gcloud metastore services create SERVICE \ --location=LOCATION \ --consumer-subnetworks="projects/PROJECT_ID/regions/LOCATION/subnetworks/SUBNET1, projects/PROJECT_ID/regions/LOCATION/subnetworks/SUBNET2"
or
gcloud metastore services create SERVICE \ --location=LOCATION \ --network-config-from-file=NETWORK_CONFIG_FROM_FILE
Verify that the creation was successful.
REST
Follow the API instructions to create a service by using the API Explorer.
In the create
request parameters, use the Network Config
filed to
configure Private Service Connect. You can specify 1 to 5 subnetworks.
"network_config": { "consumers": [ {"subnetwork": "projects/PROJECT_ID/regions/LOCATION/subnetworks/SUBNET1"}, {"subnetwork": "projects/PROJECT_ID/regions/LOCATION/subnetworks/SUBNET2"} ] }