Deploy a Dataproc Metastore service with a Dataproc cluster

Deploy a Dataproc Metastore service with a Dataproc cluster

This page shows you how to create a Dataproc Metastore service and a Dataproc cluster that uses the service as its Hive metastore.

---

For step-by-step guidance on this task directly in console, click Guide me:

Guide me

---

The following sections take you through the same steps as clicking Guide me.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

5. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

6. Enable the Dataproc Metastore API.

   Enable the API

Required Roles

To get the permission that you need to create a Dataproc Metastore, ask your administrator to grant you the following IAM roles on your project. You might not them all, depending on the level of access you require:

• Grant full control of Dataproc Metastore resources (roles/metastore.editor)
• Grant full access to all Dataproc Metastore resources, including IAM policy administration (roles/metastore.admin)

For more information about granting roles, see Manage access.

This predefined role contains the metastore.services.create permission, which is required to create a Dataproc Metastore. You might also be able to get this permission with custom roles or other predefined roles.

For more information about specific Dataproc Metastore roles and permissions, see Manage Dataproc access with IAM.

Create a Dataproc Metastore service

The following instructions demonstrate how to create a Dataproc Metastore service using the Google Cloud console, the gcloud CLI, or the Dataproc Metastore API.

Console

1. In the console, open the Create service page:

   Open the Create service page in the console

   

2. In the Service name field, enter example-service.

3. Select the Data location. For information on selecting a region, see Cloud locations.

4. For other service configuration options, use the provided defaults.

5. To create and start the service, click the Submit button.

Your new service appears in the Service list.

gcloud

Run the following gcloud metastore services create command to create a service:

 gcloud metastore services create example-service \
     --location=LOCATION
 

Replace LOCATION with the Compute Engine region where the service is to be created. Make sure that Dataproc Metastore is available in the location.

REST

Follow the API instructions to create a service by using the API Explorer.

Create a Dataproc cluster that uses the service

After you create a service, you can create and attach a Dataproc cluster that uses the service as its Hive metastore.

The Dataproc image and Dataproc Metastore Hive version must be compatible. Check the following image versioning pages to ensure that the Hive version is compatible:

• Dataproc 2.0.x release versions
• Dataproc 1.5.x release versions
• Dataproc 1.4.x release versions

For more information, see Dataproc Image version list.

Console

1. In the console, open the Dataproc Create a cluster page:

   Open the Create a cluster page in the console

2. In the Cluster Name field, enter example-cluster.

3. On the Region and Zone menus, select a region and zone for the cluster. You can select a distinct region, to isolate resources and metadata storage locations within the specified region. If you select a distinct region, you can select "No preference" for the zone to let Dataproc pick a zone within the selected region for your cluster (see Dataproc Auto zone placement).

   Note: We require that you create the Dataproc cluster in the same region as the Dataproc Metastore for optimal network connectivity.

4. Use the provided defaults for all the other options.

5. Click on the Customize cluster tab.

6. In the Network configuration section, select the same network specified during the metastore service creation.

7. In the Dataproc Metastore section, select example-service.

8. Click Create to create the cluster.

Your new cluster appears in the Clusters list. Cluster status is listed as "Provisioning" until the cluster is ready to use. Its status then changes to "Running."

gcloud

Run the following gcloud dataproc clusters create command to create a cluster:

 gcloud dataproc clusters create example-cluster \
    --dataproc-metastore=projects/PROJECT_ID/locations/LOCATION/services/example-service \
    --region=LOCATION
 

Replace PROJECT_ID with the project ID of the project you created your Dataproc Metastore service in.

Replace LOCATION with the region you specified for the Dataproc Metastore service.

Note: We require that you create the Dataproc cluster in the same region as the Dataproc Metastore service for optimal network connectivity. In addition, the Dataproc cluster must be on the same network as the Dataproc Metastore service.

REST

Follow the API instructions to create a cluster by using the API Explorer.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the console, go to the Manage resources page.

   Go to Manage resources

2. If the project that you plan to delete is attached to an organization, expand the Organization list in the Name column.
3. In the project list, select the project that you want to delete, and then click Delete.
4. In the dialog, type the project ID, and then click Shut down to delete the project.

Alternatively, you can delete the resources used in this tutorial:

1. Delete the Dataproc Metastore service.

   Console

   1. In the console, open the Dataproc Metastore page:

      Open Dataproc Metastore in the console

   2. On the left of the service name, select example-service by checking the box.

   3. At the top of the Dataproc Metastore page, click Delete to delete the service.

   4. On the dialog, click Delete to confirm the deletion.

   Your service no longer appears in the Service list.

   gcloud

   Run the following gcloud metastore services delete command to delete a service:

    gcloud metastore services delete example-service \
        --location=LOCATION
    

   Replace LOCATION with the Compute Engine region where the service was created.

   REST

   Follow the API instructions to delete a service by using the API Explorer.

   All deletions succeed immediately.

2. Delete the Cloud Storage bucket for the Dataproc Metastore service.

   Note: Deleting the Dataproc Metastore service does not delete its bucket.

3. Delete the Dataproc cluster that used the Dataproc Metastore service.

What's next

• What is Dataproc Metastore?
• Create a service
• Import metadata into a service