Access gRPC endpoints for Dataproc Metastore

This page explains how to grant a Google Cloud user account or a service account access to a Dataproc Metastore service that uses the gRPC endpoint protocol.

About granting gRPC metadata roles

When granting an account access to your metadata, consider the following concepts:

  • The level of access to provide. The level of access you grant controls how much metadata that an account can access. For example, you can choose to provide access to metadata stored in a specific database, a specific table, or you can provide access to your entire project.

  • The principal that requires access. You use IAM principals (identities) to run your jobs. For example, you can run Dataproc cluster jobs with user accounts or service accounts (typically the Dataproc VM service account).

    For more information about the accounts you can use with Dataproc Metastore, see Dataproc service accounts.

Depending on the scope of control required, grant your principal one of the following predefined IAM roles:

  • To grant full access to the metadata resources. Metadata Owner role (roles/metastore.metadataOwner)
  • To grant read-write access to metadata: Metadata Editor role roles/metastore.metadataEditor)
  • To grant read access to the metadata: Metadata Viewer role (roles/metastore.metadataViewer)

Before you begin

  • Enable Dataproc Metastore in your project.
  • Create a metastore service that uses the gRPC protocol.

  • Understand networking requirements specific to your project.

    • gRPC and Virtual Private Cloud (VPC) requirements. If you're using gRPC, you don't have to configure a Shared VPC or set any extra network configurations. By default, gRPC endpoints can be reached from any VPC.

    However, there's one exception. If your project is using a VPC-SC service perimeter, then gRPC endpoints can only be reached from a VPC belonging to projects in the perimeter. For more information, see VPC Service Controls with Dataproc Metastore.

Required Roles

To get the permissions that you need to grant a principal access to Dataproc Metastore metadata, ask your administrator to grant you the following IAM roles on your project, while following the principle of least privilege:

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to grant a principal access to Dataproc Metastore metadata. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to grant a principal access to Dataproc Metastore metadata:

  • resourcemanager.projects.get
  • resourcemanager.projects.getIamPolicy
  • resourcemanager.projects.setIamPolicy

You might also be able to get these permissions with custom roles or other predefined roles.

For more information about specific Dataproc Metastore roles and permissions, see Dataproc Metastore IAM overview.

Grant a principal account access to metadata

You can grant a principal account access to metadata at the project level, service level, database level, or table level.

Grant access at the project level

To grant access to all Dataproc Metastore metadata at the project level, you must grant a metadata role to your principal account.

gcloud CLI

To grant metadata roles to all Dataproc Metastore services in a specified project, run the following gcloud projects add-iam-policy-binding command:

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member=PRINCIPAL \
    --role=METASTORE_ROLE

Replace the following:

  • PROJECT_ID: The Google Cloud project ID that you want to grant metadata access to.
  • PRINCIPAL: The type and email ID (email address) of the principal.
    • For user accounts: user:EMAIL_ID
    • For service accounts: serviceAccount:EMAIL_ID
    • For Google Groups: group:EMAIL_ID
    • For other principal types: Concepts related to identity
  • METASTORE_ROLE: One of the following roles, depending on the scope of access that you want to grant the principal: roles/metastore.metadataViewer, roles/metastore.metadataEditor, or roles/metastore.metadataOwner.

Grant access at the service level

To grant access to all Dataproc Metastore metadata at the service level, you must grant a metadata role to your principal account.

gcloud CLI

To grant metadata roles at the granularity of a single Dataproc Metastore metastore service, run the following gcloud metastore services add-iam-policy-binding command:

gcloud metastore services add-iam-policy-binding SERVICE_ID \
  --location=LOCATION \
  --member=PRINCIPAL \
  --role=METASTORE_ROLE

Replace the following:

  • SERVICE_ID: The ID or fully qualified identifier for your Dataproc Metastore service.
  • LOCATION: The region of the Dataproc Metastore that you're granting access to.
  • PRINCIPAL: The type and email ID (email address) of the principal account:
    • For user accounts: user:EMAIL_ID
    • For service accounts: serviceAccount:EMAIL_ID
    • For Google Groups: group:EMAIL_ID
    • For other principal types: Concepts related to identity
  • METASTORE_ROLE: One of the following roles, depending on the scope of access that you want to grant the principal: roles/metastore.metadataViewer,roles/metastore.metadataEditor, or roles/metastore.metadataOwner.

Grant access at the database level

To grant access to all Dataproc Metastore metadata in a specific database, you must add a metadata role to your principal account.

gcloud CLI

To grant metadata roles at the granularity of a specific database, run the following gcloud metastore services databases add-iam-policy-binding command:

gcloud metastore services databases add-iam-policy-binding DATABASE_ID \
  --project=PROJECT \
  --location=LOCATION \
  --service=SERVICE_ID \
  --member=PRINCIPAL \
  --role=METASTORE_ROLE

Replace the following:

  • DATABASE_ID: The ID of the database that you're granting metadata access to. You get this ID from your database schema.
  • PROJECT: The Google Cloud project ID that contains the Dataproc Metastore service that you're granting metadata access to.
  • LOCATION: The region of the Dataproc Metastore service that you're granting access to.
  • SERVICE_ID: The ID or fully qualified identifier for your Dataproc Metastore service.
  • PRINCIPAL: The type and email ID (email address) of the principal account:
    • For user accounts: user:EMAIL_ID
    • For service accounts: serviceAccount:EMAIL_ID
    • For Google Groups: group:EMAIL_ID
    • For other principal types: Concepts related to identity
  • METASTORE_ROLE: One of the following roles, depending on the scope of access that you want to grant the principal: roles/metastore.metadataViewer, roles/metastore.metadataEditor, or roles/metastore.metadataOwner.

Grant access at the table level

To grant access to all Dataproc Metastore metadata in a specific table, you must grant a metadata role to your principal account.

gcloud CLI

To grant metadata roles at the granularity of a table, run the following gcloud metastore services databases tables add-iam-policy-binding command:

gcloud metastore services databases tables add-iam-policy-binding TABLE_ID \
  --database=DATABASE_ID \
  --project=PROJECT \
  --location=LOCATION \
  --service=SERVICE_ID \
  --member=PRINCIPAL \
  --role=METASTORE_ROLE

Replace the following:

  • TABLE_ID: The ID of the table that you're granting access to. You get this ID from your database schema.
  • DATABASE_ID: The ID of the database that contains the table that you're granting metadata access to. You get this ID from your database schema.
  • PROJECT: The Google Cloud project ID that contains the Dataproc Metastore service that you're granting metadata access to.
  • LOCATION: The region of the Dataproc Metastore service that you're granting metadata access to.
  • SERVICE_ID: The ID or fully qualified identifier for your Dataproc Metastore service.
  • PRINCIPAL: The type and email ID (email address) of the principal account:
    • For user accounts: user:EMAIL_ID
    • For service accounts: serviceAccount:EMAIL_ID
    • For Google Groups: group:EMAIL_ID
  • METASTORE_ROLE: One of the following roles, depending on the scope of access you want to grant the principal: roles/metastore.metadataViewer, roles/metastore.metadataEditor, or roles/metastore.metadataOwner.

After you grant access to your metadata

After you grant the required roles to your service accounts, you can connect your Dataproc Metastore to a Dataproc cluster. Your cluster then uses the Dataproc Metastore service as its Hive Metastore.

What's next