Manage data assets using Data Catalog

Data Catalog is a feature of Dataplex that automatically catalogs metadata about Bigtable resources. This feature helps your organization search for and manage your Bigtable data assets.

Metadata is automatically synced from Bigtable to Data Catalog for the following Bigtable resources:

  • Instances
  • Clusters
  • Tables, including column family details

Data Catalog is a fully managed metadata management service within Dataplex. It provides unified view and tagging mechanisms for technical and business metadata. To learn more about Data Catalog and why you should use it, see What is Data Catalog.

Benefits of using Data Catalog with Bigtable

Data Catalog information about your data can help facilitate analysis, data reuse, application development, and data management.

You can search entries in Data Catalog by resource name or by details. For example, you can search for an instance name to find the entry for an instance or search for a column family to find the entry for a table. You can also use a faceted search, applying filters to narrow down your search. For more information, see Search and view data assets with Data Catalog.

Tagging data assets can help you understand and catalog the data that is available in your Bigtable tables to make the data more discoverable and searchable by other Data Catalog users, and as part of your larger data estate. You use tags to add your own metadata to your cataloged metadata, and then you can use those tags to conduct searches.

Get started with Data Catalog

To get started using Data Catalog to manage your Bigtable data assets, complete the following tasks:

  1. If you are new to Data Catalog, enable the API.

  2. Grant IAM roles to principals who require access to Bigtable entries in Data Catalog.

  3. Learn how to search for data assets by reading How to search with Data Catalog.

  4. Review Data Catalog search syntax to understand how to find your metadata.

  5. Read the concepts on tagging Data Catalog entries at Tags and tag templates

Ways to search for Bigtable metadata

You can search synced Bigtable metadata using Data Catalog.

In addition to standard qualifiers, you can search for Bigtable resources with the following:

  • Instance

    • Bigtable instance name
    • Bigtable instance ID
    • Cluster location
    • Cluster ID

  • Table

    • Table ID
    • Column family name
    • Bigtable instance name
    • Bigtable instance ID

Required roles

Viewing, tagging, and searching Bigtable resources in Data Catalog have different permission requirements. In general, grant principals in your organization the minimum role required to accomplish a task.

Permission to view public and private tags

To get the permissions that you need to view public and private tags on Bigtable resources, ask your administrator to grant you the following IAM roles:

These predefined roles contain the permissions required to view public and private tags. The exact permissions that are required are the following:

  • bigtable.instances.get
  • bigtable.clusters.get
  • bigtable.tables.get
  • datacatalog.tagTemplates.getTag

Permission to search Bigtable resources

To get the permissions that you need to search Bigtable resources in Data Catalog, ask your administrator to grant you the following IAM roles:

These predefined roles contain the permissions required to search Bigtable resources. The exact permissions that are required are the following:

  • bigtable.instances.get
  • bigtable.clusters.get
  • bigtable.tables.get

Permission to attach tags to Bigtable resources

To get the permissions that you need to attach tags to Bigtable resources in Data Catalog, ask your administrator to grant you the following IAM roles:

These predefined roles contain the permissions required to add public and private tags. The exact permissions that are required are the following:

  • bigtable.instances.update
  • bigtable.clusters.update
  • bigtable.tables.update
  • datacatalog.tagTemplates.use
  • datacatalog.entries.updateTag

(Optional) Create a custom role

You can create a custom role to grant to users who require complete access to all Bigtable metadata in Data Catalog across your project or organization. Then you can make any user or service account that needs access a principal of the custom role. To learn more about custom roles, see Roles and permissions.

Choose a meaningful name for the role, such as Bigtable Data Catalog Editor, and then assign only the principals who need access to the role. For setup instructions, see Creating a custom role.

Grant the following permissions to the custom role:

  • bigtable.instances.get
  • bigtable.instances.update
  • bigtable.clusters.get
  • bigtable.clusters.update
  • bigtable.tables.get
  • bigtable.tables.update
  • datacatalog.tagTemplates.getTag
  • datacatalog.tagTemplates.use

For details on Data Catalog permissions, see Roles to search Google Cloud resources. For descriptions of Bigtable permissions, see Permissions.

Known constraints

The following section lists some constraints that you must note before using Data Catalog to manage your Bigtable metadata.

Location constraints

If your organization has set up location constraints, metadata entries for your Bigtable resources might not be synced to Data Catalog. Any entries created before enforcing the location constraints are retained by Data Catalog.

To allow Data Catalog to load resources, we recommend adding the global multi-region to your location policy.

Column family name exception

The Data Catalog API rejects requests where the column family name in your request includes a period (.). For example, if the column family name is personal.info.

What's next