This document provides guidance and best practices for using Dataplex Universal Catalog.
Choose a project for your lake
When you select the project in which to host your lake, consider the following factors:
The project must belong to the same VPC Service Controls perimeter as the data destined to be within the lake.
The lake service account requires administrator permissions on the Cloud Storage buckets or BigQuery datasets. Dataplex Universal Catalog creates external tables in BigQuery for tables discovered in Cloud Storage. Dataplex Universal Catalog also makes available BigQuery table metadata, and tables discovered in the Cloud Storage bucket, in a Dataproc Metastore service. The Dataproc Metastore is located within the data lake project.
Cloud Storage settings and limitations
Region: Dataplex Universal Catalog supports single region and multi-region buckets in some Google Cloud regions.
Storage class: Cloud Storage buckets of all storage classes are supported (Standard, Nearline, Coldline, Archive). Additional data retrieval costs might incur for accessing or scanning Nearline, Coldline, or Archive data.
Bucket ACL: Dataplex Universal Catalog supports Cloud Storage buckets with uniform access controls only. Fine-grained access controls aren't supported.
Requester Pays: Cloud Storage buckets with the Requester Pays feature enabled aren't supported.
Security and permissions guidance
Dataplex Universal Catalog requires adding the Dataplex Universal Catalog service accounts as an administrative service account on managed buckets and datasets.
Dataplex Universal Catalog enables analysts to access Cloud Storage buckets and BigQuery datasets across many projects. To enable this access, Dataplex Universal Catalog requires adding the Dataplex Universal Catalog service accounts with administrative controls to these projects.
For Discovery, Dataplex Universal Catalog adds the Dataproc Metastore service account to the Cloud Storage buckets. If you have your own Dataproc Metastore cluster, you might want to make the Dataplex Universal Catalog lake use your Dataproc Metastore service, which is an option when you create your lake.
If you choose to add a Cloud Storage bucket with fine-grained access to a lake, Dataplex Universal Catalog will provide full access to that bucket through the lake because Dataplex Universal Catalog permissions are propagated to all objects in the bucket. If you require fine-grained access, it's recommended that you split the data in your bucket into multiple buckets.