Build a data mesh

You can use Dataplex Universal Catalog to build a data mesh architecture. This quickstart shows you how to use Dataplex Universal Catalog features, such as a lake, zones, and assets, to build a data mesh.

A data mesh is an organizational and technical approach that decentralizes data ownership among domain data owners. These owners provide the data as a product in a standard way and facilitate communication among different parts of the organization to distribute datasets across different locations. Learn more about data mesh architectures.

Objectives

In this guide, you use the Dataplex Universal Catalog entities to build a data mesh architecture:

  • Create a Dataplex Universal Catalog lake that acts as the domain for your data mesh.
  • Add zones to your lake that represents individual teams within each domain and provide managed data contracts.
  • Attach assets that map to data stored in Cloud Storage.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator.

New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

  1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  2. Verify that billing is enabled for your Google Cloud project.

  3. Enable the Dataplex API.

    Enable the API

  4. Create a Dataproc Metastore service.

Create a Cloud Storage bucket

You need a Cloud Storage bucket to store the data assets of your data mesh.

To create a Cloud Storage bucket, follow the instructions in create a Cloud Storage bucket. When doing so, note the following:

  • Name your bucket.
  • For Location type, choose Region, and select us-central1 (Iowa) from the menu.

Create a domain

  1. In the Google Cloud console, go to the Dataplex Universal Catalog Lakes page.

    Go to Lakes

  2. Click Create to create a new lake, which acts as your data mesh.

  3. In the Display name field, enter My data mesh.

  4. For Region, select us-central1.

  5. Select the Dataproc Metastore service that you created and configured earlier as the associated metastore.

  6. Click Create.

Create zones in your lake

After creating a domain by creating a Dataplex Universal Catalog lake, you can host managed data contracts and individual teams within the domain by using zones. There are two types of zones:

  • Raw zones are typically used to store data in any format from external sources in Cloud Storage. Raw zones are useful for data that requires further processing before it's ready for consumption.

  • Curated zones are used for structured data in Cloud Storage that must conform to certain file formats, and are organized in a hive-compatible directory layout. They are most useful for data that's ready for consumption and analysis.

Each domain (for example, sales, customers, products) should have at least a raw zone and a curated zone.

Additional zones are used to manage data contracts between teams or to provide a more granular breakdown for teams within a given domain. For example, inventory management within the product domain. Data owners are able to manage the data within their domain and access it.

  1. In the Google Cloud console, navigate to the Dataplex Universal Catalog Manage view.

  2. Click the name of the lake (My data mesh) you want to add a zone to.

  3. In the Zones tab, click Add Zone.

  4. In the Display name field, enter My sub domain. Dataplex Universal Catalog automatically generates an ID for your zone.

  5. For Type, select Raw zone.

  6. Click Create.

Attach assets to your zones

Attach data assets to your zone. A data asset, the storage resources that contain your data, can be a Cloud Storage bucket or a BigQuery dataset. This is the final step in creating your data mesh architecture.

  1. In the Dataplex Universal Catalog Manage view, click the lake you created (My data mesh).

  2. In the Zones tab, click the zone (My sub domain) to add the asset to.

  3. In the Assets tab, click Add assets

  4. Click Add an Asset.

  5. For Type, select Cloud Storage bucket.

  6. In the Display name field , enter Data mesh asset. Dataplex Universal Catalog automatically generates an asset ID for you.

  7. In the Bucket field, click Browse.

    1. Select your bucket from the list.
    2. Click Select.

  8. Click Done and then click Continue.

  9. Click Continue to accept the default Advanced settings.

  10. Click Submit.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

  1. In the Google Cloud console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete your data mesh architecture

  1. In the Google Cloud console, navigate to the Dataplex Universal Catalog Manage view.

  2. For the lake that you want to delete, click View more, and then click Delete.

  3. To confirm the action, enter delete and click Delete lake.

What's next