Build a data mesh

---

You can use Dataplex to build a data mesh architecture. This guide shows you how to use Dataplex features, such as a lake, zones, and assets, to build a data mesh.

A data mesh is an organizational and technical approach that decentralizes data ownership among domain data owners. These owners provide the data as a product in a standard way and facilitate communication among different parts of the organization to distribute datasets across different locations. Learn more about data mesh architectures.

Objectives

In following this guide, you use the Dataplex entities to build a data mesh architecture:

• Create a Dataplex lake that will act as the domain for your data mesh.
• Add zones to your lake that will represent individual teams within each domain and provide managed data contracts.
• Attach assets that map to data stored in Cloud Storage.

Costs

This tutorial uses the following billable components of Google Cloud:

• Dataplex
• Cloud Storage

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

3. Enable the Dataplex API.

   Enable the Dataplex API

4. Create a Dataproc Metastore service.

   Note: Each Dataproc Metastore can only be attached to a single Dataplex lake. Enable gRPC for your metastore.

Create a Cloud Storage bucket

You need a Cloud Storage bucket to store the data assets of your data mesh.

Follow the steps to create a Cloud Storage bucket, and:

• Name your bucket.
• Under Location type, choose Region.
• Under Location, choose us-central1 (Iowa).

Create a domain

1. In the Google Cloud console, go to the Dataplex page:

   Go to Dataplex

2. Navigate to the Manage view.

3. Click Create to create a new lake, which will act as your data mesh.

4. Under Display name, enter My data mesh.

   Note: Dataplex automatically generates a lake ID for you.

5. Under Region, select us-central1.

   Note: The region you select for your data mesh determines the location of the data (not including attached assets) managed by Dataplex. The same region is used when Dataplex creates resources in other services, but not for data contained within assets.

6. Select the Dataproc Metastore service you previously created and configured as the associated metastore.

7. Click Create.

Create zones in your lake

After creating a domain by creating a Dataplex lake, you can host managed data contracts and individual teams within the domain by using zones. There are two types of zones:

• Raw zones are typically used to store data in any format from external sources in Cloud Storage and are useful for data that requires further processing before it is ready for consumption.

• Curated zones are used for structured data in Cloud Storage that must conform to certain file formats, and are organized in a hive-compatible directory layout. They are most useful for data that's ready for consumption and analysis.

Each domain (for example, sales, customers, products) should have a raw zone and a curated zone, at least.

Additional zones are used to manage data contracts between teams or to provide a more granular breakdown for teams within a given domain (for example, inventory management within the product domain). Data owners are able to manage the data within their domain and access it.

1. In Dataplex in the Google Cloud console, navigate to the Manage view.

2. Click the name of the lake (My data mesh) you'd like to add a zone to.

3. In the Zones tab, click Add Zone.

4. Under Display name, enter My sub domain. Dataplex automatically generates an ID for your zone.

   NOTE: The zone name will become the name of a BigQuery dataset. Therefore, all zones hosted in the same Google Cloud project must have a unique ID, even if they exist within different lakes.

5. Under Type, select Raw zone.

6. Click Create.

Attach assets to your zones

Attach data assets to your zone. A data asset, the storage resources that contain your data, can be a Cloud Storage bucket or a BigQuery dataset. This is the final step in creating your data mesh architecture.

1. In the Dataplex Manage view, click the lake you created (My data mesh).

2. In the Zones tab, click the zone (My sub domain) to add the asset to.

3. In the Assets tab, click Add assets

4. Click Add an Asset.

5. Under Type, select Cloud Storage bucket.

6. In Display name, enter Data mesh asset. Dataplex automatically generates an asset ID for you.

7. Next to Bucket, click Browse.

8. Click Done. Click Continue.

   1. Select your bucket from the list.
   2. Click Select.

9. Click Continue to accept the default Advanced settings.

10. Click Submit to add your Cloud Storage bucket as a data asset to your zone.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete your data mesh architecture

1. In Dataplex in the Google Cloud console, navigate to the Manage view.

2. Find the name of the Dataplex lake you want to remove.

3. Click the three dot menu to the right of the lake you wish to delete and click Delete.

4. Type "delete" and click Delete lake to confirm.

What's next

• Learn about data processing tasks
• Learn about discovering data
• Learn about using data quality tasks