What is data mesh?

Data mesh isn't just about a new set of tools or technologies; it's a shift in how companies think about their data. There are four core principles that guide the data mesh approach. These principles are what make the approach so effective at solving the problems of traditional, centralized data architectures.

In a traditional data architecture, a single central team, like an IT or data engineering team, is responsible for all data. In a data mesh, data ownership is spread out to the business domains that create the data. For example, a sales team would own the customer data they generate, and a marketing team would own the campaign data they create. This makes teams more responsible and accountable for the data they produce.

With domain-oriented ownership, the teams that create data must also treat it like a product. Just as a company would provide a high-quality product to a customer, a data domain team needs to provide high-quality data to other teams that need it. This means the data is easy to discover, understand, and use. It also has to be trustworthy, secure, and well-documented with built-in access controls so the right people only access the data that is intended for their use case.

To make treating data as a product possible, a data mesh uses a self-serve platform. This platform is a set of tools and services that allows data domain teams to easily create and manage their data products without needing help from a central data team. It can be a simple, easy-to-use platform that automates many of the technical tasks involved in data management, like data storage, security, and governance.

Since data is decentralized and spread across many different teams, there needs to be a way to ensure everyone follows the same rules. This is where federated computational governance comes in. It's a model where a small, central team sets the global rules and standards for all data. However, the enforcement of these rules is handled by the data domain teams themselves. This combines the best of both worlds: centralized policies with decentralized execution.

The data mesh approach was created to solve some of the common problems with traditional data architectures. These models, such as data warehouses or data lakes owned by individual departments or teams, can create data silos and governance risks, especially as a company grows. Data mesh tackles these issues by distributing ownership and empowering individual teams while still maintaining central controls for governing and monitoring the data across domains.

Dataplex Universal Catalog acts as a unified data fabric and provides a central governance layer over your data mesh. It can help you discover, manage, and govern your distributed data across various environments, ensuring you have a single source of truth for metadata and policies. To get started, you'll need to create a Dataplex lake. A Dataplex lake is a top-level container that holds your data and is typically mapped to a business domain.

Here are the steps to create a lake:

1. In the Google Cloud console, navigate to the Dataplex Universal Catalog Lakes page.
2. After clicking "Create," name your new lake something descriptive, such as "Sales Data Domain" or "Marketing Data Mesh."
3. Choose a region for your lake.
4. Once the lake is created, you can add zones. A zone is a subdomain within your lake that represents a specific team or data contract. For example, within the "Sales Data Domain" lake, you might create a "Raw" zone for unprocessed data and a "Curated" zone for cleaned, production-ready data.
5. After creating zones, you can attach assets to them. An asset is the actual data stored in a service like Cloud Storage or BigQuery. You simply point the Dataplex zone to the location of your data.

Dataplex then automatically scans these assets to discover and catalog metadata.

A key part of the "data as a product" principle is making data easily discoverable. BigQuery data sharing allows you to build a data product marketplace. This lets domain teams securely share data products with other teams without copying or moving the data. It can help data consumers find the data they need and provides them with a clear, well-defined interface to access it.

Google Cloud's serverless services empower domain teams to create and manage their own data products with minimal overhead. BigQuery is a powerful, serverless data warehouse that allows teams to analyze large datasets quickly and efficiently. Dataflow is a serverless data processing service that can be used to build and automate data pipelines for data products. These services reduce the need for a central data engineering team to manage infrastructure, making domain teams more autonomous and agile.

Federated computational governance is the principle of having a central team define global rules, but allowing domain teams to enforce them. Google Cloud's Identity and Access Management (IAM) conditions provide the tools to implement this. IAM conditions allow for attribute-based access control (ABAC), where you can set up fine-grained permissions based on data attributes. For example, you can create a policy that only allows a user to access customer data from their specific region, helping ensure compliance with data sovereignty regulations like GDPR.