Use data lineage with Google Cloud systems

Data lineage displays the relations between your project's resources and the processes that created them.

You can view data lineage details in the Google Cloud console or retrieve them by using the Data Lineage API.

Tip: When you use data lineage with Dataplex Universal Catalog, be aware of the differences that are described in About data lineage in Data Catalog.

Caution: Data lineage is enabled on a per-project basis, not a per-service basis. After you enable the Data Lineage API, lineage information is automatically reported for multiple Google Cloud services in the project, depending on their product-level lineage control. For more details, see Data lineage considerations.

Depending on the volume of data being processed, it takes time for data lineage to display a graph. For most jobs it takes three hours, and for some jobs it can take up to 24 hours.

Lineage is captured across projects. When you view lineage that is generated from multiple projects, you can view the aggregated lineage information in any of the relevant projects.

Roles and permissions

To view data lineage, you need specific Identity and Access Management (IAM) permissions. Lineage information is captured across projects, so you need permissions in multiple projects.

• When viewing lineage in Dataplex Universal Catalog, BigQuery, or Vertex AI: you need permissions to view lineage information in the project where you are viewing it.

• When viewing lineage that was recorded in other projects: you need permissions to view lineage information in those projects where it was recorded.

To view data lineage, you need the following roles or permissions. Grant these roles to the principal (for example, the user or service account) on the relevant resources (for example, projects or BigQuery tables).

• Data Lineage Viewer (roles/datalineage.viewer): grants permission to view lineage in the Google Cloud console and use the Data Lineage API. You must apply this role on projects where lineage is recorded and on projects where lineage is viewed. For more information, see Project types.

• Dataplex Universal Catalog Viewer (roles/dataplex.catalogViewer): grants permission to view metadata for assets cataloged in Dataplex Universal Catalog. Grant this role on projects where catalog entries are stored to let users see entry details in the lineage graph. Alternatively, you can grant a viewer role on the source system. For more information, see Search for data assets with Dataplex Universal Catalog.

• Other BigQuery permissions:

  • To view details of BigQuery assets in lineage graph, such as table definitions or view queries, grant the BigQuery Data Viewer (roles/bigquery.dataViewer) role.
  • To view the SQL query of a BigQuery job that produced lineage, grant the bigquery.jobs.get permission. This permission is included in roles like the BigQuery Admin (roles/bigquery.admin) role and the BigQuery Resource Viewer (roles/bigquery.resourceViewer) role.

Data lineage tracks lineage information automatically when you enable the Data Lineage API. You don't need any administrator or editor roles to capture lineage for your data assets.

For more information about granting roles, see Manage access. You can assign a role at a higher folder or organization level. For more information, see Grant or revoke a single role.

Types of data lineage views

You can view lineage information as a graph or a list. The lineage graph displays table-level lineage by default. For BigQuery jobs, you can view column-level lineage in both graph and list views.

The following view types are available:

• Graph view: displays lineage as an interactive graph, letting you explore relationships between data assets and columns by expanding nodes.

• List view: displays lineage in a tabular format, providing simplified and detailed representations of table-level and column-level lineage. You can customize columns and export lineage data from this view.

The key elements in the graph are described as follows:

• Nodes: represent the data entities. In the table-level view, a node shows the table name and its columns. In the column-level view, each node represents a specific table and its columns that have lineage.

• Edges: the lines that connect nodes and represent the processes that occur between them. Edges can feature icons or labels to provide more information about the transformation:

  • Icons: In table-level view, icons appear on edges to represent the transformation process. When you manually explore the graph, icons on edges represent the source system of the process (for example, BigQuery or Vertex AI). If multiple processes are involved, a 'multiple processes' icon is displayed. If the process source system is unknown, a gear icon is used. When you apply filters, a gear icon is used for all processes.
  • Labels: In column-level view, edges are labeled to describe the type of dependency between columns, such as Exact copy or Other.

Enable data lineage

Enable data lineage to begin automatically tracking lineage information for supported systems. You must enable the Data Lineage API in both the project where you view lineage, and the projects in which lineage is recorded. For more information, see Project types.

1. To capture lineage information, complete the following steps:

   1. In the Google Cloud console, on the Project selector page, select the project in which you want to record lineage.

      Go to Project selector

   2. Enable the Data Lineage API.

      Enable the Data Lineage API

   3. Repeat the previous steps for each project in which you want to record lineage.

2. In the project where you view lineage, enable the Data Lineage API and the Dataplex API.

   Enable the APIs

View lineage in Dataplex Universal Catalog

You can view data lineage information in the Dataplex Universal Catalog web interface.

To view the lineage, follow these instructions:

1. In the Google Cloud console, go to the Dataplex Universal Catalog Search page.

   Go to Search

2. Select Dataplex Universal Catalog as the search mode.

3. Search for the entry you want to view, and then click it. For more information, see Search for resources in Dataplex Universal Catalog.

4. Click the Lineage tab.

   The default Graph view opens, showing table-level lineage across systems and regions. For more information, see Lineage graph view.

5. To manually explore the lineage graph, click Expand next to a node to load five more nodes at a time.

   For more information, see Manually explore the lineage graph.

6. Click a node in the Graph view.

   The Details panel opens with information about the asset, such as fully qualified name and type. For more information, see Node details.

7. Click an edge with a process icon in the Graph view.

   The Query panel opens. For more information, see Inspect transformation logic and Audit and history of runs.

   • To inspect transformation logic, click the Details tab.

   • To see audit and history of runs, click the Runs tab.

8. In the Lineage explorer panel, select filter criteria—for example, Direction, Dependency type, or Time range—and then click Apply.

   This opens a focused view within a specific region (Preview). This view automatically expands the graph up to three levels of nodes. For more information, see Apply filters for a focused lineage view.

9. In the focused Graph view, select a node, and then in the node's details panel, click Visualize Path to visualize the lineage path from the selected node back to the root entry (only in focused view).

   For more information, see Lineage path visualization.

10. To view column-level lineage (only for BigQuery jobs), do one of the following:

    • In a focused Graph view, click the column icon on a table.

      

      Column icon

    • In the Lineage explorer panel, filter by column name, and click Apply.

    For more information, see Column-level lineage.

11. Click Reset.

    This action removes all applied filters and takes you to the beginning of the graph view.

12. Click List to switch to the list view.

    The List view offers simplified and detailed tabular representations of lineage for both table-level and column-level lineage, synchronized with the Graph view. By default, simplified list view is displayed, and you can toggle to detailed list view for analyzing individual source-target relationships. You can configure which columns are displayed and export lineage data. For more information, see Lineage list view.

View lineage in BigQuery

You can view data lineage information in the BigQuery web interface.

To view the lineage, follow these instructions:

1. In the Google Cloud console, go to the BigQuery page.

   Open the BigQuery page

2. Open the table for which you want to see the data lineage.

3. Click the Lineage tab.

   The default Graph view opens, showing table-level lineage across systems and regions. For more information, see Lineage graph view.

4. To manually explore the lineage graph, click Expand next to a node to load five more nodes at a time.

   For more information, see Manually explore the lineage graph.

5. Click a node in the Graph view.

   The Details panel opens with information about the asset, such as fully qualified name and type. For more information, see Node details.

6. Click an edge with a process icon in the Graph view.

   The Query panel opens. For more information, see Inspect transformation logic and Audit and history of runs.

   • To inspect transformation logic, click the Details tab.

   • To see audit and history of runs, click the Runs tab.

7. In the Lineage explorer panel, select filter criteria—for example, Direction, Dependency type, or Time range—and then click Apply.

   This opens a focused view within a specific region (Preview). This view automatically expands the graph up to three levels of nodes. For more information, see Apply filters for a focused lineage view.

8. In the focused Graph view, select a node, and then in the node's details panel, click Visualize Path to visualize the lineage path from the selected node back to the root entry (only in focused view).

   For more information, see Lineage path visualization.

9. To view column-level lineage (only for BigQuery jobs), do one of the following:

   • In a focused Graph view, click the column icon on a table.

     

     Column icon

   • In the Lineage explorer panel, filter by column name, and click Apply.

   For more information, see Column-level lineage.

10. Click Reset.

    This action removes all applied filters and takes you to the beginning of the graph view.

11. Click List to switch to the list view.

    The List view offers simplified and detailed tabular representations of lineage for both table-level and column-level lineage, synchronized with the Graph view. By default, simplified list view is displayed, and you can toggle to detailed list view for analyzing individual source-target relationships. You can configure which columns are displayed and export lineage data. For more information, see Lineage list view.

View lineage in Vertex AI

Systems like Vertex AI Pipelines generate lineage data for Vertex AI models and datasets. You can view data lineage information in the Vertex AI web interface.

Note: To view the lineage of Vertex AI operations, such as the training, test, and evaluation data used or hyperparameters applied, see Track the lineage of pipeline artifacts.

View lineage for a managed dataset in Vertex AI

To view the lineage for a dataset, follow these instructions:

1. In the Google Cloud console, go to the Datasets page.

   Open the Datasets page

2. Click the dataset for which you want to see the data lineage.

3. Click the Lineage tab.

   The default Graph view opens, showing table-level lineage across systems and regions. For more information, see Lineage graph view.

4. To manually explore the lineage graph, click Expand next to a node to load five more nodes at a time.

   For more information, see Manually explore the lineage graph.

5. Click a node in the Graph view.

   The Details panel opens with information about the asset, such as fully qualified name and type. For more information, see Node details.

6. Click an edge with a process icon in the Graph view.

   The Query panel opens. For more information, see Inspect transformation logic and Audit and history of runs.

   • To inspect transformation logic, click the Details tab.

   • To see audit and history of runs, click the Runs tab.

7. In the Lineage explorer panel, select filter criteria—for example, Direction, Dependency type, or Time range—and then click Apply.

   This opens a focused view within a specific region (Preview). This view automatically expands the graph up to three levels of nodes. For more information, see Apply filters for a focused lineage view.

8. In the focused Graph view, select a node, and then in the node's details panel, click Visualize Path to visualize the lineage path from the selected node back to the root entry (only in focused view).

   For more information, see Lineage path visualization.

9. To view column-level lineage (only for BigQuery jobs), do one of the following:

   • In a focused Graph view, click the column icon on a table.

     

     Column icon

   • In the Lineage explorer panel, filter by column name, and click Apply.

   For more information, see Column-level lineage.

10. Click Reset.

    This action removes all applied filters and takes you to the beginning of the graph view.

11. Click List to switch to the list view.

    The List view offers simplified and detailed tabular representations of lineage for both table-level and column-level lineage, synchronized with the Graph view. By default, simplified list view is displayed, and you can toggle to detailed list view for analyzing individual source-target relationships. You can configure which columns are displayed and export lineage data. For more information, see Lineage list view.

Note: On the lineage graph, Vertex AI processes are preceded by the . These processes include Vertex AI Pipelines components and templates, and Vertex ML Metadata artifacts, datasets, and models. If the process is cataloged in Dataplex Universal Catalog with a fully qualified domain name (FQN), you can click the entry in the lineage graph to view its details. Alternatively, to view the lineage graph details in Vertex AI, click Open in Vertex AI in the Details panel.

View lineage for a model in Vertex AI

To view the lineage for a model, follow these instructions:

1. In the Google Cloud console, go to the Model Registry page.

   Open the Model Registry page

2. Click the model for which you want to see the data lineage.

3. Click the Lineage tab.

   The default Graph view opens, showing table-level lineage across systems and regions. For more information, see Lineage graph view.

4. To manually explore the lineage graph, click Expand next to a node to load five more nodes at a time.

   For more information, see Manually explore the lineage graph.

5. Click a node in the Graph view.

   The Details panel opens with information about the asset, such as fully qualified name and type. For more information, see Node details.

6. Click an edge with a process icon in the Graph view.

   The Query panel opens. For more information, see Inspect transformation logic and Audit and history of runs.

   • To inspect transformation logic, click the Details tab.

   • To see audit and history of runs, click the Runs tab.

7. In the Lineage explorer panel, select filter criteria—for example, Direction, Dependency type, or Time range—and then click Apply.

   This opens a focused view within a specific region (Preview). This view automatically expands the graph up to three levels of nodes. For more information, see Apply filters for a focused lineage view.

8. In the focused Graph view, select a node, and then in the node's details panel, click Visualize Path to visualize the lineage path from the selected node back to the root entry (only in focused view).

   For more information, see Lineage path visualization.

9. To view column-level lineage (only for BigQuery jobs), do one of the following:

   • In a focused Graph view, click the column icon on a table.

     

     Column icon

   • In the Lineage explorer panel, filter by column name, and click Apply.

   For more information, see Column-level lineage.

10. Click Reset.

    This action removes all applied filters and takes you to the beginning of the graph view.

11. Click List to switch to the list view.

    The List view offers simplified and detailed tabular representations of lineage for both table-level and column-level lineage, synchronized with the Graph view. By default, simplified list view is displayed, and you can toggle to detailed list view for analyzing individual source-target relationships. You can configure which columns are displayed and export lineage data. For more information, see Lineage list view.

Note: On the lineage graph, Vertex AI processes are preceded by the . These processes include Vertex AI Pipelines components and templates, and Vertex ML Metadata artifacts, datasets, and models. If the process is cataloged in Dataplex Universal Catalog with a fully qualified domain name (FQN), you can click the entry in the lineage graph to view its details. Alternatively, to view the lineage graph details in Vertex AI, click Open in Vertex AI in the Details panel.

What's next

• Track data lineage for a BigQuery table's copy and query jobs.

• Learn about data lineage information model.

• Learn about data lineage considerations.

• Learn about data lineage audit logging.

• Learn how to troubleshoot data lineage.

• Learn how to integrate with OpenLineage.