Manage metadata of Dataform repositories with Dataplex

This document describes using Dataplex to view metadata of Dataform repositories and annotate them with aspects.

Dataplex lets you store and manage your metadata. Dataform repositories are available in Dataplex by default, without any additional configuration. Dataplex automatically retrieves the following metadata from Dataform repositories:

  • Data asset name
  • Data asset parent
  • Data asset location
  • Data asset type
  • Corresponding Google Cloud project
  • Third-party repository source
  • Service account

Before you begin

  1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  2. Enable the Dataplex API.

    Enable the API

  3. Ensure that you have the necessary Identity and Access Management (IAM) permissions.

Required roles

To manage Dataform repositories in Dataplex, you need the predefined Dataform roles that give you access to Dataform resources, and required Dataplex roles.

Manage metadata in Dataplex

Dataplex logs Dataform repositories as entries with the following entry values:

System entry group
The system entry group for Dataform repositories is @dataform. To view details of repository entries in Dataplex, you need to view the dataform system entry group. For instructions about how to view a list of all entries in an entry group, see View details of an entry group in the Dataplex documentation.
System entry type
The system entry type for Dataform repositories is dataform-repository. To view details of Dataform repositories, you need to view the dataform-repository system entry type. For instructions about how to view details of a selected entry type, see View details of an entry type in the Dataplex documentation. For instructions about how to view details of a selected entry, see View details of an entry in the Dataplex documentation.

Search for assets

For instructions about how to search for data assets in Dataplex, see Search for data assets in Dataplex in the Dataplex documentation.

Annotate repositories

To provide additional context to Dataform repositories in Dataplex, you can annotate Dataform entries with aspects.

The system aspect type for Dataform repositories is dataform-repository.

For instructions about how to annotate entries with aspects, see Manage aspects and enrich metadata in the Dataplex documentation.

Limitations

Managing Dataform repositories in Dataplex is available with the following limitations:

  • You cannot view or manage Dataform file-level assets in Dataplex—for example, tables, or views.

Locations

You can use Dataplex to manage Dataform repositories from all Dataform locations.

Quotas

Managing Dataform repositories in Dataplex is subject to Dataplex quotas and limits.

Pricing

Managing Dataform repositories in Dataplex is subject to Dataplex pricing.

What's next