Manage aspects and enrich metadata

This document describes how to create and manage aspect types, and annotate entries with aspects.

Dataplex Catalog describes the entries by a set of aspects. You can describe your entries with additional metadata by using aspects and aspect types.

For more information, see Dataplex Catalog overview.

Aspects

Aspects let you capture metadata within entries. Adding aspects to an entry helps provide meaningful context to anyone who needs to use the asset. You can use aspects to store business metadata (for example, data classification) and technical metadata (for example, schema).

Aspects are considered to be parts of the entry resource and not separate resources. When you modify an aspect, it involves modifying the entry containing the aspect.

You can specify aspects at entry-level for describing an entry, or at column-level for describing a column in an entry.

Every aspect is an instance of an aspect type. An aspect type defines a template for its aspects. Every aspect type contains a set of fields. When you create aspects, you must provide values for those fields.

For a given entry, there can be at most one aspect associated with the entry, per aspect type. You can have multiple aspects associated with entry columns per aspect type.

Categories of aspects

Aspects are categorized into the following:

  • Required aspects: aspects which are mandatory upon creation of an entry. Such aspects are defined by the entry type of a given entry. All entries belonging to an entry type must always have all of the required aspects that are defined by that entry type.

    Dataplex manages the required aspects (for example, schema) for system entries.

    Note the following:

    • You can associate required aspects only with entries and not with the columns of an entry.

    • You can't delete the required aspects from an entry.

    • You can read the required aspects of system entries, but can't modify them.

  • Optional aspects: You can associate optional aspects with entries or with entry columns. You can populate optional aspects either at the time of entry creation, or later by updating the entry.

    You can delete optional aspects after they have been populated.

Aspect types

Aspect types are reusable resources that provide templates for aspects.

Categories of aspect types

Aspect types are categorized into the following:

  • Custom aspect types: aspect types that you create in Dataplex Catalog.

  • System aspect types: aspect types that Dataplex provides, uses, and manages.

    System aspect types are further categorized into reusable and restricted. The following table describes the categories of system aspect types, and the list of aspect types that Dataplex provides for each of the categories:

    Category of system aspect type Description Aspect types that Dataplex provides
    Reusable system aspect type You can use these aspect types to create or modify aspects.
    • generic
    • storage
    Restricted system aspect type Dataplex manages these aspect types.
    You can read aspects under these aspect types, but can't create or edit aspects under these aspect types.
    • bigquery-connection
    • bigquery-dataset
    • bigquery-model
    • bigquery-routine
    • bigquery-table
    • bigquery-view
    • cloudsql-database
    • cloudsql-instance
    • cloudsql-schema
    • cloudsql-table
    • cloudsql-view
    • storage
    • sql-access
    • storage-bucket
    • storage-folder

Aspect types can be global or regional. You can create custom aspect types either in a specific regional location or as global resource. System aspect types are always global.

Global aspect types are used to create aspects for entries in any region. The content of global aspect types is replicated across all regions. Regional aspect types are used to create aspects for entries in the same region.

The location of an aspect type impacts the scope of its applicability. For more information, see Project and location constraints.

Before you begin

Before you create and manage aspect types and aspects, complete the tasks described in this section.

Required roles

To get the permissions that you need to create and manage aspect types and aspects, ask your administrator to grant you the following IAM roles on the resource:

  • Full set of permissions on all Dataplex Catalog resources including aspect types: Dataplex Catalog Admin (roles/dataplex.catalogAdmin)
  • Create and manage all Dataplex Catalog resources including aspect types: Dataplex Catalog Editor (roles/dataplex.catalogEditor)
  • Full set of permissions on custom aspect types (except for permissions to use aspect types to create or edit entries): Dataplex Aspect Type Owner (roles/dataplex.aspectTypeOwner)
  • View aspects types and IAM policies associated with them: Dataplex Catalog Viewer (roles/dataplex.catalogViewer)
  • Use aspect types to create and modify entries with the corresponding aspects: Dataplex Aspect Type User (roles/dataplex.aspectTypeUser)
  • Add aspects of some of the system aspect types, such as schema, overview, contacts: Dataplex Entry Owner (roles/dataplex.entryOwner)

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

For more information, see Dataplex IAM roles.

Enable the API

Enable the Dataplex API in your Dataplex project.

Enable the API

Create an aspect type

Console

  1. In the Google Cloud console, go to the Dataplex Catalog page.

    Go to Catalog

  2. Click the Aspect types & tag templates > Custom tab.

  3. Click Create aspect type (Dataplex Catalog).

  4. In the Create aspect type window, enter the following:

    1. Optional: In the Display name field, enter a name for the aspect type.
    2. In the Aspect type ID field, enter a unique ID for the aspect type.
    3. Optional: In the Description field, enter a description for the aspect type.
    4. In the Location field, select a location for the aspect type. You can't modify the location of an aspect type after you create it.

  5. Optional: Define a template for your aspect type.

    In the Templates, click Add field. In the New field section, enter the following:

    1. In the Name field, enter a name.
    2. Optional: In the Display name field, enter a display name.
    3. Optional: In the Description field, enter a description.

    4. In the Type field, select a data type for the field. Based on your selection, the next set of fields and options are displayed:

      • If you selected Text as the data type, follow these steps:

        1. In the Text type field, select the type of text.
        2. In the Text values field, provide a hint for the text field. To do this, click Add value and enter the hint. You can add multiple hints for a text field.
        3. Click Done.

      • If you selected Enum as the data type, add an enum value:

        1. Click Add an enum value.
        2. In the Value field, enter an enum value. You can add multiple enum values.
        3. Click Done.

      • If you selected Array as the data type, in the Array item section, define the types of items to be present in the array:

        1. Click Add array item.
        2. In the Name field, enter a name for the array items.
        3. Optional: In the Display name field, enter a display name for the array items.
        4. Optional: In the Description field, enter a description for the array items.

        5. In the Type field, select a data type for the array items.

          Based on your selection, the next set of fields and options are displayed. They are similar to the options described for the data types Text, Enum, Map, Array, and Record elsewhere in this section.

        6. Click Done.

      • If you selected Map as the data type, in the Map value section, define the types of values to be present in the map:

        1. Click Add map value.
        2. In the Name field, enter a name for the map.
        3. Optional: In the Display name field, enter a display name for the map.
        4. Optional: In the Description field, enter a description for the map.

        5. In the Type field, select a data type for the map.

          Based on your selection, the next set of fields and options are displayed. They are similar to the options described for the data types Text, Enum, Map, Array, and Record elsewhere in this section.

        6. Click Done.

      • If you selected Record as the data type, enter the following:

        1. In the Record ID field, enter a unique ID that other record fields can use to refer to this record. See the Example for using the Record ID and Record reference fields section of this document.
        2. Optional: If you want to add a reference to another record from this template, use the Record reference field. You can't modify this after you create the aspect type. See the Example for using the Record ID and Record reference fields section of this document.

        3. In the Record fields section, you can define a complex object with multiple nested fields. To do this, click Add record field item, and specify the following:

        4. In the Name field, enter a name for the record field.

        5. Optional: In the Display name field, enter a display name for the record field.

        6. Optional: In the Description field, enter a description for the record field.

        7. In the Type field, select a data type.

          Based on your selection, the next set of fields and options are displayed. They are similar to the options described for the data types Text, Enum, Map, Array, and Record earlier in this section.

        8. Click Done.

    5. To make the field mandatory for an aspect of this type, select Is required. For more information about required aspects and optional aspects, see the categories of aspects section of this document.

    6. Click Done.

    7. To add multiple fields, click Add field and repeat the previous steps.

  6. Optional: In the Labels section, add arbitrary labels as key-value pairs to your resources:

    1. Click Add label.
    2. In the Key field, enter a key.
    3. In the Value field, enter a value for the key.
    4. To add more labels, click Add label and repeat the steps.

  7. Click Save.

gcloud

To create an aspect type, use the gcloud dataplex aspect-types create command.

REST

To create an aspect type, use the aspectType.create method.

After you create an aspect type, you can add aspects to entries.

Example for using the Record ID and Record reference fields

You can use the Record ID and Record reference fields for recursive references. The following example shows how to use these fields:

Consider an aspect type called Employee, with the following fields:

  • Name (type:Text)
  • Start date (type:Date & time)
  • Designation (type:Text)
  • Current address (type:Record)
  • Permanent address (type:Record)

The two address fields Current address and Permanent address are of the same data type Record. To avoid duplication, you can set the Record ID and Record reference values when defining these fields.

When you define the field Current address, you can specify Record ID as address-field. For Permanent address, you can specify the same value (address-field) for Record reference. For example:

  • Name (type:Text)
  • Start date (type:Date & time)
  • Designation (type:Text)
  • Current address (type:Record, Record ID:address-field)
  • Permanent address (type:Record, Record reference:address-field)

This way, you don't need to duplicate the fields of another address.

Add aspects to an entry

After you create an aspect type, you can create aspects of that type. To add aspects to an entry, you must update the entry, as aspects are stored within entries.

Note the following:

  • You can add aspects to an entry or to the columns of an entry.
  • You can edit the required aspects only for custom entries. You can't delete the required aspects.

  • You can edit and delete the optional aspects for both custom entries and system entries.

Console

  1. In the Google Cloud console, go to the Dataplex Search page.

    Go to Search

  2. For Choose search platform, select Dataplex Catalog as the search mode.

  3. Search for the entry that you want to add aspects to, and click the entry. The entry details page opens.

  4. To add aspects to the entry, follow these steps:

    1. Click the Details tab.
    2. To add required aspects or optional aspects to the entry, in the Tags & aspects section, click Add for the respective category.
      You can't add required aspects if the entry type of the selected entry has no required aspects defined.
    3. Search for and select the aspect you want to add.
    4. In the Add aspect window, enter the values for the fields.
    5. Click Save.

  5. To add aspects to a column of the entry, follow these steps:

    1. On the entry details page, click the Schema tab.
    2. Select the columns to which you want to add aspects.
    3. Click Add aspect.
    4. Search for and select the aspect you want to add.
    5. In the Add aspect window, enter the values for the fields.
    6. Click Save.

gcloud

To add aspects to an entry or to a column of an entry, use the gcloud dataplex entries update command.

REST

To add aspects to an entry or to a column of an entry, use the entry.patch method.

Creating and using aspects in an entry where the respective aspect type and the entry are in different Google Cloud organizations isn't supported.

Manage existing aspects for an entry

This section describes how to update and delete the existing aspects for an entry.

Update an aspect

You can edit the optional aspects for both custom entries and system entries. You can edit the required aspects only for custom entries.

Console

  1. In the Google Cloud console, go to the Dataplex Search page.

    Go to Search

  2. For Choose search platform, select Dataplex Catalog as the search mode.

  3. Search for the entry whose aspects you want to update, and click the entry.

    The entry details page opens.

  4. Click the Details tab.

  5. In the Tags & aspects section, expand the aspect or tag that you want to update, and click Edit.

  6. In the Edit aspect window, update the required fields.

  7. Click Save.

gcloud

To update aspects for an entry or a column of an entry, use the gcloud dataplex entries update command.

REST

To update aspects for an entry or a column of an entry, use the entry.update method.

Delete an aspect

Console

  1. In the Google Cloud console, go to the Dataplex Search page.

    Go to Search

  2. For Choose search platform, select Dataplex Catalog as the search mode.

  3. Search for the entry whose aspects you want to delete, and click the entry.

    The entry details page opens.

  4. Click the Details tab.

  5. In the Tags & aspects section, expand the aspect or tag that you want to delete, click Delete.

  6. Click Confirm.

gcloud

To delete an aspect for an entry, use the gcloud dataplex entries update command.

REST

To delete an aspect for an entry, use the entry.update method.

Manage aspect types

This section describes how to view, update, and delete aspect types.

For more information about managing Data Catalog tag templates, see Tags and tag templates.

View the list of available aspect types and tag templates

Console

  1. In the Google Cloud console, go to the Dataplex Catalog page.

    Go to Catalog

  2. Click the Aspect types & tag templates tab.

    This page lists all the available Dataplex Catalog aspects types and Data Catalog tag templates.

  3. To view custom aspect types and tag templates, click the Custom tab. In the Custom tab, the Catalog source column displays where the resource resides—Dataplex Catalog or Data Catalog.

    To view system aspect types, click the System tab.

    For more information about custom and system aspect types, see the categories of aspect types section of this document.

  4. Optional: To view the list of aspect types and tag templates in your selected project, click the Custom tab, and then click the Show from all projects toggle to the off position.

    The Show from all projects toggle is on by default, and the list includes Dataplex Catalog resources from your selected organization and Data Catalog resources from all the organizations that you can access.

gcloud

To list all the available aspect types, use the gcloud dataplex aspect-types list command.

REST

To list all the available aspect types, use the aspectTypes.list method.

View details of an aspect type or tag template

Console

  1. In the Google Cloud console, go to the Dataplex Catalog page.

    Go to Catalog

  2. Click the Aspect types & tag templates tab.

  3. To view the details of a Dataplex Catalog aspect type, click the aspect type. The aspect type details page opens, where you can access details, such as display name, aspect type ID, description, project ID, location, labels, creation date, and last modified date.

    • To view the structure of the selected aspect type, click the Template tab.
    • To view the list of 10 related entries created recently, click the Sample entries tab.
    • To search for all related entries, click Show all related entries in Search. This button appears only if there's at least one related entry.

  4. To view the details of a Data Catalog tag template, click the tag template. The tag template details page opens in Data Catalog, where you can access details, such as display name, template ID, project ID, location, visibility, and labels.

gcloud

To get the details of an aspect type, use the gcloud dataplex aspect-types describe command.

REST

To get the details of an aspect type, use the aspectTypes.get method.

Update an aspect type

You can update the display name, description, template fields, and labels of an aspect type. You can't delete an existing field in a template.

You can't update the aspect type ID and location after you create the aspect type.

Console

  1. In the Google Cloud console, go to the Dataplex Catalog page.

    Go to Catalog

  2. Click the Aspect types & tag templates tab.

  3. Click the aspect type that you want to update.

  4. On the Aspect type details page, click Edit.

  5. Edit the display name, description, template fields, and labels, as required.

  6. Optional: To mark a field in the aspect type as deprecated, follow these steps:

    1. In the Template section, expand the field.
    2. Select Is Deprecated.
    3. In the Deprecation reason field, enter a reason for deprecating the selected field.
    4. Click Done.

  7. Click Save.

gcloud

To update an aspect type, use the gcloud dataplex aspect-types update command.

REST

To update an aspect type, use the aspectTypes.patch method.

Delete an aspect type

Console

  1. In the Google Cloud console, go to the Dataplex Catalog page.

    Go to Catalog

  2. Click the Aspect types & tag templates tab.

  3. Click the aspect type that you want to delete.

  4. On the Aspect type details page, click Delete. Confirm when prompted.

gcloud

To delete an aspect type, use the gcloud dataplex aspect-types delete command.

REST

To delete an existing aspect type, use the aspectTypes.delete method.

What's next