Package google.cloud.datacatalog.v1

Index

DataCatalog

Data Catalog API service allows you to discover, understand, and manage your data.

CreateEntry

rpc CreateEntry(CreateEntryRequest) returns (Entry)

Creates an entry.

You can create entries only with 'FILESET', 'CLUSTER', 'DATA_STREAM', or custom types. Data Catalog automatically creates entries with other types during metadata ingestion from integrated systems.

You must enable the Data Catalog API in the project identified by the parent parameter. For more information, see Data Catalog resource project.

An entry group can have a maximum of 100,000 entries.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

CreateEntryGroup

rpc CreateEntryGroup(CreateEntryGroupRequest) returns (EntryGroup)

Creates an entry group.

An entry group contains logically related entries together with Cloud Identity and Access Management policies. These policies specify users who can create, edit, and view entries within entry groups.

Data Catalog automatically creates entry groups with names that start with the @ symbol for the following resources:

  • BigQuery entries (@bigquery)
  • Pub/Sub topics (@pubsub)
  • Dataproc Metastore services (@dataproc_metastore_{SERVICE_NAME_HASH})

You can create your own entry groups for Cloud Storage fileset entries and custom entries together with the corresponding IAM policies. User-created entry groups can't contain the @ symbol, it is reserved for automatically created groups.

Entry groups, like entries, can be searched.

A maximum of 10,000 entry groups may be created per organization across all locations.

You must enable the Data Catalog API in the project identified by the parent parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

CreateTag

rpc CreateTag(CreateTagRequest) returns (Tag)

Creates a tag and assigns it to:

  • An Entry if the method name is projects.locations.entryGroups.entries.tags.create.
  • Or EntryGroupif the method name is projects.locations.entryGroups.tags.create.

Note: The project identified by the parent parameter for the tag and the tag template used to create the tag must be in the same organization.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

CreateTagTemplate

rpc CreateTagTemplate(CreateTagTemplateRequest) returns (TagTemplate)

Creates a tag template.

You must enable the Data Catalog API in the project identified by the parent parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

CreateTagTemplateField

rpc CreateTagTemplateField(CreateTagTemplateFieldRequest) returns (TagTemplateField)

Creates a field in a tag template.

You must enable the Data Catalog API in the project identified by the parent parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

DeleteEntry

rpc DeleteEntry(DeleteEntryRequest) returns (Empty)

Deletes an existing entry.

You can delete only the entries created by the CreateEntry method.

You must enable the Data Catalog API in the project identified by the name parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

DeleteEntryGroup

rpc DeleteEntryGroup(DeleteEntryGroupRequest) returns (Empty)

Deletes an entry group.

You must enable the Data Catalog API in the project identified by the name parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

DeleteTag

rpc DeleteTag(DeleteTagRequest) returns (Empty)

Deletes a tag.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

DeleteTagTemplate

rpc DeleteTagTemplate(DeleteTagTemplateRequest) returns (Empty)

Deletes a tag template and all tags that use it.

You must enable the Data Catalog API in the project identified by the name parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

DeleteTagTemplateField

rpc DeleteTagTemplateField(DeleteTagTemplateFieldRequest) returns (Empty)

Deletes a field in a tag template and all uses of this field from the tags based on this template.

You must enable the Data Catalog API in the project identified by the name parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

GetEntry

rpc GetEntry(GetEntryRequest) returns (Entry)

Gets an entry.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

GetEntryGroup

rpc GetEntryGroup(GetEntryGroupRequest) returns (EntryGroup)

Gets an entry group.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

GetIamPolicy

rpc GetIamPolicy(GetIamPolicyRequest) returns (Policy)

Gets the access control policy for a resource.

May return:

  • ANOT_FOUND error if the resource doesn't exist or you don't have the permission to view it.
  • An empty policy if the resource exists but doesn't have a set policy.

Supported resources are:

  • Tag templates
  • Entry groups

Note: This method doesn't get policies from Google Cloud Platform resources ingested into Data Catalog.

To call this method, you must have the following Google IAM permissions:

  • datacatalog.tagTemplates.getIamPolicy to get policies on tag templates.
  • datacatalog.entryGroups.getIamPolicy to get policies on entry groups.
Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

GetTagTemplate

rpc GetTagTemplate(GetTagTemplateRequest) returns (TagTemplate)

Gets a tag template.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ListEntries

rpc ListEntries(ListEntriesRequest) returns (ListEntriesResponse)

Lists entries.

Note: Currently, this method can list only custom entries. To get a list of both custom and automatically created entries, use SearchCatalog.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ListEntryGroups

rpc ListEntryGroups(ListEntryGroupsRequest) returns (ListEntryGroupsResponse)

Lists entry groups.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ListTags

rpc ListTags(ListTagsRequest) returns (ListTagsResponse)

Lists tags assigned to an Entry. The columns in the response are lowercased.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

LookupEntry

rpc LookupEntry(LookupEntryRequest) returns (Entry)

Gets an entry by its target resource name.

The resource name comes from the source Google Cloud Platform service.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ModifyEntryContacts

rpc ModifyEntryContacts(ModifyEntryContactsRequest) returns (Contacts)

Modifies contacts, part of the business context of an Entry.

To call this method, you must have the datacatalog.entries.updateContacts IAM permission on the corresponding project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ModifyEntryOverview

rpc ModifyEntryOverview(ModifyEntryOverviewRequest) returns (EntryOverview)

Modifies entry overview, part of the business context of an Entry.

To call this method, you must have the datacatalog.entries.updateOverview IAM permission on the corresponding project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ReconcileTags

rpc ReconcileTags(ReconcileTagsRequest) returns (Operation)

ReconcileTags creates or updates a list of tags on the entry. If the ReconcileTagsRequest.force_delete_missing parameter is set, the operation deletes tags not included in the input tag list.

ReconcileTags returns a long-running operation resource that can be queried with Operations.GetOperation to return ReconcileTagsMetadata and a ReconcileTagsResponse message.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

RenameTagTemplateField

rpc RenameTagTemplateField(RenameTagTemplateFieldRequest) returns (TagTemplateField)

Renames a field in a tag template.

You must enable the Data Catalog API in the project identified by the name parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

RenameTagTemplateFieldEnumValue

rpc RenameTagTemplateFieldEnumValue(RenameTagTemplateFieldEnumValueRequest) returns (TagTemplateField)

Renames an enum value in a tag template.

Within a single enum field, enum values must be unique.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

RetrieveConfig

rpc RetrieveConfig(RetrieveConfigRequest) returns (OrganizationConfig)

Retrieves the configuration related to the migration from Data Catalog to Dataplex for a specific organization, including all the projects under it which have a separate configuration set.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

IAM Permissions

Requires the following IAM permission on the name resource:

  • datacatalog.migrationConfig.get

For more information, see the IAM documentation.

RetrieveEffectiveConfig

rpc RetrieveEffectiveConfig(RetrieveEffectiveConfigRequest) returns (MigrationConfig)

Retrieves the effective configuration related to the migration from Data Catalog to Dataplex for a specific organization or project. If there is no specific configuration set for the resource, the setting is checked hierarchicahlly through the ancestors of the resource, starting from the resource itself.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

IAM Permissions

Requires the following IAM permission on the name resource:

  • datacatalog.migrationConfig.get

For more information, see the IAM documentation.

SearchCatalog

rpc SearchCatalog(SearchCatalogRequest) returns (SearchCatalogResponse)

Searches Data Catalog for multiple resources like entries and tags that match a query.

This is a Custom Method that doesn't return all information on a resource, only its ID and high level fields. To get more information, you can subsequently call specific get methods.

Note: Data Catalog search queries don't guarantee full recall. Results that match your query might not be returned, even in subsequent result pages. Additionally, returned (and not returned) results can vary if you repeat search queries.

For more information, see Data Catalog search syntax.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

SetConfig

rpc SetConfig(SetConfigRequest) returns (MigrationConfig)

Sets the configuration related to the migration to Dataplex for an organization or project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

IAM Permissions

Requires the following IAM permission on the name resource:

  • datacatalog.migrationConfig.set

For more information, see the IAM documentation.

SetIamPolicy

rpc SetIamPolicy(SetIamPolicyRequest) returns (Policy)

Sets an access control policy for a resource. Replaces any existing policy.

Supported resources are:

  • Tag templates
  • Entry groups

Note: This method sets policies only within Data Catalog and can't be used to manage policies in BigQuery, Pub/Sub, Dataproc Metastore, and any external Google Cloud Platform resources synced with the Data Catalog.

To call this method, you must have the following Google IAM permissions:

  • datacatalog.tagTemplates.setIamPolicy to set policies on tag templates.
  • datacatalog.entryGroups.setIamPolicy to set policies on entry groups.
Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

StarEntry

rpc StarEntry(StarEntryRequest) returns (StarEntryResponse)

Marks an Entry as starred by the current user. Starring information is private to each user.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

TestIamPermissions

rpc TestIamPermissions(TestIamPermissionsRequest) returns (TestIamPermissionsResponse)

Gets your permissions on a resource.

Returns an empty set of permissions if the resource doesn't exist.

Supported resources are:

  • Tag templates
  • Entry groups

Note: This method gets policies only within Data Catalog and can't be used to get policies from BigQuery, Pub/Sub, Dataproc Metastore, and any external Google Cloud Platform resources ingested into Data Catalog.

No Google IAM permissions are required to call this method.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

UnstarEntry

rpc UnstarEntry(UnstarEntryRequest) returns (UnstarEntryResponse)

Marks an Entry as NOT starred by the current user. Starring information is private to each user.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

UpdateEntry

rpc UpdateEntry(UpdateEntryRequest) returns (Entry)

Updates an existing entry.

You must enable the Data Catalog API in the project identified by the entry.name parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

UpdateEntryGroup

rpc UpdateEntryGroup(UpdateEntryGroupRequest) returns (EntryGroup)

Updates an entry group.

You must enable the Data Catalog API in the project identified by the entry_group.name parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

UpdateTag

rpc UpdateTag(UpdateTagRequest) returns (Tag)

Updates an existing tag.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

UpdateTagTemplate

rpc UpdateTagTemplate(UpdateTagTemplateRequest) returns (TagTemplate)

Updates a tag template.

You can't update template fields with this method. These fields are separate resources with their own create, update, and delete methods.

You must enable the Data Catalog API in the project identified by the tag_template.name parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

UpdateTagTemplateField

rpc UpdateTagTemplateField(UpdateTagTemplateFieldRequest) returns (TagTemplateField)

Updates a field in a tag template.

You can't update the field type with this method.

You must enable the Data Catalog API in the project identified by the name parameter. For more information, see Data Catalog resource project.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

PolicyTagManager

Policy Tag Manager API service allows you to manage your policy tags and taxonomies.

Policy tags are used to tag BigQuery columns and apply additional access control policies. A taxonomy is a hierarchical grouping of policy tags that classify data along a common axis.

CreatePolicyTag

rpc CreatePolicyTag(CreatePolicyTagRequest) returns (PolicyTag)

Creates a policy tag in a taxonomy.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

CreateTaxonomy

rpc CreateTaxonomy(CreateTaxonomyRequest) returns (Taxonomy)

Creates a taxonomy in a specified project.

The taxonomy is initially empty, that is, it doesn't contain policy tags.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

DeletePolicyTag

rpc DeletePolicyTag(DeletePolicyTagRequest) returns (Empty)

Deletes a policy tag together with the following:

  • All of its descendant policy tags, if any
  • Policies associated with the policy tag and its descendants
  • References from BigQuery table schema of the policy tag and its descendants
Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

DeleteTaxonomy

rpc DeleteTaxonomy(DeleteTaxonomyRequest) returns (Empty)

Deletes a taxonomy, including all policy tags in this taxonomy, their associated policies, and the policy tags references from BigQuery columns.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

GetIamPolicy

rpc GetIamPolicy(GetIamPolicyRequest) returns (Policy)

Gets the IAM policy for a policy tag or a taxonomy.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

GetPolicyTag

rpc GetPolicyTag(GetPolicyTagRequest) returns (PolicyTag)

Gets a policy tag.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

GetTaxonomy

rpc GetTaxonomy(GetTaxonomyRequest) returns (Taxonomy)

Gets a taxonomy.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ListPolicyTags

rpc ListPolicyTags(ListPolicyTagsRequest) returns (ListPolicyTagsResponse)

Lists all policy tags in a taxonomy.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ListTaxonomies

rpc ListTaxonomies(ListTaxonomiesRequest) returns (ListTaxonomiesResponse)

Lists all taxonomies in a project in a particular location that you have a permission to view.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

SetIamPolicy

rpc SetIamPolicy(SetIamPolicyRequest) returns (Policy)

Sets the IAM policy for a policy tag or a taxonomy.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

TestIamPermissions

rpc TestIamPermissions(TestIamPermissionsRequest) returns (TestIamPermissionsResponse)

Returns your permissions on a specified policy tag or taxonomy.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

UpdatePolicyTag

rpc UpdatePolicyTag(UpdatePolicyTagRequest) returns (PolicyTag)

Updates a policy tag, including its display name, description, and parent policy tag.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

UpdateTaxonomy

rpc UpdateTaxonomy(UpdateTaxonomyRequest) returns (Taxonomy)

Updates a taxonomy, including its display name, description, and activated policy types.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

PolicyTagManagerSerialization

Policy Tag Manager Serialization API service allows you to manipulate your policy tags and taxonomies in a serialized format.

Taxonomy is a hierarchical group of policy tags.

ExportTaxonomies

rpc ExportTaxonomies(ExportTaxonomiesRequest) returns (ExportTaxonomiesResponse)

Exports taxonomies in the requested type and returns them, including their policy tags. The requested taxonomies must belong to the same project.

This method generates SerializedTaxonomy protocol buffers with nested policy tags that can be used as input for ImportTaxonomies calls.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ImportTaxonomies

rpc ImportTaxonomies(ImportTaxonomiesRequest) returns (ImportTaxonomiesResponse)

Creates new taxonomies (including their policy tags) in a given project by importing from inlined or cross-regional sources.

For a cross-regional source, new taxonomies are created by copying from a source in another region.

For an inlined source, taxonomies and policy tags are created in bulk using nested protocol buffer structures.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

ReplaceTaxonomy

rpc ReplaceTaxonomy(ReplaceTaxonomyRequest) returns (Taxonomy)

Replaces (updates) a taxonomy and all its policy tags.

The taxonomy and its entire hierarchy of policy tags must be represented literally by SerializedTaxonomy and the nested SerializedPolicyTag messages.

This operation automatically does the following:

  • Deletes the existing policy tags that are missing from the SerializedPolicyTag.
  • Creates policy tags that don't have resource names. They are considered new.
  • Updates policy tags with valid resources names accordingly.
Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

BigQueryConnectionSpec

Specification for the BigQuery connection.

Fields
connection_type

ConnectionType

The type of the BigQuery connection.

has_credential

bool

True if there are credentials attached to the BigQuery connection; false otherwise.

Union field connection_spec.

connection_spec can be only one of the following:

cloud_sql

CloudSqlBigQueryConnectionSpec

Specification for the BigQuery connection to a Cloud SQL instance.

ConnectionType

The type of the BigQuery connection.

Enums
CONNECTION_TYPE_UNSPECIFIED Unspecified type.
CLOUD_SQL Cloud SQL connection.

BigQueryDateShardedSpec

Specification for a group of BigQuery tables with the [prefix]YYYYMMDD name pattern.

For more information, see Introduction to partitioned tables.

Fields
dataset

string

Output only. The Data Catalog resource name of the dataset entry the current table belongs to. For example:

projects/{PROJECT_ID}/locations/{LOCATION}/entrygroups/{ENTRY_GROUP_ID}/entries/{ENTRY_ID}.

table_prefix

string

Output only. The table name prefix of the shards.

The name of any given shard is [table_prefix]YYYYMMDD. For example, for the MyTable20180101 shard, the table_prefix is MyTable.

shard_count

int64

Output only. Total number of shards.

latest_shard_resource

string

Output only. BigQuery resource name of the latest shard.

BigQueryRoutineSpec

Fields specific for BigQuery routines.

Fields
imported_libraries[]

string

Paths of the imported libraries.

BigQueryTableSpec

Describes a BigQuery table.

Fields
table_source_type

TableSourceType

Output only. The table source type.

Union field type_spec. Output only. type_spec can be only one of the following:
view_spec

ViewSpec

Table view specification. Populated only if the table_source_type is BIGQUERY_VIEW.

table_spec

TableSpec

Specification of a BigQuery table. Populated only if the table_source_type is BIGQUERY_TABLE.

BusinessContext

Business Context of the entry.

Fields
entry_overview

EntryOverview

Entry overview fields for rich text descriptions of entries.

contacts

Contacts

Contact people for the entry.

CatalogUIExperience

Configuration related to the opt-in status for the UI switch to Dataplex.

Enums
CATALOG_UI_EXPERIENCE_UNSPECIFIED Default value. The default UI is Dataplex.
CATALOG_UI_EXPERIENCE_ENABLED The UI is Dataplex.
CATALOG_UI_EXPERIENCE_DISABLED The UI is Data Catalog.

CloudBigtableInstanceSpec

Specification that applies to Instance entries that are part of CLOUD_BIGTABLE system. (user_specified_type)

Fields
cloud_bigtable_cluster_specs[]

CloudBigtableClusterSpec

The list of clusters for the Instance.

CloudBigtableClusterSpec

Spec that applies to clusters of an Instance of Cloud Bigtable.

Fields
display_name

string

Name of the cluster.

location

string

Location of the cluster, typically a Cloud zone.

type

string

Type of the resource. For a cluster this would be "CLUSTER".

linked_resource

string

A link back to the parent resource, in this case Instance.

CloudBigtableSystemSpec

Specification that applies to all entries that are part of CLOUD_BIGTABLE system (user_specified_type)

Fields
instance_display_name

string

Display name of the Instance. This is user specified and different from the resource name.

CloudSqlBigQueryConnectionSpec

Specification for the BigQuery connection to a Cloud SQL instance.

Fields
instance_id

string

Cloud SQL instance ID in the format of project:location:instance.

database

string

Database name.

type

DatabaseType

Type of the Cloud SQL database.

DatabaseType

Supported Cloud SQL database types.

Enums
DATABASE_TYPE_UNSPECIFIED Unspecified database type.
POSTGRES Cloud SQL for PostgreSQL.
MYSQL Cloud SQL for MySQL.

ColumnSchema

A column within a schema. Columns can be nested inside other columns.

Fields
column

string

Required. Name of the column.

Must be a UTF-8 string without dots (.). The maximum size is 64 bytes.

type

string

Required. Type of the column.

Must be a UTF-8 string with the maximum size of 128 bytes.

description

string

Optional. Description of the column. Default value is an empty string.

The description must be a UTF-8 string with the maximum size of 2000 bytes.

mode

string

Optional. A column's mode indicates whether values in this column are required, nullable, or repeated.

Only NULLABLE, REQUIRED, and REPEATED values are supported. Default mode is NULLABLE.

default_value

string

Optional. Default value for the column.

ordinal_position

int32

Optional. Ordinal position

highest_indexing_type

IndexingType

Optional. Most important inclusion of this column.

subcolumns[]

ColumnSchema

Optional. Schema of sub-columns. A column can have zero or more sub-columns.

range_element_type

FieldElementType

Optional. The subtype of the RANGE, if the type of this field is RANGE. If the type is RANGE, this field is required. Possible values for the field element type of a RANGE include: * DATE * DATETIME * TIMESTAMP

gc_rule

string

Optional. Garbage collection policy for the column or column family. Applies to systems like Cloud Bigtable.

Union field system_spec. Information only applying for columns in Entries from a specific system. system_spec can be only one of the following:
looker_column_spec

LookerColumnSpec

Looker specific column info of this column.

FieldElementType

Represents the type of a field element.

Fields
type

string

Required. The type of a field element. See ColumnSchema.type.

IndexingType

Specifies inclusion of the column in an index

Enums
INDEXING_TYPE_UNSPECIFIED Unspecified.
INDEXING_TYPE_NONE Column not a part of an index.
INDEXING_TYPE_NON_UNIQUE Column Part of non unique index.
INDEXING_TYPE_UNIQUE Column part of unique index.
INDEXING_TYPE_PRIMARY_KEY Column part of the primary key.

LookerColumnSpec

Column info specific to Looker System.

Fields
type

LookerColumnType

Looker specific column type of this column.

LookerColumnType

Column type in Looker.

Enums
LOOKER_COLUMN_TYPE_UNSPECIFIED Unspecified.
DIMENSION Dimension.
DIMENSION_GROUP Dimension group - parent for Dimension.
FILTER Filter.
MEASURE Measure.
PARAMETER Parameter.

CommonUsageStats

Common statistics on the entry's usage.

They can be set on any system.

Fields
view_count

int64

View count in source system.

Contacts

Contact people for the entry.

Fields
people[]

Person

The list of contact people for the entry.

Person

A contact person for the entry.

Fields
designation

string

Designation of the person, for example, Data Steward.

email

string

Email of the person in the format of john.doe@xyz, <john.doe@xyz>, or John Doe<john.doe@xyz>.

CreateEntryGroupRequest

Request message for CreateEntryGroup.

Fields
parent

string

Required. The names of the project and location that the new entry group belongs to.

Note: The entry group itself and its child resources might not be stored in the location specified in its name.

entry_group_id

string

Required. The ID of the entry group to create.

The ID must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), and must start with a letter or underscore. The maximum size is 64 bytes when encoded in UTF-8.

entry_group

EntryGroup

The entry group to create. Defaults to empty.

CreateEntryRequest

Request message for CreateEntry.

Fields
parent

string

Required. The name of the entry group this entry belongs to.

Note: The entry itself and its child resources might not be stored in the location specified in its name.

entry_id

string

Required. The ID of the entry to create.

The ID must contain only letters (a-z, A-Z), numbers (0-9), and underscores (_). The maximum size is 64 bytes when encoded in UTF-8.

entry

Entry

Required. The entry to create.

CreatePolicyTagRequest

Request message for CreatePolicyTag.

Fields
parent

string

Required. Resource name of the taxonomy that the policy tag will belong to.

Authorization requires the following IAM permission on the specified resource parent:

  • datacatalog.taxonomies.update
policy_tag

PolicyTag

The policy tag to create.

CreateTagRequest

Request message for CreateTag.

Fields
parent

string

Required. The name of the resource to attach this tag to.

Tags can be attached to entries or entry groups. An entry can have up to 1000 attached tags.

Note: The tag and its child resources might not be stored in the location specified in its name.

tag

Tag

Required. The tag to create.

CreateTagTemplateFieldRequest

Request message for CreateTagTemplateField.

Fields
parent

string

Required. The name of the project and the template location region.

tag_template_field_id

string

Required. The ID of the tag template field to create.

Note: Adding a required field to an existing template is not allowed.

Field IDs can contain letters (both uppercase and lowercase), numbers (0-9), underscores (_) and dashes (-). Field IDs must be at least 1 character long and at most 128 characters long. Field IDs must also be unique within their template.

tag_template_field

TagTemplateField

Required. The tag template field to create.

CreateTagTemplateRequest

Request message for CreateTagTemplate.

Fields
parent

string

Required. The name of the project and the template location region.

tag_template_id

string

Required. The ID of the tag template to create.

The ID must contain only lowercase letters (a-z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum size is 64 bytes when encoded in UTF-8.

tag_template

TagTemplate

Required. The tag template to create.

CreateTaxonomyRequest

Request message for CreateTaxonomy.

Fields
parent

string

Required. Resource name of the project that the taxonomy will belong to.

Authorization requires the following IAM permission on the specified resource parent:

  • datacatalog.taxonomies.create
taxonomy

Taxonomy

The taxonomy to create.

CrossRegionalSource

Cross-regional source used to import an existing taxonomy into a different region.

Fields
taxonomy

string

Required. The resource name of the source taxonomy to import.

DataSource

Physical location of an entry.

Fields
service

Service

Service that physically stores the data.

resource

string

Full name of a resource as defined by the service. For example:

//bigquery.googleapis.com/projects/{PROJECT_ID}/locations/{LOCATION}/datasets/{DATASET_ID}/tables/{TABLE_ID}

source_entry

string

Output only. Data Catalog entry name, if applicable.

Union field properties.

properties can be only one of the following:

storage_properties

StorageProperties

Detailed properties of the underlying storage.

Service

Name of a service that stores the data.

Enums
SERVICE_UNSPECIFIED Default unknown service.
CLOUD_STORAGE Google Cloud Storage service.
BIGQUERY BigQuery service.

DataSourceConnectionSpec

Specification that applies to a data source connection. Valid only for entries with the DATA_SOURCE_CONNECTION type. Only one of internal specs can be set at the time, and cannot be changed later.

Fields
bigquery_connection_spec

BigQueryConnectionSpec

Output only. Fields specific to BigQuery connections.

DatabaseTableSpec

Specification that applies to a table resource. Valid only for entries with the TABLE type.

Fields
type

TableType

Type of this table.

dataplex_table

DataplexTableSpec

Output only. Fields specific to a Dataplex table and present only in the Dataplex table entries.

database_view_spec

DatabaseViewSpec

Spec what aplies to tables that are actually views. Not set for "real" tables.

DatabaseViewSpec

Specification that applies to database view.

Fields
view_type

ViewType

Type of this view.

Union field source_definition. Definition of the view. source_definition can be only one of the following:
base_table

string

Name of a singular table this view reflects one to one.

sql_query

string

SQL query used to generate this view.

ViewType

Concrete type of the view.

Enums
VIEW_TYPE_UNSPECIFIED Default unknown view type.
STANDARD_VIEW Standard view.
MATERIALIZED_VIEW Materialized view.

TableType

Type of the table.

Enums
TABLE_TYPE_UNSPECIFIED Default unknown table type.
NATIVE Native table.
EXTERNAL External table.

DataplexExternalTable

External table registered by Dataplex. Dataplex publishes data discovered from an asset into multiple other systems (BigQuery, DPMS) in form of tables. We call them "external tables". External tables are also synced into the Data Catalog. This message contains pointers to those external tables (fully qualified name, resource name et cetera) within the Data Catalog.

Fields
system

IntegratedSystem

Service in which the external table is registered.

fully_qualified_name

string

Fully qualified name (FQN) of the external table.

google_cloud_resource

string

Google Cloud resource name of the external table.

data_catalog_entry

string

Name of the Data Catalog entry representing the external table.

DataplexFilesetSpec

Entry specyfication for a Dataplex fileset.

Fields
dataplex_spec

DataplexSpec

Common Dataplex fields.

DataplexSpec

Common Dataplex fields.

Fields
asset

string

Fully qualified resource name of an asset in Dataplex, to which the underlying data source (Cloud Storage bucket or BigQuery dataset) of the entity is attached.

data_format

PhysicalSchema

Format of the data.

compression_format

string

Compression format of the data, e.g., zip, gzip etc.

project_id

string

Project ID of the underlying Cloud Storage or BigQuery data. Note that this may not be the same project as the correspondingly Dataplex lake / zone / asset.

DataplexTableSpec

Entry specification for a Dataplex table.

Fields
external_tables[]

DataplexExternalTable

List of external tables registered by Dataplex in other systems based on the same underlying data.

External tables allow to query this data in those systems.

dataplex_spec

DataplexSpec

Common Dataplex fields.

user_managed

bool

Indicates if the table schema is managed by the user or not.

DatasetSpec

Specification that applies to a dataset. Valid only for entries with the DATASET type.

Fields
Union field system_spec. Fields specific to the source system. system_spec can be only one of the following:
vertex_dataset_spec

VertexDatasetSpec

Vertex AI Dataset specific fields

DeleteEntryGroupRequest

Request message for DeleteEntryGroup.

Fields
name

string

Required. The name of the entry group to delete.

force

bool

Optional. If true, deletes all entries in the entry group.

DeleteEntryRequest

Request message for DeleteEntry.

Fields
name

string

Required. The name of the entry to delete.

DeletePolicyTagRequest

Request message for DeletePolicyTag.

Fields
name

string

Required. Resource name of the policy tag to delete.

Note: All of its descendant policy tags are also deleted.

Authorization requires the following IAM permission on the specified resource name:

  • datacatalog.taxonomies.update

DeleteTagRequest

Request message for DeleteTag.

Fields
name

string

Required. The name of the tag to delete.

DeleteTagTemplateFieldRequest

Request message for DeleteTagTemplateField.

Fields
name

string

Required. The name of the tag template field to delete.

force

bool

Required. If true, deletes this field from any tags that use it.

Currently, true is the only supported value.

DeleteTagTemplateRequest

Request message for DeleteTagTemplate.

Fields
name

string

Required. The name of the tag template to delete.

force

bool

Required. If true, deletes all tags that use this template.

Currently, true is the only supported value.

DeleteTaxonomyRequest

Request message for DeleteTaxonomy.

Fields
name

string

Required. Resource name of the taxonomy to delete.

Note: All policy tags in this taxonomy are also deleted.

Authorization requires the following IAM permission on the specified resource name:

  • datacatalog.taxonomies.delete

DumpItem

Wrapper for any item that can be contained in the dump.

Fields

Union field item.

item can be only one of the following:

tagged_entry

TaggedEntry

Entry and its tags.

Entry

Entry metadata. A Data Catalog entry represents another resource in Google Cloud Platform (such as a BigQuery dataset or a Pub/Sub topic) or outside of it. You can use the linked_resource field in the entry resource to refer to the original resource ID of the source system.

An entry resource contains resource details, for example, its schema. Additionally, you can attach flexible metadata to an entry in the form of a Tag.

Fields
name

string

Output only. Identifier. The resource name of an entry in URL format.

Note: The entry itself and its child resources might not be stored in the location specified in its name.

linked_resource

string

The resource this metadata entry refers to.

For Google Cloud Platform resources, linked_resource is the Full Resource Name. For example, the linked_resource for a table resource from BigQuery is:

//bigquery.googleapis.com/projects/{PROJECT_ID}/datasets/{DATASET_ID}/tables/{TABLE_ID}

Output only when the entry is one of the types in the EntryType enum.

For entries with a user_specified_type, this field is optional and defaults to an empty string.

The resource string must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), periods (.), colons (:), slashes (/), dashes (-), and hashes (#). The maximum size is 200 bytes when encoded in UTF-8.

fully_qualified_name

string

Fully Qualified Name (FQN) of the resource. Set automatically for entries representing resources from synced systems. Settable only during creation, and read-only later. Can be used for search and lookup of the entries.

display_name

string

Display name of an entry.

The maximum size is 500 bytes when encoded in UTF-8. Default value is an empty string.

description

string

Entry description that can consist of several sentences or paragraphs that describe entry contents.

The description must not contain Unicode non-characters as well as C0 and C1 control codes except tabs (HT), new lines (LF), carriage returns (CR), and page breaks (FF). The maximum size is 2000 bytes when encoded in UTF-8. Default value is an empty string.

business_context

BusinessContext

Business Context of the entry. Not supported for BigQuery datasets

schema

Schema

Schema of the entry. An entry might not have any schema attached to it.

source_system_timestamps

SystemTimestamps

Timestamps from the underlying resource, not from the Data Catalog entry.

Output only when the entry has a system listed in the IntegratedSystem enum. For entries with user_specified_system, this field is optional and defaults to an empty timestamp.

usage_signal

UsageSignal

Resource usage statistics.

labels

map<string, string>

Cloud labels attached to the entry.

In Data Catalog, you can create and modify labels attached only to custom entries. Synced entries have unmodifiable labels that come from the source system.

data_source

DataSource

Output only. Physical location of the entry.

personal_details

PersonalDetails

Output only. Additional information related to the entry. Private to the current user.

Union field entry_type. Required. Entry type. entry_type can be only one of the following:
type

EntryType

The type of the entry.

For details, see EntryType.

user_specified_type

string

Custom entry type that doesn't match any of the values allowed for input and listed in the EntryType enum.

When creating an entry, first check the type values in the enum. If there are no appropriate types for the new entry, provide a custom value, for example, my_special_type.

The user_specified_type string has the following limitations:

  • Is case insensitive.
  • Must begin with a letter or underscore.
  • Can only contain letters, numbers, and underscores.
  • Must be at least 1 character and at most 64 characters long.
Union field system. The source system of the entry. system can be only one of the following:
integrated_system

IntegratedSystem

Output only. Indicates the entry's source system that Data Catalog integrates with, such as BigQuery, Pub/Sub, or Dataproc Metastore.

user_specified_system

string

Indicates the entry's source system that Data Catalog doesn't automatically integrate with.

The user_specified_system string has the following limitations:

  • Is case insensitive.
  • Must begin with a letter or underscore.
  • Can only contain letters, numbers, and underscores.
  • Must be at least 1 character and at most 64 characters long.
Union field system_spec. System specification. Can be used as a complement for spec, when some metadata is relevant for all entries existing within given system system_spec can be only one of the following:
sql_database_system_spec

SqlDatabaseSystemSpec

Specification that applies to a relational database system. Only settable when user_specified_system is equal to SQL_DATABASE

looker_system_spec

LookerSystemSpec

Specification that applies to Looker sysstem. Only settable when user_specified_system is equal to LOOKER

cloud_bigtable_system_spec

CloudBigtableSystemSpec

Specification that applies to Cloud Bigtable system. Only settable when integrated_system is equal to CLOUD_BIGTABLE

Union field type_spec. Type specification. type_spec can be only one of the following:
gcs_fileset_spec

GcsFilesetSpec

Specification that applies to a Cloud Storage fileset. Valid only for entries with the FILESET type.

bigquery_table_spec

BigQueryTableSpec

Output only. Specification that applies to a BigQuery table. Valid only for entries with the TABLE type.

bigquery_date_sharded_spec

BigQueryDateShardedSpec

Output only. Specification for a group of BigQuery tables with the [prefix]YYYYMMDD name pattern.

For more information, see Introduction to partitioned tables.

Union field spec. Type- and system-specific information. Specifications for types contain fields common to all entries of a given type, and sub-specifications with fields specific to a given source system.

When extending the API with new types and systems, use this field instead of the legacy type_spec. spec can be only one of the following:

database_table_spec

DatabaseTableSpec

Specification that applies to a table resource. Valid only for entries with the TABLE or EXPLORE type.

data_source_connection_spec

DataSourceConnectionSpec

Specification that applies to a data source connection. Valid only for entries with the DATA_SOURCE_CONNECTION type.

routine_spec

RoutineSpec

Specification that applies to a user-defined function or procedure. Valid only for entries with the ROUTINE type.

dataset_spec

DatasetSpec

Specification that applies to a dataset.

fileset_spec

FilesetSpec

Specification that applies to a fileset resource. Valid only for entries with the FILESET type.

service_spec

ServiceSpec

Specification that applies to a Service resource.

model_spec

ModelSpec

Model specification.

feature_online_store_spec

FeatureOnlineStoreSpec

FeatureonlineStore spec for Vertex AI Feature Store.

EntryGroup

Entry group metadata.

An EntryGroup resource represents a logical grouping of zero or more Data Catalog Entry resources.

Fields
name

string

Identifier. The resource name of the entry group in URL format.

Note: The entry group itself and its child resources might not be stored in the location specified in its name.

display_name

string

A short name to identify the entry group, for example, "analytics data - jan 2011". Default value is an empty string.

description

string

Entry group description. Can consist of several sentences or paragraphs that describe the entry group contents. Default value is an empty string.

data_catalog_timestamps

SystemTimestamps

Output only. Timestamps of the entry group. Default value is empty.

transferred_to_dataplex

bool

Optional. When set to [true], it means DataCatalog EntryGroup was transferred to Dataplex Catalog Service. It makes EntryGroup and its Entries to be read-only in DataCatalog. However, new Tags on EntryGroup and its Entries can be created. After setting the flag to [true] it cannot be unset.

EntryOverview

Entry overview fields for rich text descriptions of entries.

Fields
overview

string

Entry overview with support for rich text.

The overview must only contain Unicode characters, and should be formatted using HTML. The maximum length is 10 MiB as this value holds HTML descriptions including encoded images. The maximum length of the text without images is 100 KiB.

EntryType

Metadata automatically ingested from Google Cloud resources like BigQuery tables or Pub/Sub topics always uses enum values from EntryType as the type of entry.

Other sources of metadata like Hive or Oracle databases can identify the type by either using one of the enum values from EntryType (for example, FILESET for a Cloud Storage fileset) or specifying a custom value using the Entry field user_specified_type. For more information, see Surface files from Cloud Storage with fileset entries or Create custom entries for your data sources.

Enums
ENTRY_TYPE_UNSPECIFIED Default unknown type.
TABLE The entry type that has a GoogleSQL schema, including logical views.
MODEL

The type of models.

For more information, see Supported models in BigQuery ML.

DATA_STREAM An entry type for streaming entries. For example, a Pub/Sub topic.
FILESET An entry type for a set of files or objects. For example, a Cloud Storage fileset.
CLUSTER A group of servers that work together. For example, a Kafka cluster.
DATABASE A database.
DATA_SOURCE_CONNECTION Connection to a data source. For example, a BigQuery connection.
ROUTINE Routine, for example, a BigQuery routine.
LAKE A Dataplex lake.
ZONE A Dataplex zone.
SERVICE A service, for example, a Dataproc Metastore service.
DATABASE_SCHEMA Schema within a relational database.
DASHBOARD A Dashboard, for example from Looker.
EXPLORE

A Looker Explore.

For more information, see Looker Explore API.

LOOK

A Looker Look.

For more information, see Looker Look API.

FEATURE_ONLINE_STORE Feature Online Store resource in Vertex AI Feature Store.
FEATURE_VIEW Feature View resource in Vertex AI Feature Store.
FEATURE_GROUP Feature Group resource in Vertex AI Feature Store.

ExportTaxonomiesRequest

Request message for ExportTaxonomies.

Fields
parent

string

Required. Resource name of the project that the exported taxonomies belong to.

Authorization requires the following IAM permission on the specified resource parent:

  • datacatalog.taxonomies.get
taxonomies[]

string

Required. Resource names of the taxonomies to export.

Union field destination. Required. Export destination for taxonomies. destination can be only one of the following:
serialized_taxonomies

bool

Serialized export taxonomies that contain all the policy tags as nested protocol buffers.

ExportTaxonomiesResponse

Response message for ExportTaxonomies.

Fields
taxonomies[]

SerializedTaxonomy

List of taxonomies and policy tags as nested protocol buffers.

FeatureOnlineStoreSpec

Detail description of the source information of a Vertex Feature Online Store.

Fields
storage_type

StorageType

Output only. Type of underelaying storage for the FeatureOnlineStore.

StorageType

Type of underlaying storage type.

Enums
STORAGE_TYPE_UNSPECIFIED Should not be used.
BIGTABLE Underlsying storgae is Bigtable.
OPTIMIZED Underlaying is optimized online server (Lightning).

FieldType

Fields
Union field type_decl. Required. type_decl can be only one of the following:
primitive_type

PrimitiveType

Primitive types, such as string, boolean, etc.

enum_type

EnumType

An enum type.

EnumType

Fields
allowed_values[]

EnumValue

The set of allowed values for this enum.

This set must not be empty and can include up to 100 allowed values. The display names of the values in this set must not be empty and must be case-insensitively unique within this set.

The order of items in this set is preserved. This field can be used to create, remove, and reorder enum values. To rename enum values, use the RenameTagTemplateFieldEnumValue method.

EnumValue

Fields
display_name

string

Required. The display name of the enum value. Must not be an empty string.

The name must contain only Unicode letters, numbers (0-9), underscores (_), dashes (-), spaces ( ), and can't start or end with spaces. The maximum length is 200 characters.

PrimitiveType

Enums
PRIMITIVE_TYPE_UNSPECIFIED The default invalid value for a type.
DOUBLE A double precision number.
STRING An UTF-8 string.
BOOL A boolean value.
TIMESTAMP A timestamp.
RICHTEXT A Richtext description.

FilesetSpec

Specification that applies to a fileset. Valid only for entries with the 'FILESET' type.

Fields
dataplex_fileset

DataplexFilesetSpec

Fields specific to a Dataplex fileset and present only in the Dataplex fileset entries.

GcsFileSpec

Specification of a single file in Cloud Storage.

Fields
file_path

string

Required. Full file path. Example: gs://bucket_name/a/b.txt.

gcs_timestamps

SystemTimestamps

Output only. Creation, modification, and expiration timestamps of a Cloud Storage file.

size_bytes

int64

Output only. File size in bytes.

GcsFilesetSpec

Describes a Cloud Storage fileset entry.

Fields
file_patterns[]

string

Required. Patterns to identify a set of files in Google Cloud Storage.

For more information, see Wildcard Names.

Note: Currently, bucket wildcards are not supported.

Examples of valid file_patterns:

  • gs://bucket_name/dir/*: matches all files in bucket_name/dir directory
  • gs://bucket_name/dir/**: matches all files in bucket_name/dir and all subdirectories
  • gs://bucket_name/file*: matches files prefixed by file in bucket_name
  • gs://bucket_name/??.txt: matches files with two characters followed by .txt in bucket_name
  • gs://bucket_name/[aeiou].txt: matches files that contain a single vowel character followed by .txt in bucket_name
  • gs://bucket_name/[a-m].txt: matches files that contain a, b, ... or m followed by .txt in bucket_name
  • gs://bucket_name/a/*/b: matches all files in bucket_name that match the a/*/b pattern, such as a/c/b, a/d/b
  • gs://another_bucket/a.txt: matches gs://another_bucket/a.txt

You can combine wildcards to match complex sets of files, for example:

gs://bucket_name/[a-m]??.j*g

sample_gcs_file_specs[]

GcsFileSpec

Output only. Sample files contained in this fileset, not all files contained in this fileset are represented here.

GetEntryGroupRequest

Request message for GetEntryGroup.

Fields
name

string

Required. The name of the entry group to get.

read_mask

FieldMask

The fields to return. If empty or omitted, all fields are returned.

GetEntryRequest

Request message for GetEntry.

Fields
name

string

Required. The name of the entry to get.

GetPolicyTagRequest

Request message for GetPolicyTag.

Fields
name

string

Required. Resource name of the policy tag.

Authorization requires the following IAM permission on the specified resource name:

  • datacatalog.taxonomies.get

GetTagTemplateRequest

Request message for GetTagTemplate.

Fields
name

string

Required. The name of the tag template to get.

GetTaxonomyRequest

Request message for GetTaxonomy.

Fields
name

string

Required. Resource name of the taxonomy to get.

Authorization requires the following IAM permission on the specified resource name:

  • datacatalog.taxonomies.get

ImportTaxonomiesRequest

Request message for ImportTaxonomies.

Fields
parent

string

Required. Resource name of project that the imported taxonomies will belong to.

Authorization requires the following IAM permission on the specified resource parent:

  • datacatalog.taxonomies.create
Union field source. Source taxonomies to import. source can be only one of the following:
inline_source

InlineSource

Inline source taxonomy to import.

cross_regional_source

CrossRegionalSource

Cross-regional source taxonomy to import.

ImportTaxonomiesResponse

Response message for ImportTaxonomies.

Fields
taxonomies[]

Taxonomy

Imported taxonomies.

InlineSource

Inline source containing taxonomies to import.

Fields
taxonomies[]

SerializedTaxonomy

Required. Taxonomies to import.

IntegratedSystem

This enum lists all the systems that Data Catalog integrates with.

Enums
INTEGRATED_SYSTEM_UNSPECIFIED Default unknown system.
BIGQUERY BigQuery.
CLOUD_PUBSUB Cloud Pub/Sub.
DATAPROC_METASTORE Dataproc Metastore.
DATAPLEX Dataplex.
CLOUD_SPANNER Cloud Spanner
CLOUD_BIGTABLE Cloud Bigtable
CLOUD_SQL Cloud Sql
LOOKER Looker
VERTEX_AI Vertex AI

ListEntriesRequest

Request message for ListEntries.

Fields
parent

string

Required. The name of the entry group that contains the entries to list.

Can be provided in URL format.

page_size

int32

The maximum number of items to return. Default is 10. Maximum limit is 1000. Throws an invalid argument if page_size is more than 1000.

page_token

string

Pagination token that specifies the next page to return. If empty, the first page is returned.

read_mask

FieldMask

The fields to return for each entry. If empty or omitted, all fields are returned.

For example, to return a list of entries with only the name field, set read_mask to only one path with the name value.

ListEntriesResponse

Response message for ListEntries.

Fields
entries[]

Entry

Entry details.

next_page_token

string

Pagination token of the next results page. Empty if there are no more items in results.

ListEntryGroupsRequest

Request message for ListEntryGroups.

Fields
parent

string

Required. The name of the location that contains the entry groups to list.

Can be provided as a URL.

page_size

int32

Optional. The maximum number of items to return.

Default is 10. Maximum limit is 1000. Throws an invalid argument if page_size is greater than 1000.

page_token

string

Optional. Pagination token that specifies the next page to return. If empty, returns the first page.

ListEntryGroupsResponse

Response message for ListEntryGroups.

Fields
entry_groups[]

EntryGroup

Entry group details.

next_page_token

string

Pagination token to specify in the next call to retrieve the next page of results. Empty if there are no more items.

ListPolicyTagsRequest

Request message for ListPolicyTags.

Fields
parent

string

Required. Resource name of the taxonomy to list the policy tags of.

Authorization requires the following IAM permission on the specified resource parent:

  • datacatalog.taxonomies.get
page_size

int32

The maximum number of items to return. Must be a value between 1 and 1000 inclusively. If not set, defaults to 50.

page_token

string

The pagination token of the next results page. If not set, returns the first page.

The token is returned in the response to a previous list request.

ListPolicyTagsResponse

Response message for ListPolicyTags.

Fields
policy_tags[]

PolicyTag

The policy tags that belong to the taxonomy.

next_page_token

string

Pagination token of the next results page. Empty if there are no more results in the list.

ListTagsRequest

Request message for ListTags.

Fields
parent

string

Required. The name of the Data Catalog resource to list the tags of.

The resource can be an Entry or an EntryGroup (without /entries/{entries} at the end).

page_size

int32

The maximum number of tags to return. Default is 10. Maximum limit is 1000.

page_token

string

Pagination token that specifies the next page to return. If empty, the first page is returned.

ListTagsResponse

Response message for ListTags.

Fields
tags[]

Tag

Tag details.

next_page_token

string

Pagination token of the next results page. Empty if there are no more items in results.

ListTaxonomiesRequest

Request message for ListTaxonomies.

Fields
parent

string

Required. Resource name of the project to list the taxonomies of.

Authorization requires the following IAM permission on the specified resource parent:

  • datacatalog.taxonomies.list
page_size

int32

The maximum number of items to return. Must be a value between 1 and 1000 inclusively. If not set, defaults to 50.

page_token

string

The pagination token of the next results page. If not set, the first page is returned.

The token is returned in the response to a previous list request.

filter

string

Supported field for filter is 'service' and value is 'dataplex'. Eg: service=dataplex.

ListTaxonomiesResponse

Response message for ListTaxonomies.

Fields
taxonomies[]

Taxonomy

Taxonomies that the project contains.

next_page_token

string

Pagination token of the next results page. Empty if there are no more results in the list.

LookerSystemSpec

Specification that applies to entries that are part LOOKER system (user_specified_type)

Fields
parent_instance_id

string

ID of the parent Looker Instance. Empty if it does not exist. Example value: someinstance.looker.com

parent_instance_display_name

string

Name of the parent Looker Instance. Empty if it does not exist.

parent_model_id

string

ID of the parent Model. Empty if it does not exist.

parent_model_display_name

string

Name of the parent Model. Empty if it does not exist.

parent_view_id

string

ID of the parent View. Empty if it does not exist.

parent_view_display_name

string

Name of the parent View. Empty if it does not exist.

LookupEntryRequest

Request message for LookupEntry.

Fields
project

string

Project where the lookup should be performed. Required to lookup entry that is not a part of DPMS or DATAPLEX integrated_system using its fully_qualified_name. Ignored in other cases.

location

string

Location where the lookup should be performed. Required to lookup entry that is not a part of DPMS or DATAPLEX integrated_system using its fully_qualified_name. Ignored in other cases.

Union field target_name. Required. A full name, SQL name, or a fully qualified name of a Google Cloud Platform resource. target_name can be only one of the following:
linked_resource

string

The full name of the Google Cloud Platform resource the Data Catalog entry represents. For more information, see Full Resource Name.

Full names are case-sensitive. For example:

  • //bigquery.googleapis.com/projects/{PROJECT_ID}/datasets/{DATASET_ID}/tables/{TABLE_ID}
  • //pubsub.googleapis.com/projects/{PROJECT_ID}/topics/{TOPIC_ID}
sql_resource

string

The SQL name of the entry. SQL names are case-sensitive.

Examples:

  • pubsub.topic.{PROJECT_ID}.{TOPIC_ID}
  • pubsub.topic.{PROJECT_ID}.`{TOPIC.ID.SEPARATED.WITH.DOTS}`
  • bigquery.table.{PROJECT_ID}.{DATASET_ID}.{TABLE_ID}
  • bigquery.dataset.{PROJECT_ID}.{DATASET_ID}
  • datacatalog.entry.{PROJECT_ID}.{LOCATION_ID}.{ENTRY_GROUP_ID}.{ENTRY_ID}

Identifiers (*_ID) should comply with the Lexical structure in Standard SQL.

fully_qualified_name

string

Fully Qualified Name (FQN) of the resource.

FQNs take two forms:

  • For non-regionalized resources:

{SYSTEM}:{PROJECT}.{PATH_TO_RESOURCE_SEPARATED_WITH_DOTS}

  • For regionalized resources:

{SYSTEM}:{PROJECT}.{LOCATION_ID}.{PATH_TO_RESOURCE_SEPARATED_WITH_DOTS}

Example for a DPMS table:

dataproc_metastore:{PROJECT_ID}.{LOCATION_ID}.{INSTANCE_ID}.{DATABASE_ID}.{TABLE_ID}

ManagingSystem

This enum describes all the systems that manage Taxonomy and PolicyTag resources in DataCatalog.

Enums
MANAGING_SYSTEM_UNSPECIFIED Default value
MANAGING_SYSTEM_DATAPLEX Dataplex.
MANAGING_SYSTEM_OTHER Other

MigrationConfig

The configuration related to the migration to Dataplex applied to an organization or project. It is the response message for SetConfig and RetrieveEffectiveConfig.

Fields
tag_template_migration

TagTemplateMigration

Opt-in status for the migration of Tag Templates to Dataplex.

catalog_ui_experience

CatalogUIExperience

Opt-in status for the UI switch to Dataplex.

ModelSpec

Specification that applies to a model. Valid only for entries with the MODEL type.

Fields
Union field system_spec. System spec system_spec can be only one of the following:
vertex_model_spec

VertexModelSpec

Specification for vertex model resources.

ModifyEntryContactsRequest

Request message for ModifyEntryContacts.

Fields
name

string

Required. The full resource name of the entry.

contacts

Contacts

Required. The new value for the Contacts.

ModifyEntryOverviewRequest

Request message for ModifyEntryOverview.

Fields
name

string

Required. The full resource name of the entry.

entry_overview

EntryOverview

Required. The new value for the Entry Overview.

OrganizationConfig

The configuration related to the migration from Data Catalog to Dataplex that has been applied to an organization and any projects under it. It is the response message for RetrieveConfig.

Fields
config

map<string, MigrationConfig>

Map of organizations and project resource names and their configuration. The format for the map keys is organizations/{organizationId} or projects/{projectId}.

PersonalDetails

Entry metadata relevant only to the user and private to them.

Fields
starred

bool

True if the entry is starred by the user; false otherwise.

star_time

Timestamp

Set if the entry is starred; unset otherwise.

PhysicalSchema

Native schema used by a resource represented as an entry. Used by query engines for deserializing and parsing source data.

Fields

Union field schema.

schema can be only one of the following:

avro

AvroSchema

Schema in Avro JSON format.

thrift

ThriftSchema

Schema in Thrift format.

protobuf

ProtobufSchema

Schema in protocol buffer format.

parquet

ParquetSchema

Marks a Parquet-encoded data source.

orc

OrcSchema

Marks an ORC-encoded data source.

csv

CsvSchema

Marks a CSV-encoded data source.

AvroSchema

Schema in Avro JSON format.

Fields
text

string

JSON source of the Avro schema.

CsvSchema

This type has no fields.

Marks a CSV-encoded data source.

OrcSchema

This type has no fields.

Marks an ORC-encoded data source.

ParquetSchema

This type has no fields.

Marks a Parquet-encoded data source.

ProtobufSchema

Schema in protocol buffer format.

Fields
text

string

Protocol buffer source of the schema.

ThriftSchema

Schema in Thrift format.

Fields
text

string

Thrift IDL source of the schema.

PolicyTag

Denotes one policy tag in a taxonomy, for example, SSN.

Policy tags can be defined in a hierarchy. For example:

+ Geolocation
  + LatLong
  + City
  + ZipCode

Where the "Geolocation" policy tag contains three children.

Fields
name

string

Identifier. Resource name of this policy tag in the URL format.

The policy tag manager generates unique taxonomy IDs and policy tag IDs.

display_name

string

Required. User-defined name of this policy tag.

The name can't start or end with spaces and must be unique within the parent taxonomy, contain only Unicode letters, numbers, underscores, dashes and spaces, and be at most 200 bytes long when encoded in UTF-8.

description

string

Description of this policy tag. If not set, defaults to empty.

The description must contain only Unicode characters, tabs, newlines, carriage returns and page breaks, and be at most 2000 bytes long when encoded in UTF-8.

parent_policy_tag

string

Resource name of this policy tag's parent policy tag. If empty, this is a top level tag. If not set, defaults to an empty string.

For example, for the "LatLong" policy tag in the example above, this field contains the resource name of the "Geolocation" policy tag, and, for "Geolocation", this field is empty.

child_policy_tags[]

string

Output only. Resource names of child policy tags of this policy tag.

ReconcileTagsMetadata

Long-running operation metadata message returned by the ReconcileTags.

Fields
state

ReconciliationState

State of the reconciliation operation.

errors

map<string, Status>

Maps the name of each tagged column (or empty string for a sole entry) to tagging operation status.

ReconciliationState

Enum holding possible states of the reconciliation operation.

Enums
RECONCILIATION_STATE_UNSPECIFIED Default value. This value is unused.
RECONCILIATION_QUEUED The reconciliation has been queued and awaits for execution.
RECONCILIATION_IN_PROGRESS The reconciliation is in progress.
RECONCILIATION_DONE The reconciliation has been finished.

ReconcileTagsRequest

Request message for ReconcileTags.

Fields
parent

string

Required. Name of Entry to be tagged.

tag_template

string

Required. The name of the tag template, which is used for reconciliation.

force_delete_missing

bool

If set to true, deletes entry tags related to a tag template not listed in the tags source from an entry. If set to false, unlisted tags are retained.

tags[]

Tag

A list of tags to apply to an entry. A tag can specify a tag template, which must be the template specified in the ReconcileTagsRequest. The sole entry and each of its columns must be mentioned at most once.

ReconcileTagsResponse

Long-running operation response message returned by ReconcileTags.

Fields
created_tags_count

int64

Number of tags created in the request.

updated_tags_count

int64

Number of tags updated in the request.

deleted_tags_count

int64

Number of tags deleted in the request.

RenameTagTemplateFieldEnumValueRequest

Request message for RenameTagTemplateFieldEnumValue.

Fields
name

string

Required. The name of the enum field value.

new_enum_value_display_name

string

Required. The new display name of the enum value. For example, my_new_enum_value.

RenameTagTemplateFieldRequest

Request message for RenameTagTemplateField.

Fields
name

string

Required. The name of the tag template field.

new_tag_template_field_id

string

Required. The new ID of this tag template field. For example, my_new_field.

ReplaceTaxonomyRequest

Request message for ReplaceTaxonomy.

Fields
name

string

Required. Resource name of the taxonomy to update.

Authorization requires the following IAM permission on the specified resource name:

  • datacatalog.taxonomies.update
serialized_taxonomy

SerializedTaxonomy

Required. Taxonomy to update along with its child policy tags.

RetrieveConfigRequest

Request message for RetrieveConfig.

Fields
name

string

Required. The organization whose config is being retrieved.

RetrieveEffectiveConfigRequest

Request message for RetrieveEffectiveConfig.

Fields
name

string

Required. The resource whose effective config is being retrieved.

RoutineSpec

Specification that applies to a routine. Valid only for entries with the ROUTINE type.

Fields
routine_type

RoutineType

The type of the routine.

language

string

The language the routine is written in. The exact value depends on the source system. For BigQuery routines, possible values are:

  • SQL
  • JAVASCRIPT
routine_arguments[]

Argument

Arguments of the routine.

return_type

string

Return type of the argument. The exact value depends on the source system and the language.

definition_body

string

The body of the routine.

Union field system_spec. Contains fields specific to the source system. system_spec can be only one of the following:
bigquery_routine_spec

BigQueryRoutineSpec

Fields specific for BigQuery routines.

Argument

Input or output argument of a function or stored procedure.

Fields
name

string

The name of the argument. A return argument of a function might not have a name.

mode

Mode

Specifies whether the argument is input or output.

type

string

Type of the argument. The exact value depends on the source system and the language.

Mode

The input or output mode of the argument.

Enums
MODE_UNSPECIFIED Unspecified mode.
IN The argument is input-only.
OUT The argument is output-only.
INOUT The argument is both an input and an output.

RoutineType

The fine-grained type of the routine.

Enums
ROUTINE_TYPE_UNSPECIFIED Unspecified type.
SCALAR_FUNCTION Non-builtin permanent scalar function.
PROCEDURE Stored procedure.

Schema

Represents a schema, for example, a BigQuery, GoogleSQL, or Avro schema.

Fields
columns[]

ColumnSchema

The unified GoogleSQL-like schema of columns.

The overall maximum number of columns and nested columns is 10,000. The maximum nested depth is 15 levels.

SearchCatalogRequest

Request message for SearchCatalog.

Fields
scope

Scope

Required. The scope of this search request.

The scope is invalid if include_org_ids, include_project_ids are empty AND include_gcp_public_datasets is set to false. In this case, the request returns an error.

query

string

Optional. The query string with a minimum of 3 characters and specific syntax. For more information, see Data Catalog search syntax.

An empty query string returns all data assets (in the specified scope) that you have access to.

A query string can be a simple xyz or qualified by predicates:

  • name:x
  • column:y
  • description:z
page_size

int32

Upper bound on the number of results you can get in a single response.

Can't be negative or 0, defaults to 10 in this case. The maximum number is 1000. If exceeded, throws an "invalid argument" exception.

page_token

string

Optional. Pagination token that, if specified, returns the next page of search results. If empty, returns the first page.

This token is returned in the SearchCatalogResponse.next_page_token field of the response to a previous SearchCatalogRequest call.

order_by

string

Specifies the order of results.

Currently supported case-sensitive values are:

  • relevance that can only be descending
  • last_modified_timestamp [asc|desc] with descending (desc) as default
  • default that can only be descending

Search queries don't guarantee full recall. Results that match your query might not be returned, even in subsequent result pages. Additionally, returned (and not returned) results can vary if you repeat search queries. If you are experiencing recall issues and you don't have to fetch the results in any specific order, consider setting this parameter to default.

If this parameter is omitted, it defaults to the descending relevance.

Scope

The criteria that select the subspace used for query matching.

Fields
include_org_ids[]

string

The list of organization IDs to search within.

To find your organization ID, follow the steps from Creating and managing organizations.

include_project_ids[]

string

The list of project IDs to search within.

For more information on the distinction between project names, IDs, and numbers, see Projects.

include_gcp_public_datasets

bool

If true, include Google Cloud public datasets in search results. By default, they are excluded.

See Google Cloud Public Datasets for more information.

restricted_locations[]

string

Optional. The list of locations to search within. If empty, all locations are searched.

Returns an error if any location in the list isn't one of the Supported regions.

If a location is unreachable, its name is returned in the SearchCatalogResponse.unreachable field. To get additional information on the error, repeat the search request and set the location name as the value of this parameter.

starred_only

bool

Optional. If true, search only among starred entries.

By default, all results are returned, starred or not.

include_public_tag_templates
(deprecated)

bool

Optional. This field is deprecated. The search mechanism for public and private tag templates is the same.

SearchCatalogResponse

Response message for SearchCatalog.

Fields
results[]

SearchCatalogResult

Search results.

total_size

int32

The approximate total number of entries matched by the query.

next_page_token

string

Pagination token that can be used in subsequent calls to retrieve the next page of results.

unreachable[]

string

Unreachable locations. Search results don't include data from those locations.

To get additional information on an error, repeat the search request and restrict it to specific locations by setting the SearchCatalogRequest.scope.restricted_locations parameter.

SearchCatalogResult

Result in the response to a search request.

Each result captures details of one entry that matches the search.

Fields
search_result_type

SearchResultType

Type of the search result.

You can use this field to determine which get method to call to fetch the full resource.

search_result_subtype

string

Sub-type of the search result.

A dot-delimited full type of the resource. The same type you specify in the type search predicate.

Examples: entry.table, entry.dataStream, tagTemplate.

relative_resource_name

string

The relative name of the resource in URL format.

Examples:

  • projects/{PROJECT_ID}/locations/{LOCATION_ID}/entryGroups/{ENTRY_GROUP_ID}/entries/{ENTRY_ID}
  • projects/{PROJECT_ID}/tagTemplates/{TAG_TEMPLATE_ID}
linked_resource

string

The full name of the Google Cloud resource the entry belongs to.

For more information, see Full Resource Name.

Example:

//bigquery.googleapis.com/projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_ID

modify_time

Timestamp

The last modification timestamp of the entry in the source system.

fully_qualified_name

string

Fully qualified name (FQN) of the resource.

FQNs take two forms:

  • For non-regionalized resources:

{SYSTEM}:{PROJECT}.{PATH_TO_RESOURCE_SEPARATED_WITH_DOTS}

  • For regionalized resources:

{SYSTEM}:{PROJECT}.{LOCATION_ID}.{PATH_TO_RESOURCE_SEPARATED_WITH_DOTS}

Example for a DPMS table:

dataproc_metastore:PROJECT_ID.LOCATION_ID.INSTANCE_ID.DATABASE_ID.TABLE_ID

display_name

string

The display name of the result.

description

string

Entry description that can consist of several sentences or paragraphs that describe entry contents.

Union field system. The source system of the entry. Applicable only when the search_result_type is ENTRY. system can be only one of the following:
integrated_system

IntegratedSystem

Output only. The source system that Data Catalog automatically integrates with, such as BigQuery, Cloud Pub/Sub, or Dataproc Metastore.

user_specified_system

string

Custom source system that you can manually integrate Data Catalog with.

SearchResultType

The resource types that can be returned in search results.

Enums
SEARCH_RESULT_TYPE_UNSPECIFIED Default unknown type.
ENTRY An Entry.
TAG_TEMPLATE A TagTemplate.
ENTRY_GROUP An EntryGroup.

SerializedPolicyTag

A nested protocol buffer that represents a policy tag and all its descendants.

Fields
policy_tag

string

Resource name of the policy tag.

This field is ignored when calling ImportTaxonomies.

display_name

string

Required. Display name of the policy tag. At most 200 bytes when encoded in UTF-8.

description

string

Description of the serialized policy tag. At most 2000 bytes when encoded in UTF-8. If not set, defaults to an empty description.

child_policy_tags[]

SerializedPolicyTag

Children of the policy tag, if any.

SerializedTaxonomy

A nested protocol buffer that represents a taxonomy and the hierarchy of its policy tags. Used for taxonomy replacement, import, and export.

Fields
display_name

string

Required. Display name of the taxonomy. At most 200 bytes when encoded in UTF-8.

description

string

Description of the serialized taxonomy. At most 2000 bytes when encoded in UTF-8. If not set, defaults to an empty description.

policy_tags[]

SerializedPolicyTag

Top level policy tags associated with the taxonomy, if any.

activated_policy_types[]

PolicyType

A list of policy types that are activated per taxonomy.

ServiceSpec

Specification that applies to a Service resource. Valid only for entries with the SERVICE type.

Fields
Union field system_spec. System spec system_spec can be only one of the following:
cloud_bigtable_instance_spec

CloudBigtableInstanceSpec

Specification that applies to Instance entries of CLOUD_BIGTABLE system.

SetConfigRequest

Request message for SetConfig.

Fields
name

string

Required. The organization or project whose config is being specified.

Union field configuration. The configuration field to set. configuration can be only one of the following:
tag_template_migration

TagTemplateMigration

Opt-in status for the migration of Tag Templates to Dataplex.

catalog_ui_experience

CatalogUIExperience

Opt-in status for the UI switch to Dataplex.

SqlDatabaseSystemSpec

Specification that applies to entries that are part SQL_DATABASE system (user_specified_type)

Fields
sql_engine

string

SQL Database Engine. enum SqlEngine { UNDEFINED = 0; MY_SQL = 1; POSTGRE_SQL = 2; SQL_SERVER = 3; } Engine of the enclosing database instance.

database_version

string

Version of the database engine.

instance_host

string

Host of the SQL database enum InstanceHost { UNDEFINED = 0; SELF_HOSTED = 1; CLOUD_SQL = 2; AMAZON_RDS = 3; AZURE_SQL = 4; } Host of the enclousing database instance.

StarEntryRequest

Request message for StarEntry.

Fields
name

string

Required. The name of the entry to mark as starred.

StarEntryResponse

This type has no fields.

Response message for StarEntry. Empty for now

StorageProperties

Details the properties of the underlying storage.

Fields
file_pattern[]

string

Patterns to identify a set of files for this fileset.

Examples of a valid file_pattern:

  • gs://bucket_name/dir/*: matches all files in the bucket_name/dir directory
  • gs://bucket_name/dir/**: matches all files in the bucket_name/dir and all subdirectories recursively
  • gs://bucket_name/file*: matches files prefixed by file in bucket_name
  • gs://bucket_name/??.txt: matches files with two characters followed by .txt in bucket_name
  • gs://bucket_name/[aeiou].txt: matches files that contain a single vowel character followed by .txt in bucket_name
  • gs://bucket_name/[a-m].txt: matches files that contain a, b, ... or m followed by .txt in bucket_name
  • gs://bucket_name/a/*/b: matches all files in bucket_name that match the a/*/b pattern, such as a/c/b, a/d/b
  • gs://another_bucket/a.txt: matches gs://another_bucket/a.txt
file_type

string

File type in MIME format, for example, text/plain.

SystemTimestamps

Timestamps associated with this resource in a particular system.

Fields
create_time

Timestamp

Creation timestamp of the resource within the given system.

update_time

Timestamp

Timestamp of the last modification of the resource or its metadata within a given system.

Note: Depending on the source system, not every modification updates this timestamp. For example, BigQuery timestamps every metadata modification but not data or permission changes.

expire_time

Timestamp

Output only. Expiration timestamp of the resource within the given system.

Currently only applicable to BigQuery resources.

TableSourceType

Table source type.

Enums
TABLE_SOURCE_TYPE_UNSPECIFIED Default unknown type.
BIGQUERY_VIEW Table view.
BIGQUERY_TABLE BigQuery native table.
BIGQUERY_MATERIALIZED_VIEW BigQuery materialized view.

TableSpec

Normal BigQuery table specification.

Fields
grouped_entry

string

Output only. If the table is date-sharded, that is, it matches the [prefix]YYYYMMDD name pattern, this field is the Data Catalog resource name of the date-sharded grouped entry. For example:

projects/{PROJECT_ID}/locations/{LOCATION}/entrygroups/{ENTRY_GROUP_ID}/entries/{ENTRY_ID}.

Otherwise, grouped_entry is empty.

Tag

Tags contain custom metadata and are attached to Data Catalog resources. Tags conform with the specification of their tag template.

See Data Catalog IAM for information on the permissions needed to create or view tags.

Fields
name

string

Identifier. The resource name of the tag in URL format where tag ID is a system-generated identifier.

Note: The tag itself might not be stored in the location specified in its name.

template

string

Required. The resource name of the tag template this tag uses. Example:

projects/{PROJECT_ID}/locations/{LOCATION}/tagTemplates/{TAG_TEMPLATE_ID}

This field cannot be modified after creation.

template_display_name

string

Output only. The display name of the tag template.

fields

map<string, TagField>

Required. Maps the ID of a tag field to its value and additional information about that field.

Tag template defines valid field IDs. A tag must have at least 1 field and at most 500 fields.

dataplex_transfer_status

DataplexTransferStatus

Output only. Denotes the transfer status of the Tag Template.

Union field scope. The scope within the parent resource that this tag is attached to. If not provided, the tag is attached to the parent resource itself.

Deleting the scope from the parent resource deletes all tags attached to that scope.

These fields cannot be updated after creation. scope can be only one of the following:

column

string

Resources like entry can have schemas associated with them. This scope allows you to attach tags to an individual column based on that schema.

To attach a tag to a nested column, separate column names with a dot (.). Example: column.nested_column.

TagField

Contains the value and additional information on a field within a Tag.

Fields
display_name

string

Output only. The display name of this field.

order

int32

Output only. The order of this field with respect to other fields in this tag. Can be set by Tag.

For example, a higher value can indicate a more important field. The value can be negative. Multiple fields can have the same order, and field orders within a tag don't have to be sequential.

Union field kind. Required. The value of this field. kind can be only one of the following:
double_value

double

The value of a tag field with a double type.

string_value

string

The value of a tag field with a string type.

The maximum length is 2000 UTF-8 characters.

bool_value

bool

The value of a tag field with a boolean type.

timestamp_value

Timestamp

The value of a tag field with a timestamp type.

enum_value

EnumValue

The value of a tag field with an enum type.

This value must be one of the allowed values listed in this enum.

richtext_value

string

The value of a tag field with a rich text type.

The maximum length is 10 MiB as this value holds HTML descriptions including encoded images. The maximum length of the text without images is 100 KiB.

EnumValue

An enum value.

Fields
display_name

string

The display name of the enum value.

TagTemplate

A tag template defines a tag that can have one or more typed fields.

The template is used to create tags that are attached to Google Cloud resources. Tag template roles provide permissions to create, edit, and use the template. For example, see the TagTemplate User role that includes a permission to use the tag template to tag resources.

Fields
name

string

Identifier. The resource name of the tag template in URL format.

Note: The tag template itself and its child resources might not be stored in the location specified in its name.

display_name

string

Display name for this template. Defaults to an empty string.

The name must contain only Unicode letters, numbers (0-9), underscores (_), dashes (-), spaces ( ), and can't start or end with spaces. The maximum length is 200 characters.

is_publicly_readable

bool

Indicates whether tags created with this template are public. Public tags do not require tag template access to appear in [ListTags][google.cloud.datacatalog.v1.ListTags] API response.

Additionally, you can search for a public tag by value with a simple search query in addition to using a tag: predicate.

fields

map<string, TagTemplateField>

Required. Map of tag template field IDs to the settings for the field. This map is an exhaustive list of the allowed fields. The map must contain at least one field and at most 500 fields.

The keys to this map are tag template field IDs. The IDs have the following limitations:

  • Can contain uppercase and lowercase letters, numbers (0-9) and underscores (_).
  • Must be at least 1 character and at most 64 characters long.
  • Must start with a letter or underscore.
dataplex_transfer_status

DataplexTransferStatus

Optional. Transfer status of the TagTemplate

DataplexTransferStatus

This enum describes TagTemplate transfer status to Dataplex service.

Enums
DATAPLEX_TRANSFER_STATUS_UNSPECIFIED Default value. TagTemplate and its tags are only visible and editable in DataCatalog.
MIGRATED

TagTemplate and its tags are auto-copied to Dataplex service. Visible in both services. Editable in DataCatalog, read-only in Dataplex. Deprecated: Individual TagTemplate migration is deprecated in favor of organization or project wide TagTemplate migration opt-in.

TRANSFERRED TagTemplate and its tags are auto-copied to Dataplex service. Visible in both services. Editable in Dataplex, read-only in DataCatalog.

TagTemplateField

The template for an individual field within a tag template.

Fields
name

string

Identifier. The resource name of the tag template field in URL format. Example:

projects/{PROJECT_ID}/locations/{LOCATION}/tagTemplates/{TAG_TEMPLATE}/fields/{FIELD}

Note: The tag template field itself might not be stored in the location specified in its name.

The name must contain only letters (a-z, A-Z), numbers (0-9), or underscores (_), and must start with a letter or underscore. The maximum length is 64 characters.

display_name

string

The display name for this field. Defaults to an empty string.

The name must contain only Unicode letters, numbers (0-9), underscores (_), dashes (-), spaces ( ), and can't start or end with spaces. The maximum length is 200 characters.

type

FieldType

Required. The type of value this tag field can contain.

is_required

bool

If true, this field is required. Defaults to false.

description

string

The description for this field. Defaults to an empty string.

order

int32

The order of this field with respect to other fields in this tag template.

For example, a higher value can indicate a more important field. The value can be negative. Multiple fields can have the same order and field orders within a tag don't have to be sequential.

TagTemplateMigration

Configuration related to the opt-in status for the migration of TagTemplates to Dataplex.

Enums
TAG_TEMPLATE_MIGRATION_UNSPECIFIED Default value. Migration of Tag Templates from Data Catalog to Dataplex is not performed.
TAG_TEMPLATE_MIGRATION_ENABLED Migration of Tag Templates from Data Catalog to Dataplex is enabled.
TAG_TEMPLATE_MIGRATION_DISABLED Migration of Tag Templates from Data Catalog to Dataplex is disabled.

TaggedEntry

Wrapper containing Entry and information about Tags that should and should not be attached to it.

Fields
present_tags[]

Tag

Optional. Tags that should be ingested into the Data Catalog. Caller should populate template name, column and fields.

absent_tags[]

Tag

Optional. Tags that should be deleted from the Data Catalog. Caller should populate template name and column only.

Union field entry. Required. Entry to be ingested. entry can be only one of the following:
v1_entry

Entry

Non-encrypted Data Catalog v1 Entry.

Taxonomy

A taxonomy is a collection of hierarchical policy tags that classify data along a common axis.

For example, a "data sensitivity" taxonomy might contain the following policy tags:

+ PII
  + Account number
  + Age
  + SSN
  + Zipcode
+ Financials
  + Revenue

A "data origin" taxonomy might contain the following policy tags:

+ User data
+ Employee data
+ Partner data
+ Public data
Fields
name

string

Identifier. Resource name of this taxonomy in URL format.

Note: Policy tag manager generates unique taxonomy IDs.

display_name

string

Required. User-defined name of this taxonomy.

The name can't start or end with spaces, must contain only Unicode letters, numbers, underscores, dashes, and spaces, and be at most 200 bytes long when encoded in UTF-8.

The taxonomy display name must be unique within an organization.

description

string

Optional. Description of this taxonomy. If not set, defaults to empty.

The description must contain only Unicode characters, tabs, newlines, carriage returns, and page breaks, and be at most 2000 bytes long when encoded in UTF-8.

policy_tag_count

int32

Output only. Number of policy tags in this taxonomy.

taxonomy_timestamps

SystemTimestamps

Output only. Creation and modification timestamps of this taxonomy.

activated_policy_types[]

PolicyType

Optional. A list of policy types that are activated for this taxonomy. If not set, defaults to an empty list.

service

Service

Output only. Identity of the service which owns the Taxonomy. This field is only populated when the taxonomy is created by a Google Cloud service. Currently only 'DATAPLEX' is supported.

PolicyType

Defines policy types where the policy tags can be used for.

Enums
POLICY_TYPE_UNSPECIFIED Unspecified policy type.
FINE_GRAINED_ACCESS_CONTROL Fine-grained access control policy that enables access control on tagged sub-resources.

Service

The source system of the Taxonomy.

Fields
name

ManagingSystem

The Google Cloud service name.

identity

string

The service agent for the service.

UnstarEntryRequest

Request message for UnstarEntry.

Fields
name

string

Required. The name of the entry to mark as not starred.

UnstarEntryResponse

This type has no fields.

Response message for UnstarEntry. Empty for now

UpdateEntryGroupRequest

Request message for UpdateEntryGroup.

Fields
entry_group

EntryGroup

Required. Updates for the entry group. The name field must be set.

update_mask

FieldMask

Names of fields whose values to overwrite on an entry group.

If this parameter is absent or empty, all modifiable fields are overwritten. If such fields are non-required and omitted in the request body, their values are emptied.

UpdateEntryRequest

Request message for UpdateEntry.

Fields
entry

Entry

Required. Updates for the entry. The name field must be set.

update_mask

FieldMask

Names of fields whose values to overwrite on an entry.

If this parameter is absent or empty, all modifiable fields are overwritten. If such fields are non-required and omitted in the request body, their values are emptied.

You can modify only the fields listed below.

For entries with type DATA_STREAM:

  • schema

For entries with type FILESET:

  • schema
  • display_name
  • description
  • gcs_fileset_spec
  • gcs_fileset_spec.file_patterns

For entries with user_specified_type:

  • schema
  • display_name
  • description
  • user_specified_type
  • user_specified_system
  • linked_resource
  • source_system_timestamps

UpdatePolicyTagRequest

Request message for UpdatePolicyTag.

Fields
policy_tag

PolicyTag

The policy tag to update. You can update only its description, display name, and parent policy tag fields.

Authorization requires the following IAM permission on the specified resource policyTag:

  • datacatalog.taxonomies.update
update_mask

FieldMask

Specifies the fields to update.

You can update only display name, description, and parent policy tag. If not set, defaults to all updatable fields. For more information, see FieldMask.

UpdateTagRequest

Request message for UpdateTag.

Fields
tag

Tag

Required. The updated tag. The "name" field must be set.

update_mask

FieldMask

Names of fields whose values to overwrite on a tag. Currently, a tag has the only modifiable field with the name fields.

In general, if this parameter is absent or empty, all modifiable fields are overwritten. If such fields are non-required and omitted in the request body, their values are emptied.

UpdateTagTemplateFieldRequest

Request message for UpdateTagTemplateField.

Fields
name

string

Required. The name of the tag template field.

tag_template_field

TagTemplateField

Required. The template to update.

update_mask

FieldMask

Optional. Names of fields whose values to overwrite on an individual field of a tag template. The following fields are modifiable:

  • display_name
  • type.enum_type
  • is_required

If this parameter is absent or empty, all modifiable fields are overwritten. If such fields are non-required and omitted in the request body, their values are emptied with one exception: when updating an enum type, the provided values are merged with the existing values. Therefore, enum values can only be added, existing enum values cannot be deleted or renamed.

Additionally, updating a template field from optional to required is not allowed.

UpdateTagTemplateRequest

Request message for UpdateTagTemplate.

Fields
tag_template

TagTemplate

Required. The template to update. The name field must be set.

update_mask

FieldMask

Names of fields whose values to overwrite on a tag template. Currently, only display_name and is_publicly_readable can be overwritten.

If this parameter is absent or empty, all modifiable fields are overwritten. If such fields are non-required and omitted in the request body, their values are emptied.

Note: Updating the is_publicly_readable field may require up to 12 hours to take effect in search results.

UpdateTaxonomyRequest

Request message for UpdateTaxonomy.

Fields
taxonomy

Taxonomy

The taxonomy to update. You can update only its description, display name, and activated policy types.

Authorization requires the following IAM permission on the specified resource taxonomy:

  • datacatalog.taxonomies.update
update_mask

FieldMask

Specifies fields to update. If not set, defaults to all fields you can update.

For more information, see FieldMask.

UsageSignal

The set of all usage signals that Data Catalog stores.

Note: Usually, these signals are updated daily. In rare cases, an update may fail but will be performed again on the next day.

Fields
update_time

Timestamp

The end timestamp of the duration of usage statistics.

usage_within_time_range

map<string, UsageStats>

Output only. BigQuery usage statistics over each of the predefined time ranges.

Supported time ranges are {"24H", "7D", "30D"}.

common_usage_within_time_range

map<string, CommonUsageStats>

Common usage statistics over each of the predefined time ranges.

Supported time ranges are {"24H", "7D", "30D", "Lifetime"}.

favorite_count

int64

Favorite count in the source system.

UsageStats

Detailed statistics on the entry's usage.

Usage statistics have the following limitations:

Fields
total_completions

float

The number of successful uses of the underlying entry.

total_failures

float

The number of failed attempts to use the underlying entry.

total_cancellations

float

The number of cancelled attempts to use the underlying entry.

total_execution_time_for_completions_millis

float

Total time spent only on successful uses, in milliseconds.

VertexDatasetSpec

Specification for vertex dataset resources.

Fields
data_item_count

int64

The number of DataItems in this Dataset. Only apply for non-structured Dataset.

data_type

DataType

Type of the dataset.

DataType

Type of data stored in the dataset.

Enums
DATA_TYPE_UNSPECIFIED Should not be used.
TABLE Structured data dataset.
IMAGE Image dataset which supports ImageClassification, ImageObjectDetection and ImageSegmentation problems.
TEXT Document dataset which supports TextClassification, TextExtraction and TextSentiment problems.
VIDEO Video dataset which supports VideoClassification, VideoObjectTracking and VideoActionRecognition problems.
CONVERSATION Conversation dataset which supports conversation problems.
TIME_SERIES TimeSeries dataset.
DOCUMENT Document dataset which supports DocumentAnnotation problems.
TEXT_TO_SPEECH TextToSpeech dataset which supports TextToSpeech problems.
TRANSLATION Translation dataset which supports Translation problems.
STORE_VISION Store Vision dataset which is used for HITL integration.
ENTERPRISE_KNOWLEDGE_GRAPH Enterprise Knowledge Graph dataset which is used for HITL labeling integration.
TEXT_PROMPT Text prompt dataset which supports Large Language Models.

VertexModelSourceInfo

Detail description of the source information of a Vertex model.

Fields
source_type

ModelSourceType

Type of the model source.

copy

bool

If this Model is copy of another Model. If true then source_type pertains to the original.

ModelSourceType

Source of the model.

Enums
MODEL_SOURCE_TYPE_UNSPECIFIED Should not be used.
AUTOML The Model is uploaded by automl training pipeline.
CUSTOM The Model is uploaded by user or custom training pipeline.
BQML The Model is registered and sync'ed from BigQuery ML.
MODEL_GARDEN The Model is saved or tuned from Model Garden.
GENIE The Model is saved or tuned from Genie.
CUSTOM_TEXT_EMBEDDING The Model is uploaded by text embedding finetuning pipeline.
MARKETPLACE The Model is saved or tuned from Marketplace.

VertexModelSpec

Specification for vertex model resources.

Fields
version_id

string

The version ID of the model.

version_aliases[]

string

User provided version aliases so that a model version can be referenced via alias

version_description

string

The description of this version.

vertex_model_source_info

VertexModelSourceInfo

Source of a Vertex model.

container_image_uri

string

URI of the Docker image to be used as the custom container for serving predictions.

ViewSpec

Table view specification.

Fields
view_query

string

Output only. The query that defines the table view.