- Resource: Entry
- EntryType
- IntegratedSystem
- GcsFilesetSpec
- GcsFileSpec
- BigQueryTableSpec
- TableSourceType
- ViewSpec
- TableSpec
- BigQueryDateShardedSpec
- DatabaseTableSpec
- TableType
- DataplexTableSpec
- DataplexExternalTable
- DataplexSpec
- PhysicalSchema
- AvroSchema
- ThriftSchema
- ProtobufSchema
- ParquetSchema
- OrcSchema
- CsvSchema
- DataSourceConnectionSpec
- BigQueryConnectionSpec
- ConnectionType
- CloudSqlBigQueryConnectionSpec
- DatabaseType
- RoutineSpec
- RoutineType
- Argument
- Mode
- BigQueryRoutineSpec
- FilesetSpec
- DataplexFilesetSpec
- BusinessContext
- EntryOverview
- Contacts
- Person
- Schema
- ColumnSchema
- UsageSignal
- UsageStats
- DataSource
- Service
- StorageProperties
- PersonalDetails
- Methods
Resource: Entry
Entry metadata. A Data Catalog entry represents another resource in Google Cloud Platform (such as a BigQuery dataset or a Pub/Sub topic) or outside of it. You can use the linkedResource
field in the entry resource to refer to the original resource ID of the source system.
An entry resource contains resource details, for example, its schema. Additionally, you can attach flexible metadata to an entry in the form of a Tag
.
JSON representation | |
---|---|
{ "name": string, "linkedResource": string, "fullyQualifiedName": string, "displayName": string, "description": string, "businessContext": { object ( |
Fields | ||
---|---|---|
name |
Output only. The resource name of an entry in URL format. Note: The entry itself and its child resources might not be stored in the location specified in its name. |
|
linkedResource |
The resource this metadata entry refers to. For Google Cloud Platform resources,
Output only when the entry is one of the types in the For entries with a The resource string must contain only letters (a-z, A-Z), numbers (0-9), underscores (_), periods (.), colons (:), slashes (/), dashes (-), and hashes (#). The maximum size is 200 bytes when encoded in UTF-8. |
|
fullyQualifiedName |
Fully qualified name (FQN) of the resource. Set automatically for entries representing resources from synced systems. Settable only during creation and read-only afterwards. Can be used for search and lookup of the entries. FQNs take two forms:
Example for a DPMS table:
|
|
displayName |
Display name of an entry. The name must contain only Unicode letters, numbers (0-9), underscores (_), dashes (-), spaces ( ), and can't start or end with spaces. The maximum size is 200 bytes when encoded in UTF-8. Default value is an empty string. |
|
description |
Entry description that can consist of several sentences or paragraphs that describe entry contents. The description must not contain Unicode non-characters as well as C0 and C1 control codes except tabs (HT), new lines (LF), carriage returns (CR), and page breaks (FF). The maximum size is 2000 bytes when encoded in UTF-8. Default value is an empty string. |
|
businessContext |
Business Context of the entry. Not supported for BigQuery datasets |
|
schema |
Schema of the entry. An entry might not have any schema attached to it. |
|
sourceSystemTimestamps |
Timestamps from the underlying resource, not from the Data Catalog entry. Output only when the entry has a type listed in the |
|
usageSignal |
Output only. Resource usage statistics. |
|
labels |
Cloud labels attached to the entry. In Data Catalog, you can create and modify labels attached only to custom entries. Synced entries have unmodifiable labels that come from the source system. |
|
dataSource |
Output only. Physical location of the entry. |
|
personalDetails |
Output only. Additional information related to the entry. Private to the current user. |
|
Union field entry_type . Required. Entry type. entry_type can be only one of the following: |
||
type |
The type of the entry. Only used for entries with types listed in the Currently, only |
|
userSpecifiedType |
Custom entry type that doesn't match any of the values allowed for input and listed in the When creating an entry, first check the type values in the enum. If there are no appropriate types for the new entry, provide a custom value, for example, The
|
|
Union field system . The source system of the entry. system can be only one of the following: |
||
integratedSystem |
Output only. Indicates the entry's source system that Data Catalog integrates with, such as BigQuery, Pub/Sub, or Dataproc Metastore. |
|
userSpecifiedSystem |
Indicates the entry's source system that Data Catalog doesn't automatically integrate with. The
|
|
Union field type_spec . Type specification. type_spec can be only one of the following: |
||
gcsFilesetSpec |
Specification that applies to a Cloud Storage fileset. Valid only for entries with the |
|
bigqueryTableSpec |
Specification that applies to a BigQuery table. Valid only for entries with the |
|
bigqueryDateShardedSpec |
Specification for a group of BigQuery tables with the For more information, see Introduction to partitioned tables. |
|
Union field When extending the API with new types and systems, use this field instead of the legacy |
||
databaseTableSpec |
Specification that applies to a table resource. Valid only for entries with the |
|
dataSourceConnectionSpec |
Specification that applies to a data source connection. Valid only for entries with the |
|
routineSpec |
Specification that applies to a user-defined function or procedure. Valid only for entries with the |
|
filesetSpec |
Specification that applies to a fileset resource. Valid only for entries with the |
EntryType
The enum field that lists all the types of entry resources in Data Catalog. For example, a BigQuery table entry has the TABLE
type.
Enums | |
---|---|
ENTRY_TYPE_UNSPECIFIED |
Default unknown type. |
TABLE |
Output only. The entry type that has a GoogleSQL schema, including logical views. |
MODEL |
Output only. The type of models. For more information, see Supported models in BigQuery ML. |
DATA_STREAM |
An entry type for streaming entries. For example, a Pub/Sub topic. |
FILESET |
An entry type for a set of files or objects. For example, a Cloud Storage fileset. |
CLUSTER |
A group of servers that work together. For example, a Kafka cluster. |
DATABASE |
A database. |
DATA_SOURCE_CONNECTION |
Output only. Connection to a data source. For example, a BigQuery connection. |
ROUTINE |
Output only. Routine, for example, a BigQuery routine. |
LAKE |
A Dataplex lake. |
ZONE |
A Dataplex zone. |
SERVICE |
A service, for example, a Dataproc Metastore service. |
IntegratedSystem
This enum lists all the systems that Data Catalog integrates with.
Enums | |
---|---|
INTEGRATED_SYSTEM_UNSPECIFIED |
Default unknown system. |
BIGQUERY |
BigQuery. |
CLOUD_PUBSUB |
Cloud Pub/Sub. |
DATAPROC_METASTORE |
Dataproc Metastore. |
DATAPLEX |
Dataplex. |
GcsFilesetSpec
Describes a Cloud Storage fileset entry.
JSON representation | |
---|---|
{
"filePatterns": [
string
],
"sampleGcsFileSpecs": [
{
object ( |
Fields | |
---|---|
filePatterns[] |
Required. Patterns to identify a set of files in Google Cloud Storage. For more information, see Wildcard Names. Note: Currently, bucket wildcards are not supported. Examples of valid
You can combine wildcards to match complex sets of files, for example:
|
sampleGcsFileSpecs[] |
Output only. Sample files contained in this fileset, not all files contained in this fileset are represented here. |
GcsFileSpec
Specification of a single file in Cloud Storage.
JSON representation | |
---|---|
{
"filePath": string,
"gcsTimestamps": {
object ( |
Fields | |
---|---|
filePath |
Required. Full file path. Example: |
gcsTimestamps |
Output only. Creation, modification, and expiration timestamps of a Cloud Storage file. |
sizeBytes |
Output only. File size in bytes. |
BigQueryTableSpec
Describes a BigQuery table.
JSON representation | |
---|---|
{ "tableSourceType": enum ( |
Fields | ||
---|---|---|
tableSourceType |
Output only. The table source type. |
|
Union field type_spec . Output only. type_spec can be only one of the following: |
||
viewSpec |
Table view specification. Populated only if the |
|
tableSpec |
Specification of a BigQuery table. Populated only if the |
TableSourceType
Table source type.
Enums | |
---|---|
TABLE_SOURCE_TYPE_UNSPECIFIED |
Default unknown type. |
BIGQUERY_VIEW |
Table view. |
BIGQUERY_TABLE |
BigQuery native table. |
BIGQUERY_MATERIALIZED_VIEW |
BigQuery materialized view. |
ViewSpec
Table view specification.
JSON representation | |
---|---|
{ "viewQuery": string } |
Fields | |
---|---|
viewQuery |
Output only. The query that defines the table view. |
TableSpec
Normal BigQuery table specification.
JSON representation | |
---|---|
{ "groupedEntry": string } |
Fields | |
---|---|
groupedEntry |
Output only. If the table is date-sharded, that is, it matches the
Otherwise, |
BigQueryDateShardedSpec
Specification for a group of BigQuery tables with the [prefix]YYYYMMDD
name pattern.
For more information, see Introduction to partitioned tables.
JSON representation | |
---|---|
{ "dataset": string, "tablePrefix": string, "shardCount": string, "latestShardResource": string } |
Fields | |
---|---|
dataset |
Output only. The Data Catalog resource name of the dataset entry the current table belongs to. For example:
|
tablePrefix |
Output only. The table name prefix of the shards. The name of any given shard is |
shardCount |
Output only. Total number of shards. |
latestShardResource |
Output only. BigQuery resource name of the latest shard. |
DatabaseTableSpec
Specification that applies to a table resource. Valid only for entries with the TABLE
type.
JSON representation | |
---|---|
{ "type": enum ( |
Fields | |
---|---|
type |
Type of this table. |
dataplexTable |
Fields specific to a Dataplex table and present only in the Dataplex table entries. |
TableType
Type of the table.
Enums | |
---|---|
TABLE_TYPE_UNSPECIFIED |
Default unknown table type. |
NATIVE |
Native table. |
EXTERNAL |
External table. |
DataplexTableSpec
Entry specification for a Dataplex table.
JSON representation | |
---|---|
{ "externalTables": [ { object ( |
Fields | |
---|---|
externalTables[] |
List of external tables registered by Dataplex in other systems based on the same underlying data. External tables allow to query this data in those systems. |
dataplexSpec |
Common Dataplex fields. |
userManaged |
Indicates if the table schema is managed by the user or not. |
DataplexExternalTable
External table registered by Dataplex. Dataplex publishes data discovered from an asset into multiple other systems (BigQuery, DPMS) in form of tables. We call them "external tables". External tables are also synced into the Data Catalog. This message contains pointers to those external tables (fully qualified name, resource name et cetera) within the Data Catalog.
JSON representation | |
---|---|
{
"system": enum ( |
Fields | |
---|---|
system |
Service in which the external table is registered. |
fullyQualifiedName |
Fully qualified name (FQN) of the external table. |
googleCloudResource |
Google Cloud resource name of the external table. |
dataCatalogEntry |
Name of the Data Catalog entry representing the external table. |
DataplexSpec
Common Dataplex fields.
JSON representation | |
---|---|
{
"asset": string,
"dataFormat": {
object ( |
Fields | |
---|---|
asset |
Fully qualified resource name of an asset in Dataplex, to which the underlying data source (Cloud Storage bucket or BigQuery dataset) of the entity is attached. |
dataFormat |
Format of the data. |
compressionFormat |
Compression format of the data, e.g., zip, gzip etc. |
projectId |
Project ID of the underlying Cloud Storage or BigQuery data. Note that this may not be the same project as the correspondingly Dataplex lake / zone / asset. |
PhysicalSchema
Native schema used by a resource represented as an entry. Used by query engines for deserializing and parsing source data.
JSON representation | |
---|---|
{ // Union field |
Fields | ||
---|---|---|
Union field
|
||
avro |
Schema in Avro JSON format. |
|
thrift |
Schema in Thrift format. |
|
protobuf |
Schema in protocol buffer format. |
|
parquet |
Marks a Parquet-encoded data source. |
|
orc |
Marks an ORC-encoded data source. |
|
csv |
Marks a CSV-encoded data source. |
AvroSchema
Schema in Avro JSON format.
JSON representation | |
---|---|
{ "text": string } |
Fields | |
---|---|
text |
JSON source of the Avro schema. |
ThriftSchema
Schema in Thrift format.
JSON representation | |
---|---|
{ "text": string } |
Fields | |
---|---|
text |
Thrift IDL source of the schema. |
ProtobufSchema
Schema in protocol buffer format.
JSON representation | |
---|---|
{ "text": string } |
Fields | |
---|---|
text |
Protocol buffer source of the schema. |
ParquetSchema
Marks a Parquet-encoded data source.
OrcSchema
Marks an ORC-encoded data source.
CsvSchema
Marks a CSV-encoded data source.
DataSourceConnectionSpec
Specification that applies to a data source connection. Valid only for entries with the DATA_SOURCE_CONNECTION
type.
JSON representation | |
---|---|
{
"bigqueryConnectionSpec": {
object ( |
Fields | |
---|---|
bigqueryConnectionSpec |
Fields specific to BigQuery connections. |
BigQueryConnectionSpec
Specification for the BigQuery connection.
JSON representation | |
---|---|
{ "connectionType": enum ( |
Fields | |
---|---|
connectionType |
The type of the BigQuery connection. |
hasCredential |
True if there are credentials attached to the BigQuery connection; false otherwise. |
cloudSql |
Specification for the BigQuery connection to a Cloud SQL instance. |
ConnectionType
The type of the BigQuery connection.
Enums | |
---|---|
CONNECTION_TYPE_UNSPECIFIED |
Unspecified type. |
CLOUD_SQL |
Cloud SQL connection. |
CloudSqlBigQueryConnectionSpec
Specification for the BigQuery connection to a Cloud SQL instance.
JSON representation | |
---|---|
{
"instanceId": string,
"database": string,
"type": enum ( |
Fields | |
---|---|
instanceId |
Cloud SQL instance ID in the format of |
database |
Database name. |
type |
Type of the Cloud SQL database. |
DatabaseType
Supported Cloud SQL database types.
Enums | |
---|---|
DATABASE_TYPE_UNSPECIFIED |
Unspecified database type. |
POSTGRES |
Cloud SQL for PostgreSQL. |
MYSQL |
Cloud SQL for MySQL. |
RoutineSpec
Specification that applies to a routine. Valid only for entries with the ROUTINE
type.
JSON representation | |
---|---|
{ "routineType": enum ( |
Fields | |
---|---|
routineType |
The type of the routine. |
language |
The language the routine is written in. The exact value depends on the source system. For BigQuery routines, possible values are:
|
routineArguments[] |
Arguments of the routine. |
returnType |
Return type of the argument. The exact value depends on the source system and the language. |
definitionBody |
The body of the routine. |
bigqueryRoutineSpec |
Fields specific for BigQuery routines. |
RoutineType
The fine-grained type of the routine.
Enums | |
---|---|
ROUTINE_TYPE_UNSPECIFIED |
Unspecified type. |
SCALAR_FUNCTION |
Non-builtin permanent scalar function. |
PROCEDURE |
Stored procedure. |
Argument
Input or output argument of a function or stored procedure.
JSON representation | |
---|---|
{
"name": string,
"mode": enum ( |
Fields | |
---|---|
name |
The name of the argument. A return argument of a function might not have a name. |
mode |
Specifies whether the argument is input or output. |
type |
Type of the argument. The exact value depends on the source system and the language. |
Mode
The input or output mode of the argument.
Enums | |
---|---|
MODE_UNSPECIFIED |
Unspecified mode. |
IN |
The argument is input-only. |
OUT |
The argument is output-only. |
INOUT |
The argument is both an input and an output. |
BigQueryRoutineSpec
Fields specific for BigQuery routines.
JSON representation | |
---|---|
{ "importedLibraries": [ string ] } |
Fields | |
---|---|
importedLibraries[] |
Paths of the imported libraries. |
FilesetSpec
Specification that applies to a fileset. Valid only for entries with the 'FILESET' type.
JSON representation | |
---|---|
{
"dataplexFileset": {
object ( |
Fields | |
---|---|
dataplexFileset |
Fields specific to a Dataplex fileset and present only in the Dataplex fileset entries. |
DataplexFilesetSpec
Entry specyfication for a Dataplex fileset.
JSON representation | |
---|---|
{
"dataplexSpec": {
object ( |
Fields | |
---|---|
dataplexSpec |
Common Dataplex fields. |
BusinessContext
Business Context of the entry.
JSON representation | |
---|---|
{ "entryOverview": { object ( |
Fields | |
---|---|
entryOverview |
Entry overview fields for rich text descriptions of entries. |
contacts |
Contact people for the entry. |
EntryOverview
Entry overview fields for rich text descriptions of entries.
JSON representation | |
---|---|
{ "overview": string } |
Fields | |
---|---|
overview |
Entry overview with support for rich text. The overview must only contain Unicode characters, and should be formatted using HTML. The maximum length is 10 MiB as this value holds HTML descriptions including encoded images. The maximum length of the text without images is 100 KiB. |
Contacts
Contact people for the entry.
JSON representation | |
---|---|
{
"people": [
{
object ( |
Fields | |
---|---|
people[] |
The list of contact people for the entry. |
Person
A contact person for the entry.
JSON representation | |
---|---|
{ "designation": string, "email": string } |
Fields | |
---|---|
designation |
Designation of the person, for example, Data Steward. |
email |
Email of the person in the format of |
Schema
Represents a schema, for example, a BigQuery, GoogleSQL, or Avro schema.
JSON representation | |
---|---|
{
"columns": [
{
object ( |
Fields | |
---|---|
columns[] |
The unified GoogleSQL-like schema of columns. The overall maximum number of columns and nested columns is 10,000. The maximum nested depth is 15 levels. |
ColumnSchema
A column within a schema. Columns can be nested inside other columns.
JSON representation | |
---|---|
{
"column": string,
"type": string,
"description": string,
"mode": string,
"subcolumns": [
{
object ( |
Fields | |
---|---|
column |
Required. Name of the column. Must be a UTF-8 string without dots (.). The maximum size is 64 bytes. |
type |
Required. Type of the column. Must be a UTF-8 string with the maximum size of 128 bytes. |
description |
Optional. Description of the column. Default value is an empty string. The description must be a UTF-8 string with the maximum size of 2000 bytes. |
mode |
Optional. A column's mode indicates whether values in this column are required, nullable, or repeated. Only |
subcolumns[] |
Optional. Schema of sub-columns. A column can have zero or more sub-columns. |
UsageSignal
The set of all usage signals that Data Catalog stores.
Note: Usually, these signals are updated daily. In rare cases, an update may fail but will be performed again on the next day.
JSON representation | |
---|---|
{
"updateTime": string,
"usageWithinTimeRange": {
string: {
object ( |
Fields | |
---|---|
updateTime |
The end timestamp of the duration of usage statistics. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
usageWithinTimeRange |
Usage statistics over each of the predefined time ranges. Supported time ranges are |
UsageStats
Detailed statistics on the entry's usage.
Usage statistics have the following limitations:
- Only BigQuery tables have them.
- They only include BigQuery query jobs.
- They might be underestimated because wildcard table references are not yet counted. For more information, see Querying multiple tables using a wildcard table
JSON representation | |
---|---|
{ "totalCompletions": number, "totalFailures": number, "totalCancellations": number, "totalExecutionTimeForCompletionsMillis": number } |
Fields | |
---|---|
totalCompletions |
The number of successful uses of the underlying entry. |
totalFailures |
The number of failed attempts to use the underlying entry. |
totalCancellations |
The number of cancelled attempts to use the underlying entry. |
totalExecutionTimeForCompletionsMillis |
Total time spent only on successful uses, in milliseconds. |
DataSource
Physical location of an entry.
JSON representation | |
---|---|
{ "service": enum ( |
Fields | |
---|---|
service |
Service that physically stores the data. |
resource |
Full name of a resource as defined by the service. For example:
|
sourceEntry |
Output only. Data Catalog entry name, if applicable. |
storageProperties |
Detailed properties of the underlying storage. |
Service
Name of a service that stores the data.
Enums | |
---|---|
SERVICE_UNSPECIFIED |
Default unknown service. |
CLOUD_STORAGE |
Google Cloud Storage service. |
BIGQUERY |
BigQuery service. |
StorageProperties
Details the properties of the underlying storage.
JSON representation | |
---|---|
{ "filePattern": [ string ], "fileType": string } |
Fields | |
---|---|
filePattern[] |
Patterns to identify a set of files for this fileset. Examples of a valid
|
fileType |
File type in MIME format, for example, |
PersonalDetails
Entry metadata relevant only to the user and private to them.
JSON representation | |
---|---|
{ "starred": boolean, "starTime": string } |
Fields | |
---|---|
starred |
True if the entry is starred by the user; false otherwise. |
starTime |
Set if the entry is starred; unset otherwise. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
Methods |
|
---|---|
|
Creates an entry. |
|
Deletes an existing entry. |
|
Gets an entry. |
|
Gets the access control policy for a resource. |
|
Lists entries. |
|
Modifies contacts, part of the business context of an Entry . |
|
Modifies entry overview, part of the business context of an Entry . |
|
Updates an existing entry. |
|
Marks an Entry as starred by the current user. |
|
Gets your permissions on a resource. |
|
Marks an Entry as NOT starred by the current user. |