This document describes the syntax for Data Catalog search queries.
In its simplest form, a search query comprises a single predicate. Such a
predicate can match several pieces of metadata:
A substring of a name, display name, and description of a data asset
Exact type of a data asset
A substring of a column name (or nested column
name) in the schema of a data asset
A substring of a project ID
For example, the predicate foo matches the following entities:
Data asset with the foo.bar name
Data asset with the Foo Bar display name
Data asset with the description This is the foo script.
Data asset with the exact foo type
Column foo_bar in the schema of a data asset
Nested column foo_bar in the schema of a data asset
You can qualify a predicate by prefixing it with a key that restricts the
matching to a specific piece of metadata.
An equal sign (=) restricts the search to an exact match. A colon (:)
after the key matches the predicate to either a substring or token within
the value in search results.
name=foo selects entities with the exact foo name.
name:foo selects entities with names that contain the foo substring:
foo1 and barfoo.
description:foo selects entities with the foo token in the description:
bar and foo.
Data Catalog supports the following qualifiers:
Matches x as a substring of the data asset ID.
Match x as a substring of the data asset display name.
Matches x as a substring of the column name (or nested column name) in the schema of the data asset. Currently, you can search for a nested column by its path using the AND logical operator. For example, column:(foo bar) matches a nested column with the foo.bar path.
Matches x as a token in the data asset description.
Matches BigQuery data assets that have a label (with some value) and the label key has bar as a substring.
Matches x as a token in the value of a label bar attached to a BigQuery data asset.
Matches data assets of a specific object type or subtype. Subtypes can be added with the format <type>.<sub-type>. Types and subtypes include:
type=table matches all tables.
type=dataset matches all BigQuery datasets.
type=table.view or type=view matches all views.
type=tag_template matches all tag templates.
type=entry_group matches all entry groups.
type=data_stream matches all Pub/Sub topics.
Matches data assets within Cloud projects that match bar as a substring in the ID.
Matches x as a substring of the hierarchical path of a BigQuery data asset. The path has the format <project_id>.<dataset_name>. For example, parent:foo.bar matches all tables and views of a dataset with the path project-foo.bar-dataset.
Matches data assets within a Cloud organization with the exact ID value of number.
Matches all data assets from a specified system. Systems include:
system=bigquery matches all data assets from BigQuery.
system=cloud_pubsub matches all data assets from Pub/Sub.
system=dataproc_metastore matches all data assets from Dataproc Metastore.
system=data_catalog matches all data assets created in Data Catalog.
Matches data assets where x matches any substring in <tag_template_project_id>.<tag_template_id>.<tag_field_id>. Examples:
tag:data_owner matches data assets that have the data_owner tag.
tag:data_gov_template matches data assets that have been tagged with the data_gov_template tag template.
tag:mycloudproject.data_gov_template matches data assets tagged with the data_gov_template template in the mycloudproject project.
First, matches the key to any substring of the tag field ID, tag template ID, or Cloud project ID of a tag template. Then, matches val to the tag value of the key depending on the tag field type. The type-dependent <operator> sets permitted for tag values are:
string: ":" Note: The colon in this string search denotes an exact token match, not a substring.
boolean and enum: "="
double: "=", "<", ">", "<=", ">="
timestamp: ":", "=", "<", ">", "<=", ">="
string: tag:data_owner:@mail.com matches data assets that have @mail.com values.
boolean: tag:data_gov_template.hasPII=true matches hasPII boolean tags in the data_gov_template that are true.
double: tag:datascore=9 matches data assets with datascore double tags that have value 9.
timestamp: tag:expiredDate:2019-01-01 matches data assets that have an expiredDate tag of 2019-01-01.
timestamp: tag:expiredDate<2019-02 matches data assets that have an expiredDate tag prior to 2019-02-01T00:00:00.
Finds data assets that were created within, prior to, or after a given date or time. Examples:
createtime:2019-01-01 matches data assets created on 2019-01-01.
createtime<2019-02 matches data assets created prior to 2019-02-01T00:00:00.
createtime>2019-02 matches data assets created after 2019-02-01T00:00:00.
Finds data assets that were updated within, prior to, or after a given date or time. Examples:
updatetime:2019-01-01 matches data assets updated on 2019-01-01.
updatetime<2019-02 matches data assets updated prior to 2019-02-01T00:00:00.
updatetime>2019-02 matches data assets updated after 2019-02-01T00:00:00.
Match x as a substring of the policy tag display name. Finds all assets using matching policy tag or its descendants.
A query may be comprised of several predicates with logical operators. If you don't specify an operator, logical AND is implied. For example, foo bar returns entities that match both predicate foo and predicate bar.
Logical AND and logical OR are supported, for example, foo OR bar.
You can negate a predicate with a - or NOT prefix. For example, -name:foo returns
all entities with names that do not match the predicate foo.
An abbreviated search syntax is also available, using | for OR operators and , for AND operators.
For example, to search for entries inside one of many projects using the OR operator, you can use:
projectid:pid1 OR projectid:pid2 OR projectid:pid3 OR projectid:pid4