This document shows you how to export an asset snapshot from your organization, folder, or project to a BigQuery table, and then run data analysis on your inventory. BigQuery provides a SQL-like experience for users to analyze data and produce meaningful insights without the use of custom scripts.
Before you begin
Enable the Cloud Asset Inventory API in the project you're running Cloud Asset Inventory commands from.
Make sure your account has the correct role to call the Cloud Asset Inventory API. For individual permissions for each call type, see Permissions.
Create a BigQuery dataset to export to, if you don't have one already.
Limitations
When exporting BigQuery table data, not all fields are supported by Cloud Asset Inventory.
Frequently changing asset fields such as
numBytes,numLongTermBytes,numPhysicalBytes, andnumRowsmight be exported with anullvalue.Exporting to BigQuery clustered tables isn't supported.
BigQuery tables encrypted with custom Cloud Key Management Service (Cloud KMS) keys aren't supported.
Appending the export output to an existing table isn't supported unless you export to a partitioned table. The destination table must be empty or you must overwrite it. To overwrite it, use the
--output-bigquery-forceflag with the gcloud CLI, or use"force": truewith the REST API.Google Kubernetes Engine (GKE) resource types, except for
container.googleapis.com/Clusterandcontainer.googleapis.com/NodePool, are not supported when exporting to separate tables for each resource type.Cloud Asset Inventory rejects export requests if a previous request to the same destination started less than 15 minutes ago and is still running. However, if an export has taken longer than 15 minutes to complete, it is marked as done and new export requests to the same destination are permitted.
The
ACCESS_POLICYcontent type can only be exported at the organization level.If the table you're exporting to already exists and is in the process of being exported to, a
400error is returned.
BigQuery schemas used for export
Every BigQuery table is defined by a schema that describes the column names, data types, and other information. Setting the content type for an export determines the schema for your table:
Resource or unspecified: When you set the content type to
RESOURCEor don't specify it, and you set theper-asset-typeflag tofalseor don't use it, you create a BigQuery table that has the following schema.Resource schema
[ { "name": "name", "type": "STRING", "mode": "NULLABLE" }, { "name": "asset_type", "type": "STRING", "mode": "NULLABLE" }, { "name": "resource", "type": "RECORD", "mode": "NULLABLE", "fields": [ { "name": "version", "type": "STRING", "mode": "NULLABLE" }, { "name": "discovery_document_uri", "type": "STRING", "mode": "NULLABLE" }, { "name": "discovery_name", "type": "STRING", "mode": "NULLABLE" }, { "name": "resource_url", "type": "STRING", "mode": "NULLABLE" }, { "name": "parent", "type": "STRING", "mode": "NULLABLE" }, { "name": "data", "type": "STRING", "mode": "NULLABLE" }, { "name": "location", "type": "STRING", "mode": "NULLABLE" } ] }, { "name": "ancestors", "type": "STRING", "mode": "REPEATED" }, { "name": "update_time", "type": "TIMESTAMP", "mode": "NULLABLE" } ]
The
resource.datacolumn is the resource metadata represented as a JSON string.When you set the content type to
RESOURCEor don't set the content type, and set theper-asset-typeflag totrue, you create separate tables for each asset type. The schema of each table includes RECORD-type columns mapped to the nested fields in theResource.datafield of that asset type (up to the 15 nested levels that BigQuery supports). For example tables, see export-assets-examples in the Google Cloud console.IAM policy: When you set the content type to
IAM_POLICYin the REST API oriam-policyin the gcloud CLI, you create a BigQuery table that has the following schema.IAM policy schema
[ { "name": "name", "mode": "NULLABLE", "type": "STRING" }, { "name": "asset_type", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "version", "mode": "NULLABLE", "type": "INTEGER" }, { "fields": [ { "name": "role", "mode": "NULLABLE", "type": "STRING" }, { "name": "members", "mode": "REPEATED", "type": "STRING" }, { "fields": [ { "name": "expression", "mode": "NULLABLE", "type": "STRING" }, { "name": "title", "mode": "NULLABLE", "type": "STRING" }, { "name": "description", "mode": "NULLABLE", "type": "STRING" }, { "name": "location", "mode": "NULLABLE", "type": "STRING" } ], "name": "condition", "mode": "NULLABLE", "type": "RECORD" } ], "name": "bindings", "mode": "REPEATED", "type": "RECORD" }, { "fields": [ { "name": "service", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "log_type", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "exempted_members", "mode": "REPEATED", "type": "STRING" } ], "name": "audit_log_configs", "mode": "REPEATED", "type": "RECORD" } ], "name": "audit_configs", "mode": "REPEATED", "type": "RECORD" }, { "name": "etag", "mode": "NULLABLE", "type": "STRING" } ], "name": "iam_policy", "mode": "NULLABLE", "type": "RECORD" }, { "name": "ancestors", "mode": "REPEATED", "type": "STRING" }, { "name": "update_time", "mode": "NULLABLE", "type": "TIMESTAMP" } ]
Organization policy: When you set the content type to
ORG_POLICYin the REST API ororg-policyin the gcloud CLI, you create a BigQuery table that has the following schema.Organization policy schema
[ { "name": "name", "mode": "NULLABLE", "type": "STRING" }, { "name": "asset_type", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "version", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "constraint", "mode": "NULLABLE", "type": "STRING" }, { "name": "etag", "mode": "NULLABLE", "type": "STRING" }, { "name": "update_time", "mode": "NULLABLE", "type": "TIMESTAMP" }, { "fields": [ { "name": "allowed_values", "mode": "REPEATED", "type": "STRING" }, { "name": "denied_values", "mode": "REPEATED", "type": "STRING" }, { "name": "all_values", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "suggested_value", "mode": "NULLABLE", "type": "STRING" }, { "name": "inherit_from_parent", "mode": "NULLABLE", "type": "BOOLEAN" } ], "name": "list_policy", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "enforced", "mode": "NULLABLE", "type": "BOOLEAN" } ], "name": "boolean_policy", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "_present", "mode": "NULLABLE", "type": "BOOLEAN" } ], "name": "restore_default", "mode": "NULLABLE", "type": "RECORD" } ], "name": "org_policy", "mode": "REPEATED", "type": "RECORD" }, { "name": "ancestors", "mode": "REPEATED", "type": "STRING" }, { "name": "update_time", "mode": "NULLABLE", "type": "TIMESTAMP" } ]
VPCSC policy: When you set content type to
ACCESS_POLICYin the REST API oraccess-policyin the gcloud CLI, you create a BigQuery table that has the following schema.VPCSC policy schema
[ { "name": "name", "mode": "NULLABLE", "type": "STRING" }, { "name": "asset_type", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "name", "mode": "NULLABLE", "type": "STRING" }, { "name": "parent", "mode": "NULLABLE", "type": "STRING" }, { "name": "title", "mode": "NULLABLE", "type": "STRING" }, { "name": "scopes", "mode": "REPEATED", "type": "STRING" }, { "name": "etag", "mode": "NULLABLE", "type": "STRING" } ], "name": "access_policy", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "name", "mode": "NULLABLE", "type": "STRING" }, { "name": "title", "mode": "NULLABLE", "type": "STRING" }, { "name": "description", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "fields": [ { "name": "ip_subnetworks", "mode": "REPEATED", "type": "STRING" }, { "fields": [ { "name": "require_screenlock", "mode": "NULLABLE", "type": "BOOLEAN" }, { "name": "allowed_encryption_statuses", "mode": "REPEATED", "type": "INTEGER" }, { "fields": [ { "name": "os_type", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "minimum_version", "mode": "NULLABLE", "type": "STRING" }, { "name": "require_verified_chrome_os", "mode": "NULLABLE", "type": "BOOLEAN" } ], "name": "os_constraints", "mode": "REPEATED", "type": "RECORD" }, { "name": "allowed_device_management_levels", "mode": "REPEATED", "type": "INTEGER" }, { "name": "require_admin_approval", "mode": "NULLABLE", "type": "BOOLEAN" }, { "name": "require_corp_owned", "mode": "NULLABLE", "type": "BOOLEAN" } ], "name": "device_policy", "mode": "NULLABLE", "type": "RECORD" }, { "name": "required_access_levels", "mode": "REPEATED", "type": "STRING" }, { "name": "negate", "mode": "NULLABLE", "type": "BOOLEAN" }, { "name": "members", "mode": "REPEATED", "type": "STRING" }, { "name": "regions", "mode": "REPEATED", "type": "STRING" } ], "name": "conditions", "mode": "REPEATED", "type": "RECORD" }, { "name": "combining_function", "mode": "NULLABLE", "type": "INTEGER" } ], "name": "basic", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "fields": [ { "name": "expression", "mode": "NULLABLE", "type": "STRING" }, { "name": "title", "mode": "NULLABLE", "type": "STRING" }, { "name": "description", "mode": "NULLABLE", "type": "STRING" }, { "name": "location", "mode": "NULLABLE", "type": "STRING" } ], "name": "expr", "mode": "NULLABLE", "type": "RECORD" } ], "name": "custom", "mode": "NULLABLE", "type": "RECORD" } ], "name": "access_level", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "name", "mode": "NULLABLE", "type": "STRING" }, { "name": "title", "mode": "NULLABLE", "type": "STRING" }, { "name": "description", "mode": "NULLABLE", "type": "STRING" }, { "name": "perimeter_type", "mode": "NULLABLE", "type": "INTEGER" }, { "fields": [ { "name": "resources", "mode": "REPEATED", "type": "STRING" }, { "name": "access_levels", "mode": "REPEATED", "type": "STRING" }, { "name": "restricted_services", "mode": "REPEATED", "type": "STRING" }, { "fields": [ { "name": "enable_restriction", "mode": "NULLABLE", "type": "BOOLEAN" }, { "name": "allowed_services", "mode": "REPEATED", "type": "STRING" } ], "name": "vpc_accessible_services", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "fields": [ { "fields": [ { "name": "access_level", "mode": "NULLABLE", "type": "STRING" }, { "name": "resource", "mode": "NULLABLE", "type": "STRING" } ], "name": "sources", "mode": "REPEATED", "type": "RECORD" }, { "name": "identities", "mode": "REPEATED", "type": "STRING" }, { "name": "identity_type", "mode": "NULLABLE", "type": "INTEGER" } ], "name": "ingress_from", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "fields": [ { "name": "service_name", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "method", "mode": "NULLABLE", "type": "STRING" }, { "name": "permission", "mode": "NULLABLE", "type": "STRING" } ], "name": "method_selectors", "mode": "REPEATED", "type": "RECORD" } ], "name": "operations", "mode": "REPEATED", "type": "RECORD" }, { "name": "resources", "mode": "REPEATED", "type": "STRING" } ], "name": "ingress_to", "mode": "NULLABLE", "type": "RECORD" } ], "name": "ingress_policies", "mode": "REPEATED", "type": "RECORD" }, { "fields": [ { "fields": [ { "name": "identities", "mode": "REPEATED", "type": "STRING" }, { "name": "identity_type", "mode": "NULLABLE", "type": "INTEGER" } ], "name": "egress_from", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "resources", "mode": "REPEATED", "type": "STRING" }, { "fields": [ { "name": "service_name", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "method", "mode": "NULLABLE", "type": "STRING" }, { "name": "permission", "mode": "NULLABLE", "type": "STRING" } ], "name": "method_selectors", "mode": "REPEATED", "type": "RECORD" } ], "name": "operations", "mode": "REPEATED", "type": "RECORD" }, { "name": "external_resources", "mode": "REPEATED", "type": "STRING" } ], "name": "egress_to", "mode": "NULLABLE", "type": "RECORD" } ], "name": "egress_policies", "mode": "REPEATED", "type": "RECORD" } ], "name": "status", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "resources", "mode": "REPEATED", "type": "STRING" }, { "name": "access_levels", "mode": "REPEATED", "type": "STRING" }, { "name": "restricted_services", "mode": "REPEATED", "type": "STRING" }, { "fields": [ { "name": "enable_restriction", "mode": "NULLABLE", "type": "BOOLEAN" }, { "name": "allowed_services", "mode": "REPEATED", "type": "STRING" } ], "name": "vpc_accessible_services", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "fields": [ { "fields": [ { "name": "access_level", "mode": "NULLABLE", "type": "STRING" }, { "name": "resource", "mode": "NULLABLE", "type": "STRING" } ], "name": "sources", "mode": "REPEATED", "type": "RECORD" }, { "name": "identities", "mode": "REPEATED", "type": "STRING" }, { "name": "identity_type", "mode": "NULLABLE", "type": "INTEGER" } ], "name": "ingress_from", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "fields": [ { "name": "service_name", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "method", "mode": "NULLABLE", "type": "STRING" }, { "name": "permission", "mode": "NULLABLE", "type": "STRING" } ], "name": "method_selectors", "mode": "REPEATED", "type": "RECORD" } ], "name": "operations", "mode": "REPEATED", "type": "RECORD" }, { "name": "resources", "mode": "REPEATED", "type": "STRING" } ], "name": "ingress_to", "mode": "NULLABLE", "type": "RECORD" } ], "name": "ingress_policies", "mode": "REPEATED", "type": "RECORD" }, { "fields": [ { "fields": [ { "name": "identities", "mode": "REPEATED", "type": "STRING" }, { "name": "identity_type", "mode": "NULLABLE", "type": "INTEGER" } ], "name": "egress_from", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "resources", "mode": "REPEATED", "type": "STRING" }, { "fields": [ { "name": "service_name", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "method", "mode": "NULLABLE", "type": "STRING" }, { "name": "permission", "mode": "NULLABLE", "type": "STRING" } ], "name": "method_selectors", "mode": "REPEATED", "type": "RECORD" } ], "name": "operations", "mode": "REPEATED", "type": "RECORD" }, { "name": "external_resources", "mode": "REPEATED", "type": "STRING" } ], "name": "egress_to", "mode": "NULLABLE", "type": "RECORD" } ], "name": "egress_policies", "mode": "REPEATED", "type": "RECORD" } ], "name": "spec", "mode": "NULLABLE", "type": "RECORD" }, { "name": "use_explicit_dry_run_spec", "mode": "NULLABLE", "type": "BOOLEAN" } ], "name": "service_perimeter", "mode": "NULLABLE", "type": "RECORD" }, { "name": "ancestors", "mode": "REPEATED", "type": "STRING" }, { "name": "update_time", "mode": "NULLABLE", "type": "TIMESTAMP" } ]
OSConfig instance inventory: When you set content type to
OS_INVENTORYin the REST API oros-inventoryin the gcloud CLI, you create a BigQuery table that has the following schema.OS inventory schema
[ { "name": "name", "mode": "NULLABLE", "type": "STRING" }, { "name": "asset_type", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "name", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "hostname", "mode": "NULLABLE", "type": "STRING" }, { "name": "long_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "short_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "kernel_version", "mode": "NULLABLE", "type": "STRING" }, { "name": "kernel_release", "mode": "NULLABLE", "type": "STRING" }, { "name": "osconfig_agent_version", "mode": "NULLABLE", "type": "STRING" } ], "name": "os_info", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "key", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "id", "mode": "NULLABLE", "type": "STRING" }, { "name": "origin_type", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "create_time", "mode": "NULLABLE", "type": "TIMESTAMP" }, { "name": "update_time", "mode": "NULLABLE", "type": "TIMESTAMP" }, { "name": "type", "mode": "NULLABLE", "type": "INTEGER" }, { "fields": [ { "fields": [ { "name": "package_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" } ], "name": "yum_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "package_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" } ], "name": "apt_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "package_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" } ], "name": "zypper_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "package_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" } ], "name": "googet_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "patch_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "category", "mode": "NULLABLE", "type": "STRING" }, { "name": "severity", "mode": "NULLABLE", "type": "STRING" }, { "name": "summary", "mode": "NULLABLE", "type": "STRING" } ], "name": "zypper_patch", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "title", "mode": "NULLABLE", "type": "STRING" }, { "name": "description", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "id", "mode": "NULLABLE", "type": "STRING" }, { "name": "name", "mode": "NULLABLE", "type": "STRING" } ], "name": "categories", "mode": "REPEATED", "type": "RECORD" }, { "name": "kb_article_ids", "mode": "REPEATED", "type": "STRING" }, { "name": "support_url", "mode": "NULLABLE", "type": "STRING" }, { "name": "more_info_urls", "mode": "REPEATED", "type": "STRING" }, { "name": "update_id", "mode": "NULLABLE", "type": "STRING" }, { "name": "revision_number", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "last_deployment_change_time", "mode": "NULLABLE", "type": "TIMESTAMP" } ], "name": "wua_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "caption", "mode": "NULLABLE", "type": "STRING" }, { "name": "description", "mode": "NULLABLE", "type": "STRING" }, { "name": "hot_fix_id", "mode": "NULLABLE", "type": "STRING" }, { "name": "install_time", "mode": "NULLABLE", "type": "TIMESTAMP" } ], "name": "qfe_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "package_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" } ], "name": "cos_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "display_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "display_version", "mode": "NULLABLE", "type": "STRING" }, { "name": "publisher", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "year", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "month", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "day", "mode": "NULLABLE", "type": "INTEGER" } ], "name": "install_date", "mode": "NULLABLE", "type": "RECORD" }, { "name": "help_link", "mode": "NULLABLE", "type": "STRING" } ], "name": "windows_application", "mode": "NULLABLE", "type": "RECORD" } ], "name": "installed_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "fields": [ { "name": "package_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" } ], "name": "yum_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "package_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" } ], "name": "apt_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "package_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" } ], "name": "zypper_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "package_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" } ], "name": "googet_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "patch_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "category", "mode": "NULLABLE", "type": "STRING" }, { "name": "severity", "mode": "NULLABLE", "type": "STRING" }, { "name": "summary", "mode": "NULLABLE", "type": "STRING" } ], "name": "zypper_patch", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "title", "mode": "NULLABLE", "type": "STRING" }, { "name": "description", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "id", "mode": "NULLABLE", "type": "STRING" }, { "name": "name", "mode": "NULLABLE", "type": "STRING" } ], "name": "categories", "mode": "REPEATED", "type": "RECORD" }, { "name": "kb_article_ids", "mode": "REPEATED", "type": "STRING" }, { "name": "support_url", "mode": "NULLABLE", "type": "STRING" }, { "name": "more_info_urls", "mode": "REPEATED", "type": "STRING" }, { "name": "update_id", "mode": "NULLABLE", "type": "STRING" }, { "name": "revision_number", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "last_deployment_change_time", "mode": "NULLABLE", "type": "TIMESTAMP" } ], "name": "wua_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "caption", "mode": "NULLABLE", "type": "STRING" }, { "name": "description", "mode": "NULLABLE", "type": "STRING" }, { "name": "hot_fix_id", "mode": "NULLABLE", "type": "STRING" }, { "name": "install_time", "mode": "NULLABLE", "type": "TIMESTAMP" } ], "name": "qfe_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "package_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "architecture", "mode": "NULLABLE", "type": "STRING" }, { "name": "version", "mode": "NULLABLE", "type": "STRING" } ], "name": "cos_package", "mode": "NULLABLE", "type": "RECORD" }, { "fields": [ { "name": "display_name", "mode": "NULLABLE", "type": "STRING" }, { "name": "display_version", "mode": "NULLABLE", "type": "STRING" }, { "name": "publisher", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "year", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "month", "mode": "NULLABLE", "type": "INTEGER" }, { "name": "day", "mode": "NULLABLE", "type": "INTEGER" } ], "name": "install_date", "mode": "NULLABLE", "type": "RECORD" }, { "name": "help_link", "mode": "NULLABLE", "type": "STRING" } ], "name": "windows_application", "mode": "NULLABLE", "type": "RECORD" } ], "name": "available_package", "mode": "NULLABLE", "type": "RECORD" } ], "name": "value", "mode": "NULLABLE", "type": "RECORD" } ], "name": "items", "mode": "REPEATED", "type": "RECORD" }, { "name": "update_time", "mode": "NULLABLE", "type": "TIMESTAMP" } ], "name": "os_inventory", "mode": "NULLABLE", "type": "RECORD" }, { "name": "ancestors", "mode": "REPEATED", "type": "STRING" }, { "name": "update_time", "mode": "NULLABLE", "type": "TIMESTAMP" } ]
Relationship: When you set content type to
RELATIONSHIPin the REST API orrelationshipin the gcloud CLI, you create a BigQuery table that has the following schema.Relationship schema
[ { "name": "name", "mode": "NULLABLE", "type": "STRING" }, { "name": "asset_type", "mode": "NULLABLE", "type": "STRING" }, { "fields": [ { "name": "asset", "mode": "NULLABLE", "type": "STRING" }, { "name": "asset_type", "mode": "NULLABLE", "type": "STRING" }, { "name": "ancestors", "mode": "REPEATED", "type": "STRING" }, { "name": "relationship_type", "mode": "NULLABLE", "type": "STRING" } ], "name": "related_asset", "mode": "NULLABLE", "type": "RECORD" }, { "name": "ancestors", "mode": "REPEATED", "type": "STRING" }, { "name": "update_time", "mode": "NULLABLE", "type": "TIMESTAMP" } ]
Export an asset snapshot
gcloud
gcloud asset export \ --SCOPE \ --billing-project=BILLING_PROJECT_ID \ --asset-types=ASSET_TYPE_1,ASSET_TYPE_2,... \ --content-type=CONTENT_TYPE \ --relationship-types=RELATIONSHIP_TYPE_1,RELATIONSHIP_TYPE_2,... \ --snapshot-time="SNAPSHOT_TIME" \ --bigquery-table=projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \ --output-bigquery-force
Provide the following values:
-
SCOPE: Use one of the following values:-
project=PROJECT_ID, wherePROJECT_IDis the ID of the project that has the asset metadata you want to export. -
folder=FOLDER_ID, whereFOLDER_IDis the ID of the folder that has the asset metadata you want to export.How to find the ID of a Google Cloud folder
Google Cloud console
To find the ID of a Google Cloud folder, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Search for your folder name. The folder ID is shown next to the folder name.
gcloud CLI
You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:
gcloud resource-manager folders list \ --organization=$(gcloud organizations describe ORGANIZATION_NAME \ --format="value(name.segment(1))") \ --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \ --format="value(ID)"
Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the
--formatflag to see more information about the found folders.The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:
gcloud resource-manager folders list --folder=FOLDER_ID
-
-
organization=ORGANIZATION_ID, whereORGANIZATION_IDis the ID of the organization that has the asset metadata you want to export.How to find the ID of a Google Cloud organization
Google Cloud console
To find the ID of a Google Cloud organization, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Click the All tab. The organization ID is shown next to the organization name.
gcloud CLI
You can retrieve the ID of a Google Cloud organization with the following command:
gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"
-
-
-
BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project. ASSET_TYPE_#: Optional. A comma-separated list of searchable asset types. RE2-compatible regular expressions are supported. If the regular expression doesn't match any supported asset type, anINVALID_ARGUMENTerror is returned. When--asset-typesisn't specified, all asset types are returned.CONTENT_TYPE: Optional. The content type of the metadata you want to retrieve. When--content-typeisn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.-
RELATIONSHIP_TYPE_#: Optional. Requires access to the Security Command Center Premium or Enterprise tier, or [Gemini Cloud Assist](/products/gemini/cloud-assist). A comma-separated list of asset relationship types you want to retrieve. You must setCONTENT_TYPEtoRELATIONSHIPfor this to work. -
SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in gcloud topic datetime format. The value must be no more than 35 days in the past. When--snapshot-timeisn't specified, a snapshot is taken at the current time. -
BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to. -
DATASET_ID: The ID of the BigQuery dataset. -
TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created.
The --output-bigquery-force flag overwrites the destination table if it exists.
See the gcloud CLI reference for all options.
Example
Run the following command to export your resource metadata as it was on January 30,
2024 in the my-project project, to the BigQuery table
my-table.
gcloud asset export \ --project=my-project \ --content-type=resource \ --snapshot-time="2024-01-30" \ --bigquery-table=projects/my-project/datasets/my-dataset/tables/my-table \ --output-bigquery-force
Example response
Export in progress for root asset [projects/my-project]. Use [gcloud asset operations describe projects/000000000000/operations/ExportAssets/RESOURCE/00000000000000000000000000000000] to check the status of the operation.
REST
HTTP method and URL:
POST https://cloudasset.googleapis.com/v1/SCOPE_PATH:exportAssets
Headers:
X-Goog-User-Project: BILLING_PROJECT_ID
Request JSON body:
{ "contentType": "CONTENT_TYPE", "relationshipTypes": [ "RELATIONSHIP_TYPE_1", "RELATIONSHIP_TYPE_2", "..." ], "readTime": "SNAPSHOT_TIME", "outputConfig": { "bigqueryDestination": { "dataset": "projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID", "table": "TABLE_NAME", "force": true } } }
Provide the following values:
-
SCOPE_PATH: Use one of the following values:The allowed values are:
-
projects/PROJECT_ID, wherePROJECT_IDis the ID of the project that has the asset metadata you want to export. -
projects/PROJECT_NUMBER, wherePROJECT_NUMBERis the number of the project that has the asset metadata you want to export.How to find a Google Cloud project number
Google Cloud console
To find a Google Cloud project number, complete the following steps:
-
Go to the Welcome page in the Google Cloud console.
- Click the switcher list box in the menu bar.
-
Select your organization from the list box, and then search for your project name. The project name, project number, and project ID are shown near the Welcome heading.
Up to 4,000 resources are displayed. If you don't see the project you're looking for, go to the Manage resources page and filter the list using the name of that project.
gcloud CLI
You can retrieve a Google Cloud project number with the following command:
gcloud projects describe PROJECT_ID --format="value(projectNumber)"
-
-
folders/FOLDER_ID, whereFOLDER_IDis the ID of the folder that has the asset metadata you want to export.How to find the ID of a Google Cloud folder
Google Cloud console
To find the ID of a Google Cloud folder, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Search for your folder name. The folder ID is shown next to the folder name.
gcloud CLI
You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:
gcloud resource-manager folders list \ --organization=$(gcloud organizations describe ORGANIZATION_NAME \ --format="value(name.segment(1))") \ --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \ --format="value(ID)"
Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the
--formatflag to see more information about the found folders.The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:
gcloud resource-manager folders list --folder=FOLDER_ID
-
-
organizations/ORGANIZATION_ID, whereORGANIZATION_IDis the ID of the organization that has the asset metadata you want to export.How to find the ID of a Google Cloud organization
Google Cloud console
To find the ID of a Google Cloud organization, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Click the All tab. The organization ID is shown next to the organization name.
gcloud CLI
You can retrieve the ID of a Google Cloud organization with the following command:
gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"
-
-
-
BILLING_PROJECT_ID: The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project. CONTENT_TYPE: The content type of the metadata you want to retrieve. WhencontentTypeisn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.-
RELATIONSHIP_TYPE_#: Optional. Requires access to the Security Command Center Premium or Enterprise tier, or [Gemini Cloud Assist](/products/gemini/cloud-assist). A comma-separated list of asset relationship types you want to retrieve. You must setCONTENT_TYPEtoRELATIONSHIPfor this to work. -
SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in RFC 3339 format. The value must be no more than 35 days in the past. WhenreadTimeisn't specified, a snapshot is taken at the current time. -
BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to. -
DATASET_ID: The ID of the BigQuery dataset. -
TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created.
The "force": true key-value pair overwrites the destination table if it exists.
See the REST reference for all options.
Command examples
Run one of the following commands to export your resource metadata as it was on
January 30, 2024 in the my-project project, to the BigQuery table
my-table.
curl (Linux, macOS, or Cloud Shell)
curl -X POST \ -H "X-Goog-User-Project: BILLING_PROJECT_ID" \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "Content-Type: application/json; charset=utf-8" \ -d '{ "contentType": "RESOURCE", "readTime": "2024-01-30T00:00:00Z", "outputConfig": { "bigqueryDestination": { "dataset": "projects/my-project/datasets/my-dataset", "table": "my-table", "force": true } } }' \ https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets
PowerShell (Windows)
$cred = gcloud auth print-access-token $headers = @{ "X-Goog-User-Project" = "BILLING_PROJECT_ID"; "Authorization" = "Bearer $cred" } $body = @" { "contentType": "RESOURCE", "readTime": "2024-01-30T00:00:00Z", "outputConfig": { "bigqueryDestination": { "dataset": "projects/my-project/datasets/my-dataset", "table": "my-table", "force": true } } } "@ Invoke-WebRequest ` -Method POST ` -Headers $headers ` -ContentType: "application/json; charset=utf-8" ` -Body $body ` -Uri "https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets" | Select-Object -Expand Content
Example response
{ "name": "projects/000000000000/operations/ExportAssets/RESOURCE/00000000000000000000000000000000", "metadata": { "@type": "type.googleapis.com/google.cloud.asset.v1.ExportAssetsRequest", "parent": "projects/000000000000", "readTime": "2024-01-30T00:00:00Z", "contentType": "RESOURCE", "outputConfig": { "bigqueryDestination": { "dataset": "projects/my-project/datasets/my-dataset", "table": "my-table", "force": true } } } }
Go
To learn how to install and use the client library for Cloud Asset Inventory, see Cloud Asset Inventory client libraries.
To authenticate to Cloud Asset Inventory, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To learn how to install and use the client library for Cloud Asset Inventory, see Cloud Asset Inventory client libraries.
To authenticate to Cloud Asset Inventory, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
To learn how to install and use the client library for Cloud Asset Inventory, see Cloud Asset Inventory client libraries.
To authenticate to Cloud Asset Inventory, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
To learn how to install and use the client library for Cloud Asset Inventory, see Cloud Asset Inventory client libraries.
To authenticate to Cloud Asset Inventory, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Export an asset snapshot to separate tables for each asset type
You can export assets to separate BigQuery tables for each asset
type with the --per-asset-type flag in the gcloud CLI, and
"separateTablesPerAssetType": true in REST API requests.
In this mode, each table's name is TABLE_NAME concatenated
with _ (underscore) and the asset type name. Non-alphanumeric characters are
replaced with _.
If exporting to any table fails, the entire export operation fails and returns the first error. Results of previous successful exports persist.
The following types are packed in a JSON string to overcome a compatibility issue between Proto3 and BigQuery types.
google.protobuf.Timestampgoogle.protobuf.Durationgoogle.protobuf.FieldMaskgoogle.protobuf.ListValuegoogle.protobuf.Valuegoogle.protobuf.Structgoogle.api.*
gcloud
gcloud asset export \ --SCOPE \ --billing-project=BILLING_PROJECT_ID \ --asset-types=ASSET_TYPE_1,ASSET_TYPE_2,... \ --content-type=CONTENT_TYPE \ --relationship-types=RELATIONSHIP_TYPE_1,RELATIONSHIP_TYPE_2,... \ --snapshot-time="SNAPSHOT_TIME" \ --bigquery-table=projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \ --per-asset-type \ --output-bigquery-force
Provide the following values:
-
SCOPE: Use one of the following values:-
project=PROJECT_ID, wherePROJECT_IDis the ID of the project that has the asset metadata you want to export. -
folder=FOLDER_ID, whereFOLDER_IDis the ID of the folder that has the asset metadata you want to export.How to find the ID of a Google Cloud folder
Google Cloud console
To find the ID of a Google Cloud folder, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Search for your folder name. The folder ID is shown next to the folder name.
gcloud CLI
You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:
gcloud resource-manager folders list \ --organization=$(gcloud organizations describe ORGANIZATION_NAME \ --format="value(name.segment(1))") \ --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \ --format="value(ID)"
Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the
--formatflag to see more information about the found folders.The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:
gcloud resource-manager folders list --folder=FOLDER_ID
-
-
organization=ORGANIZATION_ID, whereORGANIZATION_IDis the ID of the organization that has the asset metadata you want to export.How to find the ID of a Google Cloud organization
Google Cloud console
To find the ID of a Google Cloud organization, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Click the All tab. The organization ID is shown next to the organization name.
gcloud CLI
You can retrieve the ID of a Google Cloud organization with the following command:
gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"
-
-
-
BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project. ASSET_TYPE_#: Optional. A comma-separated list of searchable asset types. RE2-compatible regular expressions are supported. If the regular expression doesn't match any supported asset type, anINVALID_ARGUMENTerror is returned. When--asset-typesisn't specified, all asset types are returned.CONTENT_TYPE: Optional. The content type of the metadata you want to retrieve. When--content-typeisn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.-
RELATIONSHIP_TYPE_#: Optional. Requires access to the Security Command Center Premium or Enterprise tier, or [Gemini Cloud Assist](/products/gemini/cloud-assist). A comma-separated list of asset relationship types you want to retrieve. You must setCONTENT_TYPEtoRELATIONSHIPfor this to work. -
SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in gcloud topic datetime format. The value must be no more than 35 days in the past. When--snapshot-timeisn't specified, a snapshot is taken at the current time. -
BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to. -
DATASET_ID: The ID of the BigQuery dataset. -
TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created.
The --output-bigquery-force flag overwrites the destination table if it exists.
See the gcloud CLI reference for all options.
Example
Run the following command to export your resource metadata as it was on January 30,
2024 in the my-project project, to multiple BigQuery tables that have
my-table as a prefix.
gcloud asset export \ --project=my-project \ --content-type=resource \ --snapshot-time="2024-01-30" \ --bigquery-table=projects/my-project/datasets/my-dataset/tables/my-table \ --per-asset-type \ --output-bigquery-force
REST
HTTP method and URL:
POST https://cloudasset.googleapis.com/v1/SCOPE_PATH:exportAssets
Headers:
X-Goog-User-Project: BILLING_PROJECT_ID
Request JSON body:
{ "contentType": "CONTENT_TYPE", "relationshipTypes": [ "RELATIONSHIP_TYPE_1", "RELATIONSHIP_TYPE_2", "..." ], "readTime": "SNAPSHOT_TIME", "outputConfig": { "bigqueryDestination": { "dataset": "projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID", "table": "TABLE_NAME", "force": true, "separateTablesPerAssetType": true } } }
Provide the following values:
-
SCOPE_PATH: Use one of the following values:The allowed values are:
-
projects/PROJECT_ID, wherePROJECT_IDis the ID of the project that has the asset metadata you want to export. -
projects/PROJECT_NUMBER, wherePROJECT_NUMBERis the number of the project that has the asset metadata you want to export.How to find a Google Cloud project number
Google Cloud console
To find a Google Cloud project number, complete the following steps:
-
Go to the Welcome page in the Google Cloud console.
- Click the switcher list box in the menu bar.
-
Select your organization from the list box, and then search for your project name. The project name, project number, and project ID are shown near the Welcome heading.
Up to 4,000 resources are displayed. If you don't see the project you're looking for, go to the Manage resources page and filter the list using the name of that project.
gcloud CLI
You can retrieve a Google Cloud project number with the following command:
gcloud projects describe PROJECT_ID --format="value(projectNumber)"
-
-
folders/FOLDER_ID, whereFOLDER_IDis the ID of the folder that has the asset metadata you want to export.How to find the ID of a Google Cloud folder
Google Cloud console
To find the ID of a Google Cloud folder, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Search for your folder name. The folder ID is shown next to the folder name.
gcloud CLI
You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:
gcloud resource-manager folders list \ --organization=$(gcloud organizations describe ORGANIZATION_NAME \ --format="value(name.segment(1))") \ --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \ --format="value(ID)"
Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the
--formatflag to see more information about the found folders.The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:
gcloud resource-manager folders list --folder=FOLDER_ID
-
-
organizations/ORGANIZATION_ID, whereORGANIZATION_IDis the ID of the organization that has the asset metadata you want to export.How to find the ID of a Google Cloud organization
Google Cloud console
To find the ID of a Google Cloud organization, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Click the All tab. The organization ID is shown next to the organization name.
gcloud CLI
You can retrieve the ID of a Google Cloud organization with the following command:
gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"
-
-
-
BILLING_PROJECT_ID: The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project. CONTENT_TYPE: The content type of the metadata you want to retrieve. WhencontentTypeisn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.-
RELATIONSHIP_TYPE_#: Optional. Requires access to the Security Command Center Premium or Enterprise tier, or [Gemini Cloud Assist](/products/gemini/cloud-assist). A comma-separated list of asset relationship types you want to retrieve. You must setCONTENT_TYPEtoRELATIONSHIPfor this to work. -
SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in RFC 3339 format. The value must be no more than 35 days in the past. WhenreadTimeisn't specified, a snapshot is taken at the current time. -
BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to. -
DATASET_ID: The ID of the BigQuery dataset. -
TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created.
The "force": true key-value pair overwrites the destination table if it exists.
Command examples
Run one of the following commands to export your resource metadata as it was on
January 30, 2024 in the my-project project, to multiple BigQuery tables that
have my-table as a prefix.
curl (Linux, macOS, or Cloud Shell)
curl -X POST \ -H "X-Goog-User-Project: BILLING_PROJECT_ID" \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "Content-Type: application/json; charset=utf-8" \ -d '{ "contentType": "RESOURCE", "readTime": "2024-01-30T00:00:00Z", "outputConfig": { "bigqueryDestination": { "dataset": "projects/my-project/datasets/my-dataset", "table": "my-table", "force": true, "separateTablesPerAssetType": true } } }' \ https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets
PowerShell (Windows)
$cred = gcloud auth print-access-token $headers = @{ "X-Goog-User-Project" = "BILLING_PROJECT_ID"; "Authorization" = "Bearer $cred" } $body = @" { "contentType": "RESOURCE", "readTime": "2024-01-30T00:00:00Z", "outputConfig": { "bigqueryDestination": { "dataset": "projects/my-project/datasets/my-dataset", "table": "my-table", "force": true, "separateTablesPerAssetType": true } } } "@ Invoke-WebRequest ` -Method POST ` -Headers $headers ` -ContentType: "application/json; charset=utf-8" ` -Body $body ` -Uri "https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets" | Select-Object -Expand Content
Export an asset snapshot to a time-unit column partitioned table
You can export assets in a project to
time-unit column partitioned tables.
The exported snapshot is stored in a BigQuery table named
TABLE_NAME with daily granularity and two additional
timestamp columns, readTime and requestTime, one of which you specify as the
partition column with the PARTITION_KEY value.
To export assets in a project to partitioned tables, make one the following requests.
gcloud
gcloud asset export \ --SCOPE \ --billing-project=BILLING_PROJECT_ID \ --asset-types=ASSET_TYPE_1,ASSET_TYPE_2,... \ --content-type=CONTENT_TYPE \ --relationship-types=RELATIONSHIP_TYPE_1,RELATIONSHIP_TYPE_2,... \ --snapshot-time="SNAPSHOT_TIME" \ --bigquery-table=projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \ --partition-key=PARTITION_KEY \ --output-bigquery-force
Provide the following values:
-
SCOPE: Use one of the following values:-
project=PROJECT_ID, wherePROJECT_IDis the ID of the project that has the asset metadata you want to export. -
folder=FOLDER_ID, whereFOLDER_IDis the ID of the folder that has the asset metadata you want to export.How to find the ID of a Google Cloud folder
Google Cloud console
To find the ID of a Google Cloud folder, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Search for your folder name. The folder ID is shown next to the folder name.
gcloud CLI
You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:
gcloud resource-manager folders list \ --organization=$(gcloud organizations describe ORGANIZATION_NAME \ --format="value(name.segment(1))") \ --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \ --format="value(ID)"
Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the
--formatflag to see more information about the found folders.The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:
gcloud resource-manager folders list --folder=FOLDER_ID
-
-
organization=ORGANIZATION_ID, whereORGANIZATION_IDis the ID of the organization that has the asset metadata you want to export.How to find the ID of a Google Cloud organization
Google Cloud console
To find the ID of a Google Cloud organization, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Click the All tab. The organization ID is shown next to the organization name.
gcloud CLI
You can retrieve the ID of a Google Cloud organization with the following command:
gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"
-
-
-
BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project. ASSET_TYPE_#: Optional. A comma-separated list of searchable asset types. RE2-compatible regular expressions are supported. If the regular expression doesn't match any supported asset type, anINVALID_ARGUMENTerror is returned. When--asset-typesisn't specified, all asset types are returned.CONTENT_TYPE: Optional. The content type of the metadata you want to retrieve. When--content-typeisn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.-
RELATIONSHIP_TYPE_#: Optional. Requires access to the Security Command Center Premium or Enterprise tier, or [Gemini Cloud Assist](/products/gemini/cloud-assist). A comma-separated list of asset relationship types you want to retrieve. You must setCONTENT_TYPEtoRELATIONSHIPfor this to work. -
SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in gcloud topic datetime format. The value must be no more than 35 days in the past. When--snapshot-timeisn't specified, a snapshot is taken at the current time. -
BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to. -
DATASET_ID: The ID of the BigQuery dataset. -
TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created. -
PARTITION_KEY: The partition key column when exporting to BigQuery partitioned tables. Valid values areread-timeandrequest-time.
The --output-bigquery-force flag overwrites data in the corresponding partition in
the destination table. Data in different partitions remains intact.
If --output-bigquery-force isn't specified, exported data is appended to the
corresponding partition.
The export operation fails if a schema update or attempt to append data fails. This includes if the destination table already exists and doesn't have the schema the export expects.
See the gcloud CLI reference for all options.
Example
Run the following command to export your resource metadata as it was on January 30,
2024 in the my-project project, to the BigQuery table
my-table.
gcloud asset export \ --project=projects/my-project \ --content-type=resource \ --snapshot-time="2024-01-30" \ --bigquery-table=projects/my-project/datasets/my-dataset/tables/my-table \ --partition-key=my-partition-key \ --output-bigquery-force
REST
HTTP method and URL:
POST https://cloudasset.googleapis.com/v1/SCOPE_PATH:exportAssets
Headers:
X-Goog-User-Project: BILLING_PROJECT_ID
Request JSON body:
{ "contentType": "CONTENT_TYPE", "relationshipTypes": [ "RELATIONSHIP_TYPE_1", "RELATIONSHIP_TYPE_2", "..." ], "readTime": "SNAPSHOT_TIME", "outputConfig": { "bigqueryDestination": { "dataset": "projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID", "table": "TABLE_NAME", "partitionSpec": { "partitionKey": "PARTITION_KEY" }, "force": true, } } }
Provide the following values:
-
SCOPE_PATH: Use one of the following values:The allowed values are:
-
projects/PROJECT_ID, wherePROJECT_IDis the ID of the project that has the asset metadata you want to export. -
projects/PROJECT_NUMBER, wherePROJECT_NUMBERis the number of the project that has the asset metadata you want to export.How to find a Google Cloud project number
Google Cloud console
To find a Google Cloud project number, complete the following steps:
-
Go to the Welcome page in the Google Cloud console.
- Click the switcher list box in the menu bar.
-
Select your organization from the list box, and then search for your project name. The project name, project number, and project ID are shown near the Welcome heading.
Up to 4,000 resources are displayed. If you don't see the project you're looking for, go to the Manage resources page and filter the list using the name of that project.
gcloud CLI
You can retrieve a Google Cloud project number with the following command:
gcloud projects describe PROJECT_ID --format="value(projectNumber)"
-
-
folders/FOLDER_ID, whereFOLDER_IDis the ID of the folder that has the asset metadata you want to export.How to find the ID of a Google Cloud folder
Google Cloud console
To find the ID of a Google Cloud folder, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Search for your folder name. The folder ID is shown next to the folder name.
gcloud CLI
You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:
gcloud resource-manager folders list \ --organization=$(gcloud organizations describe ORGANIZATION_NAME \ --format="value(name.segment(1))") \ --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \ --format="value(ID)"
Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the
--formatflag to see more information about the found folders.The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:
gcloud resource-manager folders list --folder=FOLDER_ID
-
-
organizations/ORGANIZATION_ID, whereORGANIZATION_IDis the ID of the organization that has the asset metadata you want to export.How to find the ID of a Google Cloud organization
Google Cloud console
To find the ID of a Google Cloud organization, complete the following steps:
-
Go to the Google Cloud console.
- Click the switcher list box in the menu bar.
- Select your organization from the list box.
- Click the All tab. The organization ID is shown next to the organization name.
gcloud CLI
You can retrieve the ID of a Google Cloud organization with the following command:
gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"
-
-
-
BILLING_PROJECT_ID: The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project. CONTENT_TYPE: The content type of the metadata you want to retrieve. WhencontentTypeisn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.-
RELATIONSHIP_TYPE_#: Optional. Requires access to the Security Command Center Premium or Enterprise tier, or [Gemini Cloud Assist](/products/gemini/cloud-assist). A comma-separated list of asset relationship types you want to retrieve. You must setCONTENT_TYPEtoRELATIONSHIPfor this to work. -
SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in RFC 3339 format. The value must be no more than 35 days in the past. WhenreadTimeisn't specified, a snapshot is taken at the current time. -
BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to. -
DATASET_ID: The ID of the BigQuery dataset. -
TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created. -
PARTITION_KEY: The partition key column when exporting to BigQuery partitioned tables. Valid values areREAD_TIMEandREQUEST_TIME.
The "force": true key-value pair overwrites data in the corresponding partition in
the destination table. Data in different partitions remains intact.
If force isn't set or set to false, exported data is appended to the
corresponding partition.
The export operation fails if a schema update or attempt to append data fails. This includes if the destination table already exists and doesn't have the schema the export expects.
Command examples
Run one of the following commands to export your resource metadata as it was on
January 30, 2024 in the my-project project, to the BigQuery table
my-table.
curl (Linux, macOS, or Cloud Shell)
curl -X POST \ -H "X-Goog-User-Project: BILLING_PROJECT_ID" \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "Content-Type: application/json; charset=utf-8" \ -d '{ "contentType": "RESOURCE", "readTime": "2024-01-30T00:00:00Z", "outputConfig": { "bigqueryDestination": { "dataset": "projects/my-project/datasets/my-dataset", "table": "my-table", "partitionSpec": { "partitionKey": "my-partition-key" }, "force": true, } } }' \ https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets
PowerShell (Windows)
$cred = gcloud auth print-access-token $headers = @{ "X-Goog-User-Project" = "BILLING_PROJECT_ID"; "Authorization" = "Bearer $cred" } $body = @" { "contentType": "RESOURCE", "readTime": "2024-01-30T00:00:00Z", "outputConfig": { "bigqueryDestination": { "dataset": "projects/my-project/datasets/my-dataset", "table": "my-table", "partitionSpec": { "partitionKey": "my-partition-key" }, "force": true, } } } "@ Invoke-WebRequest ` -Method POST ` -Headers $headers ` -ContentType: "application/json; charset=utf-8" ` -Body $body ` -Uri "https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets" | Select-Object -Expand Content
Check the status of an export
Exports take time to complete. To check if an export is done, you can query the operation using its operation ID.
Be aware that even if your export is done, someone might have made another export request to the same destination as a different operation. New export requests to the same destination can be made after a previous request has finished, or if more than 15 minutes has elapsed. Export requests made outside of these conditions are rejected by Cloud Asset Inventory.
gcloud
To view the status of your export, complete the following instructions:
Get the
OPERATION_PATH, which includes the operation ID, from the response to your export request. TheOPERATION_PATHis shown in the response to the export, which is formatted as follows:projects/PROJECT_NUMBER/operations/ExportAssets/CONTENT_TYPE/OPERATION_IDTo check the status of your export, run following command with the
OPERATION_PATH:gcloud asset operations describe OPERATION_PATH
REST
To view the status of your export, complete the following instructions:
Get the
OPERATION_PATH, which includes the operation ID, from the response to your export request. TheOPERATION_PATHis shown as the value of thenamefield in the response to the export, which is formatted as follows:projects/PROJECT_NUMBER/operations/ExportAssets/CONTENT_TYPE/OPERATION_IDTo check the status of your export, make the following request.
REST
HTTP method and URL:
GET https://cloudasset.googleapis.com/v1/OPERATION_PATH
Command examples
curl (Linux, macOS, or Cloud Shell)
curl -X GET \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ https://cloudasset.googleapis.com/v1/OPERATION_PATH
PowerShell (Windows)
$cred = gcloud auth print-access-token $headers = @{ "Authorization" = "Bearer $cred" } Invoke-WebRequest ` -Method GET ` -Headers $headers ` -Uri "https://cloudasset.googleapis.com/v1/OPERATION_PATH" | Select-Object -Expand Content
View an asset snapshot in BigQuery
Console
Go to the BigQuery Studio page in the Google Cloud console.
To display the tables and views in the dataset, open the navigation panel. In the Resources section, select your project to expand it, and then select a dataset.
From the list, select your table.
Select Details and note the value in Number of rows. You might need this value to control the starting point for your results using the gcloud CLI or REST API.
To view a sample set of data, select Preview.
REST
To browse your table's data, call
tabledata.list. In the
tableId parameter, specify the name of your table.
You can configure the following optional parameters to control the output.
maxResultsis the maximum number of results to return.selectedFieldsis a comma-separated list of columns to return; If unspecified, all columns are returned.startIndexis the zero-based index of the starting row to read.
Values are returned wrapped in a JSON object that you must parse, as described
in the tabledata.list
reference documentation.
Query an asset snapshot in BigQuery
After you export your snapshot to BigQuery, you can run queries on your asset metadata.
By default, BigQuery runs interactive, or on-demand, query jobs, which means that the query is executed as soon as possible. Interactive queries count towards your concurrent rate limit and your daily limit.
Query results are saved to either a temporary or permanent table. You can choose to append or overwrite data in an existing table or to create a new table, if none exists with the same name.
To run an interactive query that writes the output to a temporary table, complete the following steps.
Console
Go to the BigQuery Studio page in the Google Cloud console.
Select Compose new query.
In the Query editor text area, enter a valid BigQuery SQL query.
Optional: To change the data processing location, complete the following steps.
Select More, and then select Query settings.
Under Processing location, select Auto-select, and then choose your data's location.
To update the query settings, select Save.
Select Run.
REST
To start a new job, call the
jobs.insertmethod. In the job resource, set the following parameters.In the
configurationfield, set thequeryfield to a JobConfigurationQuery that describes the BigQuery query job.In the
jobReferencefield, set thelocationfield appropriately for your job.
To poll for results, call
getQueryResults. Poll untiljobCompleteequalstrue. You can check for errors and warnings in theerrorslist.
Additional SQL query examples
This section provides example SQL queries to analyze your asset metadata after you've exported it to BigQuery. See standard SQL query syntax for more information.
Query available columns directly
To find the quantity of each asset type, run the following query:
SELECT asset_type, COUNT(*) AS asset_count
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`
GROUP BY asset_type
ORDER BY asset_count DESC
Work with repeated fields
To find Identity and Access Management (IAM) policies that grant access to Gmail
accounts, run the following query. BigQuery uses UNNEST to
flatten
repeated fields into a table
that you can query directly:
SELECT name, asset_type, bindings.role
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`
JOIN UNNEST(iam_policy.bindings) AS bindings
JOIN UNNEST(bindings.members) AS principals
WHERE principals like "%@gmail.com"
To find an organization, folder, or project that allows creating using a public IP address, run the following query. This query is useful because allowing public IP addresses with Cloud SQL instances can introduce vulnerabilities unless SSL or a proxy is configured:
SELECT name
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`
JOIN UNNEST(org_policy) AS op
WHERE
op.constraint = "constraints/sql.restrictPublicIp"
AND (op.boolean_policy IS NULL OR op.boolean_policy.enforced = FALSE);
To find an organization, folder, or project in the same VPC Service Controls service perimeter of a project, run the following query:
SELECT service_perimeter.title, service_perimeter.status.resources
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`
CROSS JOIN UNNEST(service_perimeter.status.resources) as resource
WHERE resource = "projects/PROJECT_ID";
Work with JSON strings
To find open firewall rules, run the following query. Learn more about JSON functions used in BigQuery.
CREATE TEMP FUNCTION json2array(json STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS """
return JSON.parse(json).map(x=>JSON.stringify(x));
""";
SELECT firewall.name, firewall.resource.parent, JSON_EXTRACT(firewall.resource.data, '$.sourceRanges') AS sourceRanges
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME` AS firewall
JOIN UNNEST(json2array(JSON_EXTRACT(firewall.resource.data, '$.sourceRanges'))) AS source_ranges
WHERE asset_type="compute.googleapis.com/Firewall" AND JSON_EXTRACT(firewall.resource.data, '$.sourceRanges') IS NOT NULL AND JSON_EXTRACT_SCALAR(source_ranges, '$') = "0.0.0.0/0"
By separating tables for each resource type, you can find open firewall rules with an easier and faster query.
SELECT firewall.name, firewall.resource.parent, sourceRanges
FROM `PROJECT_ID.DATASET_ID.STRUCTURED_INSTANCE_TABLE_NAME` AS firewall
JOIN UNNEST(firewall.resource.data.sourceRanges) AS sourceRanges
WHERE sourceRanges = "0.0.0.0/0";
Join tables of different resource types
To join tables of different resource types, run the following query. The
following example shows how to find all subnetworks that have no VM attached.
First, the query finds all subnetworks. Then, from that list, it selects those
subnetworks whose selfLink values are not present.
CREATE TEMP FUNCTION json2array(json STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS """
return JSON.parse(json).map(x=>JSON.stringify(x));
""";
SELECT name, JSON_EXTRACT(subnetwork.resource.data, '$.selfLink') AS selflink
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME` AS subnetwork
WHERE asset_type = "compute.googleapis.com/Subnetwork" AND (JSON_EXTRACT(subnetwork.resource.data, '$.selfLink') NOT IN
(SELECT DISTINCT JSON_EXTRACT(network_interfaces, '$.subnetwork')
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME` as instance
JOIN UNNEST(json2array(JSON_EXTRACT(instance.resource.data, '$.networkInterfaces'))) AS network_interfaces
WHERE asset_type ="compute.googleapis.com/Instance"
AND JSON_EXTRACT(instance.resource.data, '$.networkInterfaces') IS NOT NULL
)) IS NULL
By separating tables for each resource type, you can find all subnetworks that have no VM attached with an easier and faster query.
SELECT name, subnetwork.resource.data.selfLink
FROM `PROJECT_ID.DATASET_ID.STRUCTURED_SUBNETWORK_TABLE_NAME` AS subnetwork
WHERE
(
subnetwork.resource.data.selfLink
NOT IN (
SELECT DISTINCT networkInterface.subnetwork
FROM `PROJECT_ID.DATASET_ID.STRUCTURED_INSTANCE_TABLE_NAME` as instance
JOIN
UNNEST(instance.resource.data.networkInterfaces) AS networkInterface
WHERE
networkInterface IS NOT NULL
)
) IS NULL;
Find vulnerable Dataproc clusters due to CVE-2021-44228
CREATE TEMP FUNCTION vulnerable_version(imageVersion STRING)
RETURNS BOOL
LANGUAGE js AS r"""
const version_regexp = /(?<major>\d+)(?:\.)(?<minor>\d+)(?:\.)?(?<sub>\d+)?/g;
let match = version_regexp.exec(imageVersion);
if(match.groups.major < 1){
return true;
}
if (match.groups.major == 1){
if (match.groups.minor < 3){
return true;
}
if(match.groups.minor == 3 && match.groups.sub < 95){
return true;
}
if(match.groups.minor == 4 && match.groups.sub < 77){
return true;
}
if(match.groups.minor == 5 && match.groups.sub < 53){
return true;
}
}
if (match.groups.major == 2 && match.groups.minor == 0 && match.groups.sub < 27){
return true;
}
return false;
""";
SELECT
c.name,
c.resource.data.projectId AS project_id,
c.resource.data.clusterName AS cluster_name,
c.resource.data.config.softwareConfig.imageVersion AS image_version,
c.resource.data.status.state AS cluster_state,
vulnerable_version(c.resource.data.config.softwareConfig.imageVersion) AS is_vulnerable
FROM
`PROJECT_ID.DATASET_ID.TABLE_NAME_PREFIX_dataproc_googleapis_com_Cluster` c
INNER JOIN `PROJECT_ID.DATASET_ID.TABLE_NAME_PREFIX_cloudresourcemanager_googleapis_com_Project` p
ON p.resource.data.projectId = c.resource.data.projectId
WHERE
c.resource.data.config.softwareConfig.imageVersion IS NOT NULL
AND c.resource.data.status.state = "RUNNING"
AND p.resource.data.lifecycleState = "ACTIVE";