Export asset metadata to BigQuery

This document shows you how to export an asset snapshot from your organization, folder, or project to a BigQuery table, and then run data analysis on your inventory. BigQuery provides a SQL-like experience for users to analyze data and produce meaningful insights without the use of custom scripts.

Before you begin

  1. Enable the Cloud Asset Inventory API in the project you're running Cloud Asset Inventory commands from.

    Enable the Cloud Asset Inventory API

  2. Make sure your account has the correct role to call the Cloud Asset Inventory API. For individual permissions for each call type, see Permissions.

  3. Create a BigQuery dataset to export to, if you don't have one already.

Limitations

  • When exporting BigQuery table data, not all fields are supported by Cloud Asset Inventory.

  • Frequently changing asset fields such as numBytes, numLongTermBytes, numPhysicalBytes, and numRows might be exported with a null value.

  • Exporting to BigQuery clustered tables isn't supported.

  • BigQuery tables encrypted with custom Cloud Key Management Service (Cloud KMS) keys aren't supported.

  • Appending the export output to an existing table isn't supported unless you export to a partitioned table. The destination table must be empty or you must overwrite it. To overwrite it, use the --output-bigquery-force flag with the gcloud CLI, or use "force": true with the REST API.

  • Google Kubernetes Engine (GKE) resource types, except for container.googleapis.com/Cluster and container.googleapis.com/NodePool, are not supported when exporting to separate tables for each resource type.

  • Cloud Asset Inventory rejects export requests if a previous request to the same destination started less than 15 minutes ago and is still running. However, if an export has taken longer than 15 minutes to complete, it is marked as done and new export requests to the same destination are permitted.

  • The ACCESS_POLICY content type can only be exported at the organization level.

  • If the table you're exporting to already exists and is in the process of being exported to, a 400 error is returned.

BigQuery schemas used for export

Every BigQuery table is defined by a schema that describes the column names, data types, and other information. Setting the content type for an export determines the schema for your table:

  • Resource or unspecified: When you set the content type to RESOURCE or don't specify it, and you set the per-asset-type flag to false or don't use it, you create a BigQuery table that has the following schema.

    Resource schema

    [
  {
    "name": "name",
    "type": "STRING",
    "mode": "NULLABLE"
  },
  {
    "name": "asset_type",
    "type": "STRING",
    "mode": "NULLABLE"
  },
  {
    "name": "resource",
    "type": "RECORD",
    "mode": "NULLABLE",
    "fields": [
      {
        "name": "version",
        "type": "STRING",
        "mode": "NULLABLE"
      },
      {
        "name": "discovery_document_uri",
        "type": "STRING",
        "mode": "NULLABLE"
      },
      {
        "name": "discovery_name",
        "type": "STRING",
        "mode": "NULLABLE"
      },
      {
        "name": "resource_url",
        "type": "STRING",
        "mode": "NULLABLE"
      },
      {
        "name": "parent",
        "type": "STRING",
        "mode": "NULLABLE"
      },
      {
        "name": "data",
        "type": "STRING",
        "mode": "NULLABLE"
      },
      {
        "name": "location",
        "type": "STRING",
        "mode": "NULLABLE"
      }
    ]
  },
  {
    "name": "ancestors",
    "type": "STRING",
    "mode": "REPEATED"
  },
  {
    "name": "update_time",
    "type": "TIMESTAMP",
    "mode": "NULLABLE"
  }
]

    The resource.data column is the resource metadata represented as a JSON string.

    When you set the content type to RESOURCE or don't set the content type, and set the per-asset-type flag to true, you create separate tables for each asset type. The schema of each table includes RECORD-type columns mapped to the nested fields in the Resource.data field of that asset type (up to the 15 nested levels that BigQuery supports). For example tables, see export-assets-examples in the Google Cloud console.

  • IAM policy: When you set the content type to IAM_POLICY in the REST API or iam-policy in the gcloud CLI, you create a BigQuery table that has the following schema.

    IAM policy schema

    [
  {
    "name": "name",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "asset_type",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "fields": [
      {
        "name": "version",
        "mode": "NULLABLE",
        "type": "INTEGER"
      },
      {
        "fields": [
          {
            "name": "role",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "members",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "fields": [
              {
                "name": "expression",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "title",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "description",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "location",
                "mode": "NULLABLE",
                "type": "STRING"
              }
            ],
            "name": "condition",
            "mode": "NULLABLE",
            "type": "RECORD"
          }
        ],
        "name": "bindings",
        "mode": "REPEATED",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "service",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "fields": [
              {
                "name": "log_type",
                "mode": "NULLABLE",
                "type": "INTEGER"
              },
              {
                "name": "exempted_members",
                "mode": "REPEATED",
                "type": "STRING"
              }
            ],
            "name": "audit_log_configs",
            "mode": "REPEATED",
            "type": "RECORD"
          }
        ],
        "name": "audit_configs",
        "mode": "REPEATED",
        "type": "RECORD"
      },
      {
        "name": "etag",
        "mode": "NULLABLE",
        "type": "STRING"
      }
    ],
    "name": "iam_policy",
    "mode": "NULLABLE",
    "type": "RECORD"
  },
  {
    "name": "ancestors",
    "mode": "REPEATED",
    "type": "STRING"
  },
  {
    "name": "update_time",
    "mode": "NULLABLE",
    "type": "TIMESTAMP"
  }
]

  • Organization policy: When you set the content type to ORG_POLICY in the REST API or org-policy in the gcloud CLI, you create a BigQuery table that has the following schema.

    Organization policy schema

    [
  {
    "name": "name",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "asset_type",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "fields": [
      {
        "name": "version",
        "mode": "NULLABLE",
        "type": "INTEGER"
      },
      {
        "name": "constraint",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "etag",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "update_time",
        "mode": "NULLABLE",
        "type": "TIMESTAMP"
      },
      {
        "fields": [
          {
            "name": "allowed_values",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "name": "denied_values",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "name": "all_values",
            "mode": "NULLABLE",
            "type": "INTEGER"
          },
          {
            "name": "suggested_value",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "inherit_from_parent",
            "mode": "NULLABLE",
            "type": "BOOLEAN"
          }
        ],
        "name": "list_policy",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "enforced",
            "mode": "NULLABLE",
            "type": "BOOLEAN"
          }
        ],
        "name": "boolean_policy",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "_present",
            "mode": "NULLABLE",
            "type": "BOOLEAN"
          }
        ],
        "name": "restore_default",
        "mode": "NULLABLE",
        "type": "RECORD"
      }
    ],
    "name": "org_policy",
    "mode": "REPEATED",
    "type": "RECORD"
  },
  {
    "name": "ancestors",
    "mode": "REPEATED",
    "type": "STRING"
  },
  {
    "name": "update_time",
    "mode": "NULLABLE",
    "type": "TIMESTAMP"
  }
]

  • VPCSC policy: When you set content type to ACCESS_POLICY in the REST API or access-policy in the gcloud CLI, you create a BigQuery table that has the following schema.

    VPCSC policy schema

    [
  {
    "name": "name",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "asset_type",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "fields": [
      {
        "name": "name",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "parent",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "title",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "scopes",
        "mode": "REPEATED",
        "type": "STRING"
      },
      {
        "name": "etag",
        "mode": "NULLABLE",
        "type": "STRING"
      }
    ],
    "name": "access_policy",
    "mode": "NULLABLE",
    "type": "RECORD"
  },
  {
    "fields": [
      {
        "name": "name",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "title",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "description",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "fields": [
              {
                "name": "ip_subnetworks",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "fields": [
                  {
                    "name": "require_screenlock",
                    "mode": "NULLABLE",
                    "type": "BOOLEAN"
                  },
                  {
                    "name": "allowed_encryption_statuses",
                    "mode": "REPEATED",
                    "type": "INTEGER"
                  },
                  {
                    "fields": [
                      {
                        "name": "os_type",
                        "mode": "NULLABLE",
                        "type": "INTEGER"
                      },
                      {
                        "name": "minimum_version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "require_verified_chrome_os",
                        "mode": "NULLABLE",
                        "type": "BOOLEAN"
                      }
                    ],
                    "name": "os_constraints",
                    "mode": "REPEATED",
                    "type": "RECORD"
                  },
                  {
                    "name": "allowed_device_management_levels",
                    "mode": "REPEATED",
                    "type": "INTEGER"
                  },
                  {
                    "name": "require_admin_approval",
                    "mode": "NULLABLE",
                    "type": "BOOLEAN"
                  },
                  {
                    "name": "require_corp_owned",
                    "mode": "NULLABLE",
                    "type": "BOOLEAN"
                  }
                ],
                "name": "device_policy",
                "mode": "NULLABLE",
                "type": "RECORD"
              },
              {
                "name": "required_access_levels",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "name": "negate",
                "mode": "NULLABLE",
                "type": "BOOLEAN"
              },
              {
                "name": "members",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "name": "regions",
                "mode": "REPEATED",
                "type": "STRING"
              }
            ],
            "name": "conditions",
            "mode": "REPEATED",
            "type": "RECORD"
          },
          {
            "name": "combining_function",
            "mode": "NULLABLE",
            "type": "INTEGER"
          }
        ],
        "name": "basic",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "fields": [
              {
                "name": "expression",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "title",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "description",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "location",
                "mode": "NULLABLE",
                "type": "STRING"
              }
            ],
            "name": "expr",
            "mode": "NULLABLE",
            "type": "RECORD"
          }
        ],
        "name": "custom",
        "mode": "NULLABLE",
        "type": "RECORD"
      }
    ],
    "name": "access_level",
    "mode": "NULLABLE",
    "type": "RECORD"
  },
  {
    "fields": [
      {
        "name": "name",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "title",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "description",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "perimeter_type",
        "mode": "NULLABLE",
        "type": "INTEGER"
      },
      {
        "fields": [
          {
            "name": "resources",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "name": "access_levels",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "name": "restricted_services",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "fields": [
              {
                "name": "enable_restriction",
                "mode": "NULLABLE",
                "type": "BOOLEAN"
              },
              {
                "name": "allowed_services",
                "mode": "REPEATED",
                "type": "STRING"
              }
            ],
            "name": "vpc_accessible_services",
            "mode": "NULLABLE",
            "type": "RECORD"
          },
          {
            "fields": [
              {
                "fields": [
                  {
                    "fields": [
                      {
                        "name": "access_level",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "resource",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "sources",
                    "mode": "REPEATED",
                    "type": "RECORD"
                  },
                  {
                    "name": "identities",
                    "mode": "REPEATED",
                    "type": "STRING"
                  },
                  {
                    "name": "identity_type",
                    "mode": "NULLABLE",
                    "type": "INTEGER"
                  }
                ],
                "name": "ingress_from",
                "mode": "NULLABLE",
                "type": "RECORD"
              },
              {
                "fields": [
                  {
                    "fields": [
                      {
                        "name": "service_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "fields": [
                          {
                            "name": "method",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "permission",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "method_selectors",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      }
                    ],
                    "name": "operations",
                    "mode": "REPEATED",
                    "type": "RECORD"
                  },
                  {
                    "name": "resources",
                    "mode": "REPEATED",
                    "type": "STRING"
                  }
                ],
                "name": "ingress_to",
                "mode": "NULLABLE",
                "type": "RECORD"
              }
            ],
            "name": "ingress_policies",
            "mode": "REPEATED",
            "type": "RECORD"
          },
          {
            "fields": [
              {
                "fields": [
                  {
                    "name": "identities",
                    "mode": "REPEATED",
                    "type": "STRING"
                  },
                  {
                    "name": "identity_type",
                    "mode": "NULLABLE",
                    "type": "INTEGER"
                  }
                ],
                "name": "egress_from",
                "mode": "NULLABLE",
                "type": "RECORD"
              },
              {
                "fields": [
                  {
                    "name": "resources",
                    "mode": "REPEATED",
                    "type": "STRING"
                  },
                  {
                    "fields": [
                      {
                        "name": "service_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "fields": [
                          {
                            "name": "method",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "permission",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "method_selectors",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      }
                    ],
                    "name": "operations",
                    "mode": "REPEATED",
                    "type": "RECORD"
                  },
                  {
                    "name": "external_resources",
                    "mode": "REPEATED",
                    "type": "STRING"
                  }
                ],
                "name": "egress_to",
                "mode": "NULLABLE",
                "type": "RECORD"
              }
            ],
            "name": "egress_policies",
            "mode": "REPEATED",
            "type": "RECORD"
          }
        ],
        "name": "status",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "resources",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "name": "access_levels",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "name": "restricted_services",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "fields": [
              {
                "name": "enable_restriction",
                "mode": "NULLABLE",
                "type": "BOOLEAN"
              },
              {
                "name": "allowed_services",
                "mode": "REPEATED",
                "type": "STRING"
              }
            ],
            "name": "vpc_accessible_services",
            "mode": "NULLABLE",
            "type": "RECORD"
          },
          {
            "fields": [
              {
                "fields": [
                  {
                    "fields": [
                      {
                        "name": "access_level",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "resource",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "sources",
                    "mode": "REPEATED",
                    "type": "RECORD"
                  },
                  {
                    "name": "identities",
                    "mode": "REPEATED",
                    "type": "STRING"
                  },
                  {
                    "name": "identity_type",
                    "mode": "NULLABLE",
                    "type": "INTEGER"
                  }
                ],
                "name": "ingress_from",
                "mode": "NULLABLE",
                "type": "RECORD"
              },
              {
                "fields": [
                  {
                    "fields": [
                      {
                        "name": "service_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "fields": [
                          {
                            "name": "method",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "permission",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "method_selectors",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      }
                    ],
                    "name": "operations",
                    "mode": "REPEATED",
                    "type": "RECORD"
                  },
                  {
                    "name": "resources",
                    "mode": "REPEATED",
                    "type": "STRING"
                  }
                ],
                "name": "ingress_to",
                "mode": "NULLABLE",
                "type": "RECORD"
              }
            ],
            "name": "ingress_policies",
            "mode": "REPEATED",
            "type": "RECORD"
          },
          {
            "fields": [
              {
                "fields": [
                  {
                    "name": "identities",
                    "mode": "REPEATED",
                    "type": "STRING"
                  },
                  {
                    "name": "identity_type",
                    "mode": "NULLABLE",
                    "type": "INTEGER"
                  }
                ],
                "name": "egress_from",
                "mode": "NULLABLE",
                "type": "RECORD"
              },
              {
                "fields": [
                  {
                    "name": "resources",
                    "mode": "REPEATED",
                    "type": "STRING"
                  },
                  {
                    "fields": [
                      {
                        "name": "service_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "fields": [
                          {
                            "name": "method",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "permission",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "method_selectors",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      }
                    ],
                    "name": "operations",
                    "mode": "REPEATED",
                    "type": "RECORD"
                  },
                  {
                    "name": "external_resources",
                    "mode": "REPEATED",
                    "type": "STRING"
                  }
                ],
                "name": "egress_to",
                "mode": "NULLABLE",
                "type": "RECORD"
              }
            ],
            "name": "egress_policies",
            "mode": "REPEATED",
            "type": "RECORD"
          }
        ],
        "name": "spec",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "name": "use_explicit_dry_run_spec",
        "mode": "NULLABLE",
        "type": "BOOLEAN"
      }
    ],
    "name": "service_perimeter",
    "mode": "NULLABLE",
    "type": "RECORD"
  },
  {
    "name": "ancestors",
    "mode": "REPEATED",
    "type": "STRING"
  },
  {
    "name": "update_time",
    "mode": "NULLABLE",
    "type": "TIMESTAMP"
  }
]

  • OSConfig instance inventory: When you set content type to OS_INVENTORY in the REST API or os-inventory in the gcloud CLI, you create a BigQuery table that has the following schema.

    OS inventory schema

    [
  {
    "name": "name",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "asset_type",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "fields": [
      {
        "name": "name",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "name": "hostname",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "long_name",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "short_name",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "version",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "architecture",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "kernel_version",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "kernel_release",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "osconfig_agent_version",
            "mode": "NULLABLE",
            "type": "STRING"
          }
        ],
        "name": "os_info",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "key",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "fields": [
              {
                "name": "id",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "origin_type",
                "mode": "NULLABLE",
                "type": "INTEGER"
              },
              {
                "name": "create_time",
                "mode": "NULLABLE",
                "type": "TIMESTAMP"
              },
              {
                "name": "update_time",
                "mode": "NULLABLE",
                "type": "TIMESTAMP"
              },
              {
                "name": "type",
                "mode": "NULLABLE",
                "type": "INTEGER"
              },
              {
                "fields": [
                  {
                    "fields": [
                      {
                        "name": "package_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "architecture",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "yum_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "package_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "architecture",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "apt_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "package_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "architecture",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "zypper_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "package_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "architecture",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "googet_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "patch_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "category",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "severity",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "summary",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "zypper_patch",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "title",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "description",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "fields": [
                          {
                            "name": "id",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "categories",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      },
                      {
                        "name": "kb_article_ids",
                        "mode": "REPEATED",
                        "type": "STRING"
                      },
                      {
                        "name": "support_url",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "more_info_urls",
                        "mode": "REPEATED",
                        "type": "STRING"
                      },
                      {
                        "name": "update_id",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "revision_number",
                        "mode": "NULLABLE",
                        "type": "INTEGER"
                      },
                      {
                        "name": "last_deployment_change_time",
                        "mode": "NULLABLE",
                        "type": "TIMESTAMP"
                      }
                    ],
                    "name": "wua_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "caption",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "description",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "hot_fix_id",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "install_time",
                        "mode": "NULLABLE",
                        "type": "TIMESTAMP"
                      }
                    ],
                    "name": "qfe_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "package_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "architecture",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "cos_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "display_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "display_version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "publisher",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "fields": [
                          {
                            "name": "year",
                            "mode": "NULLABLE",
                            "type": "INTEGER"
                          },
                          {
                            "name": "month",
                            "mode": "NULLABLE",
                            "type": "INTEGER"
                          },
                          {
                            "name": "day",
                            "mode": "NULLABLE",
                            "type": "INTEGER"
                          }
                        ],
                        "name": "install_date",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "name": "help_link",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "windows_application",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  }
                ],
                "name": "installed_package",
                "mode": "NULLABLE",
                "type": "RECORD"
              },
              {
                "fields": [
                  {
                    "fields": [
                      {
                        "name": "package_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "architecture",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "yum_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "package_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "architecture",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "apt_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "package_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "architecture",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "zypper_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "package_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "architecture",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "googet_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "patch_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "category",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "severity",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "summary",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "zypper_patch",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "title",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "description",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "fields": [
                          {
                            "name": "id",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "categories",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      },
                      {
                        "name": "kb_article_ids",
                        "mode": "REPEATED",
                        "type": "STRING"
                      },
                      {
                        "name": "support_url",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "more_info_urls",
                        "mode": "REPEATED",
                        "type": "STRING"
                      },
                      {
                        "name": "update_id",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "revision_number",
                        "mode": "NULLABLE",
                        "type": "INTEGER"
                      },
                      {
                        "name": "last_deployment_change_time",
                        "mode": "NULLABLE",
                        "type": "TIMESTAMP"
                      }
                    ],
                    "name": "wua_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "caption",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "description",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "hot_fix_id",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "install_time",
                        "mode": "NULLABLE",
                        "type": "TIMESTAMP"
                      }
                    ],
                    "name": "qfe_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "package_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "architecture",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "cos_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "display_name",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "display_version",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "name": "publisher",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      },
                      {
                        "fields": [
                          {
                            "name": "year",
                            "mode": "NULLABLE",
                            "type": "INTEGER"
                          },
                          {
                            "name": "month",
                            "mode": "NULLABLE",
                            "type": "INTEGER"
                          },
                          {
                            "name": "day",
                            "mode": "NULLABLE",
                            "type": "INTEGER"
                          }
                        ],
                        "name": "install_date",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "name": "help_link",
                        "mode": "NULLABLE",
                        "type": "STRING"
                      }
                    ],
                    "name": "windows_application",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  }
                ],
                "name": "available_package",
                "mode": "NULLABLE",
                "type": "RECORD"
              }
            ],
            "name": "value",
            "mode": "NULLABLE",
            "type": "RECORD"
          }
        ],
        "name": "items",
        "mode": "REPEATED",
        "type": "RECORD"
      },
      {
        "name": "update_time",
        "mode": "NULLABLE",
        "type": "TIMESTAMP"
      }
    ],
    "name": "os_inventory",
    "mode": "NULLABLE",
    "type": "RECORD"
  },
  {
    "name": "ancestors",
    "mode": "REPEATED",
    "type": "STRING"
  },
  {
    "name": "update_time",
    "mode": "NULLABLE",
    "type": "TIMESTAMP"
  }
]

  • Relationship: When you set content type to RELATIONSHIP in the REST API or relationship in the gcloud CLI, you create a BigQuery table that has the following schema.

    Relationship schema

    [
  {
    "name": "name",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "name": "asset_type",
    "mode": "NULLABLE",
    "type": "STRING"
  },
  {
    "fields": [
      {
        "name": "asset",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "asset_type",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "ancestors",
        "mode": "REPEATED",
        "type": "STRING"
      },
      {
        "name": "relationship_type",
        "mode": "NULLABLE",
        "type": "STRING"
      }
    ],
    "name": "related_asset",
    "mode": "NULLABLE",
    "type": "RECORD"
  },
  {
    "name": "ancestors",
    "mode": "REPEATED",
    "type": "STRING"
  },
  {
    "name": "update_time",
    "mode": "NULLABLE",
    "type": "TIMESTAMP"
  }
]

Export an asset snapshot

gcloud

gcloud asset export \
    --SCOPE \
    --billing-project=BILLING_PROJECT_ID \
    --asset-types=ASSET_TYPE_1,ASSET_TYPE_2,... \
    --content-type=CONTENT_TYPE \
    --relationship-types=RELATIONSHIP_TYPE_1,RELATIONSHIP_TYPE_2,... \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --output-bigquery-force

Provide the following values:

  • SCOPE: Use one of the following values:

    • project=PROJECT_ID, where PROJECT_ID is the ID of the project that has the asset metadata you want to export.

    • folder=FOLDER_ID, where FOLDER_ID is the ID of the folder that has the asset metadata you want to export.

      How to find the ID of a Google Cloud folder

      Google Cloud console

      To find the ID of a Google Cloud folder, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Search for your folder name. The folder ID is shown next to the folder name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:

      gcloud resource-manager folders list \
    --organization=$(gcloud organizations describe ORGANIZATION_NAME \
      --format="value(name.segment(1))") \
    --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
    --format="value(ID)"

      Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the --format flag to see more information about the found folders.

      The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:

      gcloud resource-manager folders list --folder=FOLDER_ID

    • organization=ORGANIZATION_ID, where ORGANIZATION_ID is the ID of the organization that has the asset metadata you want to export.

      How to find the ID of a Google Cloud organization

      Google Cloud console

      To find the ID of a Google Cloud organization, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Click the All tab. The organization ID is shown next to the organization name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud organization with the following command:

      gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project.

  • ASSET_TYPE_#: Optional. A comma-separated list of searchable asset types. RE2-compatible regular expressions are supported. If the regular expression doesn't match any supported asset type, an INVALID_ARGUMENT error is returned. When --asset-types isn't specified, all asset types are returned.
  • CONTENT_TYPE: Optional. The content type of the metadata you want to retrieve. When --content-type isn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.
  • RELATIONSHIP_TYPE_#: Optional. Only available for Security Command Center Premium and Enterprise tier subscribers. A comma-separated list of asset relationship types you want to retrieve. You must set CONTENT_TYPE to RELATIONSHIP for this to work.
  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in gcloud topic datetime format. The value must be no more than 35 days in the past. When --snapshot-time isn't specified, a snapshot is taken at the current time.
  • BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to.
  • DATASET_ID: The ID of the BigQuery dataset.
  • TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created.

The --output-bigquery-force flag overwrites the destination table if it exists.

See the gcloud CLI reference for all options.

Example

Run the following command to export your resource metadata as it was on January 30, 2024 in the my-project project, to the BigQuery table my-table.

gcloud asset export \
    --project=my-project \
    --content-type=resource \
    --snapshot-time="2024-01-30" \
    --bigquery-table=projects/my-project/datasets/my-dataset/tables/my-table \
    --output-bigquery-force

Example response

Export in progress for root asset [projects/my-project].
Use [gcloud asset operations describe projects/000000000000/operations/ExportAssets/RESOURCE/00000000000000000000000000000000] to check the status of the operation.

REST

HTTP method and URL:

POST https://cloudasset.googleapis.com/v1/SCOPE_PATH:exportAssets

Headers:

X-Goog-User-Project: BILLING_PROJECT_ID

Request JSON body:

{
  "contentType": "CONTENT_TYPE",
  "relationshipTypes": [
    "RELATIONSHIP_TYPE_1",
    "RELATIONSHIP_TYPE_2",
    "..."
  ],
  "readTime": "SNAPSHOT_TIME",
  "outputConfig": {
    "bigqueryDestination": {
      "dataset": "projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID",
      "table": "TABLE_NAME",
      "force": true
    }
  }
}

Provide the following values:

  • SCOPE_PATH: Use one of the following values:

    The allowed values are:

    • projects/PROJECT_ID, where PROJECT_ID is the ID of the project that has the asset metadata you want to export.

    • projects/PROJECT_NUMBER, where PROJECT_NUMBER is the number of the project that has the asset metadata you want to export.

      How to find a Google Cloud project number

      Google Cloud console

      To find a Google Cloud project number, complete the following steps:

      1. Go to the Welcome page in the Google Cloud console.

        Go to Welcome

      2. Click the switcher list box in the menu bar.

      3. Select your organization from the list box, and then search for your project name. The project name, project number, and project ID are shown near the Welcome heading.

        Up to 4,000 resources are displayed. If you don't see the project you're looking for, go to the Manage resources page and filter the list using the name of that project.

      gcloud CLI

      You can retrieve a Google Cloud project number with the following command:

      gcloud projects describe PROJECT_ID --format="value(projectNumber)"

    • folders/FOLDER_ID, where FOLDER_ID is the ID of the folder that has the asset metadata you want to export.

      How to find the ID of a Google Cloud folder

      Google Cloud console

      To find the ID of a Google Cloud folder, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Search for your folder name. The folder ID is shown next to the folder name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:

      gcloud resource-manager folders list \
    --organization=$(gcloud organizations describe ORGANIZATION_NAME \
      --format="value(name.segment(1))") \
    --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
    --format="value(ID)"

      Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the --format flag to see more information about the found folders.

      The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:

      gcloud resource-manager folders list --folder=FOLDER_ID

    • organizations/ORGANIZATION_ID, where ORGANIZATION_ID is the ID of the organization that has the asset metadata you want to export.

      How to find the ID of a Google Cloud organization

      Google Cloud console

      To find the ID of a Google Cloud organization, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Click the All tab. The organization ID is shown next to the organization name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud organization with the following command:

      gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

  • BILLING_PROJECT_ID: The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project.

  • CONTENT_TYPE: The content type of the metadata you want to retrieve. When contentType isn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.
  • RELATIONSHIP_TYPE_#: Optional. Only available for Security Command Center Premium and Enterprise tier subscribers. A comma-separated list of asset relationship types you want to retrieve. You must set CONTENT_TYPE to RELATIONSHIP for this to work.
  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in RFC 3339 format. The value must be no more than 35 days in the past. When readTime isn't specified, a snapshot is taken at the current time.
  • BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to.
  • DATASET_ID: The ID of the BigQuery dataset.
  • TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created.

The "force": true key-value pair overwrites the destination table if it exists.

See the REST reference for all options.

Command examples

Run one of the following commands to export your resource metadata as it was on January 30, 2024 in the my-project project, to the BigQuery table my-table.

curl (Linux, macOS, or Cloud Shell)

curl -X POST \
     -H "X-Goog-User-Project: BILLING_PROJECT_ID" \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d '{
            "contentType": "RESOURCE",
            "readTime": "2024-01-30T00:00:00Z",
            "outputConfig": {
              "bigqueryDestination": {
                "dataset": "projects/my-project/datasets/my-dataset",
                "table": "my-table",
                "force": true
              }
            }
          }' \
     https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets

PowerShell (Windows)

$cred = gcloud auth print-access-token

$headers = @{ 
  "X-Goog-User-Project" = "BILLING_PROJECT_ID";
  "Authorization" = "Bearer $cred"
}


$body = @"
{
  "contentType": "RESOURCE",
  "readTime": "2024-01-30T00:00:00Z",
  "outputConfig": {
    "bigqueryDestination": {
      "dataset": "projects/my-project/datasets/my-dataset",
      "table": "my-table",
      "force": true
    }
  }
}
"@

Invoke-WebRequest `
  -Method POST `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -Body $body `
  -Uri "https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets" | Select-Object -Expand Content

Example response

{
  "name": "projects/000000000000/operations/ExportAssets/RESOURCE/00000000000000000000000000000000",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.asset.v1.ExportAssetsRequest",
    "parent": "projects/000000000000",
    "readTime": "2024-01-30T00:00:00Z",
    "contentType": "RESOURCE",
    "outputConfig": {
      "bigqueryDestination": {
        "dataset": "projects/my-project/datasets/my-dataset",
        "table": "my-table",
        "force": true
      }
    }
  }
}

Go

To learn how to install and use the client library for Cloud Asset Inventory, see Cloud Asset Inventory client libraries.

To authenticate to Cloud Asset Inventory, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 


// Sample asset-quickstart exports assets to given bigquery table.
package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"strings"

	asset "cloud.google.com/go/asset/apiv1"
	"cloud.google.com/go/asset/apiv1/assetpb"
)

func main() {
	ctx := context.Background()
	projectID := os.Getenv("GOOGLE_CLOUD_PROJECT")
	client, err := asset.NewClient(ctx)
	if err != nil {
		log.Fatalf("asset.NewClient: %v", err)
	}
	defer client.Close()
	datasetID := strings.Replace(fmt.Sprintf("%s-for-assets", projectID), "-", "_", -1)
	dataset := fmt.Sprintf("projects/%s/datasets/%s", projectID, datasetID)
	req := &assetpb.ExportAssetsRequest{
		Parent: fmt.Sprintf("projects/%s", projectID),
		OutputConfig: &assetpb.OutputConfig{
			Destination: &assetpb.OutputConfig_BigqueryDestination{
				BigqueryDestination: &assetpb.BigQueryDestination{
					Dataset: dataset,
					Table:   "test",
					Force:   true,
				},
			},
		},
	}
	op, err := client.ExportAssets(ctx, req)
	if err != nil {
		log.Fatalf("ExportAssets: %v", err)
	}
	resp, err := op.Wait(ctx)
	if err != nil {
		log.Fatalf("Wait: %v", err)
	}
	fmt.Print(resp)
}

Java

To learn how to install and use the client library for Cloud Asset Inventory, see Cloud Asset Inventory client libraries.

To authenticate to Cloud Asset Inventory, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 

// Imports the Google Cloud client library

import com.google.cloud.ServiceOptions;
import com.google.cloud.asset.v1.AssetServiceClient;
import com.google.cloud.asset.v1.BigQueryDestination;
import com.google.cloud.asset.v1.ContentType;
import com.google.cloud.asset.v1.ExportAssetsRequest;
import com.google.cloud.asset.v1.ExportAssetsRequest.Builder;
import com.google.cloud.asset.v1.ExportAssetsResponse;
import com.google.cloud.asset.v1.OutputConfig;
import com.google.cloud.asset.v1.PartitionSpec;
import com.google.cloud.asset.v1.ProjectName;
import java.io.IOException;
import java.util.Arrays;
import java.util.concurrent.ExecutionException;

public class ExportAssetsBigqueryExample {

  // Use the default project Id.
  private static final String projectId = ServiceOptions.getDefaultProjectId();

  /** 
   * Export assets to BigQuery for a project.

   * @param bigqueryDataset which dataset the results will be exported to
   * @param bigqueryTable which table the results will be exported to
   * @param contentType determines the schema for the table
   * @param assetTypes a list of asset types to export. if empty, export all.
   * @param isPerType separate BigQuery tables for each resource type
   */
  public static void exportBigQuery(String bigqueryDataset, String bigqueryTable,
      ContentType contentType, String[] assetTypes, boolean isPerType)
      throws IOException, IllegalArgumentException, InterruptedException, ExecutionException {
    try (AssetServiceClient client = AssetServiceClient.create()) {
      ProjectName parent = ProjectName.of(projectId);
      OutputConfig outputConfig;
      // Outputs to per-type BigQuery table.
      if (isPerType) {
        outputConfig =
            OutputConfig.newBuilder()
                .setBigqueryDestination(
                    BigQueryDestination.newBuilder()
                        .setDataset(bigqueryDataset)
                        .setTable(bigqueryTable)
                        .setForce(true)
                        .setSeparateTablesPerAssetType(true)
                        .setPartitionSpec(
                            PartitionSpec.newBuilder()
                                .setPartitionKey(PartitionSpec.PartitionKey.READ_TIME)
                                .build())
                        .build())
                .build();
      } else {
        outputConfig =
            OutputConfig.newBuilder()
                .setBigqueryDestination(
                    BigQueryDestination.newBuilder()
                        .setDataset(bigqueryDataset)
                        .setTable(bigqueryTable)
                        .setForce(true)
                        .build())
                .build();
      }
      Builder exportAssetsRequestBuilder = ExportAssetsRequest.newBuilder()
          .setParent(parent.toString()).setContentType(contentType).setOutputConfig(outputConfig);
      if (assetTypes.length > 0) {
        exportAssetsRequestBuilder.addAllAssetTypes(Arrays.asList(assetTypes));
      }
      ExportAssetsRequest request = exportAssetsRequestBuilder.build();
      ExportAssetsResponse response = client.exportAssetsAsync(request).get();
      System.out.println(response);
    }
  }
}

Node.js

To learn how to install and use the client library for Cloud Asset Inventory, see Cloud Asset Inventory client libraries.

To authenticate to Cloud Asset Inventory, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 

/**
 * TODO(developer): Uncomment these variables before running the sample.
 */
// const dataSet = 'projects/project_id/datasets/dataset_id';
// const table = 'mytable';

const {AssetServiceClient} = require('@google-cloud/asset');
const client = new AssetServiceClient();

async function exportAssetsBigquery() {
  const projectId = await client.getProjectId();
  const projectResource = client.projectPath(projectId);
  const dataset = dataSet;

  const request = {
    parent: projectResource,
    outputConfig: {
      bigqueryDestination: {
        dataset: `projects/${projectId}/${dataset}`,
        table: table,
        force: true,
      },
    },
  };

  // Handle the operation using the promise pattern.
  const [operation] = await client.exportAssets(request);

  // Operation#promise starts polling for the completion of the operation.
  const [result] = await operation.promise();

  // Do things with with the response.
  console.log(result);
}

exportAssetsBigquery();

Python

To learn how to install and use the client library for Cloud Asset Inventory, see Cloud Asset Inventory client libraries.

To authenticate to Cloud Asset Inventory, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 

from google.cloud import asset_v1

# TODO project_id = 'Your Google Cloud Project ID'
# TODO dataset = 'Your BigQuery dataset path'
# TODO table = 'Your BigQuery table name'
# TODO content_type ="Content type to export"

client = asset_v1.AssetServiceClient()
parent = f"projects/{project_id}"
output_config = asset_v1.OutputConfig()
output_config.bigquery_destination.dataset = dataset
output_config.bigquery_destination.table = table
output_config.bigquery_destination.force = True
response = client.export_assets(
    request={
        "parent": parent,
        "content_type": content_type,
        "output_config": output_config,
    }
)
print(response.result())

Export an asset snapshot to separate tables for each asset type

You can export assets to separate BigQuery tables for each asset type with the --per-asset-type flag in the gcloud CLI, and "separateTablesPerAssetType": true in REST API requests.

In this mode, each table's name is TABLE_NAME concatenated with _ (underscore) and the asset type name. Non-alphanumeric characters are replaced with _.

If exporting to any table fails, the entire export operation fails and returns the first error. Results of previous successful exports persist.

The following types are packed in a JSON string to overcome a compatibility issue between Proto3 and BigQuery types.

  • google.protobuf.Timestamp

  • google.protobuf.Duration

  • google.protobuf.FieldMask

  • google.protobuf.ListValue

  • google.protobuf.Value

  • google.protobuf.Struct

  • google.api.*

gcloud

gcloud asset export \
    --SCOPE \
    --billing-project=BILLING_PROJECT_ID \
    --asset-types=ASSET_TYPE_1,ASSET_TYPE_2,... \
    --content-type=CONTENT_TYPE \
    --relationship-types=RELATIONSHIP_TYPE_1,RELATIONSHIP_TYPE_2,... \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --per-asset-type \
    --output-bigquery-force

Provide the following values:

  • SCOPE: Use one of the following values:

    • project=PROJECT_ID, where PROJECT_ID is the ID of the project that has the asset metadata you want to export.

    • folder=FOLDER_ID, where FOLDER_ID is the ID of the folder that has the asset metadata you want to export.

      How to find the ID of a Google Cloud folder

      Google Cloud console

      To find the ID of a Google Cloud folder, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Search for your folder name. The folder ID is shown next to the folder name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:

      gcloud resource-manager folders list \
    --organization=$(gcloud organizations describe ORGANIZATION_NAME \
      --format="value(name.segment(1))") \
    --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
    --format="value(ID)"

      Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the --format flag to see more information about the found folders.

      The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:

      gcloud resource-manager folders list --folder=FOLDER_ID

    • organization=ORGANIZATION_ID, where ORGANIZATION_ID is the ID of the organization that has the asset metadata you want to export.

      How to find the ID of a Google Cloud organization

      Google Cloud console

      To find the ID of a Google Cloud organization, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Click the All tab. The organization ID is shown next to the organization name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud organization with the following command:

      gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project.

  • ASSET_TYPE_#: Optional. A comma-separated list of searchable asset types. RE2-compatible regular expressions are supported. If the regular expression doesn't match any supported asset type, an INVALID_ARGUMENT error is returned. When --asset-types isn't specified, all asset types are returned.
  • CONTENT_TYPE: Optional. The content type of the metadata you want to retrieve. When --content-type isn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.
  • RELATIONSHIP_TYPE_#: Optional. Only available for Security Command Center Premium and Enterprise tier subscribers. A comma-separated list of asset relationship types you want to retrieve. You must set CONTENT_TYPE to RELATIONSHIP for this to work.
  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in gcloud topic datetime format. The value must be no more than 35 days in the past. When --snapshot-time isn't specified, a snapshot is taken at the current time.
  • BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to.
  • DATASET_ID: The ID of the BigQuery dataset.
  • TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created.

The --output-bigquery-force flag overwrites the destination table if it exists.

See the gcloud CLI reference for all options.

Example

Run the following command to export your resource metadata as it was on January 30, 2024 in the my-project project, to multiple BigQuery tables that have my-table as a prefix.

gcloud asset export \
    --project=my-project \
    --content-type=resource \
    --snapshot-time="2024-01-30" \
    --bigquery-table=projects/my-project/datasets/my-dataset/tables/my-table \
    --per-asset-type \
    --output-bigquery-force

REST

HTTP method and URL:

POST https://cloudasset.googleapis.com/v1/SCOPE_PATH:exportAssets

Headers:

X-Goog-User-Project: BILLING_PROJECT_ID

Request JSON body:

{
  "contentType": "CONTENT_TYPE",
  "relationshipTypes": [
    "RELATIONSHIP_TYPE_1",
    "RELATIONSHIP_TYPE_2",
    "..."
  ],
  "readTime": "SNAPSHOT_TIME",
  "outputConfig": {
    "bigqueryDestination": {
      "dataset": "projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID",
      "table": "TABLE_NAME",
      "force": true,
      "separateTablesPerAssetType": true
    }
  }
}

Provide the following values:

  • SCOPE_PATH: Use one of the following values:

    The allowed values are:

    • projects/PROJECT_ID, where PROJECT_ID is the ID of the project that has the asset metadata you want to export.

    • projects/PROJECT_NUMBER, where PROJECT_NUMBER is the number of the project that has the asset metadata you want to export.

      How to find a Google Cloud project number

      Google Cloud console

      To find a Google Cloud project number, complete the following steps:

      1. Go to the Welcome page in the Google Cloud console.

        Go to Welcome

      2. Click the switcher list box in the menu bar.

      3. Select your organization from the list box, and then search for your project name. The project name, project number, and project ID are shown near the Welcome heading.

        Up to 4,000 resources are displayed. If you don't see the project you're looking for, go to the Manage resources page and filter the list using the name of that project.

      gcloud CLI

      You can retrieve a Google Cloud project number with the following command:

      gcloud projects describe PROJECT_ID --format="value(projectNumber)"

    • folders/FOLDER_ID, where FOLDER_ID is the ID of the folder that has the asset metadata you want to export.

      How to find the ID of a Google Cloud folder

      Google Cloud console

      To find the ID of a Google Cloud folder, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Search for your folder name. The folder ID is shown next to the folder name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:

      gcloud resource-manager folders list \
    --organization=$(gcloud organizations describe ORGANIZATION_NAME \
      --format="value(name.segment(1))") \
    --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
    --format="value(ID)"

      Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the --format flag to see more information about the found folders.

      The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:

      gcloud resource-manager folders list --folder=FOLDER_ID

    • organizations/ORGANIZATION_ID, where ORGANIZATION_ID is the ID of the organization that has the asset metadata you want to export.

      How to find the ID of a Google Cloud organization

      Google Cloud console

      To find the ID of a Google Cloud organization, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Click the All tab. The organization ID is shown next to the organization name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud organization with the following command:

      gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

  • BILLING_PROJECT_ID: The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project.

  • CONTENT_TYPE: The content type of the metadata you want to retrieve. When contentType isn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.
  • RELATIONSHIP_TYPE_#: Optional. Only available for Security Command Center Premium and Enterprise tier subscribers. A comma-separated list of asset relationship types you want to retrieve. You must set CONTENT_TYPE to RELATIONSHIP for this to work.
  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in RFC 3339 format. The value must be no more than 35 days in the past. When readTime isn't specified, a snapshot is taken at the current time.
  • BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to.
  • DATASET_ID: The ID of the BigQuery dataset.
  • TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created.

The "force": true key-value pair overwrites the destination table if it exists.

Command examples

Run one of the following commands to export your resource metadata as it was on January 30, 2024 in the my-project project, to multiple BigQuery tables that have my-table as a prefix.

curl (Linux, macOS, or Cloud Shell)

curl -X POST \
     -H "X-Goog-User-Project: BILLING_PROJECT_ID" \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d '{
            "contentType": "RESOURCE",
            "readTime": "2024-01-30T00:00:00Z",
            "outputConfig": {
              "bigqueryDestination": {
                "dataset": "projects/my-project/datasets/my-dataset",
                "table": "my-table",
                "force": true,
                "separateTablesPerAssetType": true
              }
            }
          }' \
     https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets

PowerShell (Windows)

$cred = gcloud auth print-access-token

$headers = @{ 
  "X-Goog-User-Project" = "BILLING_PROJECT_ID";
  "Authorization" = "Bearer $cred"
}


$body = @"
{
  "contentType": "RESOURCE",
  "readTime": "2024-01-30T00:00:00Z",
  "outputConfig": {
    "bigqueryDestination": {
      "dataset": "projects/my-project/datasets/my-dataset",
      "table": "my-table",
      "force": true,
      "separateTablesPerAssetType": true
    }
  }
}
"@

Invoke-WebRequest `
  -Method POST `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -Body $body `
  -Uri "https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets" | Select-Object -Expand Content

Export an asset snapshot to a time-unit column partitioned table

You can export assets in a project to time-unit column partitioned tables. The exported snapshot is stored in a BigQuery table named TABLE_NAME with daily granularity and two additional timestamp columns, readTime and requestTime, one of which you specify as the partition column with the PARTITION_KEY value.

To export assets in a project to partitioned tables, make one the following requests.

gcloud

gcloud asset export \
    --SCOPE \
    --billing-project=BILLING_PROJECT_ID \
    --asset-types=ASSET_TYPE_1,ASSET_TYPE_2,... \
    --content-type=CONTENT_TYPE \
    --relationship-types=RELATIONSHIP_TYPE_1,RELATIONSHIP_TYPE_2,... \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --partition-key=PARTITION_KEY \
    --output-bigquery-force

Provide the following values:

  • SCOPE: Use one of the following values:

    • project=PROJECT_ID, where PROJECT_ID is the ID of the project that has the asset metadata you want to export.

    • folder=FOLDER_ID, where FOLDER_ID is the ID of the folder that has the asset metadata you want to export.

      How to find the ID of a Google Cloud folder

      Google Cloud console

      To find the ID of a Google Cloud folder, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Search for your folder name. The folder ID is shown next to the folder name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:

      gcloud resource-manager folders list \
    --organization=$(gcloud organizations describe ORGANIZATION_NAME \
      --format="value(name.segment(1))") \
    --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
    --format="value(ID)"

      Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the --format flag to see more information about the found folders.

      The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:

      gcloud resource-manager folders list --folder=FOLDER_ID

    • organization=ORGANIZATION_ID, where ORGANIZATION_ID is the ID of the organization that has the asset metadata you want to export.

      How to find the ID of a Google Cloud organization

      Google Cloud console

      To find the ID of a Google Cloud organization, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Click the All tab. The organization ID is shown next to the organization name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud organization with the following command:

      gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project.

  • ASSET_TYPE_#: Optional. A comma-separated list of searchable asset types. RE2-compatible regular expressions are supported. If the regular expression doesn't match any supported asset type, an INVALID_ARGUMENT error is returned. When --asset-types isn't specified, all asset types are returned.
  • CONTENT_TYPE: Optional. The content type of the metadata you want to retrieve. When --content-type isn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.
  • RELATIONSHIP_TYPE_#: Optional. Only available for Security Command Center Premium and Enterprise tier subscribers. A comma-separated list of asset relationship types you want to retrieve. You must set CONTENT_TYPE to RELATIONSHIP for this to work.
  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in gcloud topic datetime format. The value must be no more than 35 days in the past. When --snapshot-time isn't specified, a snapshot is taken at the current time.
  • BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to.
  • DATASET_ID: The ID of the BigQuery dataset.
  • TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created.

  • PARTITION_KEY: The partition key column when exporting to BigQuery partitioned tables. Valid values areread-time and request-time.

The --output-bigquery-force flag overwrites data in the corresponding partition in the destination table. Data in different partitions remains intact.

If --output-bigquery-force isn't specified, exported data is appended to the corresponding partition.

The export operation fails if a schema update or attempt to append data fails. This includes if the destination table already exists and doesn't have the schema the export expects.

See the gcloud CLI reference for all options.

Example

Run the following command to export your resource metadata as it was on January 30, 2024 in the my-project project, to the BigQuery table my-table.

gcloud asset export \
    --project=projects/my-project \
    --content-type=resource \
    --snapshot-time="2024-01-30" \
    --bigquery-table=projects/my-project/datasets/my-dataset/tables/my-table \
    --partition-key=my-partition-key \
    --output-bigquery-force

REST

HTTP method and URL:

POST https://cloudasset.googleapis.com/v1/SCOPE_PATH:exportAssets

Headers:

X-Goog-User-Project: BILLING_PROJECT_ID

Request JSON body:

{
  "contentType": "CONTENT_TYPE",
  "relationshipTypes": [
    "RELATIONSHIP_TYPE_1",
    "RELATIONSHIP_TYPE_2",
    "..."
  ],
  "readTime": "SNAPSHOT_TIME",
  "outputConfig": {
    "bigqueryDestination": {
      "dataset": "projects/BIGQUERY_PROJECT_ID/datasets/DATASET_ID",
      "table": "TABLE_NAME",
      "partitionSpec": {
        "partitionKey": "PARTITION_KEY"
      },
      "force": true,
    }
  }
}

Provide the following values:

  • SCOPE_PATH: Use one of the following values:

    The allowed values are:

    • projects/PROJECT_ID, where PROJECT_ID is the ID of the project that has the asset metadata you want to export.

    • projects/PROJECT_NUMBER, where PROJECT_NUMBER is the number of the project that has the asset metadata you want to export.

      How to find a Google Cloud project number

      Google Cloud console

      To find a Google Cloud project number, complete the following steps:

      1. Go to the Welcome page in the Google Cloud console.

        Go to Welcome

      2. Click the switcher list box in the menu bar.

      3. Select your organization from the list box, and then search for your project name. The project name, project number, and project ID are shown near the Welcome heading.

        Up to 4,000 resources are displayed. If you don't see the project you're looking for, go to the Manage resources page and filter the list using the name of that project.

      gcloud CLI

      You can retrieve a Google Cloud project number with the following command:

      gcloud projects describe PROJECT_ID --format="value(projectNumber)"

    • folders/FOLDER_ID, where FOLDER_ID is the ID of the folder that has the asset metadata you want to export.

      How to find the ID of a Google Cloud folder

      Google Cloud console

      To find the ID of a Google Cloud folder, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Search for your folder name. The folder ID is shown next to the folder name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud folder that's located at the organization level with the following command:

      gcloud resource-manager folders list \
    --organization=$(gcloud organizations describe ORGANIZATION_NAME \
      --format="value(name.segment(1))") \
    --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
    --format="value(ID)"

      Where TOP_LEVEL_FOLDER_NAME is a partial or full string match for the folder's name. Remove the --format flag to see more information about the found folders.

      The previous command doesn't return the IDs of subfolders within folders. To do so, run the following command using a top level folder's ID:

      gcloud resource-manager folders list --folder=FOLDER_ID

    • organizations/ORGANIZATION_ID, where ORGANIZATION_ID is the ID of the organization that has the asset metadata you want to export.

      How to find the ID of a Google Cloud organization

      Google Cloud console

      To find the ID of a Google Cloud organization, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher list box in the menu bar.
      3. Select your organization from the list box.
      4. Click the All tab. The organization ID is shown next to the organization name.

      gcloud CLI

      You can retrieve the ID of a Google Cloud organization with the following command:

      gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

  • BILLING_PROJECT_ID: The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about setting the billing project.

  • CONTENT_TYPE: The content type of the metadata you want to retrieve. When contentType isn't specified, only basic information is returned, such as asset names, the last time the assets were updated, and what projects, folders, and organizations they belong to.
  • RELATIONSHIP_TYPE_#: Optional. Only available for Security Command Center Premium and Enterprise tier subscribers. A comma-separated list of asset relationship types you want to retrieve. You must set CONTENT_TYPE to RELATIONSHIP for this to work.
  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in RFC 3339 format. The value must be no more than 35 days in the past. When readTime isn't specified, a snapshot is taken at the current time.
  • BIGQUERY_PROJECT_ID: The ID of the project that the BigQuery table is in that you want to export to.
  • DATASET_ID: The ID of the BigQuery dataset.
  • TABLE_NAME: The BigQuery table you're exporting your metadata to. If it doesn't exist, it's created.

  • PARTITION_KEY: The partition key column when exporting to BigQuery partitioned tables. Valid values areREAD_TIME and REQUEST_TIME.

The "force": true key-value pair overwrites data in the corresponding partition in the destination table. Data in different partitions remains intact.

If force isn't set or set to false, exported data is appended to the corresponding partition.

The export operation fails if a schema update or attempt to append data fails. This includes if the destination table already exists and doesn't have the schema the export expects.

Command examples

Run one of the following commands to export your resource metadata as it was on January 30, 2024 in the my-project project, to the BigQuery table my-table.

curl (Linux, macOS, or Cloud Shell)

curl -X POST \
     -H "X-Goog-User-Project: BILLING_PROJECT_ID" \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d '{
            "contentType": "RESOURCE",
            "readTime": "2024-01-30T00:00:00Z",
            "outputConfig": {
              "bigqueryDestination": {
                "dataset": "projects/my-project/datasets/my-dataset",
                "table": "my-table",
                "partitionSpec": {
                  "partitionKey": "my-partition-key"
                },
                "force": true,
              }
            }
          }' \
     https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets

PowerShell (Windows)

$cred = gcloud auth print-access-token

$headers = @{ 
  "X-Goog-User-Project" = "BILLING_PROJECT_ID";
  "Authorization" = "Bearer $cred"
}


$body = @"
{
  "contentType": "RESOURCE",
  "readTime": "2024-01-30T00:00:00Z",
  "outputConfig": {
    "bigqueryDestination": {
      "dataset": "projects/my-project/datasets/my-dataset",
      "table": "my-table",
      "partitionSpec": {
        "partitionKey": "my-partition-key"
      },
      "force": true,
    }
  }
}
"@

Invoke-WebRequest `
  -Method POST `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -Body $body `
  -Uri "https://cloudasset.googleapis.com/v1/projects/my-project:exportAssets" | Select-Object -Expand Content

Check the status of an export

Exports take time to complete. To check if an export is done, you can query the operation using its operation ID.

Be aware that even if your export is done, someone might have made another export request to the same destination as a different operation. New export requests to the same destination can be made after a previous request has finished, or if more than 15 minutes has elapsed. Export requests made outside of these conditions are rejected by Cloud Asset Inventory.

gcloud

To view the status of your export, complete the following instructions:

  1. Get the OPERATION_PATH, which includes the operation ID, from the response to your export request. The OPERATION_PATH is shown in the response to the export, which is formatted as follows:

    projects/PROJECT_NUMBER/operations/ExportAssets/CONTENT_TYPE/OPERATION_ID

  2. To check the status of your export, run following command with the OPERATION_PATH:

    gcloud asset operations describe OPERATION_PATH

REST

To view the status of your export, complete the following instructions:

  1. Get the OPERATION_PATH, which includes the operation ID, from the response to your export request. The OPERATION_PATH is shown as the value of the name field in the response to the export, which is formatted as follows:

    projects/PROJECT_NUMBER/operations/ExportAssets/CONTENT_TYPE/OPERATION_ID

  2. To check the status of your export, make the following request.

    REST

    HTTP method and URL:

    GET https://cloudasset.googleapis.com/v1/OPERATION_PATH

    Command examples

    curl (Linux, macOS, or Cloud Shell)

    curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     https://cloudasset.googleapis.com/v1/OPERATION_PATH

    PowerShell (Windows)

    $cred = gcloud auth print-access-token

$headers = @{ 
  "Authorization" = "Bearer $cred"
}


Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://cloudasset.googleapis.com/v1/OPERATION_PATH" | Select-Object -Expand Content

View an asset snapshot in BigQuery

Console

  1. Go to the BigQuery Studio page in the Google Cloud console.

    Go to BigQuery Studio

  2. To display the tables and views in the dataset, open the navigation panel. In the Resources section, select your project to expand it, and then select a dataset.

  3. From the list, select your table.

  4. Select Details and note the value in Number of rows. You might need this value to control the starting point for your results using the gcloud CLI or REST API.

  5. To view a sample set of data, select Preview.

REST

To browse your table's data, call tabledata.list. In the tableId parameter, specify the name of your table.

You can configure the following optional parameters to control the output.

  • maxResults is the maximum number of results to return.

  • selectedFields is a comma-separated list of columns to return; If unspecified, all columns are returned.

  • startIndex is the zero-based index of the starting row to read.

Values are returned wrapped in a JSON object that you must parse, as described in the tabledata.list reference documentation.

Query an asset snapshot in BigQuery

After you export your snapshot to BigQuery, you can run queries on your asset metadata.

By default, BigQuery runs interactive, or on-demand, query jobs, which means that the query is executed as soon as possible. Interactive queries count towards your concurrent rate limit and your daily limit.

Query results are saved to either a temporary or permanent table. You can choose to append or overwrite data in an existing table or to create a new table, if none exists with the same name.

To run an interactive query that writes the output to a temporary table, complete the following steps.

Console

  1. Go to the BigQuery Studio page in the Google Cloud console.

    Go to BigQuery Studio

  2. Select Compose new query.

  3. In the Query editor text area, enter a valid BigQuery SQL query.

  4. Optional: To change the data processing location, complete the following steps.

    1. Select More, and then select Query settings.

    2. Under Processing location, select Auto-select, and then choose your data's location.

    3. To update the query settings, select Save.

  5. Select Run.

REST

  1. To start a new job, call the jobs.insert method. In the job resource, set the following parameters.

    • In the configuration field, set the query field to a JobConfigurationQuery that describes the BigQuery query job.

    • In the jobReference field, set the location field appropriately for your job.

  2. To poll for results, call getQueryResults. Poll until jobComplete equals true. You can check for errors and warnings in the errors list.

Additional SQL query examples

This section provides example SQL queries to analyze your asset metadata after you've exported it to BigQuery. See standard SQL query syntax for more information.

Query available columns directly

To find the quantity of each asset type, run the following query:

SELECT asset_type, COUNT(*) AS asset_count
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`
GROUP BY asset_type
ORDER BY asset_count DESC

Work with repeated fields

To find Identity and Access Management (IAM) policies that grant access to Gmail accounts, run the following query. BigQuery uses UNNEST to flatten repeated fields into a table that you can query directly:

SELECT name, asset_type, bindings.role
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`
JOIN UNNEST(iam_policy.bindings) AS bindings
JOIN UNNEST(bindings.members) AS principals
WHERE principals like "%@gmail.com"

To find an organization, folder, or project that allows creating using a public IP address, run the following query. This query is useful because allowing public IP addresses with Cloud SQL instances can introduce vulnerabilities unless SSL or a proxy is configured:

SELECT name
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`
JOIN UNNEST(org_policy) AS op
WHERE
  op.constraint = "constraints/sql.restrictPublicIp"
  AND (op.boolean_policy IS NULL OR op.boolean_policy.enforced = FALSE);

To find an organization, folder, or project in the same VPC Service Controls service perimeter of a project, run the following query:

SELECT service_perimeter.title, service_perimeter.status.resources
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME`
CROSS JOIN UNNEST(service_perimeter.status.resources) as resource
WHERE resource = "projects/PROJECT_ID";

Work with JSON strings

To find open firewall rules, run the following query. Learn more about JSON functions used in BigQuery.

CREATE TEMP FUNCTION json2array(json STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS """
  return JSON.parse(json).map(x=>JSON.stringify(x));
""";

SELECT firewall.name, firewall.resource.parent, JSON_EXTRACT(firewall.resource.data, '$.sourceRanges') AS sourceRanges
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME` AS firewall
JOIN UNNEST(json2array(JSON_EXTRACT(firewall.resource.data, '$.sourceRanges'))) AS source_ranges
WHERE asset_type="compute.googleapis.com/Firewall" AND JSON_EXTRACT(firewall.resource.data, '$.sourceRanges') IS NOT NULL AND JSON_EXTRACT_SCALAR(source_ranges, '$') = "0.0.0.0/0"

By separating tables for each resource type, you can find open firewall rules with an easier and faster query.

SELECT firewall.name, firewall.resource.parent, sourceRanges
FROM `PROJECT_ID.DATASET_ID.STRUCTURED_INSTANCE_TABLE_NAME` AS firewall
JOIN UNNEST(firewall.resource.data.sourceRanges) AS sourceRanges
WHERE sourceRanges = "0.0.0.0/0";

Join tables of different resource types

To join tables of different resource types, run the following query. The following example shows how to find all subnetworks that have no VM attached. First, the query finds all subnetworks. Then, from that list, it selects those subnetworks whose selfLink values are not present.

CREATE TEMP FUNCTION json2array(json STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS """
  return JSON.parse(json).map(x=>JSON.stringify(x));
""";

SELECT name, JSON_EXTRACT(subnetwork.resource.data, '$.selfLink') AS selflink
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME` AS subnetwork
WHERE asset_type = "compute.googleapis.com/Subnetwork" AND (JSON_EXTRACT(subnetwork.resource.data, '$.selfLink') NOT IN
(SELECT DISTINCT JSON_EXTRACT(network_interfaces, '$.subnetwork')
FROM `PROJECT_ID.DATASET_ID.TABLE_NAME` as instance
JOIN UNNEST(json2array(JSON_EXTRACT(instance.resource.data, '$.networkInterfaces'))) AS network_interfaces
WHERE asset_type ="compute.googleapis.com/Instance"
AND JSON_EXTRACT(instance.resource.data, '$.networkInterfaces') IS NOT NULL
)) IS NULL

By separating tables for each resource type, you can find all subnetworks that have no VM attached with an easier and faster query.

SELECT name, subnetwork.resource.data.selfLink
FROM `PROJECT_ID.DATASET_ID.STRUCTURED_SUBNETWORK_TABLE_NAME` AS subnetwork
WHERE
  (
    subnetwork.resource.data.selfLink
      NOT IN (
        SELECT DISTINCT networkInterface.subnetwork
        FROM `PROJECT_ID.DATASET_ID.STRUCTURED_INSTANCE_TABLE_NAME` as instance
        JOIN
          UNNEST(instance.resource.data.networkInterfaces) AS networkInterface
        WHERE
          networkInterface IS NOT NULL
      )
  ) IS NULL;

Find vulnerable Dataproc clusters due to CVE-2021-44228

CREATE TEMP FUNCTION vulnerable_version(imageVersion STRING)
RETURNS BOOL
LANGUAGE js AS r"""
    const version_regexp = /(?<major>\d+)(?:\.)(?<minor>\d+)(?:\.)?(?<sub>\d+)?/g;
    let match = version_regexp.exec(imageVersion);
    if(match.groups.major < 1){
        return true;
    }
    if (match.groups.major == 1){
        if (match.groups.minor < 3){
            return true;
        }
        if(match.groups.minor == 3 &amp;&amp; match.groups.sub < 95){
            return true;
        }
        if(match.groups.minor == 4 &amp;&amp; match.groups.sub < 77){
            return true;
        }
        if(match.groups.minor == 5 &amp;&amp; match.groups.sub < 53){
            return true;
        }
    }
    if (match.groups.major == 2 &amp;&amp; match.groups.minor == 0 &amp;&amp; match.groups.sub < 27){
        return true;
    }
    return false;
""";

SELECT
  c.name,
  c.resource.data.projectId AS project_id,
  c.resource.data.clusterName AS cluster_name,
  c.resource.data.config.softwareConfig.imageVersion AS image_version,
  c.resource.data.status.state AS cluster_state,
  vulnerable_version(c.resource.data.config.softwareConfig.imageVersion) AS is_vulnerable
FROM
  `PROJECT_ID.DATASET_ID.TABLE_NAME_PREFIX_dataproc_googleapis_com_Cluster` c
  INNER JOIN `PROJECT_ID.DATASET_ID.TABLE_NAME_PREFIX_cloudresourcemanager_googleapis_com_Project` p
    ON p.resource.data.projectId = c.resource.data.projectId
WHERE
  c.resource.data.config.softwareConfig.imageVersion IS NOT NULL
  AND c.resource.data.status.state = "RUNNING"
  AND p.resource.data.lifecycleState = "ACTIVE";