Exporting to BigQuery

This topic shows you how to export the asset metadata for your organization, folder, or project to a BigQuery table, and then run data analysis on your inventory. BigQuery provides a SQL-like experience for users to analyze data and produce meaningful insights without the use of custom scripts.

Before you begin

Before you begin, complete the following steps.

  1. Enable the Cloud Asset Inventory API on the project where you'll be running the API commands.

    Enable the Cloud Asset Inventory API

  2. Configure the permissions that are required to call the Cloud Asset Inventory API using either the gcloud CLI or the API.

  3. Complete the following steps to set up your environment.

    gcloud CLI

    To set up your environment to use the gcloud CLI to call the Cloud Asset Inventory API, install the Google Cloud CLI on your local client.

    REST

    To set up your environment to call the Cloud Asset Inventory API with the curl command, complete the following steps.

    1. Confirm that you have access to the curl command.

    2. Ensure that you grant your account one of the following roles on your project, folder, or organization.

      • Cloud Asset Viewer role (roles/cloudasset.viewer)

      • Owner basic role (roles/owner)

  4. If you are exporting metadata from one project to another, make sure the exporting project's default Cloud Asset Inventory service account exists and has the correct permissions.

  5. Create a BigQuery dataset.

Limitations

When exporting an asset snapshot, keep the following things in mind:

  • BigQuery tables encrypted with custom Cloud Key Management Service (Cloud KMS) keys are not supported.

  • Appending the export output to an existing table is not supported unless you are exporting to a partitioned table. The destination table must be empty or you must overwrite it. To overwrite it, use the --output-bigquery-force flag with the gcloud CLI, or use force with the REST API.

  • Google Kubernetes Engine (GKE) resource types, except for container.googleapis.com/Cluster and container.googleapis.com/NodePool, are not supported when exporting to separate tables per resource type.

  • If the table you're exporting to already exists and is already being exported to, a 400 error is returned.

Setting the BigQuery schema for the export

Every BigQuery table is defined by a schema that describes the column names, data types, and other information. Setting the content type during the export determines the schema for your table.

  • Resource or unspecified: When you set the content type to RESOURCE or do not specify it, and you set the per-asset-type flag to false or do not use it, you create a BigQuery table that has the following schema.

    Resource schema

    [
      {
        "name": "name",
        "type": "STRING",
        "mode": "NULLABLE"
      },
      {
        "name": "asset_type",
        "type": "STRING",
        "mode": "NULLABLE"
      },
      {
        "name": "resource",
        "type": "RECORD",
        "mode": "NULLABLE",
        "fields": [
          {
            "name": "version",
            "type": "STRING",
            "mode": "NULLABLE"
          },
          {
            "name": "discovery_document_uri",
            "type": "STRING",
            "mode": "NULLABLE"
          },
          {
            "name": "discovery_name",
            "type": "STRING",
            "mode": "NULLABLE"
          },
          {
            "name": "resource_url",
            "type": "STRING",
            "mode": "NULLABLE"
          },
          {
            "name": "parent",
            "type": "STRING",
            "mode": "NULLABLE"
          },
          {
            "name": "data",
            "type": "STRING",
            "mode": "NULLABLE"
          },
          {
            "name": "location",
            "type": "STRING",
            "mode": "NULLABLE"
          }
        ]
      },
      {
        "name": "ancestors",
        "type": "STRING",
        "mode": "REPEATED"
      },
      {
        "name": "update_time",
        "type": "TIMESTAMP",
        "mode": "NULLABLE"
      }
    ]

    The resource.data column is the resource metadata represented as a JSON string.

    When you set the content type to RESOURCE or do not set the content type, and set the per-asset-type flag to true, you create separate tables per asset type. The schema of each table includes RECORD-type columns mapped to the nested fields in the Resource.data field of that asset type (up to the 15 nested levels that BigQuery supports). For per-type BigQuery example tables, see projects/export-assets-examples/datasets/structured_export.

  • IAM policy: When you set the content type to IAM_POLICY in the REST API or iam-policy in the gcloud CLI, you create a BigQuery table that has the following schema.

    IAM policy schema

    [
      {
        "name": "name",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "asset_type",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "name": "version",
            "mode": "NULLABLE",
            "type": "INTEGER"
          },
          {
            "fields": [
              {
                "name": "role",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "members",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "fields": [
                  {
                    "name": "expression",
                    "mode": "NULLABLE",
                    "type": "STRING"
                  },
                  {
                    "name": "title",
                    "mode": "NULLABLE",
                    "type": "STRING"
                  },
                  {
                    "name": "description",
                    "mode": "NULLABLE",
                    "type": "STRING"
                  },
                  {
                    "name": "location",
                    "mode": "NULLABLE",
                    "type": "STRING"
                  }
                ],
                "name": "condition",
                "mode": "NULLABLE",
                "type": "RECORD"
              }
            ],
            "name": "bindings",
            "mode": "REPEATED",
            "type": "RECORD"
          },
          {
            "fields": [
              {
                "name": "service",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "fields": [
                  {
                    "name": "log_type",
                    "mode": "NULLABLE",
                    "type": "INTEGER"
                  },
                  {
                    "name": "exempted_members",
                    "mode": "REPEATED",
                    "type": "STRING"
                  }
                ],
                "name": "audit_log_configs",
                "mode": "REPEATED",
                "type": "RECORD"
              }
            ],
            "name": "audit_configs",
            "mode": "REPEATED",
            "type": "RECORD"
          },
          {
            "name": "etag",
            "mode": "NULLABLE",
            "type": "STRING"
          }
        ],
        "name": "iam_policy",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "name": "ancestors",
        "mode": "REPEATED",
        "type": "STRING"
      },
      {
        "name": "update_time",
        "mode": "NULLABLE",
        "type": "TIMESTAMP"
      }
    ]
  • Organization policy: When you set the content type to ORG_POLICY in the REST API or org-policy in the gcloud CLI, you create a BigQuery table that has the following schema.

    Organization policy schema

    [
      {
        "name": "name",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "asset_type",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "name": "version",
            "mode": "NULLABLE",
            "type": "INTEGER"
          },
          {
            "name": "constraint",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "etag",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "update_time",
            "mode": "NULLABLE",
            "type": "TIMESTAMP"
          },
          {
            "fields": [
              {
                "name": "allowed_values",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "name": "denied_values",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "name": "all_values",
                "mode": "NULLABLE",
                "type": "INTEGER"
              },
              {
                "name": "suggested_value",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "inherit_from_parent",
                "mode": "NULLABLE",
                "type": "BOOLEAN"
              }
            ],
            "name": "list_policy",
            "mode": "NULLABLE",
            "type": "RECORD"
          },
          {
            "fields": [
              {
                "name": "enforced",
                "mode": "NULLABLE",
                "type": "BOOLEAN"
              }
            ],
            "name": "boolean_policy",
            "mode": "NULLABLE",
            "type": "RECORD"
          },
          {
            "fields": [
              {
                "name": "_present",
                "mode": "NULLABLE",
                "type": "BOOLEAN"
              }
            ],
            "name": "restore_default",
            "mode": "NULLABLE",
            "type": "RECORD"
          }
        ],
        "name": "org_policy",
        "mode": "REPEATED",
        "type": "RECORD"
      },
      {
        "name": "ancestors",
        "mode": "REPEATED",
        "type": "STRING"
      },
      {
        "name": "update_time",
        "mode": "NULLABLE",
        "type": "TIMESTAMP"
      }
    ]
  • VPCSC policy: When you set content type to ACCESS_POLICY in the REST API or access-policy in the gcloud CLI, you create a BigQuery table that has the following schema.

    VPCSC policy schema

    [
      {
        "name": "name",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "asset_type",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "name": "name",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "parent",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "title",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "scopes",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "name": "etag",
            "mode": "NULLABLE",
            "type": "STRING"
          }
        ],
        "name": "access_policy",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "name",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "title",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "description",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "fields": [
              {
                "fields": [
                  {
                    "name": "ip_subnetworks",
                    "mode": "REPEATED",
                    "type": "STRING"
                  },
                  {
                    "fields": [
                      {
                        "name": "require_screenlock",
                        "mode": "NULLABLE",
                        "type": "BOOLEAN"
                      },
                      {
                        "name": "allowed_encryption_statuses",
                        "mode": "REPEATED",
                        "type": "INTEGER"
                      },
                      {
                        "fields": [
                          {
                            "name": "os_type",
                            "mode": "NULLABLE",
                            "type": "INTEGER"
                          },
                          {
                            "name": "minimum_version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "require_verified_chrome_os",
                            "mode": "NULLABLE",
                            "type": "BOOLEAN"
                          }
                        ],
                        "name": "os_constraints",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      },
                      {
                        "name": "allowed_device_management_levels",
                        "mode": "REPEATED",
                        "type": "INTEGER"
                      },
                      {
                        "name": "require_admin_approval",
                        "mode": "NULLABLE",
                        "type": "BOOLEAN"
                      },
                      {
                        "name": "require_corp_owned",
                        "mode": "NULLABLE",
                        "type": "BOOLEAN"
                      }
                    ],
                    "name": "device_policy",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "name": "required_access_levels",
                    "mode": "REPEATED",
                    "type": "STRING"
                  },
                  {
                    "name": "negate",
                    "mode": "NULLABLE",
                    "type": "BOOLEAN"
                  },
                  {
                    "name": "members",
                    "mode": "REPEATED",
                    "type": "STRING"
                  },
                  {
                    "name": "regions",
                    "mode": "REPEATED",
                    "type": "STRING"
                  }
                ],
                "name": "conditions",
                "mode": "REPEATED",
                "type": "RECORD"
              },
              {
                "name": "combining_function",
                "mode": "NULLABLE",
                "type": "INTEGER"
              }
            ],
            "name": "basic",
            "mode": "NULLABLE",
            "type": "RECORD"
          },
          {
            "fields": [
              {
                "fields": [
                  {
                    "name": "expression",
                    "mode": "NULLABLE",
                    "type": "STRING"
                  },
                  {
                    "name": "title",
                    "mode": "NULLABLE",
                    "type": "STRING"
                  },
                  {
                    "name": "description",
                    "mode": "NULLABLE",
                    "type": "STRING"
                  },
                  {
                    "name": "location",
                    "mode": "NULLABLE",
                    "type": "STRING"
                  }
                ],
                "name": "expr",
                "mode": "NULLABLE",
                "type": "RECORD"
              }
            ],
            "name": "custom",
            "mode": "NULLABLE",
            "type": "RECORD"
          }
        ],
        "name": "access_level",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "name",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "title",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "description",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "perimeter_type",
            "mode": "NULLABLE",
            "type": "INTEGER"
          },
          {
            "fields": [
              {
                "name": "resources",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "name": "access_levels",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "name": "restricted_services",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "fields": [
                  {
                    "name": "enable_restriction",
                    "mode": "NULLABLE",
                    "type": "BOOLEAN"
                  },
                  {
                    "name": "allowed_services",
                    "mode": "REPEATED",
                    "type": "STRING"
                  }
                ],
                "name": "vpc_accessible_services",
                "mode": "NULLABLE",
                "type": "RECORD"
              },
              {
                "fields": [
                  {
                    "fields": [
                      {
                        "fields": [
                          {
                            "name": "access_level",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "resource",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "sources",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      },
                      {
                        "name": "identities",
                        "mode": "REPEATED",
                        "type": "STRING"
                      },
                      {
                        "name": "identity_type",
                        "mode": "NULLABLE",
                        "type": "INTEGER"
                      }
                    ],
                    "name": "ingress_from",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "fields": [
                          {
                            "name": "service_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "fields": [
                              {
                                "name": "method",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              },
                              {
                                "name": "permission",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              }
                            ],
                            "name": "method_selectors",
                            "mode": "REPEATED",
                            "type": "RECORD"
                          }
                        ],
                        "name": "operations",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      },
                      {
                        "name": "resources",
                        "mode": "REPEATED",
                        "type": "STRING"
                      }
                    ],
                    "name": "ingress_to",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  }
                ],
                "name": "ingress_policies",
                "mode": "REPEATED",
                "type": "RECORD"
              },
              {
                "fields": [
                  {
                    "fields": [
                      {
                        "name": "identities",
                        "mode": "REPEATED",
                        "type": "STRING"
                      },
                      {
                        "name": "identity_type",
                        "mode": "NULLABLE",
                        "type": "INTEGER"
                      }
                    ],
                    "name": "egress_from",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "resources",
                        "mode": "REPEATED",
                        "type": "STRING"
                      },
                      {
                        "fields": [
                          {
                            "name": "service_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "fields": [
                              {
                                "name": "method",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              },
                              {
                                "name": "permission",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              }
                            ],
                            "name": "method_selectors",
                            "mode": "REPEATED",
                            "type": "RECORD"
                          }
                        ],
                        "name": "operations",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      },
                      {
                        "name": "external_resources",
                        "mode": "REPEATED",
                        "type": "STRING"
                      }
                    ],
                    "name": "egress_to",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  }
                ],
                "name": "egress_policies",
                "mode": "REPEATED",
                "type": "RECORD"
              }
            ],
            "name": "status",
            "mode": "NULLABLE",
            "type": "RECORD"
          },
          {
            "fields": [
              {
                "name": "resources",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "name": "access_levels",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "name": "restricted_services",
                "mode": "REPEATED",
                "type": "STRING"
              },
              {
                "fields": [
                  {
                    "name": "enable_restriction",
                    "mode": "NULLABLE",
                    "type": "BOOLEAN"
                  },
                  {
                    "name": "allowed_services",
                    "mode": "REPEATED",
                    "type": "STRING"
                  }
                ],
                "name": "vpc_accessible_services",
                "mode": "NULLABLE",
                "type": "RECORD"
              },
              {
                "fields": [
                  {
                    "fields": [
                      {
                        "fields": [
                          {
                            "name": "access_level",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "resource",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "sources",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      },
                      {
                        "name": "identities",
                        "mode": "REPEATED",
                        "type": "STRING"
                      },
                      {
                        "name": "identity_type",
                        "mode": "NULLABLE",
                        "type": "INTEGER"
                      }
                    ],
                    "name": "ingress_from",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "fields": [
                          {
                            "name": "service_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "fields": [
                              {
                                "name": "method",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              },
                              {
                                "name": "permission",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              }
                            ],
                            "name": "method_selectors",
                            "mode": "REPEATED",
                            "type": "RECORD"
                          }
                        ],
                        "name": "operations",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      },
                      {
                        "name": "resources",
                        "mode": "REPEATED",
                        "type": "STRING"
                      }
                    ],
                    "name": "ingress_to",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  }
                ],
                "name": "ingress_policies",
                "mode": "REPEATED",
                "type": "RECORD"
              },
              {
                "fields": [
                  {
                    "fields": [
                      {
                        "name": "identities",
                        "mode": "REPEATED",
                        "type": "STRING"
                      },
                      {
                        "name": "identity_type",
                        "mode": "NULLABLE",
                        "type": "INTEGER"
                      }
                    ],
                    "name": "egress_from",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "name": "resources",
                        "mode": "REPEATED",
                        "type": "STRING"
                      },
                      {
                        "fields": [
                          {
                            "name": "service_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "fields": [
                              {
                                "name": "method",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              },
                              {
                                "name": "permission",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              }
                            ],
                            "name": "method_selectors",
                            "mode": "REPEATED",
                            "type": "RECORD"
                          }
                        ],
                        "name": "operations",
                        "mode": "REPEATED",
                        "type": "RECORD"
                      },
                      {
                        "name": "external_resources",
                        "mode": "REPEATED",
                        "type": "STRING"
                      }
                    ],
                    "name": "egress_to",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  }
                ],
                "name": "egress_policies",
                "mode": "REPEATED",
                "type": "RECORD"
              }
            ],
            "name": "spec",
            "mode": "NULLABLE",
            "type": "RECORD"
          },
          {
            "name": "use_explicit_dry_run_spec",
            "mode": "NULLABLE",
            "type": "BOOLEAN"
          }
        ],
        "name": "service_perimeter",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "name": "ancestors",
        "mode": "REPEATED",
        "type": "STRING"
      },
      {
        "name": "update_time",
        "mode": "NULLABLE",
        "type": "TIMESTAMP"
      }
    ]
  • OSConfig instance inventory: When you set content type to OS_INVENTORY in the REST API or os-inventory in the gcloud CLI, you create a BigQuery table that has the following schema.

    OS inventory schema

    [
      {
        "name": "name",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "asset_type",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "name": "name",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "fields": [
              {
                "name": "hostname",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "long_name",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "short_name",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "version",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "architecture",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "kernel_version",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "kernel_release",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "name": "osconfig_agent_version",
                "mode": "NULLABLE",
                "type": "STRING"
              }
            ],
            "name": "os_info",
            "mode": "NULLABLE",
            "type": "RECORD"
          },
          {
            "fields": [
              {
                "name": "key",
                "mode": "NULLABLE",
                "type": "STRING"
              },
              {
                "fields": [
                  {
                    "name": "id",
                    "mode": "NULLABLE",
                    "type": "STRING"
                  },
                  {
                    "name": "origin_type",
                    "mode": "NULLABLE",
                    "type": "INTEGER"
                  },
                  {
                    "name": "create_time",
                    "mode": "NULLABLE",
                    "type": "TIMESTAMP"
                  },
                  {
                    "name": "update_time",
                    "mode": "NULLABLE",
                    "type": "TIMESTAMP"
                  },
                  {
                    "name": "type",
                    "mode": "NULLABLE",
                    "type": "INTEGER"
                  },
                  {
                    "fields": [
                      {
                        "fields": [
                          {
                            "name": "package_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "architecture",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "yum_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "package_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "architecture",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "apt_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "package_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "architecture",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "zypper_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "package_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "architecture",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "googet_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "patch_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "category",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "severity",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "summary",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "zypper_patch",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "title",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "description",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "fields": [
                              {
                                "name": "id",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              },
                              {
                                "name": "name",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              }
                            ],
                            "name": "categories",
                            "mode": "REPEATED",
                            "type": "RECORD"
                          },
                          {
                            "name": "kb_article_ids",
                            "mode": "REPEATED",
                            "type": "STRING"
                          },
                          {
                            "name": "support_url",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "more_info_urls",
                            "mode": "REPEATED",
                            "type": "STRING"
                          },
                          {
                            "name": "update_id",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "revision_number",
                            "mode": "NULLABLE",
                            "type": "INTEGER"
                          },
                          {
                            "name": "last_deployment_change_time",
                            "mode": "NULLABLE",
                            "type": "TIMESTAMP"
                          }
                        ],
                        "name": "wua_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "caption",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "description",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "hot_fix_id",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "install_time",
                            "mode": "NULLABLE",
                            "type": "TIMESTAMP"
                          }
                        ],
                        "name": "qfe_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "package_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "architecture",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "cos_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "display_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "display_version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "publisher",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "fields": [
                              {
                                "name": "year",
                                "mode": "NULLABLE",
                                "type": "INTEGER"
                              },
                              {
                                "name": "month",
                                "mode": "NULLABLE",
                                "type": "INTEGER"
                              },
                              {
                                "name": "day",
                                "mode": "NULLABLE",
                                "type": "INTEGER"
                              }
                            ],
                            "name": "install_date",
                            "mode": "NULLABLE",
                            "type": "RECORD"
                          },
                          {
                            "name": "help_link",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "windows_application",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      }
                    ],
                    "name": "installed_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  },
                  {
                    "fields": [
                      {
                        "fields": [
                          {
                            "name": "package_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "architecture",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "yum_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "package_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "architecture",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "apt_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "package_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "architecture",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "zypper_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "package_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "architecture",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "googet_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "patch_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "category",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "severity",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "summary",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "zypper_patch",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "title",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "description",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "fields": [
                              {
                                "name": "id",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              },
                              {
                                "name": "name",
                                "mode": "NULLABLE",
                                "type": "STRING"
                              }
                            ],
                            "name": "categories",
                            "mode": "REPEATED",
                            "type": "RECORD"
                          },
                          {
                            "name": "kb_article_ids",
                            "mode": "REPEATED",
                            "type": "STRING"
                          },
                          {
                            "name": "support_url",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "more_info_urls",
                            "mode": "REPEATED",
                            "type": "STRING"
                          },
                          {
                            "name": "update_id",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "revision_number",
                            "mode": "NULLABLE",
                            "type": "INTEGER"
                          },
                          {
                            "name": "last_deployment_change_time",
                            "mode": "NULLABLE",
                            "type": "TIMESTAMP"
                          }
                        ],
                        "name": "wua_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "caption",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "description",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "hot_fix_id",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "install_time",
                            "mode": "NULLABLE",
                            "type": "TIMESTAMP"
                          }
                        ],
                        "name": "qfe_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "package_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "architecture",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "cos_package",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      },
                      {
                        "fields": [
                          {
                            "name": "display_name",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "display_version",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "name": "publisher",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          },
                          {
                            "fields": [
                              {
                                "name": "year",
                                "mode": "NULLABLE",
                                "type": "INTEGER"
                              },
                              {
                                "name": "month",
                                "mode": "NULLABLE",
                                "type": "INTEGER"
                              },
                              {
                                "name": "day",
                                "mode": "NULLABLE",
                                "type": "INTEGER"
                              }
                            ],
                            "name": "install_date",
                            "mode": "NULLABLE",
                            "type": "RECORD"
                          },
                          {
                            "name": "help_link",
                            "mode": "NULLABLE",
                            "type": "STRING"
                          }
                        ],
                        "name": "windows_application",
                        "mode": "NULLABLE",
                        "type": "RECORD"
                      }
                    ],
                    "name": "available_package",
                    "mode": "NULLABLE",
                    "type": "RECORD"
                  }
                ],
                "name": "value",
                "mode": "NULLABLE",
                "type": "RECORD"
              }
            ],
            "name": "items",
            "mode": "REPEATED",
            "type": "RECORD"
          },
          {
            "name": "update_time",
            "mode": "NULLABLE",
            "type": "TIMESTAMP"
          }
        ],
        "name": "os_inventory",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "name": "ancestors",
        "mode": "REPEATED",
        "type": "STRING"
      },
      {
        "name": "update_time",
        "mode": "NULLABLE",
        "type": "TIMESTAMP"
      }
    ]
  • Relationship: When you set content type to RELATIONSHIP in the REST API or relationship in the gcloud CLI, you create a BigQuery table that has the following schema.

    Relationship schema

    [
      {
        "name": "name",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "name": "asset_type",
        "mode": "NULLABLE",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "name": "asset",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "asset_type",
            "mode": "NULLABLE",
            "type": "STRING"
          },
          {
            "name": "ancestors",
            "mode": "REPEATED",
            "type": "STRING"
          },
          {
            "name": "relationship_type",
            "mode": "NULLABLE",
            "type": "STRING"
          }
        ],
        "name": "related_asset",
        "mode": "NULLABLE",
        "type": "RECORD"
      },
      {
        "name": "ancestors",
        "mode": "REPEATED",
        "type": "STRING"
      },
      {
        "name": "update_time",
        "mode": "NULLABLE",
        "type": "TIMESTAMP"
      }
    ]

Export an asset snapshot to BigQuery

Exporting assets at a given timestamp

To export an asset snapshot at a given timestamp to a BigQuery table named TABLE_NAME, make one the following requests.

gcloud

Projects

gcloud asset export \
    --project=PROJECT_ID \
    --billing-project=BILLING_PROJECT_ID \
    --content-type=CONTENT_TYPE \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --output-bigquery-force

Provide the following values:

  • PROJECT_ID: The ID of the project whose metadata you want to export.

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Not required if you've already switched to the billing project with gcloud config set project. Read more about billing projects.

  • CONTENT_TYPE: The asset content type. The access-policy content type can only be exported for an organization. If you export it for a project or folder, an empty table is created.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time. For information on time formats, see gcloud topic datetimes.

  • DATASET_ID: The ID of the BigQuery dataset.

  • TABLE_NAME: The table you're exporting your metadata to. If it doesn't exist, it's created.

Other flags:

  • --output-bigquery-force: Overwrites the destination table if it exists.

Folders

gcloud asset export \
    --folder=FOLDER_ID \
    --billing-project=BILLING_PROJECT_ID \
    --content-type=CONTENT_TYPE \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --output-bigquery-force

Provide the following values:

  • FOLDER_ID: The ID of the folder whose metadata you want to export.

    How to find a Google Cloud folder ID

    Console

    To find a Google Cloud folder ID, complete the following steps:

    1. Go to the Google Cloud console.

      Go to the Google Cloud console

    2. Click the switcher box in the menu bar.
    3. Click the Select from box, and then select your organization.
    4. Search for your folder name. The folder ID is shown next to the folder name.

    gcloud CLI

    You can retrieve a Google Cloud folder ID that's located at the organization level with the following command:

    gcloud resource-manager folders list \
        --organization=$(gcloud organizations describe ORGANIZATION_NAME \
          --format="value(name.segment(1))") \
        --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
        --format="value(ID)"

    Where TOP_LEVEL_FOLDER_NAME can be a full or partial string match. Remove the --format option to see further information about the found folders.

    To get the ID of a folder within another folder, list the subfolders:

    gcloud resource-manager folders list --folder=FOLDER_ID

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Not required if you've already switched to the billing project with gcloud config set project. Read more about billing projects.

  • CONTENT_TYPE: The asset content type. The access-policy content type can only be exported for an organization. If you export it for a project or folder, an empty table is created.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time. For information on time formats, see gcloud topic datetimes.

  • PROJECT_ID: The ID of the project that the BigQuery table is in.

  • DATASET_ID: The ID of the BigQuery dataset.

  • TABLE_NAME: The table you're exporting your metadata to. If it doesn't exist, it's created.

Other flags:

  • --output-bigquery-force: Overwrites the destination table if it exists.

Organizations

gcloud asset export \
    --organization=ORGANIZATION_ID \
    --billing-project=BILLING_PROJECT_ID \
    --content-type=CONTENT_TYPE \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --output-bigquery-force

Provide the following values:

  • ORGANIZATION_ID: The ID of the organization whose metadata you want to export.

    How to find a Google Cloud organization ID

    Console

    To find a Google Cloud organization ID, complete the following steps:

    1. Go to the Google Cloud console.

      Go to the Google Cloud console

    2. Click the switcher box in the menu bar.
    3. Click the Select from box, and then select your organization.
    4. Click the All tab. The organization ID is shown next to the organization name.

    gcloud CLI

    You can retrieve a Google Cloud organization ID with the following command:

    gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Not required if you've already switched to the billing project with gcloud config set project. Read more about billing projects.

  • CONTENT_TYPE: The asset content type.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time. For information on time formats, see gcloud topic datetimes.

  • PROJECT_ID: The ID of the project that the BigQuery table is in.

  • DATASET_ID: The ID of the BigQuery dataset.

  • TABLE_NAME: The table you're exporting your metadata to. If it doesn't exist, it's created.

Other flags:

  • --output-bigquery-force: Overwrites the destination table if it exists.

REST

curl -X POST \
     -H "X-Goog-User-Project: BILLING_PROJECT_ID" \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json" \
     -d '{
          "contentType": "CONTENT_TYPE",
          "readTime": "SNAPSHOT_TIME",
          "outputConfig": {
            "bigqueryDestination": {
              "dataset": "projects/PROJECT_ID/datasets/DATASET_ID",
              "table": "TABLE_NAME",
              "force": true
            }
          }
         }' \
     https://cloudasset.googleapis.com/v1/SCOPE:exportAssets

Provide the following values:

  • BILLING_PROJECT_ID: The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about billing projects.

  • CONTENT_TYPE: The asset content type. The ACCESS_POLICY content type can only be exported for an organization. If you export it for a project or folder, an empty table is created.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in RFC 3339 format. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time.

  • PROJECT_ID: The ID of the project that the BigQuery table is in.

  • DATASET_ID The ID of the BigQuery dataset.

  • TABLE_NAME The table you're exporting your metadata to. If it doesn't exist, it's created.

  • SCOPE: A scope can be a project, a folder, or an organization.

    The allowed values are:

    • projects/PROJECT_ID

    • projects/PROJECT_NUMBER

      How to find a Google Cloud project number

      Console

      To find a Google Cloud project number, complete the following steps:

      1. Go to the Dashboard page in the Google Cloud console.

        Go to Dashboard

      2. Click the switcher box in the menu bar.
      3. Select your organization from the Select from box, and then search for your project name.
      4. Click the project name to switch to that project. The project number is shown in the Project info card.

      gcloud CLI

      You can retrieve a Google Cloud project number with the following command:

      gcloud projects describe PROJECT_ID --format="value(projectNumber)"

    • folders/FOLDER_ID

      How to find a Google Cloud folder ID

      Console

      To find a Google Cloud folder ID, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher box in the menu bar.
      3. Click the Select from box, and then select your organization.
      4. Search for your folder name. The folder ID is shown next to the folder name.

      gcloud CLI

      You can retrieve a Google Cloud folder ID that's located at the organization level with the following command:

      gcloud resource-manager folders list \
          --organization=$(gcloud organizations describe ORGANIZATION_NAME \
            --format="value(name.segment(1))") \
          --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
          --format="value(ID)"

      Where TOP_LEVEL_FOLDER_NAME can be a full or partial string match. Remove the --format option to see further information about the found folders.

      To get the ID of a folder within another folder, list the subfolders:

      gcloud resource-manager folders list --folder=FOLDER_ID

    • organizations/ORGANIZATION_ID

      How to find a Google Cloud organization ID

      Console

      To find a Google Cloud organization ID, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher box in the menu bar.
      3. Click the Select from box, and then select your organization.
      4. Click the All tab. The organization ID is shown next to the organization name.

      gcloud CLI

      You can retrieve a Google Cloud organization ID with the following command:

      gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

Other parameters:

  • "force": true: Overwrites the destination table if it exists.

Exporting separate tables for each resource type

To export assets to separate BigQuery tables for each resource type, use the --per-asset-type flag. Each table's name is TABLE_NAME concatenated with _ (underscore) and the asset type name. Non-alphanumeric characters are replaced with _.

Note that GKE resource types, except for container.googleapis.com/Cluster and container.googleapis.com/NodePool, are not supported for this type of export.

gcloud

Projects

gcloud asset export \
    --project=PROJECT_ID \
    --billing-project=BILLING_PROJECT_ID \
    --content-type=CONTENT_TYPE \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --output-bigquery-force \
    --per-asset-type

Provide the following values:

  • PROJECT_ID: The ID of the project whose metadata you want to export.

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Not required if you've already switched to the billing project with gcloud config set project. Read more about billing projects.

  • CONTENT_TYPE: The asset content type. The access-policy content type can only be exported for an organization. If you export it for a project or folder, an empty table is created.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time. For information on time formats, see gcloud topic datetimes.

  • DATASET_ID: The ID of the BigQuery dataset.

  • TABLE_NAME: The table you're exporting your metadata to. If it doesn't exist, it's created.

Other flags:

  • --output-bigquery-force: Overwrites the destination table if it exists.

  • --per-asset-type: Exports to multiple BigQuery tables per resource type.

Folders

gcloud asset export \
    --folder=FOLDER_ID \
    --billing-project=BILLING_PROJECT_ID \
    --content-type=CONTENT_TYPE \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --output-bigquery-force \
    --per-asset-type

Provide the following values:

  • FOLDER_ID: The ID of the folder whose metadata you want to export.

    How to find a Google Cloud folder ID

    Console

    To find a Google Cloud folder ID, complete the following steps:

    1. Go to the Google Cloud console.

      Go to the Google Cloud console

    2. Click the switcher box in the menu bar.
    3. Click the Select from box, and then select your organization.
    4. Search for your folder name. The folder ID is shown next to the folder name.

    gcloud CLI

    You can retrieve a Google Cloud folder ID that's located at the organization level with the following command:

    gcloud resource-manager folders list \
        --organization=$(gcloud organizations describe ORGANIZATION_NAME \
          --format="value(name.segment(1))") \
        --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
        --format="value(ID)"

    Where TOP_LEVEL_FOLDER_NAME can be a full or partial string match. Remove the --format option to see further information about the found folders.

    To get the ID of a folder within another folder, list the subfolders:

    gcloud resource-manager folders list --folder=FOLDER_ID

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Not required if you've already switched to the billing project with gcloud config set project. Read more about billing projects.

  • CONTENT_TYPE: The asset content type. The access-policy content type can only be exported for an organization. If you export it for a project or folder, an empty table is created.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time. For information on time formats, see gcloud topic datetimes.

  • PROJECT_ID: The ID of the project that the BigQuery table is in.

  • DATASET_ID: The ID of the BigQuery dataset.

  • TABLE_NAME: The table you're exporting your metadata to. If it doesn't exist, it's created.

Other flags:

  • --output-bigquery-force: Overwrites the destination table if it exists.

  • --per-asset-type: Exports to multiple BigQuery tables per resource type.

Organizations

gcloud asset export \
    --organization=ORGANIZATION_ID \
    --billing-project=BILLING_PROJECT_ID \
    --content-type=CONTENT_TYPE \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --output-bigquery-force \
    --per-asset-type

Provide the following values:

  • ORGANIZATION_ID: The ID of the organization you want insights for.

    How to find a Google Cloud organization ID

    Console

    To find a Google Cloud organization ID, complete the following steps:

    1. Go to the Google Cloud console.

      Go to the Google Cloud console

    2. Click the switcher box in the menu bar.
    3. Click the Select from box, and then select your organization.
    4. Click the All tab. The organization ID is shown next to the organization name.

    gcloud CLI

    You can retrieve a Google Cloud organization ID with the following command:

    gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Not required if you've already switched to the billing project with gcloud config set project. Read more about billing projects.

  • CONTENT_TYPE: The asset content type.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time. For information on time formats, see gcloud topic datetimes.

  • PROJECT_ID: The ID of the project that the BigQuery table is in.

  • DATASET_ID: The ID of the BigQuery dataset.

  • TABLE_NAME: The table you're exporting your metadata to. If it doesn't exist, it's created.

Other flags:

  • --output-bigquery-force: Overwrites the destination table if it exists.

  • --per-asset-type: Exports to multiple BigQuery tables per resource type.

REST

curl -X POST \
     -H "X-Goog-User-Project: BILLING_PROJECT_ID" \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json" \
     -d '{
          "contentType": "CONTENT_TYPE",
          "readTime": "SNAPSHOT_TIME",
          "outputConfig": {
            "bigqueryDestination": {
              "dataset": "projects/PROJECT_ID/datasets/DATASET_ID",
              "table": "TABLE_NAME",
              "force": true,
              "separateTablesPerAssetType": true
            }
          }
         }' \
     https://cloudasset.googleapis.com/v1/SCOPE:exportAssets

Provide the following values:

  • BILLING_PROJECT_ID: The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about billing projects.

  • CONTENT_TYPE: The asset content type. The ACCESS_POLICY content type can only be exported for an organization. If you export it for a project or folder, an empty table is created.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in RFC 3339 format. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time.

  • PROJECT_ID: The ID of the project that the BigQuery table is in.

  • DATASET_ID: The ID of the BigQuery dataset.

  • TABLE_NAME: The table you're exporting your metadata to. If it doesn't exist, it's created.

  • SCOPE: A scope can be a project, a folder, or an organization.

    The allowed values are:

    • projects/PROJECT_ID

    • projects/PROJECT_NUMBER

      How to find a Google Cloud project number

      Console

      To find a Google Cloud project number, complete the following steps:

      1. Go to the Dashboard page in the Google Cloud console.

        Go to Dashboard

      2. Click the switcher box in the menu bar.
      3. Select your organization from the Select from box, and then search for your project name.
      4. Click the project name to switch to that project. The project number is shown in the Project info card.

      gcloud CLI

      You can retrieve a Google Cloud project number with the following command:

      gcloud projects describe PROJECT_ID --format="value(projectNumber)"

    • folders/FOLDER_ID

      How to find a Google Cloud folder ID

      Console

      To find a Google Cloud folder ID, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher box in the menu bar.
      3. Click the Select from box, and then select your organization.
      4. Search for your folder name. The folder ID is shown next to the folder name.

      gcloud CLI

      You can retrieve a Google Cloud folder ID that's located at the organization level with the following command:

      gcloud resource-manager folders list \
          --organization=$(gcloud organizations describe ORGANIZATION_NAME \
            --format="value(name.segment(1))") \
          --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
          --format="value(ID)"

      Where TOP_LEVEL_FOLDER_NAME can be a full or partial string match. Remove the --format option to see further information about the found folders.

      To get the ID of a folder within another folder, list the subfolders:

      gcloud resource-manager folders list --folder=FOLDER_ID

    • organizations/ORGANIZATION_ID

      How to find a Google Cloud organization ID

      Console

      To find a Google Cloud organization ID, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher box in the menu bar.
      3. Click the Select from box, and then select your organization.
      4. Click the All tab. The organization ID is shown next to the organization name.

      gcloud CLI

      You can retrieve a Google Cloud organization ID with the following command:

      gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

Other parameters:

  • "force": true: Overwrites the destination table if it exists.

If exporting to any table fails, the entire export operation fails and returns the first error. Results of previous successful exports persist.

The following types are packed in a JSON string to overcome a compatibility issue between JSON3 and BigQuery types.

  • google.protobuf.Timestamp

  • google.protobuf.Duration

  • google.protobuf.FieldMask

  • google.protobuf.ListValue

  • google.protobuf.Value

  • google.protobuf.Struct

  • google.api.*

Exporting to a partitioned table

To export assets in a project to partitioned tables, define the partition key in the export request. The exported snapshot is stored in a BigQuery table named TABLE_NAME with daily granularity and two additional timestamp columns, readTime and requestTime, one of which is specified by PARTITION_KEY value.

To export assets in a project to partitioned tables, make one the following requests.

gcloud

Projects

gcloud asset export \
    --project=PROJECT_ID \
    --billing-project=BILLING_PROJECT_ID \
    --content-type=CONTENT_TYPE \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --partition-key=PARTITION_KEY \
    --output-bigquery-force

Provide the following values:

  • PROJECT_ID: The ID of the project whose metadata you want to export.

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Not required if you've already switched to the billing project with gcloud config set project. Read more about billing projects.

  • CONTENT_TYPE: The asset content type. The access-policy content type can only be exported for an organization. If you export it for a project or folder, an empty table is created.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time. For information on time formats, see gcloud topic datetimes.

  • DATASET_ID: The ID of the BigQuery dataset.

  • TABLE_NAME: The table to export your metadata to. If it doesn't exist, it's created.

  • PARTITION_KEY: The partition key column when exporting to BigQuery partitioned tables. Valid values are read-time and request-time.

Other flags:

  • --output-bigquery-force: Overwrites the destination table if it exists.

Folders

gcloud asset export \
    --folder=FOLDER_ID \
    --billing-project=BILLING_PROJECT_ID \
    --content-type=CONTENT_TYPE \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --partition-key=PARTITION_KEY \
    --output-bigquery-force

Provide the following values:

  • FOLDER_ID: The ID of the folder whose metadata you want to export.

    How to find a Google Cloud folder ID

    Console

    To find a Google Cloud folder ID, complete the following steps:

    1. Go to the Google Cloud console.

      Go to the Google Cloud console

    2. Click the switcher box in the menu bar.
    3. Click the Select from box, and then select your organization.
    4. Search for your folder name. The folder ID is shown next to the folder name.

    gcloud CLI

    You can retrieve a Google Cloud folder ID that's located at the organization level with the following command:

    gcloud resource-manager folders list \
        --organization=$(gcloud organizations describe ORGANIZATION_NAME \
          --format="value(name.segment(1))") \
        --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
        --format="value(ID)"

    Where TOP_LEVEL_FOLDER_NAME can be a full or partial string match. Remove the --format option to see further information about the found folders.

    To get the ID of a folder within another folder, list the subfolders:

    gcloud resource-manager folders list --folder=FOLDER_ID

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Not required if you've already switched to the billing project with gcloud config set project. Read more about billing projects.

  • CONTENT_TYPE: The asset content type. The access-policy content type can only be exported for an organization. If you export it for a project or folder, an empty table is created.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time. For information on time formats, see gcloud topic datetimes.

  • PROJECT_ID: The ID of the project that the BigQuery table is in.

  • DATASET_ID: The ID of the BigQuery dataset.

  • TABLE_NAME: The table you're exporting your metadata to. If it doesn't exist, it's created.

  • PARTITION_KEY: The partition key column when exporting to BigQuery partitioned tables. Valid values are read-time and request-time.

Other flags:

  • --output-bigquery-force: Overwrites the destination table if it exists.

Organization

gcloud asset export \
    --organization=ORGANIZATION_ID \
    --billing-project=BILLING_PROJECT_ID \
    --content-type=CONTENT_TYPE \
    --snapshot-time="SNAPSHOT_TIME" \
    --bigquery-table=projects/PROJECT_ID/datasets/DATASET_ID/tables/TABLE_NAME \
    --partition-key=PARTITION_KEY \
    --output-bigquery-force

Provide the following values:

  • ORGANIZATION_ID: The ID of the organization you want insights for.

    How to find a Google Cloud organization ID

    Console

    To find a Google Cloud organization ID, complete the following steps:

    1. Go to the Google Cloud console.

      Go to the Google Cloud console

    2. Click the switcher box in the menu bar.
    3. Click the Select from box, and then select your organization.
    4. Click the All tab. The organization ID is shown next to the organization name.

    gcloud CLI

    You can retrieve a Google Cloud organization ID with the following command:

    gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

  • BILLING_PROJECT_ID: Optional. The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Not required if you've already switched to the billing project with gcloud config set project. Read more about billing projects.

  • CONTENT_TYPE: The asset content type.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time. For information on time formats, see gcloud topic datetimes.

  • PROJECT_ID: The ID of the project that the BigQuery table is in.

  • DATASET_ID: The ID of the BigQuery dataset.

  • TABLE_NAME: The table you're exporting your metadata to. If it doesn't exist, it's created.

  • PARTITION_KEY: The partition key column when exporting to BigQuery partitioned tables. Valid values are read-time and request-time.

Other flags:

  • --output-bigquery-force: Overwrites the destination table if it exists.

If the output-bigquery-force flag is set to true, the corresponding partition is overwritten by the snapshot results, however data in one or more different partitions remains intact. If output-bigquery-force is unset or false, it appends the data to the corresponding partition.

The export operation fails if a schema update or attempt to append data fails. This includes if the destination table already exists and doesn't have the schema the export expects.

REST

curl -X POST \
     -H "X-Goog-User-Project: BILLING_PROJECT_ID" \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json" \
     -d '{
          "contentType": "CONTENT_TYPE",
          "readTime": "SNAPSHOT_TIME",
          "outputConfig": {
            "bigqueryDestination": {
              "dataset": "projects/PROJECT_ID/datasets/DATASET_ID",
              "table": "TABLE_NAME",
              "force": true,
              "partitionSpec": {
                "partitionKey": "PARTITION_KEY"
              }
            }
          }
         }' \
     https://cloudasset.googleapis.com/v1/SCOPE:exportAssets

Provide the following values:

  • BILLING_PROJECT_ID: The project ID that the default Cloud Asset Inventory service agent is in that has permissions to manage your BigQuery datasets and tables. Read more about billing projects.

  • PROJECT_ID: The ID of the project that the BigQuery table is in.

  • CONTENT_TYPE: The asset content type. The ACCESS_POLICY content type can only be exported for an organization. If you export it for a project or folder, an empty table is created.

  • SNAPSHOT_TIME: Optional. The time at which you want to take a snapshot of your assets, in RFC 3339 format. The value must be the current time or a time no more than 35 days in the past. When not provided, a snapshot is taken at the current time.

  • DATASET_ID: Is the ID of the BigQuery dataset.

  • TABLE_NAME: Is the table you're exporting your metadata to. If it doesn't exist, it's created.

  • PARTITION_KEY: Is the partition key column when exporting to BigQuery partitioned tables. Valid values are read-time and request-time.

  • SCOPE: A scope can be a project, a folder, or an organization.

    The allowed values are:

    • projects/PROJECT_ID

    • projects/PROJECT_NUMBER

      How to find a Google Cloud project number

      Console

      To find a Google Cloud project number, complete the following steps:

      1. Go to the Dashboard page in the Google Cloud console.

        Go to Dashboard

      2. Click the switcher box in the menu bar.
      3. Select your organization from the Select from box, and then search for your project name.
      4. Click the project name to switch to that project. The project number is shown in the Project info card.

      gcloud CLI

      You can retrieve a Google Cloud project number with the following command:

      gcloud projects describe PROJECT_ID --format="value(projectNumber)"

    • folders/FOLDER_ID

      How to find a Google Cloud folder ID

      Console

      To find a Google Cloud folder ID, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher box in the menu bar.
      3. Click the Select from box, and then select your organization.
      4. Search for your folder name. The folder ID is shown next to the folder name.

      gcloud CLI

      You can retrieve a Google Cloud folder ID that's located at the organization level with the following command:

      gcloud resource-manager folders list \
          --organization=$(gcloud organizations describe ORGANIZATION_NAME \
            --format="value(name.segment(1))") \
          --filter='"DISPLAY_NAME":"TOP_LEVEL_FOLDER_NAME"' \
          --format="value(ID)"

      Where TOP_LEVEL_FOLDER_NAME can be a full or partial string match. Remove the --format option to see further information about the found folders.

      To get the ID of a folder within another folder, list the subfolders:

      gcloud resource-manager folders list --folder=FOLDER_ID

    • organizations/ORGANIZATION_ID

      How to find a Google Cloud organization ID

      Console

      To find a Google Cloud organization ID, complete the following steps:

      1. Go to the Google Cloud console.

        Go to the Google Cloud console

      2. Click the switcher box in the menu bar.
      3. Click the Select from box, and then select your organization.
      4. Click the All tab. The organization ID is shown next to the organization name.

      gcloud CLI

      You can retrieve a Google Cloud organization ID with the following command:

      gcloud organizations describe ORGANIZATION_NAME --format="value(name.segment(1))"

Other parameters:

  • "force": true: Overwrites the destination table if it exists. This only works if the table has the same partition structure expected by the export.

    If force is set to true, the corresponding partition is overwritten by the snapshot results, while data in different partitions remains intact. If force is unset or set to false, the data is appended to the corresponding partition.

The export operation fails if a schema update or attempt to append data fails. This includes if the destination table already exists and doesn't have the schema the export expects.

Checking the status of an export

Operations are associated with an operation ID, which is a UUID. This value is potentially sensitive, as the gcloud asset operations describe command requires no additional permissions to run successfully. Ensure that you only share the operation ID with trusted users.

The server rejects requests if a previous request to the same destination started less than 15 minutes ago and is still running. Undefined results can occur if the export time is longer than 15 minutes and a consecutive request to the same destination is executed before the first request succeeds.

To check the status of an export, run the following commands.

gcloud CLI

To check the status of the export, you can run the following command. The OPERATION_PATH is displayed in the response after running the export command.

gcloud asset operations describe OPERATION_PATH

REST

To view the status of your export, run the following command with the operation ID returned in the response to your export.

  1. You can find the OPERATION_PATH in the name field of the response to the export, which is formatted as follows:

    "name": "projects/PROJECT_NUMBER/operations/ExportAssets/CONTENT_TYPE/OPERATION_ID"
    
  2. To check the status of your export, run following command with the OPERATION_PATH:

    curl -X GET \
         -H "Authorization: Bearer $(gcloud auth print-access-token)" \
         -H "Content-Type: application/json" \
         https://cloudasset.googleapis.com/v1/OPERATION_PATH
    

Viewing an asset snapshot

To view the table containing the asset snapshot metadata, complete the following steps.

Console

  1. Go to the BigQuery Studio page in the Google Cloud console.

    Go to BigQuery Studio

  2. To display the tables and views in the dataset, open the navigation panel. In the Resources section, select your project to expand it, and then select a dataset.

  3. From the list, select your table.

  4. Select Details and note the value in Number of rows. You might need this value to control the starting point for your results using the gcloud CLI or REST API.

  5. To view a sample set of data, select Preview.

REST

To browse your table's data, call tabledata.list. In the tableId parameter, specify the name of your table.

You can configure the following optional parameters to control the output.

  • maxResults is the maximum number of results to return.

  • selectedFields is a comma-separated list of columns to return; If unspecified, all columns are returned.

  • startIndex is the zero-based index of the starting row to read.

Values are returned wrapped in a JSON object that you must parse, as described in the tabledata.list reference documentation.

The export lists the assets and their resource names.

Querying an asset snapshot

After you export your snapshot to BigQuery, you can run queries on your asset metadata. See Exporting to BigQuery Sample Queries to learn more about several typical use cases.

By default, BigQuery runs interactive, or on-demand, query jobs, which means that the query is executed as soon as possible. Interactive queries count towards your concurrent rate limit and your daily limit.

Query results are saved to either a temporary or permanent table. You can choose to append or overwrite data in an existing table or to create a new table, if none exists with the same name.

To run an interactive query that writes the output to a temporary table, complete the following steps.

Console

  1. Go to the BigQuery Studio page in the Google Cloud console.

    Go to BigQuery Studio

  2. Select Compose new query.

  3. In the Query editor text area, enter a valid BigQuery SQL query.

  4. Optional: To change the data processing location, complete the following steps.

    1. Select More, and then select Query settings.

    2. Under Processing location, select Auto-select, and then choose your data's location.

    3. To update the query settings, select Save.

  5. Select Run.

REST

  1. To start a new job, call the jobs.insert method. In the job resource, set the following parameters.

    • In the configuration field, set the query field to a JobConfigurationQuery that describes the BigQuery query job.

    • In the jobReference field, set the location field appropriately for your job.

  2. To poll for results, call getQueryResults. Poll until jobComplete equals true. You can check for errors and warnings in the errors list.