Exporting asset metadata to Google Cloud Storage

This page explains how to export all the asset metadata of your project to a Cloud Storage bucket.

Before you begin

gcloud

  1. Enable the Cloud Asset API before you can use the gcloud tool to access Cloud Asset Inventory. Note that the API only needs to be enabled on the project you'll be running Cloud Asset API commands from.
    Enable the Cloud Asset Inventory API
  2. Install the Cloud SDK on your local client.
  3. Create a Cloud Storage bucket to store the exported snapshot.

api

  1. Enable the Cloud Asset API before you can use the gcloud tool to access Cloud Asset Inventory. Note that the API only needs to be enabled on the project you'll be running Cloud Asset API commands from.
    Enable the Cloud Asset Inventory API
  2. Install oauth2l on your local machine for interacting with the Google OAuth system.
  3. Confirm that you have access to the Unix curl command.
  4. Ensure that you grant one of the following roles on your project, folder, or organization.
    • roles/owner
    • roles/cloudasset.viewer
  5. Set up a Cloud Storage bucket to store exported snapshots.

Initial setup

To set up your environment to call the Cloud Asset API:

gcloud

Set up the gcloud command-line tool

To get started with the gcloud tool, review the Cloud SDK Documentation. You can get help for the tool, resources, and commands by using the --help flag:

gcloud asset --help

The help displayed with the --help flag is also available in the Cloud SDK reference for gcloud asset.

Configure an account

To call the Cloud Asset API, you need to configure either a user account or a service account.

Configuring a user account

  1. Log in with your user account using the following command. 

    gcloud auth login USER_ACCOUNT_EMAIL

  2. Optional. If the target project you want to call the Cloud Asset API on isn't the same as your Cloud Asset Inventory enabled project, specify your project with the following command. 

    gcloud asset --billing-project PROJECT_ID

  3. Grant your user account the cloudasset.viewer Cloud IAM role on the project whose metadata you want to export. This project can be the same as your Cloud Asset API enabled project. 

    gcloud projects add-iam-policy-binding EXPORT_TARGET_PROJECT_ID \
       --member user:USER_ACCOUNT_EMAIL \
       --role roles/cloudasset.viewer

Configuring a service account

This service account should be created for the project you're running Cloud Asset API commands from.

  1. If you don't already have a service account, in the project that is Cloud Asset API enabled, create a new service account with the following command. 

    gcloud iam service-accounts create SERVICE_ACCOUNT_NAME \
       --display-name "SERVICE_ACCOUNT_DISPLAY_NAME"

  2. Create a private key for your service account. 

    gcloud iam service-accounts keys create YOUR_FILE_PATH/key.json \
       --iam-account SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com

  3. Activate your service account for use with the gcloud tool with the following command. 

    gcloud auth activate-service-account SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com \
       --key-file=YOUR_FILE_PATH/key.json

  4. Grant your new service account the cloudasset.viewer Cloud IAM role on a project whose metadata you want to export. This project can be the same as your Cloud Asset API enabled project. 

    gcloud projects add-iam-policy-binding EXPORT_TARGET_PROJECT_ID \
       --member serviceAccount:SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com \
       --role roles/cloudasset.viewer

api

Downloading the credentials file

A JSON credentials file is needed to call the Cloud Asset API. Download the file by following the process below.

  1. Go to the Credentials page.

  2. Open the Create Credentials dropdown and select OAuth client ID.

  3. If you are creating a Client ID for a new project, you must set up the OAuth consent screen. The consent screen is displayed any time an application using your Client ID requests access to private data. If prompted:

    1. Click Configure consent screen and enter in the required information for your consent screen.

    2. Save your changes to return to creating your Client ID.

  4. On the Create client ID page under Application type, select Other.

  5. Enter a name for the credential, then click Create. A confirmation dialog appears with a client ID and client secret.

  6. Close the confirmation dialog and click the download icon on the right to save your new Client ID JSON file.

  7. Name and move the downloaded JSON file so that the path is ~/credentials.json.

Pereparing your environment

Prepare you environment for making calls to the Cloud Asset API by following the process below.

  1. Verify your initial setup with the following command.

    oauth2l header --json ~/credentials.json cloud-platform

    You should see an output similar to the following:

    Authorization: Bearer y29.xxxxxxx

  2. Define a shell alias to call Google REST APIs from with the following command.

    alias gcurl='curl -H "$(oauth2l header --json ~/credentials.json \
cloud-platform)" -H "Content-Type: application/json" '

Exporting an asset snapshot

To export an asset snapshot at a given timestamp, select a command that supports the level of detail you want in your exported snapshot.

Note that the Cloud Storage bucket you use to store exported metadata must be in the Cloud Asset API enabled project you're running the export from.

gcloud

The following example exports assets within a project. To export the assets of an organization or folder, use the --organization=ORGANIZATION_ID or --folder=FOLDER_ID flag in your command. Note that access-policy can only be exported for an --organization.

  1. Export asset metadata within your project with the following command. This stores the exported snapshot in a Cloud Storage bucket at gs://YOUR_BUCKET/NEW_FILE.

    Export all resource names without metadata in a project

      gcloud asset export \
     --project PROJECT_ID \
     --output-path "gs://YOUR_BUCKET/NEW_FILE"

    Export all resource metadata in a project

    gcloud asset export \
   --content-type resource \
   --project PROJECT_ID \
   --output-path "gs://YOUR_BUCKET/NEW_FILE"

    Export metadata of resource whose asset type starts with "compute.googleapis.com" in a project

    gcloud asset export \
   --content-type resource \
   --project PROJECT_ID \
   --asset-types compute.googleapis.com.* \
   --output-path "gs://YOUR_BUCKET/NEW_FILE"

    Export Cloud IAM policies in a project

    gcloud asset export \
   --content-type iam-policy \
   --project PROJECT_ID \
   --output-path "gs://YOUR_BUCKET/NEW_FILE"

    Export org policies set on a project

    gcloud asset export \
   --content-type org-policy \
   --project PROJECT_ID \
   --output-path "gs://YOUR_BUCKET/NEW_FILE"

    Export access policies in an organization

    gcloud asset export \
   --content-type access-policy \
   --organization ORGANIZATION_ID \
   --output-path "gs://YOUR_BUCKET/NEW_FILE"

  2. (Optional) Exporting assets can be a long running process for very large folders and organizations with many projects and resources. Run the command displayed in the gcloud tool that appears after running the export command to check the status of the export. Operations are associated with an OPERATION_ID value, which is a UUID. This value is potentially sensitive, as the gcloud asset operations describe command requires no additional permissions to run successfully. Ensure that you share the OPERATION_ID value only with trusted users.

    gcloud asset operations describe projects/PROJECT_ID/operations/ExportAssets/CONTENT_TYPE/OPERATION_ID

API

To export an asset snapshot for an organization, use the https://cloudasset.googleapis.com/v1/organizations/ORGANIZATION_NUMBER:exportAssets REST method in the following gcurl commands.

To export an asset snapshot for a folder, use the https://cloudasset.googleapis.com/v1/folders/FOLDER_NUMBER:exportAssets REST method.

  1. Export asset metadata within your project with the following example commands. This stores the exported snapshot in a Cloud Storage bucket at gs://YOUR_BUCKET/NEW_FILE.

    Export all resource names without metadata in a project

    gcurl -d '{"outputConfig":{"gcsDestination": \
       {"uri":"gs://YOUR_BUCKET/NEW_FILE"}}}' \
        https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER:exportAssets

    Export all resource metadata in a project

    gcurl -d '{"contentType":"RESOURCE", "outputConfig":{"gcsDestination": \
       {"uri":"gs://YOUR_BUCKET/NEW_FILE"}}}' \
        https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER:exportAssets

    Export metadata of resource whose asset type starts with "compute.googleapis.com" in a project

    gcurl -d '{"contentType":"RESOURCE", "assetTypes": "compute.googleapis.com.*", "outputConfig":{"gcsDestination": \
       {"uri":"gs://YOUR_BUCKET/NEW_FILE"}}}' \
        https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER:exportAssets

    Export Cloud IAM policies in a project

    gcurl -d '{"contentType":"IAM_POLICY", "outputConfig":{\
        "gcsDestination": {"uri":"gs://YOUR_BUCKET/NEW_FILE"}}}' \
        https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER:exportAssets

  2. (Optional) Exporting assets can be a long running process for very large folders and organizations with many projects and resources. Use the operation name returned in the response of your export to view the status of your export. Operations are associated with an OPERATION_ID value, which is a UUID. This value is potentially sensitive, as the gcloud asset operations describe command requires no additional permissions to run successfully. Ensure that you share the OPERATION_ID value only with trusted users.

    1. Get the operation number from the name field of the response.

      "name": "projects/PROJECT_NUMBER/operations/ExportAssets/CONTENT_TYPE/OPERATION_ID"

    2. Enter the operation number into the following command.

      gcurl https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/operations/ExportAssets/CONTENT_TYPE/OPERATION_ID

Viewing an asset snapshot

To view your asset snapshot:

  1. Go to the Cloud Storage Browser page.
    Open the Cloud Storage Browser page

  2. Open the new file you exported your metadata to.

The export lists the assets and their resource names.