Create and manage inventory reports

This page describes how to create and manage inventory report configurations and generated inventory reports, so you can get a high level summary of all your objects' metadata in a given bucket. For an overview of inventory reports, see the Storage Insights inventory reports overview documentation.

Before you begin

Get required roles

In order to get the required permissions for creating and managing inventory reports, ask your administrator to grant you the following IAM roles on the project or source and destination buckets with which you'll be managing inventory reports.

  • To create and manage inventory report configurations:

    • roles/storage.admin on the source bucket and destination bucket

    • roles/storageinsights.admin on the project

  • To read and download inventory reports:

    • roles/storage.objectViewer on the destination bucket

    • roles/storageinsights.viewer on the project

These predefined roles contain the permissions required to create and manage inventory reports and configurations. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

To create and manage inventory report configurations:

  • storage.buckets.get on the source bucket
  • storage.objects.list on the source bucket
  • storage.buckets.getObjectInsights on the source bucket
  • storage.buckets.get on the destination bucket
  • storage.objects.create on the destination bucket
  • storageinsights.reportConfigs.delete on the project
  • storageinsights.reportConfigs.get on the project
  • storageinsights.reportConfigs.create on the project
  • storageinsights.reportConfigs.list on the project
  • storageinsights.reportConfigs.update on the project

To read and download inventory reports:

  • storage.objects.get on the destination bucket
  • storageinsights.reportDetails.get on the project
  • storageinsights.reportDetails.list on the project

You might also be able to get these permissions with other predefined roles. To see which roles are associated with which permissions, refer to IAM roles for Cloud Storage.

For instructions on using roles to control access to buckets, see Use IAM. For instructions on using roles to control access to projects, see Manage access.

Enable the Storage Insights API

Console

To enable the storageinsights.googleapis.com API, follow the instructions in Enabling services.

Command line

To enable the Storage Insights API in your current project, run the following command:

gcloud services enable storageinsights.googleapis.com

For more details on how to enable services for a Google Cloud project, see Enabling and disabling services.

REST APIs

JSON API

Use the Google Cloud console or the Google Cloud CLI to enable the Storage Insights API.

Create an inventory report configuration

Console

To create an inventory report configuration, complete the following steps:

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the bucket you'd like to use as the source bucket.

  3. On the Bucket details page, click the Inventory reports tab.

  4. Click Create report configuration.

  5. In the Identify your report configuration section, create a display name for the inventory report configuration. The display name can contain up to 256 characters.

  6. In the Choose metadata fields section, select the metadata fields you want to include in your inventory reports.

  7. Click Continue.

  8. In the Choose a file format section, select the file format in which you'd like to generate inventory reports.

  9. In the Choose a destination bucket section, select the bucket you'd like to use as a destination bucket.

    In the Enter destination path (optional) section, you can optionally specify a destination path where inventory reports will be generated.

  10. In the Scheduling options section, specify the frequency, start date, and end date for generating reports.

  11. Click Create.

    The Report configuration details page is displayed. The metadata of generated inventory reports will appear on the page.

Command line

  1. To create an inventory report configuration, run the gcloud storage insights inventory-reports create command:

    gcloud storage insights inventory-reports create SOURCE_BUCKET_URL \
  --csv-separator=SEPARATOR \
  --csv-delimiter=DELIMITER \
  --[no]-csv-header \
  --parquet \
  --display-name=DISPLAY_NAME \
  --destination=DESTINATION_PATH \
  --metadata-fields=METADATA_FIELD \
  --schedule-starts=START_DATE \
  --schedule-repeats=FREQUENCY \
  --schedule-repeats-until=END_DATE

    Replace:

    • SOURCE_BUCKET_URL with the URL of the source bucket. For example, gs://my_example_source_bucket.

    • SEPARATOR with the character used to separate the records in the inventory report CSV file. Must be either \n or \r\n. Default value is \n. Optional. If the --csv-separator flag is used, --parquet cannot be used.

    • DELIMITER with the delimiter that separates the fields in the inventory report CSV file. The value can include one character and cannot be the same value as SEPARATOR. Default value is ,. Optional. If --csv-delimiter is used, --parquet cannot be used.

    • --[no]-csv-header with the flag that indicates whether or not headers are included in the inventory report CSV file. Use --csv-header to include headers and --no-csv-header to exclude headers. If either flag is used, --parquet cannot be used.

    • DISPLAY_NAME with the editable name of the inventory report configuration. Optional.

    • --parquet with the flag that generates inventory reports in Apache Parquet format instead of CSV. If used, --csv-delimiter, --csv-separator, and --[no-]csv-header cannot be used.

    • DESTINATION_PATH with the bucket or folder in the bucket where inventory reports will be generated. For example, gs://my_example_destination_bucket or gs://my_example_destination_bucket/path/to/inventory/report.

      When you specify a path to a folder within a bucket, you can use keywords in the path, which are substituted with their corresponding values when the report is generated. This enables you to generate reports in a Hive partitioned format so you can load or query the data in BigQuery without additional processing.

    • METADATA_FIELD with a comma-separated list of metadata fields you want included in the inventory report.

    • START_DATE with the UTC date on which you want to start generating inventory reports. For example, 2022-01-15.

    • FREQUENCY with how often you want inventory reports to be generated. Values are daily or weekly.

    • END_DATE with the UTC date after which you want to stop generating inventory reports. Must be a value after the START_DATE. For example, if you specify 2022-02-15, inventory reports will no longer be generated as of February 16, 2022.

Client libraries

Java

For more information, see the Cloud Storage Java API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries. 


import com.google.cloud.storageinsights.v1.CSVOptions;
import com.google.cloud.storageinsights.v1.CloudStorageDestinationOptions;
import com.google.cloud.storageinsights.v1.CloudStorageFilters;
import com.google.cloud.storageinsights.v1.CreateReportConfigRequest;
import com.google.cloud.storageinsights.v1.FrequencyOptions;
import com.google.cloud.storageinsights.v1.LocationName;
import com.google.cloud.storageinsights.v1.ObjectMetadataReportOptions;
import com.google.cloud.storageinsights.v1.ReportConfig;
import com.google.cloud.storageinsights.v1.StorageInsightsClient;
import com.google.common.collect.ImmutableList;
import com.google.type.Date;
import java.io.IOException;

public class CreateInventoryReportConfig {

  public static void createInventoryReportConfig(
      String projectId, String bucketLocation, String sourceBucket, String destinationBucket)
      throws IOException {
    try (StorageInsightsClient storageInsightsClient = StorageInsightsClient.create()) {
      ReportConfig reportConfig =
          ReportConfig.newBuilder()
              .setDisplayName("Example inventory report configuration")
              .setFrequencyOptions(
                  FrequencyOptions.newBuilder()
                      .setFrequency(FrequencyOptions.Frequency.WEEKLY)
                      .setStartDate(Date.newBuilder().setDay(15).setMonth(8).setYear(3022).build())
                      .setEndDate(Date.newBuilder().setDay(15).setMonth(9).setYear(3022).build())
                      .build())
              .setCsvOptions(
                  CSVOptions.newBuilder()
                      .setDelimiter(",")
                      .setRecordSeparator("\n")
                      .setHeaderRequired(true)
                      .build())
              .setObjectMetadataReportOptions(
                  ObjectMetadataReportOptions.newBuilder()
                      .addAllMetadataFields(ImmutableList.of("project", "name", "bucket"))
                      .setStorageFilters(
                          CloudStorageFilters.newBuilder().setBucket(sourceBucket).build())
                      .setStorageDestinationOptions(
                          CloudStorageDestinationOptions.newBuilder()
                              .setBucket(destinationBucket)
                              .build())
                      .build())
              .build();
      CreateReportConfigRequest request =
          CreateReportConfigRequest.newBuilder()
              .setParent(LocationName.of(projectId, bucketLocation).toString())
              .setReportConfig(reportConfig)
              .build();
      ReportConfig response = storageInsightsClient.createReportConfig(request);
      System.out.println("Created inventory report config with name " + response.getName());
    }
  }
}

REST APIs

JSON API

To create an inventory report configuration, complete the following steps:

  1. Have gcloud CLI installed and initialized, which lets you generate an access token for the Authorization header.

  2. Create a JSON file that contains the following information, where all fields are required unless noted otherwise:

    {
  "display_name": "DISPLAY_NAME",
  "frequency_options": {
    "frequency": "FREQUENCY",
    "start_date": "START_DATE",
    "end_date": "END_DATE",
  },
  "csv_options": {
    "record_separator": "RECORD_SEPARATOR",
    "delimiter": "DELIMITER",
    "header_required": HEADER_REQUIRED
  },
  "object_metadata_report_options": {
     "metadata_fields": [
      "METADATA_FIELD",
      ...
    ],
    "storage_filters": {
      "bucket": "SOURCE_BUCKET_NAME"
    },
    "storage_destination_options": {
      "bucket": "DESTINATION_BUCKET_NAME",
      "destination_path": "DESTINATION_PATH"
    }
  }
}

    Replace:

    For example, the following code sample creates an inventory report configuration named "Example inventory report configuration" that generates reports at a weekly frequency:

    {
  "display_name": "Example inventory report configuration",
  "frequency_options": {
    "frequency": "WEEKLY",
    "start_date": {
      "day": 15,
      "month": 8,
      "year": 2022
    },
    "end_date": {
      "day": 15,
      "month": 9,
      "year": 2022
    },
  },
  "csv_options": {
    "record_separator": "\n",
    "delimiter": ",",
    "header_required": true
  },
  "object_metadata_report_options": {
     "metadata_fields": [
      "project",
      "name",
      "bucket"
    ],
    "storage_filters": {
      "bucket": "example_source_bucket"
    },
    "storage_destination_options": {
      "bucket": "example_destination_bucket"
    }
  }
}

  3. To apply the inventory report configuration, use cURL to call the JSON API with an Insert ReportConfig request:

    curl -X POST --data-binary @JSON_FILE_NAME \
"https://storageinsights.googleapis.com/v1/projects/YOUR_PROJECT/locations/LOCATION/reportConfigs" \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \

    Replace:

    • JSON_FILE_NAME with the path to the JSON file you created in the previous step.

    • YOUR_PROJECT with the name of the project in which your source and destination buckets reside.

    • LOCATION with the location of the source and destination buckets. For example, us-west1.

Grant required roles to the service agent

A project-level service agent is automatically created the first time you set up an inventory report configuration. The service agent follows the naming format service-PROJECT_NUMBER@gcp-sa-storageinsights.iam.gserviceaccount.com and appears on the IAM page of the Google Cloud console when you select the Include Google-provided role grants checkbox.

To enable Storage Insights to generate and write inventory reports, ask your administrator to grant the service agent the following IAM roles:

  • roles/storage.insightsCollectorService on the source bucket, which includes the storage.buckets.getObjectInsights and storage.buckets.get permissions
  • roles/storage.objectCreator on the destination bucket, which includes the storage.objects.create permission

For instructions on granting roles, see Use IAM. You can grant the roles/storage.insightsCollectorService role by using the Google Cloud console or the Google Cloud CLI. For example: 

gcloud storage buckets add-iam-policy-binding SOURCE_BUCKET_URL \
    --member=serviceAccount:service-PROJECT_NUMBER@gcp-sa-storageinsights.iam.gserviceaccount.com \
    --role=roles/storage.insightsCollectorService

Once the service agent is granted the required permissions, it takes up to 24 hours to write the first inventory report to the destination bucket.

Edit an inventory report configuration

Once an inventory report configuration has been created, you can modify certain properties of the configuration.

Console

To edit an inventory report configuration, follow the steps below:

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the source bucket that contains the inventory report configuration you want to edit.

  3. In your source bucket's Bucket details page, click the Inventory reports tab.

  4. Click the configuration name of the inventory report configuration you want to edit.

  5. In the Report configuration details page that appears, edit the properties you desire by using the Edit button.

Command line

To edit an inventory report configuration, follow the steps below:

  1. To find the name of the inventory report configuration you want to edit, list all the inventory configurations in the source bucket by using the gcloud storage insights inventory-reports list command:

    gcloud storage insights inventory-reports list \
  --source=SOURCE_BUCKET \
  --filter=EXPRESSION \
  --page-size=SIZE \
  --sort-by=FIELD \
  --format="yaml(name)"

    Replace:

    • SOURCE_BUCKET with the URL of the source bucket that contains the inventory report configuration.

    • EXPRESSION with a boolean filter to apply to each resource item to be listed. If the expression evaluates True, the item is listed. For more details and examples of filter expressions, run $ gcloud topic filters.

    • SIZE with the maximum number of resources per page. The default is 50.

    • FIELD with a comma-separated list of resource field key names to sort by. The default order is ascending. Prefix a field with ~ for descending order on that field.

    1. Edit the inventory report configuration fields you want to update by using the gcloud storage insights inventory-reports update command. The following example updates the inventory report configuration to generate inventory reports daily in Apache Parquet format:

      gcloud storage insights inventory-reports update CONFIG_NAME \
--parquet \
--schedule-repeats="daily"

Client libraries

Java

For more information, see the Cloud Storage Java API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries. 


import com.google.cloud.storageinsights.v1.ReportConfig;
import com.google.cloud.storageinsights.v1.ReportConfigName;
import com.google.cloud.storageinsights.v1.StorageInsightsClient;
import com.google.cloud.storageinsights.v1.UpdateReportConfigRequest;
import com.google.protobuf.FieldMask;
import java.io.IOException;

public class EditInventoryReportConfig {


  public static void editInventoryReportConfig(
          String projectId, String location, String inventoryReportConfigUuid) throws IOException {
    try (StorageInsightsClient storageInsightsClient = StorageInsightsClient.create()) {
      ReportConfigName name = ReportConfigName.of(projectId, location, inventoryReportConfigUuid);
      ReportConfig reportConfig = storageInsightsClient.getReportConfig(name);

      // Set any other fields you want to update here
      ReportConfig updatedReportConfig =
              reportConfig.toBuilder().setDisplayName("Updated Display Name").build();

      storageInsightsClient.updateReportConfig(
              UpdateReportConfigRequest.newBuilder()
                      // Add any fields that you want to update to the update mask, in snake case
                      .setUpdateMask(FieldMask.newBuilder().addPaths("display_name")
                              .build())
                      .setReportConfig(updatedReportConfig).build());

      System.out.println("Edited inventory report config with name " + name);
    }
  }
}

REST APIs

JSON API

To edit an inventory report configuration, complete the following steps:

  1. Have gcloud CLI installed and initialized, which lets you generate an access token for the Authorization header.

  2. Get the name of the inventory report configuration you want to edit by using cURL to call the JSON API with a Get ReportConfig request:

    curl --request GET \
"https://storageinsights.googleapis.com/v1/projects/YOUR_PROJECT/locations/LOCATION/reportConfigs?" \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \

    Replace:

    • YOUR_PROJECT with the name of the project in which your source and destination buckets reside.

    • LOCATION with the location of the source and destination buckets. For example, us-west1.

    If you want to retrieve all inventory report configurations in a specific source bucket, append the filter query parameter to your request:

    curl --request GET \
"https://storageinsights.googleapis.com/v1/projects/YOUR_PROJECT/locations/LOCATION/reportConfigs?filter=objectMetadataReportOptions.storageFilters.bucket=BUCKET_NAME" \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \

    Replace:

    • YOUR_PROJECT with the name of the project in which your source and destination buckets reside.
    • LOCATION with the location of the source and destination buckets. For example, us-west1.
    • BUCKET_NAME with the name of the source bucket that contains the inventory report configurations you want to edit.

  3. Create a JSON file that includes changes to the properties you want to edit. The following example updates the inventory report configuration to generate inventory reports daily in Apache Parquet format:

    {
  "ReportConfig": {
    "frequency_options": {
      "frequency": "DAILY"
    },
    "parquet_options": {
    }
  }

  4. To apply the inventory report configuration, use cURL to call the JSON API with a Patch ReportConfig request:

    curl --request PATCH \
"https://storageinsights.googleapis.com/v1/projects/YOUR_PROJECT/locations/LOCATION/reportConfigs/REPORT_CONFIG_UUID" \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \
  --data-binary "@JSON_FILE_NAME.json" \

    Replace:

    • YOUR_PROJECT with the name of the project in which your source and destination buckets reside.

    • LOCATION with the location of the source and destination buckets. For example, us-west1.

    • REPORT_CONFIG_UUID with the auto-generated UUID of the inventory report configuration.

List inventory report configurations

Console

To list the inventory report configurations in a source bucket, complete the steps below:

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the source bucket that contains the inventory report configurations you want to view.

  3. On the Bucket details page, click the Inventory reports tab.

    Inventory report configurations in the source bucket appear.

Command line

You can list all the inventory configurations in a source bucket by using the gcloud storage insights inventory-reports list command:

gcloud storage insights inventory-reports list \
  --source=SOURCE_BUCKET \
  --filter=EXPRESSION \
  --page-size=SIZE \
  --sort-by=FIELD \
  --format="yaml(name)"

Replace:

  • SOURCE_BUCKET with the URL of the source bucket that contains the inventory report configuration.

  • EXPRESSION with a boolean filter to apply to each resource item to be listed. If the expression evaluates True, the item is listed. For more details and examples of filter expressions, run $ gcloud topic filters.

  • SIZE with the maximum number of resources per page. The default is 50.

  • FIELD with a comma-separated list of resource field key names to sort by. The default order is ascending. Prefix a field with ~ for descending order on that field.

Client libraries

Java

For more information, see the Cloud Storage Java API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries. 


import com.google.cloud.storageinsights.v1.LocationName;
import com.google.cloud.storageinsights.v1.ReportConfig;
import com.google.cloud.storageinsights.v1.StorageInsightsClient;
import java.io.IOException;

public class ListInventoryReportConfigs {


  public static void listInventoryReportConfigs(String projectId, String location)
      throws IOException {
    try (StorageInsightsClient storageInsightsClient = StorageInsightsClient.create()) {
      System.out.println(
          "Printing inventory report configs in project "
              + projectId
              + " and location "
              + location);
      for (ReportConfig config :
          storageInsightsClient
              .listReportConfigs(LocationName.of(projectId, location))
              .iterateAll()) {
        System.out.println(config.getName());
      }
    }
  }
}

REST APIs

JSON API

  1. Have gcloud CLI installed and initialized, which lets you generate an access token for the Authorization header.

  2. You can list all inventory report configurations in a project for a given location by using a request to list inventory report configurations:

    curl --request GET \
"https://storageinsights.googleapis.com/v1/projects/YOUR_PROJECT/locations/LOCATION/reportConfigs?" \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \

    Replace:

    • YOUR_PROJECT with the name of the project in which your source and destination buckets reside.

    • LOCATION with the location of the source and destination buckets. For example, us-west1.

    If you want to list all inventory report configurations in a specific source bucket, append the filter query parameter to your request:

    curl --request GET \
"https://storageinsights.googleapis.com/v1/projects/YOUR_PROJECT/locations/LOCATION/reportConfigs?filter=objectMetadataReportOptions.storageFilters.bucket=BUCKET_NAME" \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \

    Replace:

    • YOUR_PROJECT with the name of the project in which your source and destination buckets reside.

    • LOCATION with the location of the source and destination buckets. For example, us-west1.

    • BUCKET_NAME with the name of the source bucket that contains the inventory report configurations you want to list.

Download an inventory report

Console

Download individual reports

Inventory reports are generated and stored as objects in the destination bucket, so you can download them as you would download a regular object.

To download an inventory report, complete the following steps:

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the destination bucket that contains the inventory report you want to download.

  3. On the Bucket details page, make sure the Objects tab is selected.

  4. Click Download associated with the inventory report you want to download.

If you don't know the destination bucket, you can also download an inventory report through the inventory report configuration from which it was generated:

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the source bucket containing the inventory report configuration that generated the report you want to download.

  3. On the Bucket details page, click the configuration name of the inventory report configuration.

  4. On the Report configuration details page that appears, navigate to the Inventory report history section, then click the destination object path of the inventory report you want to download.

    The Bucket details page appears for the destination bucket that contains the inventory report.

  5. Click Download associated with the inventory report you want to download.

Download report shards

To download an inventory report that's been split into one or more shards, complete the following steps:

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the destination bucket you specified when you created the inventory report configuration.

  3. On the Bucket details page, check for the presence of a manifest file. The presence of a manifest file indicates that all the shards of an inventory report have been generated.

    An example manifest file name is fc95c52f-157a-494f-af4a-d4a53a69ba66_2022-11-30T00:00_manifest.json.

  4. In the destination bucket, click Download associated with the manifest file. Note the names of the shard files you want to download from the report_shards_file_names field.

  5. In the destination bucket, click Download associated with the shard files you want to download.

Command line

Download individual reports

To download an inventory report, complete the following steps:

  1. To list all the inventory reports that have been generated by an inventory report configuration and retrieve their REPORT_DETAIL_ID, use the gcloud storage insights inventory-reports details list command:

    gcloud storage insights inventory-reports details list CONFIG_NAME \
  --filter=EXPRESSION \
  --page-size=SIZE \
  --sort-by=FIELD

    Replace:

    • CONFIG_NAME with the unique name of the inventory report configuration, in the format projects/PROJECT/locations/LOCATION/reportConfigs/REPORT_CONFIG_UUID.

    • EXPRESSION with a boolean filter to apply to each resource item to be listed. If the expression evaluates True, then that item is listed. For more details and examples of filter expressions, run $ gcloud topic filters.

    • SIZE with the maximum number of resources per page. The default is 50.

    • FIELD with a comma-separated list of resource field key names to sort by. The default order is ascending. Prefix a field with ~ for descending order on that field.

    If successful, the command returns output similar to the following:

    REPORT_DETAIL_ID            SNAPSHOT_TIME
Report_2023-04-10T00-00     2023-04-10T00:53:03Z
Report_2023-04-12T00-00     2023-04-12T00:52:54Z
Report_2023-04-05T00-00     2023-04-05T00:53:01Z

  2. To download an inventory report, you need to first retrieve the reportPathPrefix property of the ReportDetail object. To get the reportPathPrefix of a report, use the gcloud storage insights inventory-reports details describe command:

    gcloud storage insights inventory-reports details describe REPORT_DETAIL_NAME

    Replace REPORT_DETAIL_NAME with the name of the inventory report in the format projects/PROJECT/locations/LOCATION/reportConfigs/REPORT_CONFIG_UUID/reportDetails/REPORT_DETAIL_ID.

Download report shards

To download an inventory report that's been split into one or more shards, complete the following steps:

  1. Download the inventory report manifest file by using the gcloud storage cp command:

    gcloud storage cp gs://BUCKET_NAME/MANIFEST_FILE_NAME DOWNLOAD_PATH

    Replace:

    • BUCKET_NAME with the name of the destination bucket.

    • MANIFEST_FILE_NAME with the name of the manifest file in the destination bucket, in the following naming convention:

      REPORT_CONFIG_UUID_TARGET_DATETIME_manifest.json

      Where:

      • REPORT_CONFIG_UUID is the auto-generated UUID of the inventory report configuration that generated the report shards you want to download.

      • TARGET_DATETIME is the datetime at which the inventory report was generated.

      For example, fc95c52f-157a-494f-af4a-d4a53a69ba66_2022-11-30T00:00_manifest.json.

    • DOWNLOAD_PATH with the path to your file system where you want to save the inventory report to. For example, ./example_report.csv.

  2. To download a shard file, use the gcloud storage cp command:

    gcloud storage cp gs://BUCKET_NAME/SHARD_FILE_NAME DOWNLOAD_PATH

    Replace:

    • BUCKET_NAME with the name of the destination bucket.

    • SHARD_FILE_NAME with the URL-encoded name of the shard file you want to download. For example, fc95c52f-157a-494f-af4a-d4a53a69ba66_2022-11-30T00:54_0.csv.

    • DOWNLOAD_PATH with the path to your file system where you want to save the inventory report to. For example, ./example_report.csv.

Client libraries

Java

For more information, see the Cloud Storage Java API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries. 


import com.google.cloud.storageinsights.v1.ReportConfig;
import com.google.cloud.storageinsights.v1.ReportConfigName;
import com.google.cloud.storageinsights.v1.ReportDetail;
import com.google.cloud.storageinsights.v1.StorageInsightsClient;
import java.io.IOException;

public class GetInventoryReportNames {


  public static void getInventoryReportNames(
      String projectId, String location, String reportConfigUuid) throws IOException {
    try (StorageInsightsClient storageInsightsClient = StorageInsightsClient.create()) {
      ReportConfig config =
          storageInsightsClient.getReportConfig(
              ReportConfigName.of(projectId, location, reportConfigUuid));
      String extension = config.hasCsvOptions() ? "csv" : "parquet";
      System.out.println(
          "You can use the Google Cloud Storage Client "
              + "to download the following objects from Google Cloud Storage:");
      for (ReportDetail reportDetail :
          storageInsightsClient.listReportDetails(config.getName()).iterateAll()) {
        for (long index = reportDetail.getShardsCount() - 1; index >= 0; index--) {
          System.out.println(reportDetail.getReportPathPrefix() + index + "." + extension);
        }
      }
    }
  }
}

REST APIs

JSON API

Download individual reports

To download an inventory report, complete the following steps:

  1. Have gcloud CLI installed and initialized, which lets you generate an access token for the Authorization header.

  2. List all the inventory reports generated by an inventory report configuration by using cURL to call the JSON API with a request to list inventory report details:

    curl --request GET \
"https://storageinsights.googleapis.com/v1/projects/YOUR_PROJECT/locations/LOCATION/reportConfigs/REPORT_CONFIG_UUID/reportDetails/" \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \

    Replace:

    • YOUR_PROJECT with the name of the project in which your source and destination buckets reside.

    • LOCATION with the location of the source and destination buckets. For example, us-west1.

    • REPORT_CONFIG_UUID with the auto-generated UUID of the inventory report configuration that generated the report you want to download.

  3. To get the details of an individual report, use cURL to call the JSON API with a Get ReportDetails request:

    curl --request GET \
"https://storageinsights.googleapis.com/v1/projects/YOUR_PROJECT/locations/LOCATION/reportConfigs/REPORT_CONFIG_UUID/reportDetails/REPORT_DETAIL_ID" \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \

    Replace:

    • YOUR_PROJECT with the name of the project in which your source and destination buckets reside.

    • LOCATION with the location of the source and destination buckets. For example, us-west1.

    • REPORT_CONFIG_UUID with the auto-generated UUID of the inventory report configuration that generated the report you want to download.

    • REPORT_DETAIL_ID with the name of the inventory report you want to download.

Download report shards

To download an inventory report that's been split into one or more shards, complete the following steps:

  1. Have gcloud CLI installed and initialized, which lets you generate an access token for the Authorization header.

  2. To download the manifest file of the inventory report, use cURL to call the JSON API with a GET Object request:

    curl -X GET \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  --header "Accept: application/json" \
  --header "Content-Type: application/json" \
  "https://storage.googleapis.com/storage/v1/b/BUCKET_NAME/o/MANIFEST_FILE_NAME?alt=media" \

    Replace:

    • BUCKET_NAME with the name of the destination bucket.

    • MANIFEST_FILE_NAME with the name of the manifest file in the destination bucket, in the following naming convention:

      REPORT_CONFIG_UUID_TARGET_DATETIME_manifest.json

      Where:

      • REPORT_CONFIG_UUID is the auto-generated UUID of the inventory report configuration that generated the report shards you want to download.

      • TARGET_DATETIME is the date on which the inventory report was generated.

    For example, fc95c52f-157a-494f-af4a-d4a53a69ba66_2022-11-30T00:00_manifest.json.

    The report_shards_file_names field of the manifest file contains the names of the inventory report shards you can download.

  3. To download the inventory report shard files, use curl to call the Cloud Storage JSON API with a Get Object request:

    curl -X GET \
  --header "Authorization: Bearer $(gcloud auth print-access-token)" \
  -o "DOWNLOAD_PATH" \
  "https://storage.googleapis.com/storage/v1/b/BUCKET_NAME/o/SHARD_FILE_NAME?alt=media"

    Replace:

    • DOWNLOAD_PATH with the path to your local file system where you want to save your object. For example, Desktop/dog.png.

    • BUCKET_NAME with the name of the destination bucket containing the inventory report. For example, my-bucket.

    • SHARD_FILE_NAME with the URL-encoded name of the shard file you want to download. For example, fc95c52f-157a-494f-af4a-d4a53a69ba66_2022-11-30T00:54_0.csv.

Delete an inventory report configuration

Console

To delete an inventory report configuration, complete the following steps:

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the source bucket that contains the inventory report configuration you want to delete.

  3. In your source bucket's Bucket details page, click the Inventory reports tab.

  4. Click the configuration name of the inventory report configuration you want to delete.

  5. In the Report configuration details page that appears, click Delete.

Command line

To delete an inventory report configuration, use the gcloud storage insights inventory-reports delete command:

gcloud storage insights inventory-reports delete CONFIG_NAME
  --force

Replace CONFIG_NAME with the unique name of the inventory report configuration, in the format projects/PROJECT/locations/LOCATION/reportConfigs/REPORT_CONFIG_UUID.

The --force flag deletes the metadata for all inventory reports generated by the given inventory report configuration. The inventory report objects themselves will not be deleted.

Client libraries

Java

For more information, see the Cloud Storage Java API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries. 


import com.google.cloud.storageinsights.v1.ReportConfigName;
import com.google.cloud.storageinsights.v1.StorageInsightsClient;
import java.io.IOException;

public class DeleteInventoryReportConfig {


  public static void deleteInventoryReportConfig(
      String projectId, String location, String inventoryReportConfigUuid) throws IOException {
    try (StorageInsightsClient storageInsightsClient = StorageInsightsClient.create()) {
      ReportConfigName name = ReportConfigName.of(projectId, location, inventoryReportConfigUuid);
      storageInsightsClient.deleteReportConfig(name);

      System.out.println("Deleted inventory report config with name " + name);
    }
  }
}

REST APIs

JSON API

To delete an inventory report configuration, complete the following steps:

  1. Have gcloud CLI installed and initialized, which lets you generate an access token for the Authorization header.

  2. To delete an inventory report configuration, use cURL to call the JSON API with a Delete ReportConfig request:

    curl --request DELETE \
'https://storageinsights.googleapis.com/v1/projects/YOUR_PROJECT/locations/LOCATION/reportConfigs/REPORT_CONFIG_UUID?force=true' \
  --header 'Authorization: Bearer $(gcloud auth print-access-token)' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \

    Replace:

    • YOUR_PROJECT with the name of the project in which your source and destination buckets reside.

    • LOCATION with the location of the source and destination buckets. For example, us-west1.

    • REPORT_CONFIG_UUID with the auto-generated UUID of the inventory report configuration you want to delete.

The force query parameter deletes the metadata for all inventory reports generated by the given inventory report configuration. The inventory report objects themselves will not be deleted.

Advanced: Generate inventory reports in a Hive partitioned format

When you create an inventory report configuration, you can specify a destination path, which is a location in your destination bucket where inventory reports will be generated. This enables you to generate inventory reports in a Hive partitioned format.

You specify a destination path by using placeholder keywords. Keywords in the destination path are substituted with their corresponding values when the inventory report is generated. For example, the destination path config={{report-config-id}}/date={{date}} can resolve to config=1A34-F2E456-12B456-1C3D/date=2022-05-20.

Destination path keywords

You can specify the keywords report-config-id, date, or datetime, enclosed within {{ and }}, where:

  • report-config-id is the UUID of the inventory report configuration.

  • date is the date on which the inventory report is generated. Follows the ISO 8601 format.

  • datetime is the date and time on which the inventory report is generated. Follows the ISO 8601 format.

Keywords can be specified by using any of the following tools:

  • When using the Google Cloud console, specify keywords in the Enter destination path (optional) section. This section appears when you create an inventory report configuration.

  • When using the Google Cloud CLI, specify placeholder keywords by using the --destination flag.

  • When using the JSON API, specify placeholder keywords in the storage_destination_options.destination_path field of the ReportConfig object.

Access control and security

The following recommendations and considerations relate to the security and usage of inventory reports and configurations:

  • We recommend that users with the roles/storage.admin role also have the storageinsights.reportConfigs.* permissions so they can maintain control over the usage of all resources in their buckets, such as the reading of objects and their metadata.

  • We recommend that you limit access to your inventory report resources by granting the storageinsights.reportConfigs.* permissions only to necessary individuals.

  • Once an inventory report configuration has been created, it continues to generate inventory reports even if the user who created the configuration no longer has the required permissions. To stop generating inventory reports, you can edit the end date in the inventory report configuration or delete the configuration entirely.

What's next

  • If you encounter issues with generating inventory reports, refer to Troubleshooting for help.