Monitoring asset changes

Stay organized with collections Save and categorize content based on your preferences.

This page explains how to create and manage feeds on a project.

Overview

To receive real-time notifications about resource and policy changes, you can create and subscribe to a feed. When you configure the feed, you can specify that you want to monitor changes of supported resource types, IAM policies, access policies, and organization policies within an organization, folder, project. Additionally, you can add conditions to your feed so that you only receive notifications for certain types of changes to an asset. After configuring your feed, you immediately receive notifications whenever the specified assets are changed, sent through Pub/Sub (formatted as a TemporalAsset). Real-time notifications connect to your existing workloads. With this functionality, you can merge actions, like creating a Cloud Function to reverse a resource change once detected.

Before you begin

  1. Enable the Cloud Asset API for your project.

  2. (Optional) When you need to access Cloud Asset API with service accounts, create a new service account if you don't have an existing service account within your project.

  3. Grant your user or service account permissions to call the API for real-time feeds. The following permissions are needed for each operation:

    Permission Description
    cloudasset.feeds.create and cloudasset.assets.exportResource Create feeds
    cloudasset.feeds.update and cloudasset.assets.exportResource Update feeds
    cloudasset.feeds.delete Delete feeds
    cloudasset.feeds.get Get feeds
    cloudasset.feeds.list List feeds

    The Cloud Asset Owner (roles/cloudasset.owner) role grants all permissions related to the Cloud Asset API, including the permissions listed in the preceding table. For more information about roles and permissions, see Understanding roles and Configuring permissions

  4. Create a Pub/Sub topic if you don't have an existing Pub/Sub topic.

  5. (Optional) If the Pub/Sub topic is located in a project that is not the Cloud Asset API-enabled consumer project that you call the API from, be sure that the service-API_ENABLED_PROJECT_NUMBER@gcp-sa-cloudasset.iam.gserviceaccount.com service account has the pubsub.topics.publish permission on the topic.

    To automatically create this service account and grant it the necessary permissions, you can call ExportAssets or CreateFeed.

    To manually create this service account: 

      gcloud beta services identity create --service=cloudasset.googleapis.com --project=PROJECT_ID
  gcloud projects add-iam-policy-binding PROJECT_ID --member=serviceAccount:service-API_ENABLED_PROJECT_NUMBER@gcp-sa-cloudasset.iam.gserviceaccount.com --role=roles/cloudasset.serviceAgent

    To manually grant the publish permission to the service account: 

      gcloud pubsub topics add-iam-policy-binding TOPIC_NAME --member=serviceAccount:service-API_ENABLED_PROJECT_NUMBER@gcp-sa-cloudasset.iam.gserviceaccount.com --role=roles/pubsub.publisher

Setting up your environment

GCLOUD

  1. Install the Google Cloud CLI on your local client if you don't have Google Cloud CLI installed.

  2. Update all of your installed components to the latest version if you have Google Cloud CLI installed.

  3. Enable the Resource Manager API for your project.

API

  1. Set up a new Compute Engine VM instance by going to the Create an instance page and selecting the service account within your project.

  2. Under Access scopes, select Allow full access to all Cloud APIs.

  3. Launch your instance by clicking Create.

  4. Go to the VM Instance page.

  5. Open a web SSH client connected to the instance by clicking SSH next to the instance listing.

  6. In the web SSH client, generate an auth token for your service account with the following call: 

    TOKEN=$(gcloud auth application-default print-access-token)

The following API calls assume you are creating and managing feeds on a project. If you want to create and manage feeds for an organization or a folder, swap /projects/PROJECT_NUMBER/ with /organizations/ORGANIZATION_NUMBER/ or /folders/FOLDER_NUMBER/.

Creating a feed

You can create up to 200 feeds on a parent. This limit only applies to feeds directly following that parent and does not count the feeds of its children. For example, if you have 10 projects under an organization, each project can have up to 200 feeds and the organization can also have up to 200 feeds.

To create a feed, you can use the following options:

  • FEED_ID is the unique client-assigned asset feed identifier. Required.
  • ASSET_NAME is a list of asset full names for which you want to receive change notifications. At least one of [ASSET_NAME, ASSET_TYPE] is required.
  • ASSET_TYPE is a list of asset types for which you want to receive change notifications. At least one of [ASSET_NAME, ASSET_TYPE] is required.
  • CONTENT_TYPE is the asset content type for which you want to receive change notifications. If not specified, the notification will only contain the asset name and asset type.
  • TOPIC_NAME is the name of the Pub/Sub topic to publish notifications. Required.
  • CONDITION_TITLE is the title of the condition to apply on the feed.
  • CONDITION_DESCRIPTION is the description of the condition to apply on the feed.

  • CONDITION_EXPRESSION is the expression of the condition to apply on the feed.

GCLOUD

To create a feed using the gcloud asset feeds create command for projects, folders, and organizations:

  • Projects: 

    gcloud asset feeds create FEED_ID --project=PROJECT_ID --asset-names="ASSET_NAME"
--content-type=CONTENT_TYPE --asset-types="ASSET_TYPE"
--pubsub-topic="TOPIC_NAME" --condition-title="CONDITION_TITLE"
--condition-description="CONDITION_DESCRIPTION"
--condition-expression="CONDITION_EXPRESSION"

  • Folders: 

    gcloud asset feeds create FEED_ID --folder=FOLDER_ID --asset-names="ASSET_NAME"
--content-type=CONTENT_TYPE --asset-types="ASSET_TYPE"
--pubsub-topic="TOPIC_NAME" --condition-title="CONDITION_TITLE"
--condition-description="CONDITION_DESCRIPTION"
--condition-expression="CONDITION_EXPRESSION"

  • Organizations: 

    gcloud asset feeds create FEED_ID --organization=ORGANIZATION_ID --asset-names="ASSET_NAME"
--content-type=CONTENT_TYPE --asset-types="ASSET_TYPE"
--pubsub-topic="TOPIC_NAME" --condition-title="CONDITION_TITLE"
--condition-description="CONDITION_DESCRIPTION"
--condition-expression="CONDITION_EXPRESSION"

API

To create a feed using the feeds.create() API for projects, folders, and organizations:

  • Projects: 
    curl -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -X POST \
  -d '{"feedId": "FEED_ID",
       "feed": { "assetNames": ["ASSET_NAME"],
       "assetTypes": ["ASSET_TYPE"], "contentType": "CONTENT_TYPE",
       "feedOutputConfig": {"pubsubDestination": {"topic":"TOPIC_NAME"}},
       "condition": {"title": "CONDITION_TITLE",
       "description": "CONDITION_DESCRIPTION",
       "expression": "CONDITION_EXPRESSION"}}}' \
  https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds
  • Folders: 
    curl -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -X POST \
  -d '{"feedId": "FEED_ID",
       "feed": { "assetNames": ["ASSET_NAME"],
       "assetTypes": ["ASSET_TYPE"], "contentType": "CONTENT_TYPE",
       "feedOutputConfig": {"pubsubDestination": {"topic":"TOPIC_NAME"}},
       "condition": {"title": "CONDITION_TITLE",
       "description": "CONDITION_DESCRIPTION",
       "expression": "CONDITION_EXPRESSION"}}}' \
  https://cloudasset.googleapis.com/v1/folders/FOLDER_NUMBER/feeds
  • Organizations: 
    curl -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -X POST \
  -d '{"feedId": "FEED_ID",
       "feed": { "assetNames": ["ASSET_NAME"],
       "assetTypes": ["ASSET_TYPE"], "contentType": "CONTENT_TYPE",
       "feedOutputConfig": {"pubsubDestination": {"topic":"TOPIC_NAME"}},
       "condition": {"title": "CONDITION_TITLE",
       "description": "CONDITION_DESCRIPTION",
       "expression": "CONDITION_EXPRESSION"}}}' \
  https://cloudasset.googleapis.com/v1/organizations/ORGANIZATION_NUMBER/feeds

Cloud Asset Inventory sets a notification on any asset that matches at least one of your feed's ASSET parameters AND also matches the condition expression, if specified. For example, specifying ASSET_TYPE, ASSET_NAME and CONDITION_EXPRESSION will set notifications on assets that match ASSET_TYPE or ASSET_NAME, and satisfy CONDITION_EXPRESSION.

The following commands create notifications from the quick_start_topic Pub/Sub topic when content changes within the quick_start_bucket Cloud Storage bucket or any BigQuery tables:

GCLOUD 

 gcloud asset feeds create quick_start_feed --project=PROJECT_ID --asset-names="//storage.googleapis.com/quick_start_bucket"
  --content-type=resource --asset-types="bigquery.googleapis.com/Table"
  --pubsub-topic="projects/PROJECT_ID/topics/quick_start_topic"

API 

 curl -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -X POST \
  -d '{"feedId": "quick_start_feed",
       "feed": { "assetNames": ["storage.googleapis.com/quick_start_bucket"],
       "assetTypes": ["bigquery.googleapis.com/Table"],
       "contentType": "RESOURCE",
       "feedOutputConfig": {"pubsubDestination": {"topic":"projects/PROJECT_ID/topics/quick_start_topic"}}}}' \
        https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds

Regular expressions are also supported for the ASSET_TYPE field. The following commands create notifications from the quick_start_topic Pub/Sub topic when content changes within the resources whose asset type starts with "compute.googleapis.com". Please see RE2 for all supported regular expression syntax.

GCLOUD 

 gcloud asset feeds create quick_start_feed --project=PROJECT_ID
  --content-type=resource --asset-types="compute.googleapis.com.*"
  --pubsub-topic="projects/PROJECT_ID/topics/quick_start_topic"

API 

 curl -H "Authorization: Bearer $TOKEN" 

  -H "Content-Type: application/json" -X POST 

  -d '{"feedId": "quick_start_feed",
       "feed": {
       "assetTypes": ["compute.googleapis.com.*"],
       "contentType": "RESOURCE",
       "feedOutputConfig": {"pubsubDestination": {"topic":"projects/PROJECT_ID/topics/quick_start_topic"}}}}' 

        https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds

To get a feed you created, use the following command:

GCLOUD 

 gcloud asset feeds describe FEED_ID --project=PROJECT_ID

API 

 curl -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds/FEED_ID

The feed is returned in the following format, where FULL_NAME_FEED is the feed identifier along with its resource parent:

{
  "name": "FULL_NAME_FEED",
  "assetTypes": ["ASSET_TYPES"],
  "assetNames": ["ASSET_NAMES"],
  "contentType": "CONTENT_TYPES",
  "feedOutputConfig": {
    "pubsubDestination": {
      "topic": "TOPIC_NAME"
    }
  },
  "condition": {
    "title": "CONDITION_TITLE",
    "description": "CONDITION_DESCRIPTION",
    "expression": "CONDITION_EXPRESSION"
  }
}

Receiving updates

After creating a feed, subscribe to updates from the Pub/Sub topic you specified in the feed. A new feed can take up to 10 minutes to start sending notifications. A notification is sent for every change on an asset that matches either assetNames or assetTypes, and satisfies condition in the feed.

The first message published to the Pub/Sub topic is a welcome message in the format of string, and all the following messages are in the format of TemporalAsset. Here is a sample message for RESOURCE content type.

{
  "asset": {
    "ancestors": [
      "projects/[PROJECT_NUMBER]",
      "folders/[FOLDER_NUMBER]",
      "organizations/[ORGANIZATION_NUMBER]"
    ],
    "assetType": "[ASSET_TYPE]",
    "name": "[ASSET_NAME]",
    "resource": {
      "data": {
        ...detailed resource metadata...
      },
      "discoveryDocumentUri": "[DISCOVERY_URI]",
      "discoveryName": "[DISCOVERY_NAME]",
      "location": "[LOCATION]",
      "parent": "[PARENT_ASSET_NAME]",
      "version": "[VERSION]"
    },
    "updateTime": "[UPDATE_TIME]"
  },
  "priorAsset": {
    ...prior asset information...
  },
  "priorAssetState": "[PRIOR_ASSET_STATE]",
  "window": {
    "startTime": "[UPDATE_TIME]"
  }
}

To learn more about Pub/Sub or how to set up the subscribers, see the Pub/Sub guide.

Manage the feeds

Getting a feed

To get a specific feed, use the following command:

GCLOUD 

gcloud asset feeds describe FEED_ID --project=PROJECT_ID

API 

 curl -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -X GET \
  https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds/FEED_ID

Listing the feeds

To list all the feeds for a project, folder, or organization, use the following command:

GCLOUD 

gcloud asset feeds list --project=PROJECT_ID

API 

 curl -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -X GET \
  https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds

Updating a feed

To update the attributes of a feed, you need to specify the attribute path in the update_mask and the value of that attribute. The following command updates the assetNames and topic value of a feed on a project.

GCLOUD 

gcloud asset feeds update FEED_ID --project=PROJECT_ID --add-asset-names=ASSET_NAME
   --pubsub-topic="TOPIC"

API 

 curl -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -X PATCH \
  -d '{"feed": {"assetNames": [ASSET_NAME], "feedOutputConfig": {"pubsubDestination": {"topic":TOPIC}}},
       "update_mask": {"paths": ["asset_names", "feed_output_config.pubsub_destination.topic"]}}' \
  https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds/FEED_ID

Deleting a feed

If you no longer want to be notified of asset changes, use the following command to delete a feed on a project.

GCLOUD 

gcloud asset feeds delete FEED_ID --project=PROJECT_ID

API 

 curl -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -X DELETE \
  https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds/FEED_ID

Known limitations

  • It may take up to 10 minutes for any feed creation, update, or deletion to take effect.

  • The consumer project where the feed is created must outlive the feed because the service account used to publish to the destination Pub/Sub topic is located in the consumer project. If the consumer project is deleted, Cloud Asset Inventory cannot publish to the destination. The feed will no longer function and will be deleted as soon as project deletion is permanent.

Troubleshooting

This section shows you how to troubleshoot common issues.

Failure to create or update a feed

If the creation or update of a feed fails, it might caused by an issue with permissions. Each feed creation or update request includes three parts: the consumer project used to call the API, the target parent to be monitored and destination Pub/Sub topic to receive the notification.

To create or update a feed, the following permissions are required:

  • The caller or the service account must have asset permissions in the target parent (that can be a project, folder or organization).
  • The asset service account (service-PROJECT_NUMBER@gcp-sa-cloudasset.iam.gserviceaccount.com) in the Cloud Asset API-enabled consumer project must have pubsub.topics.publish permission on the destination Pub/Sub topic.

An error message that contains does not have permission could indicate that the user or service account does not have the asset permissions. Learn more about the required permissions.

An error message that contains Fail to use [TOPIC_NAME] as feed output destination could indicate that there is an issue publishing the message to the topic you specified in the feed output destination. To resolve the issue:

  • If you are using Google Cloud CLI, make sure you are using the correct project: 
    gcloud config list project
  • Ensure you've specified the correct topic name
  • Ensure that the service account (service-[PROJECT_NUMBER]@gcp-sa-cloudasset.iam.gserviceaccount.com) has the pubsub.topics.publish permission on the topic, where [PROJECT_NUMBER] is the project number of the Cloud Asset Inventory-enabled project you plan to create the feed from.

Failure to receive resource updates or IAM policy updates

If you are not receiving notifications for resource or IAM policy updates, verifying the following configuration details can help resolve the issue:

  • Make sure that the metadata has changed on your assets. The real-time feed only sends updates when the metadata of the supported resource types has changed; operations such as uploading a new file to your Cloud Storage bucket does not trigger a metadata change.
  • Make sure your assets meet one of the criteria you specified in the feed, which are asset names and asset types.
  • Check the logs to see if there are errors when publishing updates to your topic.

Using Cloud Logging

This section describes how to set up and view Logging for Cloud Asset Inventory real-time feeds.

When real-time feeds fail to send resources or IAM policy updates through Pub/Sub, Cloud Asset Inventory logs the error status and message via Logging. Logging is enabled by default. Learn about Google Cloud's operations suite pricing.

Viewing Google Cloud's operations suite logs

To view logs, go to the Logs Explorer.

The real-time feed Logging is indexed by a Pub/Sub Topic. To see all logs, select Cloud Pub/Sub Topic > All topic ids from the first drop-down menu. To see logs for the topic specified in your feed, select a single topic ID from the list.

UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks.

Logged information

Real-time feed log entries contain following types of information:

  • General information shown in most Google Cloud logs, such as severity, project ID, project number, or timestamp.
  • Real-time feed log fields in jsonPayload, which contains asset name, feed output config, error status when publishing resource or IAM policy updates.

The following table shows what kinds of information each field contains.

Field Type and description
name

string

Full Name of the feed. The format is one of the following:
  • projects/{project_number}/feeds/{feed_id}
  • folders/{folder_number}/feeds/{feed_id}
  • organizations/{organization_number}/feeds/{client-assigned_feed_identifier}
asset_name

string

Full name of the asset to receive updates. For example: //compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1

See Resource Names for more info.

feed_output_config

FeedOutputConfig

Feed output configuration defining where the asset updates are published to.

condition

Expr

Feed condition which determines whether an asset update should be published.

error_status

Status

Status when there's a failure to publish asset updates to a feed.