Monitoring asset changes

Beta

This feature is in a pre-release state and might change or have limited support. For more information, see the product launch stages.

This page explains how to create and manage feeds on a project.

Overview

Cloud Asset Inventory allows you to monitor resource and policy changes you're subscribed to through real-time notifications. You can monitor changes of supported resource types within an organization, folder, project, or specific resources. Subscriptions are set up by creating a feed.

For example, you can monitor security sensitive resources such as firewall rules and forwarding rules. Any changes to these resources immediately sends a notification through Pub/Sub, allowing you to take quick action if needed.

Real-time notifications connect to your existing workloads. With this functionality, you can merge actions, like creating a Cloud Function to reverse a resource change once detected.

Before you begin

1. Enable the Cloud Asset API for your project.

2. Create a new service account if you don't have an existing service account within your project.

3. Grant your service account permissions to call the Real-time Feed API. The following permissions are needed.

   Permission	Description
   cloudasset.feeds.create and cloudasset.assets.exportResource	Create feeds
   cloudasset.feeds.update	Update feeds
   cloudasset.feeds.delete	Delete feeds
   cloudasset.feeds.get	Get feeds
   cloudasset.feeds.list	List feeds

   Granting your service account the cloudasset.owner role grants all permissions related to the Cloud Asset API, including the ones in the table. For more information about roles and permissions, see Understanding roles.

4. Create a Pub/Sub topic if you don't have an existing Pub/Sub topic. Make sure [e-mail] has the pubsub.topics.publish permission on the topic, where PROJECT_NUMBER is the project number of the Cloud Asset API-enabled project you plan to create the feed from. This service account by default has the pubsub.topics.publish permission on all topics within this Cloud Asset API-enabled project.

Setting up your environment

GCLOUD

1. Install the Cloud SDK on your local client if you don't have Cloud SDK installed.

2. Update all of your installed components to the latest version if you have Cloud SDK installed.

3. Enable the Cloud Resource Manager API for your project.

API

1. Set up a new Compute Engine VM instance by going to the Create an instance page and selecting the service account within your project.

2. Under Access scopes, select Allow full access to all Cloud APIs.

3. Launch your instance by clicking Create.

4. Go to the VM Instance page.

5. Open a web SSH client connected to the instance by clicking SSH next to the instance listing.

6. In the web SSH client, generate an auth token for your service account with the following call:

   TOKEN=$(gcloud auth application-default print-access-token)
   

Note that the following API calls assume you are creating and managing feeds on a project. If you want to create and manage feeds for an organization or a folder, swap /projects/PROJECT_NUMBER/ with /organizations/ORGANIZATION_NUMBER/ or /folders/FOLDER_NUMBER/.

Creating a feed

The following command creates a feed on a project.

GCLOUD

 gcloud beta asset feeds create FEED_ID --project=PROJECT_ID --asset-names="ASSET_NAME"
  --content-type=CONTENT_TYPE --asset-types="ASSET_TYPE"
  --pubsub-topic="TOPIC_NAME"
 

API

   curl -H "Authorization: Bearer $TOKEN" 

        -H "Content-Type: application/json" -X POST 

        -d '{"feedId": "FEED_ID",
             "feed": { "assetNames": ["ASSET_NAME"],
             "assetTypes": ["ASSET_TYPE"], "contentType": "CONTENT_TYPE",
             "feedOutputConfig": {"pubsubDestination": {"topic":"TOPIC_NAME"}}}}' 

        https://cloudasset.googleapis.com/v1p2beta1/projects/PROJECT_NUMBER/feeds
 

Where:

• FEED_ID is the unique client-assigned asset feed identifier.
• ASSET_NAME is a list of asset full names that you want to receive change notifications for.
• ASSET_TYPE is a list of asset types to receive change notifications for.
• CONTENT_TYPE is the asset content type to receive change notifications for.
• TOPIC_NAME is the name of the Pub/Sub topic to publish notifications.

Cloud Asset Inventory sets a notification on any asset that matches at least one of your feed's parameters. For example, specifying ASSET_TYPE and ASSET_NAME will set notifications on assets that match ASSET_TYPE or ASSET_NAME.

Note: Notifications will only be sent when the resource or policy metadata of the resource itself gets changed. In this example, changing any resource under Cloud Storage buckets or BigQuery tables will not trigger a notification.

The following commands create notifications from the quick_start_topic Pub/Sub topic when content changes within the quick_start_bucket Cloud Storage bucket or any BigQuery tables:

GCLOUD

 gcloud beta asset feeds create quick_start_feed --project=PROJECT_ID --asset-names="//storage.googleapis.com/quick_start_bucket"
  --content-type=resource --asset-types="bigquery.googleapis.com/Table"
  --pubsub-topic="projects/PROJECT_ID/topics/quick_start_topic"
 

API

 curl -H "Authorization: Bearer $TOKEN" 

  -H "Content-Type: application/json" -X POST 

  -d '{"feedId": "quick_start_feed",
       "feed": { "assetNames": ["storage.googleapis.com/quick_start_bucket"],
       "assetTypes": ["bigquery.googleapis.com/Table"],
       "contentType": "RESOURCE",
       "feedOutputConfig": {"pubsubDestination": {"topic":"projects/PROJECT_ID/topics/quick_start_topic"}}}}' 

        https://cloudasset.googleapis.com/v1p2beta1/projects/PROJECT_NUMBER/feeds
 

To get a feed you created, use the following curl command:

GCLOUD

 gcloud beta asset feeds describe FEED_ID --project=PROJECT_ID
 

API

 curl -H "Authorization: Bearer $TOKEN" 

  -H "Content-Type: application/json" 

  https://cloudasset.googleapis.com/v1p2beta1/projects/PROJECT_NUMBER/feeds/FEED_ID
 

The feed is returned in the following format:

{
  "name": "FULL_NAME_FEED",
  "assetTypes": ["ASSET_TYPES"],
  "assetNames": ["ASSET_NAMES"],
  "contentType": "CONTENT_TYPES",
  "feedOutputConfig": {
    "pubsubDestination": {
      "topic": "TOPIC_NAME"
    }
  }
}

Where:

• FULL_NAME_FEED is the feed identifier along with its resource parent.

Receiving updates

After creating a feed, subscribe to updates from the Pub/Sub topic you specified in the feed. A new feed can take up to five minutes to start sending notifications. A notification is sent for every change on an assets that match either assetNames or assetTypes in the feed.

To learn more about Pub/Sub, see the Pub/Sub guide.

Updating a feed

To update the attributes of a feed, you need to specify the attribute path in the update_mask and the value of that attribute. The following curl command updates the assetNames and topic value of a feed on a project.

GCLOUD

gcloud beta asset feeds update FEED_ID --project=PROECT_ID --add-asset-names=ASSET_NAME
   --pubsub-topic="TOPIC"

API

 curl -H "Authorization: Bearer $TOKEN" 

  -H "Content-Type: application/json" -X PATCH 

  -d '{"feed": {"assetNames": [ASSET_NAME], "feedOutputConfig": {"pubsubDestination": {"topic":TOPIC}}},
       "update_mask": {"paths": ["asset_names", "feed_output_config.pubsub_destination.topic"]}}' 

  https://cloudasset.googleapis.com/v1p2beta1/projects/PROJECT_NUMBER/feeds/FEED_ID
 

Deleting a feed

If you no longer want to be notified of asset changes, use the following curl command to delete a feed on a project.

GCLOUD

gcloud beta asset feeds delete FEED_ID --project=PROECT_ID

API

 curl -H "Authorization: Bearer $TOKEN" 

  -H "Content-Type: application/json" -X DELETE 

  https://cloudasset.googleapis.com/v1p2beta1/projects/PROJECT_NUMBER/feeds/FEED_ID
 

FAQ

Fail to create or update a feed

If you fail to create or update a feed with the error message Fail to use [TOPIC_NAME] as feed output destination, it means there is an issue publishing the message to the topic you specified in the feed output destination. To resolve the issue:

• Ensure you've specified the correct topic name
• Ensure that the service account ([e-mail]) has the pubsub.topics.publish permission on the topic, where PROJECT_NUMBER is the project number of the Cloud Asset Inventory-enabled project you plan to create the feed from.

Fail to receive updates of your resource or Cloud IAM policy update

There are couple of reasons can cause you fail to receive the updates.

• Make sure there that the metadata has changed on your assets. The real-time feed will only send updates when the metadata of the supported resource types has changed; operations such as uploading a new file to your Cloud Storage bucket will not trigger a metadata change.
• Make sure your assets meet one of the criterias you specified in the feed, which are asset names and asset types.
• Check the logs to see if there are errors when publishing updates to your topic.

Stackdriver Logging

This section describes how to set up and view Logging for Cloud Asset Inventory real-time feeds.

When real-time feeds fail to send resources or Cloud IAM policy updates through Pub/Sub, we will log the error status and message via Logging. Logging is enabled by default. For price information, please see Stackdriver pricing.

How to view Stackdriver logs

To view logs, go to the Logs Viewer.

Real-time feed Logging is indexed by a Pub/Sub Topic. To see all logs, select Cloud Pub/Sub Topic > All topic ids from the first drop-down menu. To see logs for the topic specified in your feed, select a single topic id from the list.

UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks.

What is logged

Real-time feed log entries contain following types of information:

• General information shown in most Google Cloud logs, such as severity, project ID, project number, timestamp, and so on.
• Real-time feed log fields in jsonPayload, which contains asset name, feed output config, error status when publishing resource or Cloud IAM policy updates.

Fields
name	string Full Name of the feed. The format will be one of the following: projects/{project_number}/feeds/{feed_id} folders/{folder_number}/feeds/{feed_id} organizations/{organization_number}/feeds/{client-assigned_feed_identifier}
asset_name	string Full name of the asset to receive updates. For example: //compute.googleapis.com/projects/my_project_123/zones/zone1/instances/instance1. See Resource Names for more info.
feed_output_config	FeedOutputConfig Feed output configuration defining where the asset updates are published to.
error_status	Status Status when fail to publish asset updates to Feed.