This page explains how to create and manage feeds on a project.
Overview
To receive real-time notifications about resource and policy changes, you can create and subscribe to a feed. When you configure the feed, you can specify that you want to monitor changes of supported resource types, IAM policies, access policies, and organization policies within an organization, folder, project, or for specific resources. Additionally, you can add conditions to your feed so that you only receive notifications for certain types of changes to an asset. After configuring your feed, you immediately receive notifications whenever the specified assets are changed, sent through Pub/Sub (formatted as a TemporalAsset). Real-time notifications connect to your existing workloads. With this functionality, you can merge actions, like creating a Cloud Function to reverse a resource change once detected.
Before you begin
Enable the Cloud Asset API for your project.
Create a new service account if you don't have an existing service account within your project.
Grant your service account permissions to call the API for real-time feeds. The following permissions are needed for each operation:
Permission Description cloudasset.feeds.create
andcloudasset.assets.exportResource
Create feeds cloudasset.feeds.update
andcloudasset.assets.exportResource
Update feeds cloudasset.feeds.delete
Delete feeds cloudasset.feeds.get
Get feeds cloudasset.feeds.list
List feeds Granting your service account the Cloud Asset Owner (
roles/cloudasset.owner
) role grants all permissions related to the Cloud Asset API, including the permissions listed in the preceding table. For more information about roles and permissions, see Understanding roles.Create a Pub/Sub topic if you don't have an existing Pub/Sub topic. Make sure
service-PROJECT_NUMBER@gcp-sa-cloudasset.iam.gserviceaccount.com
has thepubsub.topics.publish
permission on the topic, where PROJECT_NUMBER is the project number of the Cloud Asset API-enabled project you plan to create the feed from. This service account by default has thepubsub.topics.publish
permission on all topics within this Cloud Asset API-enabled project.The service account will be created by calling the API once, or you can use the following command:
gcloud beta services identity create --service=cloudasset.googleapis.com --project=PROJECT_ID
Setting up your environment
GCLOUD
Install the Cloud SDK on your local client if you don't have Cloud SDK installed.
Update all of your installed components to the latest version if you have Cloud SDK installed.
Enable the Resource Manager API for your project.
API
Set up a new Compute Engine VM instance by going to the Create an instance page and selecting the service account within your project.
Under Access scopes, select Allow full access to all Cloud APIs.
Launch your instance by clicking Create.
Go to the VM Instance page.
Open a web SSH client connected to the instance by clicking SSH next to the instance listing.
In the web SSH client, generate an auth token for your service account with the following call:
TOKEN=$(gcloud auth application-default print-access-token)
The following API calls assume you are creating and managing feeds on
a project. If you want to create and manage feeds for an organization or a
folder, swap /projects/PROJECT_NUMBER/
with
/organizations/ORGANIZATION_NUMBER/
or /folders/FOLDER_NUMBER/
.
Creating a feed
You can create up to 200 feeds on an asset. This limit only applies to feeds directly following that asset and does not count the feeds of its children. For example, if you have 10 projects under an organization, each project can have up to 200 feeds and the organization can also have up to 200 feeds.
To create a feed, you can use one of the following options:
- FEED_ID is the unique client-assigned asset feed identifier.
- ASSET_NAME is a list of asset full names for which you want to receive change notifications.
- ASSET_TYPE is a list of asset types for which you want to receive change notifications.
- CONTENT_TYPE is the asset content type for which you want to receive change notifications.
- TOPIC_NAME is the name of the Pub/Sub topic to publish notifications.
- CONDITION_TITLE is the title of the condition to apply on the feed.
- CONDITION_DESCRIPTION is the description of the condition to apply on the feed.
CONDITION_EXPRESSION is the expression of the condition to apply on the feed.
GCLOUD
To create a feed using the
gcloud asset feeds create
command for projects, folders, and organizations:
Projects:
gcloud asset feeds create FEED_ID --project=PROJECT_ID --asset-names="ASSET_NAME" --content-type=CONTENT_TYPE --asset-types="ASSET_TYPE" --pubsub-topic="TOPIC_NAME" --condition-title="CONDITION_TITLE" --condition-description="CONDITION_DESCRIPTION" --condition-expression="CONDITION_EXPRESSION"
Folders:
gcloud asset feeds create FEED_ID --folder=FOLDER_ID --asset-names="ASSET_NAME" --content-type=CONTENT_TYPE --asset-types="ASSET_TYPE" --pubsub-topic="TOPIC_NAME" --condition-title="CONDITION_TITLE" --condition-description="CONDITION_DESCRIPTION" --condition-expression="CONDITION_EXPRESSION"
Organizations:
gcloud asset feeds create FEED_ID --organization=ORGANIZATION_ID --asset-names="ASSET_NAME" --content-type=CONTENT_TYPE --asset-types="ASSET_TYPE" --pubsub-topic="TOPIC_NAME" --condition-title="CONDITION_TITLE" --condition-description="CONDITION_DESCRIPTION" --condition-expression="CONDITION_EXPRESSION"
API
To create a feed using the
feeds.create()
API
for projects, folders, and organizations:
- Projects:
curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" -X POST \ -d '{"feedId": "FEED_ID", "feed": { "assetNames": ["ASSET_NAME"], "assetTypes": ["ASSET_TYPE"], "contentType": "CONTENT_TYPE", "feedOutputConfig": {"pubsubDestination": {"topic":"TOPIC_NAME"}}, "condition": {"title": "CONDITION_TITLE", "description": "CONDITION_DESCRIPTION", "expression": "CONDITION_EXPRESSION"}}}' \ https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds
- Folders:
curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" -X POST \ -d '{"feedId": "FEED_ID", "feed": { "assetNames": ["ASSET_NAME"], "assetTypes": ["ASSET_TYPE"], "contentType": "CONTENT_TYPE", "feedOutputConfig": {"pubsubDestination": {"topic":"TOPIC_NAME"}}, "condition": {"title": "CONDITION_TITLE", "description": "CONDITION_DESCRIPTION", "expression": "CONDITION_EXPRESSION"}}}' \ https://cloudasset.googleapis.com/v1/folders/FOLDER_NUMBER/feeds
- Organizations:
curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" -X POST \ -d '{"feedId": "FEED_ID", "feed": { "assetNames": ["ASSET_NAME"], "assetTypes": ["ASSET_TYPE"], "contentType": "CONTENT_TYPE", "feedOutputConfig": {"pubsubDestination": {"topic":"TOPIC_NAME"}}, "condition": {"title": "CONDITION_TITLE", "description": "CONDITION_DESCRIPTION", "expression": "CONDITION_EXPRESSION"}}}' \ https://cloudasset.googleapis.com/v1/organizations/ORGANIZATION_NUMBER/feeds
Cloud Asset Inventory sets a notification on any asset that matches at least one of your feed's ASSET parameters AND also matches the condition expression, if specified. For example, specifying ASSET_TYPE, ASSET_NAME and CONDITION_EXPRESSION will set notifications on assets that match ASSET_TYPE or ASSET_NAME, and satisfy CONDITION_EXPRESSION.
The following commands create notifications from the
quick_start_topic
Pub/Sub topic when content changes within the
quick_start_bucket
Cloud Storage bucket or any BigQuery tables:
GCLOUD
gcloud asset feeds create quick_start_feed --project=PROJECT_ID --asset-names="//storage.googleapis.com/quick_start_bucket" --content-type=resource --asset-types="bigquery.googleapis.com/Table" --pubsub-topic="projects/PROJECT_ID/topics/quick_start_topic"
API
curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" -X POST \ -d '{"feedId": "quick_start_feed", "feed": { "assetNames": ["storage.googleapis.com/quick_start_bucket"], "assetTypes": ["bigquery.googleapis.com/Table"], "contentType": "RESOURCE", "feedOutputConfig": {"pubsubDestination": {"topic":"projects/PROJECT_ID/topics/quick_start_topic"}}}}' \ https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds
Regular expressions are also supported for the ASSET_TYPE
field. The following
commands create notifications from the quick_start_topic
Pub/Sub topic
when content changes within the resources whose asset type starts with
"compute.googleapis.com". Please see
RE2 for all supported regular
expression syntax.
GCLOUD
gcloud asset feeds create quick_start_feed --project=PROJECT_ID --content-type=resource --asset-types="compute.googleapis.com.*" --pubsub-topic="projects/PROJECT_ID/topics/quick_start_topic"
API
curl -H "Authorization: Bearer $TOKEN"
-H "Content-Type: application/json" -X POST
-d '{"feedId": "quick_start_feed", "feed": { "assetTypes": ["compute.googleapis.com.*"], "contentType": "RESOURCE", "feedOutputConfig": {"pubsubDestination": {"topic":"projects/PROJECT_ID/topics/quick_start_topic"}}}}'
https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds
To get a feed you created, use the following command:
GCLOUD
gcloud asset feeds describe FEED_ID --project=PROJECT_ID
API
curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds/FEED_ID
The feed is returned in the following format, where FULL_NAME_FEED is the feed identifier along with its resource parent:
{ "name": "FULL_NAME_FEED", "assetTypes": ["ASSET_TYPES"], "assetNames": ["ASSET_NAMES"], "contentType": "CONTENT_TYPES", "feedOutputConfig": { "pubsubDestination": { "topic": "TOPIC_NAME" } }, "condition": { "title": "CONDITION_TITLE", "description": "CONDITION_DESCRIPTION", "expression": "CONDITION_EXPRESSION" } }
Receiving updates
After creating a feed, subscribe to updates from the Pub/Sub topic you
specified in the feed. A new feed can take up to five minutes to start sending
notifications. A notification is sent for every change on an asset that matches
either assetNames
or assetTypes
, and satisfies condition
in the feed.
To learn more about Pub/Sub, see the Pub/Sub guide.
Updating a feed
To update the attributes of a feed, you need to specify the attribute path in
the update_mask
and the value of that attribute. The following command
updates the assetNames
and topic
value of a feed on a project.
GCLOUD
gcloud asset feeds update FEED_ID --project=PROJECT_ID --add-asset-names=ASSET_NAME --pubsub-topic="TOPIC"
API
curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" -X PATCH \ -d '{"feed": {"assetNames": [ASSET_NAME], "feedOutputConfig": {"pubsubDestination": {"topic":TOPIC}}}, "update_mask": {"paths": ["asset_names", "feed_output_config.pubsub_destination.topic"]}}' \ https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds/FEED_ID
Deleting a feed
If you no longer want to be notified of asset changes, use the following command to delete a feed on a project.
GCLOUD
gcloud asset feeds delete FEED_ID --project=PROJECT_ID
API
curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" -X DELETE \ https://cloudasset.googleapis.com/v1/projects/PROJECT_NUMBER/feeds/FEED_ID
Adding a condition to a feed
To view only a certain type of change for a given asset, you can add a condition to your feed.
The condition
object is optional. If a feed does not have a condition
object, the configured destination receives all changes to the assets.
The condition
object has the following structure:
"condition": { "title": ... "description": ... "expression": ... }
Title
and description
are optional. These fields can help you identify and
describe the condition.
The expression
field defines the expression used to evaluate the notification
in Common Expression Language (CEL). The
condition expression can contain multiple statements, and statements are
combined using logic operators, following CEL language specification.
Using CEL
Common Expression Language (CEL) is the expression language used to specify a feed condition. It is tailored to express attribute-based logic expressions. For more information, see the CEL spec and its language definition.
In a feed condition, CEL is used to make boolean decisions
based on attribute data. A condition expression consists of one or
more statements that are joined using logic operators (&&
, ||
, or !
). Each
statement expresses an attribute-based control rule that applies to the
TemporalAsset
to determine whether the notification is sent.
The following CEL features are the most important for feed conditions:
- Variables: Conditions use variables to express a given attribute, such as
temporal_asset.deleted
(of type Boolean) ortemporal_asset.asset.name
(of type String). These variables are populated with values based on the context at runtime. - Operators: Every data type, such as String, supports a set of operators
that can be used to create a logic expression. Most commonly, operators are
used to compare the value contained in a variable with a literal value, such
as
temporal_asset.asset.name == "//cloudresourcemanager.googleapis.com/projects/12345"
. In this example, if the input value oftemporal_asset.asset.name
is//cloudresourcemanager.googleapis.com/projects/12345
, then the expression evaluates totrue
. - Functions: A function is a compound operator for data types that support
more complex operations. In condition expressions, there are predefined
functions that can be used in conjunction with a given data type. For example,
temporal_asset.asset.name.contains("keyword")
uses a String contains function, and evaluates totrue
if the value oftemporal_asset.asset.name
contains "keyword". - Logical operators: Conditions supports three logical operators that can be
used to build complex logic expressions from simple expression statements:
&&
,||
, and!
. These logical operators make it possible to use multiple input variables in a condition expression. For example:temporal_asset.deleted && temporal_asset.window.start_time.getFullYear() > 2020
joins two simple statements, and requires both statements to be met in order to produce atrue
overall evaluation result.
For more information about CEL features, see the language definition.
Using condition variables
Condition variables allow you to create conditions on different attributes. Supported condition variables are:
- temporal_asset: The current Asset change in TemporalAsset
format. If the condition evaluates to true, the
TemporalAsset
will be sent to the configured destination.
Example condition expressions
The following condition expression sends notifications on creation events:
!temporal_asset.deleted &&
temporal_asset.prior_asset_state == google.cloud.asset.v1.TemporalAsset.PriorAssetState.DOES_NOT_EXIST
The following condition expression sends notifications for resources that are
located in folders 12345
and 23456
:
"folders/12345" in temporal_asset.asset.ancestors ||
"folders/23456" in temporal_asset.asset.ancestors
The following condition expression sends notifications when new allowed rules
are added to firewalls, assuming asset_type
is already set to
compute.googleapis.com/Firewall
in the feed.
size(temporal_asset.asset.resource.data.allowed) >
size(temporal_asset.prior_asset.resource.data.allowed)
The following condition expression sends notifications for VM instances with
n1-standard-1
machine type, assuming asset_type
is already set to
compute.googleapis.com/Instance
in the feed.
temporal_asset.asset.resource.data.machineType.endsWith("/machineTypes/n1-standard-1")
The following condition expression sends notifications for storage buckets with
any IAM policies for allUsers
, assuming asset_type
is set to
storage.googleapis.com/Bucket
and content_type
is set to IAM_POLICY
in the
feed.
temporal_asset.asset.iam_policy.bindings.exists(b, b.members.exists(m, m == "allUsers"))
Known limitations
Most variable names in condition expressions are represented in snake_case. The only exception is the variable names of the sub-fields of
data
in Resource object, which are represented in camelCase.Condition expressions have a length limit of 1000 characters.
Some validations on condition expressions are performed during feed creation/update time. However, such validations are not comprehensive, especially for conditions set on
temporal_asset.asset.resource.data
field, which has dynamic type. We recommend simple feed conditions with the properasset_type
specified in feed. If validation errors happen during evaluation time, the notification is not sent and errors are logged. Learn about viewing logs.For dynamic types in
temporal_asset.asset.resource.data
, conditions specified on absent fields trigger runtime errors and notification are not published. For example, for conditiontemporal_asset.asset.resource.data.name != "my_name"
, ifname
field is missing in an update, the evaluation fails and you do not receive notifications. If your condition only works in the presence of certain fields, check the existence in the condition to ensure it is properly evaluated.Static enum types could be represented as either fully qualified path names or raw integers. For example, the following expressions are valid for
prior_asset_state
:temporal_asset.prior_asset_state == google.cloud.asset.v1.TemporalAsset.PriorAssetState.DOES_NOT_EXIST
and
temporal_asset.prior_asset_state == 3
Dynamic enum types in
temporal_asset.asset.resource.data
are represented as raw strings. For example, the following expression is valid for asset typecloudresourcemanager.googleapis.com/Project
:temporal_asset.asset.resource.data.lifecycleState == "ACTIVE"
Troubleshooting
This section shows you how to troubleshoot common issues.
Failure to create or update a feed
If you fail to create or update a feed with the error message Fail to use
[TOPIC_NAME] as feed output destination
, it means there is an issue publishing
the message to the topic you specified in the feed output destination. To
resolve the issue:
- Ensure you've specified the correct topic name
- Ensure that the service account
(
service-[PROJECT_NUMBER]@gcp-sa-cloudasset.iam.gserviceaccount.com
) has thepubsub.topics.publish
permission on the topic, where[PROJECT_NUMBER]
is the project number of the Cloud Asset Inventory-enabled project you plan to create the feed from.
Failure to receive resource updates or IAM policy updates
If you are not receiving notifications for resource or IAM policy updates, verifying the following configurations could help.
- Make sure that the metadata has changed on your assets. The real-time feed only sends updates when the metadata of the supported resource types has changed; operations such as uploading a new file to your Cloud Storage bucket does not trigger a metadata change.
- Make sure your assets meet one of the criteria you specified in the feed, which are asset names and asset types.
- Check the logs to see if there are errors when publishing updates to your topic.
Using Cloud Logging
This section describes how to set up and view Logging for Cloud Asset Inventory real-time feeds.
When real-time feeds fail to send resources or IAM policy updates through Pub/Sub, Cloud Asset Inventory logs the error status and message via Logging. Logging is enabled by default. Learn about Google Cloud's operations suite pricing.
Viewing Google Cloud's operations suite logs
To view logs, go to the Logs Viewer.
The real-time feed Logging is indexed by a Pub/Sub Topic. To see all logs, select Cloud Pub/Sub Topic > All topic ids from the first drop-down menu. To see logs for the topic specified in your feed, select a single topic ID from the list.
UTF-8 encoding is enforced for log fields. Characters that are not UTF-8 characters are replaced with question marks.
Logged information
Real-time feed log entries contain following types of information:
- General information shown in most Google Cloud logs, such as severity, project ID, project number, or timestamp.
- Real-time feed log fields in
jsonPayload
, which contains asset name, feed output config, error status when publishing resource or IAM policy updates.
The following table shows what kinds of information each field contains.
Field | Type and description |
---|---|
name |
|
asset_name |
Full name of the asset to receive updates. For example:
|
feed_output_config |
Feed output configuration defining where the asset updates are published to. |
condition |
Feed condition which determines whether an asset update should be published. |
error_status |
Status when there's a failure to publish asset updates to a feed. |