Keep a better eye on your Google Cloud environment

Monitoring, managing and understanding your cloud environment can be a challenging task for large-scale organizations. We built Google Cloud Asset Inventory so IT, security, and ops admins can get easy visibility into their Google Cloud Platform (GCP) environment. Cloud Asset Inventory is a fully managed metadata inventory service that offers various services to access GCP assets and see asset history. Two new features can make it even easier for you to do continuous asset monitoring and deep asset analysis across your GCP assets. 

Real-time notification feature for continuous monitoring
Cloud Asset Inventory now brings the real-time notification feature to beta, letting you do real-time config monitoring. For example, you can get notifications as soon as a firewall rule is changed for your web front end, or if an IAM policy binding in your production project has changed. The notifications are sent through Cloud Pub/Sub, from where you can then trigger actions. 

The example diagram below shows you how to monitor an IAM policy and trigger actions using Cloud Asset Inventory. In this scenario, a Gmail account was added to an IAM policy, which is generally against organizational security policy. If real-time notifications are set up on that IAM policy, Cloud Asset Inventory will send a Cloud Pub/Sub message containing the new change as soon as the change occurs. You can then write Cloud Functions to trigger an email notification, as well as directly revert the change back. You can see the IAM policy’s previous state by getting the change history of the IAM policy through the existing Cloud Asset Inventory export history feature.

Native BigQuery export feature for in-depth asset analysis
Given high demand from customers, and the popularity of the related open source tool, we’ve launched native BigQuery export support in Cloud Asset Inventory. You can directly export your asset snapshots and write to a BigQuery table using the same API or CLI. This enables lots of in-depth asset analysis, asset validation, and rule-based scannings. 

One of our customers from Paypal has been a longtime Cloud Asset Inventory customer, and recently got a chance to adopt the BigQuery export feature. Here’s how they’ve been using it:

“With the adoption of GCP and all of the associated services, Paypal was drowning in unorganized data. With multiple organizations and thousands of projects, we needed a method to gain insight and control of our cloud usage,” says Micah Norman, cloud engineer at Paypal. He initially created a Python application that queried all of the relevant APIs individually and stored the results in CloudSQL and BigQuery. This application worked well, but since Paypal has such a large number of assets, the entire job took about three hours per run. 

“The release of the Asset Export API allowed me to cut out nearly half of the code,” says Norman. “No longer did I have to query multiple APIs for each project. Now, with a simple bash script of around 60 lines, I was able to collect all of the relevant data in seconds. The remaining code primarily dealt with reading the resulting data and storing it correctly in CloudSQL and BigQuery.”

With the most recent release of the Asset Export API, Norman was able to write directly to BigQuery from the Asset Export API, thus eliminating 40% of the remaining code. The only code remaining was rewritten in Go, and supported the collection of data external to GCP, such as G Suite data. Analysis is supported using SQL to denormalize the collected information to support reporting, auditing, and compliance efforts.

Here’s a look at how the table looks in BigQuery with Cloud Asset Inventory data:

For example, you can easily query the following common questions in BigQuery:

1. Find the quantity of each asset type:

2. Find Cloud IAM policies containing Gmail accounts as a member:

With the broad resource and policy coverage from Cloud Asset Inventory, plus the powerful query capability of BigQuery, in-depth inventory analysis has gotten so much easier. Read more about how to analyze your asset data in BigQuery.

Try these new real-time notifications and BigQuery export features for better inventory management, monitoring, and deep analysis.