Export Cloud Billing data to BigQuery

Stay organized with collections Save and categorize content based on your preferences.

Cloud Billing export to BigQuery enables you to export detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a BigQuery dataset that you specify. Then you can access your Cloud Billing data from BigQuery for detailed analysis, or use a tool like Looker Studio to visualize your data. You can also use this export method to export data to a JSON file.

Timing is important. To have access to a more comprehensive set of Google Cloud billing data for your analysis needs, we recommend that you enable Cloud Billing data export to BigQuery at the same time that you create a Cloud Billing account.

See the limitations that might impact exporting your billing data to BigQuery.

Next steps

Managing and reporting costs effectively is a critical part of financial stewardship, whether you're running a multi-billion-dollar enterprise business or small household budget. Making data-driven decisions about your Google Cloud costs and usage starts with collecting the data you'll need to inform those decisions.

Refer to the guides in this section to learn about the following tasks:

Set up Cloud Billing data export to BigQuery

To start collecting your Cloud Billing data, you must enable Cloud Billing data export to BigQuery.

The setup guide provides best practice recommendations and detailed instructions for enabling Cloud Billing data export to BigQuery. These are the following types of Cloud Billing data you can enable for export:

  • Standard usage cost data - Contains standard Cloud Billing account cost usage information, such as account ID, invoice date, services, SKUs, projects, labels, locations, cost, usage, credits, adjustments, and currency.

    Use the Standard usage export to analyze broad trends in your cost data.

  • Detailed usage cost data - Contains detailed Cloud Billing account cost usage information. Includes everything in the standard usage cost data plus resource-level cost data, like a virtual machine or SSD that generates service usage.

    Use the Detailed usage export to analyze costs at the resource level, and identify specific resources that might be driving up costs. You can view information about GKE and Compute Engine resources in the detailed report.

    To view information about GKE, enable cost allocation in detailed exports.

    Review the schema of the detailed usage cost data for further recommendations and limitations.

  • Pricing data - Contains Cloud Billing account pricing information, such as account ID, services, SKUs, products, geographic metadata, pricing units, currency, aggregation, and tiers.

Using BigQuery to store and query Cloud Billing data will incur minimal fees. For more information, see Cost of use.

See the limitations that might impact exporting your billing data to BigQuery.

Understand the Cloud Billing data tables

After you enable Cloud Billing export to BigQuery, Cloud Billing data tables are automatically created in the BigQuery dataset.

To understand the data schema of your exported content, see the reference information for the contents of the Cloud Billing data that is exported to each table in the BigQuery dataset.

Find example queries for Cloud Billing data

For tips and guidance for using SQL to run queries on your Cloud Billing data, view the example queries.

On the example queries page, you'll find various SQL examples, including the following:

Cost of use

Using BigQuery to store usage data typically incurs minimal fees.

  • Loading data into the designated dataset is free; this action takes advantage of BigQuery's pool of shared resources to load data in batches.
  • When exporting and analyzing Cloud Billing data with BigQuery, the associated cost depends on the amount of data you stream, store, and query.
  • Many partitioned table operations are free, including loading data into partitions, copying partitions, and exporting data from partitions. Though free, these operations are subject to BigQuery's quotas and limits.

Generally, querying the Detailed cost export might cost more than querying the Standard export. To optimize your costs, we recommend using the Standard export to analyze trends in your costs, and using the Detailed export to track costs at the resource level, and identify specific resources that might be driving your costs.

To get an idea of what charges to expect, see Estimating storage and query costs.

For more information on best practices for optimizing costs in BigQuery, see Control costs in BigQuery.

For detailed prices, review BigQuery pricing.

Limitations

Exporting Cloud Billing data to BigQuery is subject to the following limitations.

  • Your BigQuery datasets containing standard usage cost and pricing data only reflect Google Cloud billing data incurred from the date you set up Cloud Billing export, and after. That is, Google Cloud billing data is not added retroactively, so you won't see Cloud Billing data from before you enable export. For more details, see Data availability.

  • For your BigQuery datasets containing detailed usage cost data, you can configure your dataset to use any supported location. The type of location you configure on the dataset will impact the timing of when your Google Cloud billing data is exported to the dataset:

    • If you configure the dataset to use a multi-region location, the dataset reflects Google Cloud billing data incurred from the start of the previous month from when you first enabled the export, unless you are re-enabling the export. That is, Google Cloud billing data is added retroactively for the current and previous month.
    • If your dataset is configured to use a region location, your detailed usage cost data only reflects Google Cloud billing data incurred starting from the date you enabled Cloud Billing export, and after. That is, Google Cloud billing data is not added retroactively for non-multi-region dataset locations, so you won't see Cloud Billing data from before you enable export.
    • For more details, see Data availability.
  • When exporting detailed usage cost data, the detailed export automatically includes resource-level information about Compute Engine. To view a breakdown of Google Kubernetes Engine (GKE) cluster costs in a detailed data export, you must also enable cost allocation for GKE.

  • Customer Managed Encryption Keys (CMEK) are not supported when exporting billing data to BigQuery. If you enable CMEK for your billing data dataset, this prevents Cloud Billing from writing billing data to the appropriate tables within that dataset. Instead, you need to configure the dataset to use Google-managed key encryption.

  • Tags are limited to Compute Engine instances and might take up to an hour to propagate to BigQuery exports. If a tag has been added or removed within an hour, or if a resource has existed for less than an hour, it might not appear in the export.