Export Cloud Billing data to BigQuery

Cloud Billing export to BigQuery enables you to export detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a BigQuery dataset that you specify. Then you can access your Cloud Billing data from BigQuery for detailed analysis, or use a tool like Google Data Studio to visualize your data. You can also use this export method to export data to a JSON file.

If you use regular file export for your Cloud Billing data and are looking for information about your data exported to Cloud Storage, see Export Cloud Billing data to a file.

Timing is important. To have access to a more comprehensive set of Google Cloud billing data for your analysis needs, we recommend that you enable Cloud Billing data export to BigQuery at the same time that you create a Cloud Billing account.

See the limitations that might impact exporting your billing data to BigQuery.

Next steps

Managing and reporting costs effectively is a critical part of financial stewardship, whether you're running a multi-billion-dollar enterprise business or small household budget. Making data-driven decisions about your Google Cloud costs and usage starts with collecting the data you'll need to inform those decisions.

Refer to the guides in this section to learn about the following tasks:

Set up Cloud Billing data export to BigQuery

To start collecting your Cloud Billing data, you must enable Cloud Billing data export to BigQuery.

The setup guide provides best practice recommendations and detailed instructions for enabling Cloud Billing data export to BigQuery. These are the following types of Cloud Billing data you can enable for export:

  • Standard usage cost data - Contains standard Cloud Billing account cost usage information, such as account ID, invoice date, services, SKUs, projects, labels, locations, cost, usage, credits, adjustments, and currency.
  • Detailed usage cost data - Contains detailed Cloud Billing account cost usage information. Includes everything in the standard usage cost data plus resource-level cost data, like a virtual machine or SSD that generates service usage. Enable this if you would like to analyze your resource-level cost data along with your standard usage cost data. You do not need to enable both standard and detailed. Review the schema of the detailed usage cost data for further recommendations and limitations.
  • Pricing data - Contains Cloud Billing account pricing information, such as account ID, services, SKUs, products, geographic metadata, pricing units, currency, aggregation, and tiers.

Using BigQuery to store and query Cloud Billing data will incur minimal fees. For more information, see Cost of use.

See the limitations that might impact exporting your billing data to BigQuery.

Understand the Cloud Billing data tables

After you enable Cloud Billing export to BigQuery, Cloud Billing data tables are automatically created in the BigQuery dataset.

To understand the data schema of your exported content, see the reference information for the contents of the Cloud Billing data that is exported to each table in the BigQuery dataset.

Find example queries for Cloud Billing data

For tips and guidance for using SQL to run queries on your Cloud Billing data, view the example queries.

On the example queries page, you'll find various SQL examples, including the following:

Cost of use

Using BigQuery to store usage data will incur minimal fees.

  • Loading data into the designated dataset is free; this action takes advantage of BigQuery's pool of shared resources to load data in batches.
  • When exporting and analyzing Cloud Billing data with BigQuery, the associated cost will depend on the amount of data you stream, store, and query.

To get an idea of what charges to expect, see Estimating storage and query costs.

Summary of BigQuery pricing

Here is a summary of the BigQuery costs that you'll likely incur when exporting and analyzing your Cloud Billing data with BigQuery. For more details, review BigQuery pricing.

SKU Pricing Details
Active Storage $0.020 per GB The first 10 GB is free each month.
Long Term Storage $0.010 per GB The first 10 GB is free each month.
Analysis $5.00 per TB on-demand pricing First 1 TB per month is free.
Flat-rate pricing is available for high-volume customers.
Streaming Inserts $0.010 per 200 MB Charged per row using a 1 KB minimum size.
*NO FREE Usage for this SKU

Many partitioned table operations are free, including loading data into partitions, copying partitions, and exporting data from partitions. Though free, these operations are subject to BigQuery's quotas and limits. For information on all free operations, see Free operations on the BigQuery pricing page.

For more information on best practices for controlling costs in BigQuery, see Controlling costs in BigQuery.

Limitations

Exporting Cloud Billing data to BigQuery is subject to the following limitations.

  • If the Cloud Billing account connected to your BigQuery dataset is inactive, for example, if there are issues with your payment method, your BigQuery dataset won't have data for the time that your account is inactive.
  • Your BigQuery datasets containing standard usage cost and pricing data only reflect Google Cloud billing data incurred from the date you set up Cloud Billing export, and after. That is, Google Cloud billing data is not added retroactively, so you won't see Cloud Billing data from before you enable export. For more details, see Data availability.
  • Your BigQuery datasets containing detailed usage cost data reflect Google Cloud billing data incurred from the start of the previous month from when you first enabled the export, unless you are re-enabling it. That is, Google Cloud billing data is added retroactively only for the current and previous month. For more details, see Data availability.
  • When exporting detailed usage cost data, you must export to a dataset configured with a multi-region location, such as United States (US) or European Union (EU). If you select a dataset that is not a multi-region location, this prevents Cloud Billing from writing detailed billing data to the appropriate tables within that dataset.
  • Customer Managed Encryption Keys (CMEK) are not supported when exporting billing data to BigQuery. If you enable CMEK for your billing data dataset, this prevents Cloud Billing from writing billing data to the appropriate tables within that dataset. Instead, you need to configure the dataset to use Google-managed key encryption.