Set up Cloud Billing data export to BigQuery

This guide provides all required setup steps to enable Cloud Billing data export to BigQuery.

Overview

To export Cloud Billing data to BigQuery, you need to take the following steps:

  • Create a project where the Cloud Billing data will be stored, and enable billing on the project (if you have not already done so).
  • Configure permissions on the project and on the Cloud Billing account.
  • Enable the BigQuery Data Transfer Service API (required to export your pricing data).
  • Create a BigQuery dataset in which to store the data.
  • Enable Cloud Billing export of cost data and pricing data to be written into the dataset.

This document walks you through the details of each of these steps.

About the Cloud Console

The Google Cloud Console (visit documentation, open console) is a web UI used to provision, configure, manage, and monitor systems that use Google Cloud products. You use the Cloud Console to set up and manage Google Cloud and Cloud Billing resources.

Permissions required for this task

To enable and configure the export of Google Cloud billing data to a BigQuery dataset, you need the following permissions:

Additionally, to enable and configure export of Cloud Billing pricing data, you need the following permissions:

  • BigQuery Admin role for the Cloud project that contains the BigQuery dataset that will be used to store the Cloud Billing pricing data
  • resourcemanager.projects.update permission for the Cloud project containing the target dataset. This is included in the roles/editor role.

For more information about Google Cloud permissions, see:

Enable Cloud Billing data export to BigQuery

Complete the steps detailed in this section to enable the export of your Cloud Billing data to BigQuery.

1. Select or Create a project

Cloud Billing data exported to BigQuery is contained in a BigQuery dataset. A dataset is contained within a Cloud project you specify.

Recommendation about projects: We recommend that you create a Cloud project to contain all of your billing administration needs, including your exported Cloud Billing data. You can also use this billing administration Cloud project for things like Cloud Billing APIs access, Pub/Sub channels for programmatic budget notifications, and other Cloud Billing administration tasks.

Important: The Cloud project you select to contain your dataset should be linked to the same Cloud Billing account that contains the data that you plan to export to the BigQuery dataset. A Cloud project is linked to exactly one Cloud Billing account at a time. A Cloud Billing account is linked to one or more projects.

In the Cloud Console, on the project selector page, select or create a Cloud project.

Go to the project selector page

Learn more about creating and managing projects.

2. Verify that billing is enabled

You must make sure that billing is enabled on the Cloud project you select to contain your dataset, and that the Cloud project is linked to the same Cloud Billing account that contains the data that you plan to export to the BigQuery dataset.

3. Enable the BigQuery Data Transfer Service API

If you plan to export your Cloud Billing pricing data to BigQuery, you need to enable the BigQuery Data Transfer Service API. If you don't plan to export your Cloud Billing pricing data, you can skip to Step 4. Create a BigQuery dataset.

The BigQuery Data Transfer Service API should be enabled in the same Cloud project you are using to contain your BigQuery datasets. If you followed the recommendation in the project section, this is your billing administration Google Cloud project.

To enable the BigQuery Data Transfer Service API, do the following:

  1. Sign in to the Google Cloud Console and go to the API page.

    Go to the BigQuery Data Transfer Service API page

  2. Click Enable to enable the API.

  3. When you are prompted, select a project. You must enable the BigQuery Data Transfer Service API in the same project you are using to contain your BigQuery dataset. If you followed the recommendation in the project section, this is your billing administration Google Cloud project.

For more information on enabling APIs, see the Service Usage documentation.

4. Create a BigQuery dataset

Before you enable your Cloud Billing data to export to BigQuery, you must create at least one BigQuery dataset to manage your exported data. You can use the same dataset to contain both your daily cost detail data and your pricing data.

A dataset is contained within a Cloud project you specify. Datasets are top-level containers that are used to organize and control access to your tables and views. A table or view must belong to a dataset, so you need to create at least one dataset before loading data into BigQuery.

To create a BigQuery dataset, do the following:

  1. Sign in to the Google Cloud Console and go to the BigQuery page.

    Go to BigQuery page

  2. In the project drop-down list ( My Project ) at the top of the Google Cloud Console page, select the project you set up to contain your dataset.

  3. In the BigQuery navigation panel, click your-project-name.

  4. Click Create dataset. The Create dataset panel opens.

    1. Enter a Dataset ID.
    2. Select a Data location.

      About locations: The data location specifies the region where your data is stored. All tables within this dataset will share this location. When creating a dataset, the Default location is the US multi-region. After you create the dataset, the location cannot be changed. Learn more about locations.

    3. Select the Default table expiration.

    4. Select the Encryption option. For Cloud Billing export, select Google-managed key.

    5. To save, click Create dataset.

5. Enable Cloud Billing export to the BigQuery dataset

You enable Cloud Billing data export in the Cloud Billing section of the Cloud Console.

To enable Cloud Billing data export to BigQuery, do the following:

  1. Sign in to the Google Cloud Console.

    Sign in to Cloud Console

  2. Open the console Navigation menu (), and then select Billing.

    If you have more than one Cloud Billing account, do one of the following:

    • To manage Cloud Billing for the current Cloud project, select Go to linked billing account.
    • To locate a different Cloud Billing account, select Manage billing accounts and choose the account for which you'd like to enable Cloud Billing data export to BigQuery.
  3. In the Billing navigation menu, select Billing export.

  4. Select the BigQuery export tab (this tab is selected by default). On the BigQuery export tab, you can enable two types of data you want to export:

Export the Daily cost detail data

To enable export of your detailed Cloud Billing usage and cost data, do the following:

  1. To enable export or update the export settings, click Edit settings.
  2. From the Project list, select the project that you set up to contain your BigQuery dataset.

  3. From the Billing export dataset list, select the dataset that you set up to contain your exported Cloud Billing data.

    Tips:

    • The BigQuery API is required to export data to BigQuery. If the project you selected doesn't have the BigQuery API enabled, you will be prompted to enable it. Click Enable BigQuery API and the API will be enabled for you.
    • If the project you selected doesn't contain any BigQuery datasets, you will be prompted to create one. If necessary, follow these steps to create a new dataset.
  4. Click Save.

Export the Pricing data

To enable export of Cloud Billing pricing data, do the following:

  1. To enable export or update the export settings, click Edit settings.
  2. From the Project list, select the project that you set up to contain your BigQuery dataset.

  3. From the Billing export dataset list, select the dataset that you set up to contain your exported Cloud Billing data.

    Tips:

    • The BigQuery API is required to export data to BigQuery. If the project you selected doesn't have the BigQuery API enabled, you will be prompted to enable it. Click Enable BigQuery API and the API will be enabled for you.
    • The BigQuery Data Transfer Service API is required to export pricing data to BigQuery. If the project you selected doesn't have the BigQuery Data Transfer Service API enabled, you will be prompted to enable it. If necessary, follow these steps to enable the API
    • If the project you selected doesn't contain any BigQuery datasets, you will be prompted to create one. If necessary, follow these steps to create a new dataset.
  4. Click Save.

About dataset permissions

Cloud Billing export to BigQuery uses a service account to manage dataset permissions.

After you enable Cloud Billing export to BigQuery, Google automatically adds a service account as an owner to the dataset that you specify. The service account looks like this:

  • billing-export-bigquery@system.gserviceaccount.com

The service account is owned and managed by Google and provides the permissions that are necessary for our offline process to create a table and write Cloud Billing records to it.

Billing data tables

Shortly after enabling Cloud Billing export to BigQuery, billing data tables are automatically created in the BigQuery dataset. To learn more about these BigQuery tables and the table schemas, see Understanding the auto-created data tables in BigQuery.

Frequency of data loads

  • When you first enable the daily cost detail export to BigQuery, it might take a few hours to start seeing your Google Cloud cost data.
  • When you first enable the pricing export to BigQuery, it might take up to 48 hours to start seeing your Google Cloud pricing data.

To learn more about the frequency of the data loads into your BigQuery tables, see Frequency of data loads.

Cost and pricing reports available in the Google Cloud Console