BigQuery Monitoring using Stackdriver

This document describes how to monitor BigQuery resources using Stackdriver Monitoring, including creating charts and alerts for BigQuery metrics.

Before you begin

  • Before you use Stackdriver Monitoring, ensure that you have the following:

    • A Google Cloud Platform account.
    • A BigQuery project with billing enabled.

    One way to ensure that you have both is to complete the Quickstart Using the Web UI.

  • If you do not have a Stackdriver account, follow the steps in the Create a Stackdriver Account section of the Stackdriver Setting Up Accounts documentation.

Viewing and creating dashboards, charts, and alerts

Viewing the Stackdriver dashboard

To use Stackdriver to monitor your BigQuery project:

  1. In the Google Cloud Platform Console, select Stackdriver > Monitoring, or use the following button:

    Go to Stackdriver Monitoring

  2. Select the name of your project if it is not already selected at the top of the page.

  3. To view BigQuery resources, select Resources > BigQuery:

    Stackdriver bigquery dashboard

  4. Selecting BigQuery takes you to the default BigQuery page. On this page you’ll see a list of tables, Events and Incident reporting that are user configurable as well as charts of Projects Metrics or Dataset Metrics.

Visualizing Slots Available and Slots Allocated

To visualize the slots available and slots allocated to your project:

  1. Go to the Stackdriver dashboard for BigQuery described in Viewing the Stackdriver dashboard.

    Go to Stackdriver Monitoring for BigQuery

  2. On the Stackdriver dashboard for BigQuery, scroll down to the chart named Slot Utilization.

The Slot Utilization chart appears on both the main Stackdriver default dashboard and the Stackdriver dashboard for BigQuery.

Creating a dashboard and chart

Display the metrics collected by Stackdriver Monitoring in your own charts and dashboards:

  1. In the top menu of the Stackdriver Monitoring Console, select Dashboards > Create....

    Go to the Create Dashboard page

  2. Click Add Chart. You see the Add Chart page:

    Add chart

  3. In the Resource Type menu, select "BigQuery".

  4. In the Metric Type menu, select "Query Time - 50th Percentile". You see the chart data in the Preview section.

  5. Click Save.

Creating an alert

Create an alert by defining a condition and configuring the notification settings.

  1. In the top menu of the Stackdriver Monitoring Console, select Alerting > Create a Policy, which opens the Create new Alerting Policy page.

    Go to the Create new Alerting Policy page

  2. In the Conditions section, click Add Condition.

  3. The Select Condition Type page offers several types of conditions. For this example, choose Metric Threshold by clicking on its associated Select button.

  4. On the Add Metric Threshold Condition page:

    1. Set RESOURCE TYPE to "BigQuery".
    2. Set APPLIES TO to "All".
    3. Set IF METRIC to "Query Time (50th Percentile)".
    4. Set THRESHOLD to a number, such as 8.
    5. Leave all other default values and click Save Condition.

    Add threshold condition

  5. On the Create new Alerting Policy page, click Add Notification to configure your notification settings. You can be notified by email, SMS, and several other options.

  6. You can customize the notification message by clicking the Add Documentation button in the Documentation section.

  7. Finally, name your policy and click Save Policy.

Metrics available for visualization

The following metrics are available, time delayed up to several hours.

Resource Type Name Units Description
BigQuery Scanned Bytes Bytes per Minute Number of bytes scanned
BigQuery Scanned Bytes Billed Bytes per Minute Number of bytes sent for billing

The Scanned Bytes and Scanned Bytes Billed can differ for a couple reasons.

  • There is a minimum billing amount. If you scan less than that amount it won’t be billed.
  • If your account has some credit associated with it then these metrics may differ.
BigQuery Query Time
- 5th Percentile
- 50th Percentile
- 95th Percentile
- 99th Percentile
seconds Non-cached query execution times (s)
BigQuery Slots Allocated slots Number of slots allocated to the project at any given moment. This can also be thought of as the number of slots being utilized by that project.

Slots are allocated per billing account and multiple projects can share the same reservation of slots.

BigQuery Slots Available slots Total number of slots available to the project.

If the project shares a reservation of slots with other projects the slots being used by the other projects is not depicted.

BigQuery Dataset Stored Bytes Bytes Bytes stored in the dataset - this is broken down by table for the 100 largest tables in the dataset.
BigQuery Dataset Tables Tables Number of Tables in the Dataset
BigQuery Dataset Uploaded Billed Bytes Bytes per minute Number of bytes uploaded to any table in the dataset that were billed
BigQuery Dataset Uploaded Bytes Bytes per minute Number of bytes uploaded to any table in the dataset

The Uploaded Billed Bytes and Uploaded Bytes Billed can differ for a couple reasons.

  • There is a minimum billing amount. If you scan less than that amount it won’t be billed.
  • If your account has some credit associated with it then these metrics may differ.
BigQuery Dataset Uploaded Rows Rows per minute Number of records uploaded to any table in the dataset

Known issues

  • If no queries are running then no data will be returned for Slots Allocated or for Slots Available or for any query related variables. Zoom out to see data.

  • If queries are running in both the US and the EU then Slots Allocated and Slots Available might be incorrect.

  • Slots Allocated is reported as the average value in a time window (the width of the time window depends on the zoom level of the chart). As such, zooming in and out can change the value of Slots Allocated. Zooming in to a time window of 1 hr of less shows the true values of Slots Allocated. At this range for any time visible on the chart, avg(Slots Allocated) = Slots Allocated.

  • Very few BigQuery customers use sub-reservations. If your project uses sub-reservations and you are using Stackdriver to monitor your utilization, please contact Cloud Support for details on how Stackdriver displays Slots Allocated and Slots Available.

  • The data in Stackdriver charts pertains only to the selected project. Multi-project monitoring has not yet been implemented.

Sending feedback

There is a Send Feedback button at the bottom right of all Stackdriver windows. Please send us suggestions and bug reports.

Send feedback button

Send feedback about...

BigQuery Documentation