BigQuery Monitoring using Stackdriver

This document describes how to monitor BigQuery resources using Stackdriver Monitoring, including creating charts and alerts for BigQuery metrics.

Before you begin

  • Before you use Monitoring, ensure that you have the following:

    • A Google Cloud Platform account.
    • A BigQuery project with billing enabled.

    One way to ensure that you have both is to complete the Quickstart Using the Web UI.

  • To use Stackdriver, your project must be in a Workspace. Follow the steps in the Creating a single project Workspace.

Viewing and creating dashboards, charts, and alerts

Viewing the Stackdriver dashboard

To use Stackdriver to monitor your BigQuery project:

  1. In the Google Cloud Platform Console, select Stackdriver > Monitoring, or use the following button:

    Go to Monitoring

  2. Select the name of your project if it is not already selected at the top of the page.

  3. To view BigQuery resources, select Resources > BigQuery. On this page you'll see a list of tables, Events and Incident reporting that are user configurable as well as charts of Projects Metrics or Dataset Metrics:

    Stackdriver BigQuery dashboard

Visualizing Slots Available and Slots Allocated

To visualize the slots available and slots allocated to your project:

  1. Go to the Stackdriver dashboard for BigQuery described in Viewing the Stackdriver dashboard.

    Go to Monitoring for BigQuery

  2. On the Stackdriver dashboard for BigQuery, scroll down to the chart named Slot Utilization.

The Slot Utilization chart appears on both the main Stackdriver default dashboard and the Stackdriver dashboard for BigQuery.

Creating a dashboard and chart

Display the metrics collected by Monitoring in your own charts and dashboards:

  1. In the left-hand navigation pane of the Monitoring console, select Dashboards > Create Dashboard.

    Go to the Create Dashboard page

  2. Click Add Chart. You see the Add Chart page:

    Add chart

  3. In the Find resource type and metric panel fields:

    • For the Resource type drop-down list, select Global. You might need to expand the list of Resource types, for the Global option to be visible.
    • For the Metric drop-down list, select Query execution time.
  4. The Aggregation pane fields control how the execution time data are displayed. You can modify the default settings for these fields.

  5. Click Save.

Creating an alert

Create an alert by defining a condition and configuring the notification settings.

  1. In left-hand navigation pane of the Monitoring console, select Alerting > Create a Policy. This opens the Create New Alerting Policy page.

    Go to the Create new Alerting Policy page

  2. In the Conditions section, click Add Condition:

    1. In the Target pane, set the Find resource type and metric fields as follows:

      • For the Resource type drop-down list, select Global.
      • For the Metric drop-down list, select Query execution time.
    2. Fill in the Aggregation pane fields to capture the 50th percentile of each time series' values in the alignment period:

      • In the Aligner drop-down list, select 50th percentile.
      • In the Reducer drop-down list, select none.
      • In the Alignment Period drop-down list, select 1.
    3. Fill in the Configuration pane fields to trigger if the 50th percentile of any time series exceeds 800ms in 1 minute.

      • In the Condition triggers if drop-down list, select Any time series violates.
      • In the Condition drop-down list, select is above.
      • For the threshold, enter 800. The threshold choice is dependent on your needs. The value of 800 was selected as it was about twice the average 50th percentile.
      • In the For drop-down list, select 1.
    4. Click Save.

      Add threshold condition

  3. To configure your notification settings,click Add Notification Channel. You can elect to receive notifications by email, SMS, and several other options.

  4. Optionally, enter a notification message in the Documentation section.

  5. Finally, name your policy and click Save.

Metrics available for visualization

The following metrics are available, time delayed up to several hours.

Resource Type Name Units Description
BigQuery Scanned Bytes Bytes per Minute Number of bytes scanned
BigQuery Scanned Bytes Billed Bytes per Minute Number of bytes sent for billing

The Scanned Bytes and Scanned Bytes Billed can differ for a couple reasons.

  • There is a minimum billing amount. If you scan less than that amount it won’t be billed.
  • If your account has some credit associated with it then these metrics may differ.
BigQuery Query Time
- 5th Percentile
- 50th Percentile
- 95th Percentile
- 99th Percentile
seconds Non-cached query execution times (s)
BigQuery Slots Allocated slots Number of slots allocated to the project at any given moment. This can also be thought of as the number of slots being utilized by that project.

Slots are allocated per billing account and multiple projects can share the same reservation of slots.

BigQuery Slots Allocated By Job Type slots Number of slots allocated to the project at any given moment separated by job type. This can also be thought of as the number of slots being utilized by that project. Currently, load and export jobs are free operations, and they run in a public pool of resources.

Slots are allocated per billing account and multiple projects can share the same reservation of slots.

BigQuery Slots Available slots Total number of slots available to the project.

If the project shares a reservation of slots with other projects the slots being used by the other projects is not depicted.

BigQuery Dataset Stored Bytes Bytes Bytes stored in the dataset - For the 100 largest tables in the dataset, bytes stored is displayed for each individual table (by name). Any additional tables in the dataset (beyond the 100 largest) are reported as single sum, and the table name for the summary is an empty string.
BigQuery Dataset Tables Tables Number of Tables in the Dataset
BigQuery Dataset Uploaded Billed Bytes Bytes per minute Number of bytes uploaded to any table in the dataset that were billed
BigQuery Dataset Uploaded Bytes Bytes per minute Number of bytes uploaded to any table in the dataset

The Uploaded Billed Bytes and Uploaded Bytes Billed can differ for a couple reasons.

  • There is a minimum billing amount. If you scan less than that amount it won’t be billed.
  • If your account has some credit associated with it then these metrics may differ.
BigQuery Dataset Uploaded Rows Rows per minute Number of records uploaded to any table in the dataset

Known issues

  • If no queries are running then no data will be returned for Slots Allocated or for Slots Available or for any query related variables. Zoom out to see data.

  • If queries are running in both the US and the EU then Slots Allocated and Slots Available might be incorrect.

  • Slots Allocated is reported as the average value in a time window (the width of the time window depends on the zoom level of the chart). As such, zooming in and out can change the value of Slots Allocated. Zooming in to a time window of 1 hr of less shows the true values of Slots Allocated. At this range for any time visible on the chart, avg(Slots Allocated) = Slots Allocated.

  • The data in Stackdriver charts pertains only to the selected project. Multi-project monitoring has not yet been implemented.

Was this page helpful? Let us know how we did:

Send feedback about...

Need help? Visit our support page.