Stay organized with collections Save and categorize content based on your preferences.

Create dashboards and alerts for BigQuery

This document describes how to create charts and alerts to monitor BigQuery resources using Cloud Monitoring.

Before you begin

Before you use Cloud Monitoring, ensure that you have the following:

  • A Cloud Billing account.
  • A BigQuery project with billing enabled.

One way to ensure that you have both is to complete the Quickstart using the Google Cloud console.

View and create dashboards, charts, and alerts

View the Cloud Monitoring dashboard

To use Cloud Monitoring to monitor your BigQuery project:

  1. In the Google Cloud console, go to the Monitoring page.

    Go to Monitoring

  2. Select the name of your project if it is not already selected at the top of the page.

  3. To view BigQuery resources, select Dashboards > BigQuery. On this page you see a list of tables, events, and incident reporting that are user-configurable as well as charts of project metrics or dataset metrics.

    BigQuery dashboard.

Visualize slots available and slots allocated

To visualize the slots available and slots allocated to your project, go to the dashboard for BigQuery described in Viewing the Cloud Monitoring dashboard:

  1. In the Google Cloud console, go to the Monitoring page.

    Go to Monitoring

  2. Select Dashboards > BigQuery.

  3. On the Cloud Monitoring dashboard for BigQuery, scroll to the chart named Slot Utilization.

The Slot Utilization chart appears on both the main Cloud Monitoring default dashboard and the Cloud Monitoring dashboard for BigQuery.

Create a dashboard and chart

Display the metrics collected by Cloud Monitoring in your own charts and dashboards:

  1. In the Google Cloud console, go to the Monitoring page.

    Go to Monitoring

  2. Select Dashboards > Create Dashboard.

  3. Click Add Chart. You see the Add Chart page:

    Add chart page.

  4. In the Find resource type and metric panel fields:

    • For the Resource type drop-down list, select Global. You might need to expand the list of Resource types for the Global option to be visible.
    • For the Metric drop-down list, select Query execution time.
  5. The Aggregation pane fields control how the execution-time data are displayed. You can modify the default settings for these fields.

  6. Click Save.

View quota usage and limits

In Cloud Monitoring, you can view metrics for quota usage and limits:

  1. In the Google Cloud console, go to the Monitoring page.

    Go to Monitoring

  2. In the navigation pane, select Metrics explorer.

  3. In the toolbar, select Explorer > Configuration.

  4. In the Resource & Metric section, click Select a metric.

  5. Select Consumer Quota > Quota > Quota limit, and then click Apply.

  6. Click Add filter, and then in the Label menu, select limit_name.

  7. In the Value menu, select the quota for which you want to view the metrics.

    Metrics explorer.

Create an alert

To create an alerting policy that triggers when the 99th percentile of the execution time of a BigQuery query exceeds a user-defined limit, use the following settings.

New condition
Field

Value
Resource and Metric In the Resources menu, select BigQuery Project.
In the Metric categories menu, select Query.
In the Metrics menu, select Query execution times.
Filter
Across time series
Time series group by
priority
Across time series
Time series aggregation
99th percentile
Rolling window 5 m
Rolling window function sum
Configure alert trigger
Field

Value
Condition type Threshold
Alert trigger Any time series violates
Threshold position Above threshold
Threshold value You determine this value; however, a threshold of 60 seconds is recommended.
Retest window most recent value

Metrics available for visualization

The following metrics are available, time delayed up to several hours.

Resource type Name Units Description
BigQuery
Scanned bytes
Bytes per minute Number of bytes scanned.
BigQuery
Scanned bytes billed
Bytes per minute Number of bytes sent for billing.

Scanned bytes and scanned bytes billed can differ for a couple reasons.

  • There is a minimum billing amount. If you scan less than that amount, it is not billed.
  • If your account has some credit associated with it, then these metrics may differ.

BigQuery
Query count
Queries Queries in flight.
BigQuery
Query execution times
- 5th percentile
- 50th percentile
- 95th percentile
- 99th percentile
Seconds Non-cached query execution times.
BigQuery
Slots used by project
Slots Number of BigQuery slots currently allocated for query jobs in the project.

Slots are allocated per billing account and multiple projects can share the same reservation of slots.

BigQuery
Slots used by project and job type
Slots Number of slots allocated to the project at any time separated by job type. This can also be thought of as the number of slots being utilized by that project. Currently, load and export jobs are free operations, and they run in a public pool of resources.

Slots are allocated per billing account and multiple projects can share the same reservation of slots.

BigQuery
Slots used by project, reservation, and job type
Slots Number of BigQuery slots currently allocated for project. Slot allocation can be broken down based on reservation and job type.
BigQuery
Total slots
Slots Total number of slots available to the project.

If the project shares a reservation of slots with other projects the slots being used by the other projects is not depicted.

BigQuery
Slots used across projects in reservations
Slots Number of BigQuery slots currently allocated across projects in the reservation. Note that the metric data is only reported while at least one project has been assigned to the reservation and is consuming slots. As an alternative, consider querying reservations information from INFORMATION_SCHEMA.
BigQuery
Slots used by project in reservation
Slots Number of BigQuery slots currently allocated for project in the reservation.
BigQuery dataset
Stored bytes
Bytes Bytes stored in the dataset - For the 100 largest tables in the dataset, bytes stored is displayed for each individual table (by name). Any additional tables in the dataset (beyond the 100 largest) are reported as single sum, and the table name for the summary is an empty string.
BigQuery dataset
Table count
Tables Number of tables in the dataset.
BigQuery dataset
Uploaded billed bytes
Bytes per minute Number of bytes uploaded to any table in the dataset that were billed.
BigQuery dataset
Uploaded bytes
Bytes per minute Number of bytes uploaded to any table in the dataset.

Uploaded billed bytes and uploaded bytes billed can differ for a couple reasons.

  • There is a minimum billing amount. If you scan less than that amount, it is not billed.
  • If your account has some credit associated with it, then these metrics may differ.
BigQuery dataset Uploaded rows Rows per minute Number of records uploaded to any table in the dataset.

Known issues

  • If no queries are running, then no data is returned for slots allocated, slots available, or any query-related variables. Zoom out to see data.

  • If queries are running in both the US and the EU, then slots allocated and slots available might be incorrect.

  • Slots allocated is reported as the average value in a time window (the width of the time window depends on the zoom level of the chart). Zooming in and out can change the value of slots allocated. Zooming in to a time window of 1 hr or less shows the true values of slots allocated. At this range for any time visible on the chart, avg(slots allocated) = slots allocated.

  • The data in Cloud Monitoring charts pertains only to the selected project. Multi-project monitoring is not currently available.

  • Metrics are instantaneous values, sampled at a point in time, and might miss data points between sample intervals. For example, the job count metric is sampled every minute. The value is the number of jobs at that particular time, not the maximum number of jobs throughout the entire minute.