BigQuery monitoring

Note: Stackdriver Monitoring in the GCP Console is in beta release. Your configuration information, such as uptime checks and alerting policies, is accessible, and changeable, from the GCP Console and from the classic Stackdriver Monitoring console. This page contains some content that describes how to use the classic Stackdriver Monitoring console. For more information on the beta, see Monitoring in the GCP Console.

This document describes how to monitor BigQuery resources using Stackdriver Monitoring, including creating charts and alerts for BigQuery metrics.

Before you begin

• Before you use Stackdriver Monitoring, ensure that you have the following:

  • A Google Cloud account.
  • A BigQuery project with billing enabled.

  One way to ensure that you have both is to complete the Quickstart using the web UI.

• To use Stackdriver Monitoring, your project must be in a Workspace. Follow the steps in the Creating a single project Workspace.

Viewing and creating dashboards, charts, and alerts

Viewing the Stackdriver Monitoring dashboard

To use Stackdriver Monitoring to monitor your BigQuery project:

1. In the Google Cloud Console, go to the Monitoring page.

   Go to Monitoring

2. Select the name of your project if it is not already selected at the top of the page.

3. To view BigQuery resources, select Resources > BigQuery. On this page you'll see a list of tables, events, and incident reporting that are user configurable as well as charts of project metrics or dataset metrics.

   

Visualizing slots available and slots allocated

To visualize the slots available and slots allocated to your project, go to the dashboard for BigQuery described in Viewing the Stackdriver Monitoring dashboard:

1. In the Google Cloud Console, go to the Monitoring page.

   Go to Monitoring

2. Select Resources > BigQuery.

3. On the Stackdriver Monitoring dashboard for BigQuery, scroll to the chart named Slot Utilization.

The Slot Utilization chart appears on both the main Stackdriver Monitoring default dashboard and the Stackdriver Monitoring dashboard for BigQuery.

Creating a dashboard and chart

Display the metrics collected by Stackdriver Monitoring in your own charts and dashboards:

1. In the Google Cloud Console, go to the Monitoring page.

   Go to Monitoring

2. Select Dashboards > Create Dashboard.

3. Click Add Chart. You see the Add Chart page:

   

4. In the Find resource type and metric panel fields:

   • For the Resource type drop-down list, select Global. You might need to expand the list of Resource types, for the Global option to be visible.
   • For the Metric drop-down list, select Query execution time.

5. The Aggregation pane fields control how the execution time data are displayed. You can modify the default settings for these fields.

6. Click Save.

Creating an alert

Create an alert by defining a condition and configuring the notification settings.

1. In the Google Cloud Console, go to the Monitoring page.

   Go to Monitoring

2. Select Alerting > Create a Policy. This opens the Create New Alerting Policy page.

3. In the Conditions section, click Add Condition:

   1. In the Target pane, set the Find resource type and metric fields as follows:

      • For the Resource type drop-down list, select Global.
      • For the Metric drop-down list, select Query execution time.

   2. Fill in the Aggregation pane fields to capture the 50th percentile of each time series' values in the alignment period:

      • In the Aligner drop-down list, select 50th percentile.
      • In the Reducer drop-down list, select none.
      • In the Alignment Period drop-down list, select 1.

   3. Fill in the Configuration pane fields to trigger if the 50th percentile of any time series exceeds 800 ms in one minute.

      • In the Condition triggers if drop-down list, select Any time series violates.
      • In the Condition drop-down list, select is above.
      • For the threshold, enter 800. The threshold choice is dependent on your needs. The value of 800 was selected as it was about twice the average 50th percentile.
      • In the For drop-down list, select 1.

   4. Click Save.

      

4. To configure your notification settings,click Add Notification Channel. You can elect to receive notifications by email, SMS, and several other options.

5. Optionally, enter a notification message in the Documentation section.

6. Finally, name your policy and click Save.

Metrics available for visualization

The following metrics are available, time delayed up to several hours.

Resource type	Name	Units	Description
BigQuery	Scanned bytes	Bytes per minute	Number of bytes scanned
BigQuery	Scanned bytes billed	Bytes per minute	Number of bytes sent for billing Scanned bytes and scanned bytes billed can differ for a couple reasons. There is a minimum billing amount. If you scan less than that amount it won't be billed. If your account has some credit associated with it then these metrics may differ.
BigQuery	Query time - 5th percentile - 50th percentile - 95th percentile - 99th percentile	seconds	Non-cached query execution times (s)
BigQuery	Slots allocated	slots	Number of slots allocated to the project at any time. This can also be thought of as the number of slots being utilized by that project. Slots are allocated per billing account and multiple projects can share the same reservation of slots.
BigQuery	Slots allocated by job type	slots	Number of slots allocated to the project at any time separated by job type. This can also be thought of as the number of slots being utilized by that project. Currently, load and export jobs are free operations, and they run in a public pool of resources. Slots are allocated per billing account and multiple projects can share the same reservation of slots.
BigQuery	Slots available	slots	Total number of slots available to the project. If the project shares a reservation of slots with other projects the slots being used by the other projects is not depicted.
BigQuery dataset	Stored bytes	Bytes	Bytes stored in the dataset - For the 100 largest tables in the dataset, bytes stored is displayed for each individual table (by name). Any additional tables in the dataset (beyond the 100 largest) are reported as single sum, and the table name for the summary is an empty string.
BigQuery dataset	Tables	Tables	Number of tables in the dataset
BigQuery dataset	Uploaded billed bytes	Bytes per minute	Number of bytes uploaded to any table in the dataset that were billed
BigQuery dataset	Uploaded bytes	Bytes per minute	Number of bytes uploaded to any table in the dataset Uploaded billed bytes and uploaded bytes billed can differ for a couple reasons. There is a minimum billing amount. If you scan less than that amount it won’t be billed. If your account has some credit associated with it then these metrics may differ.
BigQuery dataset	Uploaded rows	Rows per minute	Number of records uploaded to any table in the dataset

Known issues

• If no queries are running then no data will be returned for slots allocated or for slots available or for any query related variables. Zoom out to see data.

• If queries are running in both the US and the EU, then slots allocated and slots available might be incorrect.

• Slots allocated is reported as the average value in a time window (the width of the time window depends on the zoom level of the chart). Zooming in and out can change the value of slots allocated. Zooming in to a time window of 1 hr or less shows the true values of slots allocated. At this range for any time visible on the chart, avg(slots allocated) = slots allocated.

• The data in Stackdriver Monitoring charts pertains only to the selected project. Multi-project monitoring is not currently available.