This document describes how to monitor BigQuery resources using Stackdriver Monitoring, including creating charts and alerts for BigQuery metrics.
Before you begin
Before you use Stackdriver Monitoring, ensure that you have the following:
- A Google Cloud account.
- A BigQuery project with billing enabled.
One way to ensure that you have both is to complete the Quickstart using the web UI.
To use Stackdriver Monitoring, your project must be in a Workspace. Follow the steps in the Creating a single project Workspace.
Viewing and creating dashboards, charts, and alerts
Viewing the Stackdriver Monitoring dashboard
To use Stackdriver Monitoring to monitor your BigQuery project:
In the Google Cloud Console, go to the Monitoring page.
Select the name of your project if it is not already selected at the top of the page.
To view BigQuery resources, select Resources > BigQuery. On this page you'll see a list of tables, events, and incident reporting that are user configurable as well as charts of project metrics or dataset metrics.
Visualizing slots available and slots allocated
To visualize the slots available and slots allocated to your project, go to the dashboard for BigQuery described in Viewing the Stackdriver Monitoring dashboard:
In the Google Cloud Console, go to the Monitoring page.
Select Resources > BigQuery.
On the Stackdriver Monitoring dashboard for BigQuery, scroll to the chart named Slot Utilization.
The Slot Utilization chart appears on both the main Stackdriver Monitoring default dashboard and the Stackdriver Monitoring dashboard for BigQuery.
Creating a dashboard and chart
Display the metrics collected by Stackdriver Monitoring in your own charts and dashboards:
In the Google Cloud Console, go to the Monitoring page.
Select Dashboards > Create Dashboard.
Click Add Chart. You see the Add Chart page:
In the Find resource type and metric panel fields:
- For the Resource type drop-down list, select Global. You might need to expand the list of Resource types, for the Global option to be visible.
- For the Metric drop-down list, select Query execution time.
The Aggregation pane fields control how the execution time data are displayed. You can modify the default settings for these fields.
Click Save.
Creating an alert
Create an alert by defining a condition and configuring the notification settings.
In the Google Cloud Console, go to the Monitoring page.
Select Alerting > Create a Policy. This opens the Create New Alerting Policy page.
In the Conditions section, click Add Condition:
In the Target pane, set the Find resource type and metric fields as follows:
- For the Resource type drop-down list, select Global.
- For the Metric drop-down list, select Query execution time.
Fill in the Aggregation pane fields to capture the 50th percentile of each time series' values in the alignment period:
- In the Aligner drop-down list, select 50th percentile.
- In the Reducer drop-down list, select none.
- In the Alignment Period drop-down list, select 1.
Fill in the Configuration pane fields to trigger if the 50th percentile of any time series exceeds 800 ms in one minute.
- In the Condition triggers if drop-down list, select Any time series violates.
- In the Condition drop-down list, select is above.
- For the threshold, enter
800
. The threshold choice is dependent on your needs. The value of 800 was selected as it was about twice the average 50th percentile. - In the For drop-down list, select 1.
Click Save.
To configure your notification settings,click Add Notification Channel. You can elect to receive notifications by email, SMS, and several other options.
Optionally, enter a notification message in the Documentation section.
Finally, name your policy and click Save.
Metrics available for visualization
The following metrics are available, time delayed up to several hours.
Resource type | Name | Units | Description |
---|---|---|---|
BigQuery | Scanned bytes | Bytes per minute | Number of bytes scanned |
BigQuery | Scanned bytes billed | Bytes per minute | Number of bytes sent for billing
Scanned bytes and scanned bytes billed can differ for a couple reasons.
|
BigQuery | Query time |
seconds | Non-cached query execution times (s) |
BigQuery | Slots allocated | slots | Number of slots allocated to the project at any time. This can
also be thought of as the number of slots being utilized by that project.
Slots are allocated per billing account and multiple projects can share the same reservation of slots. |
BigQuery | Slots allocated by job type | slots | Number of slots allocated to the project at any time separated by
job type.
This can also be thought of as the number of slots being utilized by that
project. Currently, load and export jobs are free operations, and they run
in a public pool of resources.
Slots are allocated per billing account and multiple projects can share the same reservation of slots. |
BigQuery | Slots available | slots | Total number of slots available to the project.
If the project shares a reservation of slots with other projects the slots being used by the other projects is not depicted. |
BigQuery dataset | Stored bytes | Bytes | Bytes stored in the dataset - For the 100 largest tables in the dataset, bytes stored is displayed for each individual table (by name). Any additional tables in the dataset (beyond the 100 largest) are reported as single sum, and the table name for the summary is an empty string. |
BigQuery dataset | Tables | Tables | Number of tables in the dataset |
BigQuery dataset | Uploaded billed bytes | Bytes per minute | Number of bytes uploaded to any table in the dataset that were billed |
BigQuery dataset | Uploaded bytes | Bytes per minute | Number of bytes uploaded to any table in the dataset
Uploaded billed bytes and uploaded bytes billed can differ for a couple reasons.
|
BigQuery dataset | Uploaded rows | Rows per minute | Number of records uploaded to any table in the dataset |
Known issues
If no queries are running then no data will be returned for slots allocated or for slots available or for any query related variables. Zoom out to see data.
If queries are running in both the US and the EU, then slots allocated and slots available might be incorrect.
Slots allocated is reported as the average value in a time window (the width of the time window depends on the zoom level of the chart). Zooming in and out can change the value of slots allocated. Zooming in to a time window of 1 hr or less shows the true values of slots allocated. At this range for any time visible on the chart,
avg(slots allocated) = slots allocated
.The data in Stackdriver Monitoring charts pertains only to the selected project. Multi-project monitoring is not currently available.