Monitoring with Cloud Monitoring

This document describes how to use the Cloud Monitoring console to monitor your Cloud Spanner instances.

The Cloud Monitoring console provides several monitoring tools for Cloud Spanner:

• A curated dashboard, which shows pre-made charts for your Cloud Spanner resources
• Custom charts, including ad-hoc charts in the Metrics Explorer as well as charts in custom dashboards
• Alerts, which notify you if a metric exceeds a threshold that you specify

If you prefer to monitor Cloud Spanner programmatically, use the Cloud Client Libraries for Cloud Monitoring to retrieve metrics.

Note: You can also monitor your instances by viewing charts in the Google Cloud Console. Use the Cloud Console to get a quick view of the most important metrics for your instance.

Before you begin

Configure a Cloud Monitoring Workspace for your project by doing the following:

1. In the Cloud Console, select your Google Cloud project.
   Go to Cloud Console
2. In the navigation pane, select Monitoring.

   If you have never used Cloud Monitoring, then on your first access of Monitoring in the Google Cloud Console, a Workspace is automatically created and your project is associated with that Workspace. Otherwise, if your project isn't associated with a Workspace, then a dialog appears and you can either create a Workspace or add your project to an existing Workspace. We recommend that you create a Workspace. After you make your selection, click Add.

Using the Cloud Monitoring curated dashboard

Cloud Monitoring provides you with a curated dashboard that summarizes key information about your Cloud Spanner instances, including:

• Incidents: User-created monitoring alerts that are open, active, or resolved
• Events: A list of Cloud Spanner audit logs (if enabled and available)
• Instances: A high-level summary of your Cloud Spanner instances, including node count, database count, and instance health
• Aggregated charts of throughput and storage use

To view the Cloud Spanner dashboard, do the following:

1. In the Google Cloud Console, select Monitoring, or use the following button:

   Go to Monitoring

2. If Resources is shown in the navigation pane, then select Resources and then select Cloud Spanner. Otherwise, select Dashboards and then select the dashboard named Cloud Spanner.

Viewing instance and database details

When you open the curated dashboard for Cloud Spanner, it shows aggregated data for all of your instances. You can view more details about a specific instance by clicking the instance's name under Instances.

The dashboard displays information such as instance metadata, databases in the instance, and charts of various metrics broken down by region.

From the instance dashboard page, you can also see charts for a specific database in the instance:

1. On the right-hand side, above the instance metrics charts, click Database metrics.

2. In the Select a breakdown drop-down list, select the database that you want to examine.

   The Monitoring console displays charts for the database.

Creating custom charts for Cloud Spanner metrics

You can use Cloud Monitoring to create custom charts for Cloud Spanner metrics. You can use the Metrics Explorer to create temporary, ad-hoc charts, or you can create charts that appear on custom dashboards.

In particular, Cloud Monitoring enables you to create a custom chart that shows whether two or more metrics are correlated with each other. For example, you can check for a correlation between CPU utilization and latency in a Cloud Spanner instance, which might indicate that your instance needs more nodes or that some of your queries are causing high CPU utilization.

To get started with this example, follow these steps:

1. In the Google Cloud Console, select Monitoring, or use the following button:

   Go to Monitoring

2. If Metrics Explorer is shown in the navigation pane, select it. Otherwise, select Resources and then select Metrics Explorer.

3. Click the View options tab, then select the Log scale on Y-axis checkbox. This option helps you compare multiple metrics when one metric has much larger values than the others.

4. In the drop-down list above the right pane, select Line.

5. Click the Metrics tab. You can now add metrics to the chart.

To add latency metrics to the chart, follow these steps:

1. In the Find resource type and metric box, enter the value spanner.googleapis.com/api/request_latencies, then click the row that appears below the box.
2. In the Filter box, enter the value instance_id, then enter the instance ID you want to examine and click Apply.
3. In the Aggregator drop-down list, click max.

4. Optional: Change the latency percentile:

   1. Click Show advanced options.

   2. Click the Aligner drop-down list, then click the latency percentile that you want to view.

      In most cases, you should look at either the 50th percentile latency, to understand the typical amount of latency, or the 99th percentile latency, to understand the latency for the slowest 1% of requests.

To add CPU utilization metrics to the chart, follow these steps:

1. Click Add metric.
2. In the Find resource type and metric box, enter the value spanner.googleapis.com/instance/cpu/utilization, then click the row that appears below the box.
3. In the Filter box, enter the value instance_id, then enter the instance ID you want to examine and click Apply.
4. In the Aggregator drop-down list, click max.

You now have a chart that shows the CPU utilization and latency metrics for a Cloud Spanner instance. If both metrics are higher than expected at the same time, you can take additional steps to correct the issue.

For more information about creating custom charts, see the Cloud Monitoring documentation.

Creating alerts for Cloud Spanner metrics

When you create a Cloud Spanner instance, you choose the number of nodes that provide compute resources for the instance. As the instance's workload changes, Cloud Spanner does not automatically adjust the number of nodes in the instance. As a result, you need to set up several alerts to ensure that the instance stays within the recommended maximums for CPU utilization and the recommended limit for storage per node.

Note: You can also create alerts for other metrics. To find out what metrics are available, see the metrics list for Cloud Spanner.

To create the recommended alerting policies, use the instructions and settings displayed in the following table:

High-priority CPU

To create an alerting policy that triggers when your high priority cpu utilization for Cloud Spanner is above a recommended threshold, use the following settings:

Steps to create an alerting policy.

To create an alerting policy, do the following:

1. In the Google Cloud Console, go to Monitoring or use the following button:
   Go to Monitoring
2. In the Monitoring navigation pane, select Alerting and then select Create Policy.
3. Click Add Condition, and then complete the Target and Configuration sections of the dialog by using the settings in the following tables. Click Add. To monitor another resource, repeat this step.
4. To advance to the notifications section, click Next.
5. (Optional) To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu and then click OK.

   If a notification channel that you want to add isn't listed, then click Manage notification channels. You are taken to the Notification channels page in a new browser tab. From this page, you can update the configured notification channels. After you have completed your updates, return to the original tab, click Refresh , and then select the notification channels to add to the alerting policy.

6. To advance to the documentation section, click Next.
7. Click Name and enter a name for the alerting policy.
8. (Optional) Click Documentation and add any information that you want included in a notification message.
9. Click Save.

Target pane Field	Value
Resource type^*	Cloud Spanner Instance
Metric	CPU Utilization by priority
Filter	instance_id = YOUR_INSTANCE_ID priority = high
Aggregator	max
Period	10 m
Advanced Aggregation	Aligner: mean

^* Selecting this resource and metric type is equivalent to entering the following value in the Find resource type and metric pane: spanner.googleapis.com/instance/cpu/utilization_by_priority.

Configuration pane Field	Value
Condition triggers if	Any time series violates
Condition	is above
Threshold	45% for multi-region instances; 65% for regional instances.
For	10 minutes

24 hour rolling average CPU

To create an alerting policy that triggers when the 24 hour rolling average of your cpu utilization for Cloud Spanner is above a recommended threshold, use the following settings:

Steps to create an alerting policy.

To create an alerting policy, do the following:

1. In the Google Cloud Console, go to Monitoring or use the following button:
   Go to Monitoring
2. In the Monitoring navigation pane, select Alerting and then select Create Policy.
3. Click Add Condition, and then complete the Target and Configuration sections of the dialog by using the settings in the following tables. Click Add. To monitor another resource, repeat this step.
4. To advance to the notifications section, click Next.
5. (Optional) To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu and then click OK.

   If a notification channel that you want to add isn't listed, then click Manage notification channels. You are taken to the Notification channels page in a new browser tab. From this page, you can update the configured notification channels. After you have completed your updates, return to the original tab, click Refresh , and then select the notification channels to add to the alerting policy.

6. To advance to the documentation section, click Next.
7. Click Name and enter a name for the alerting policy.
8. (Optional) Click Documentation and add any information that you want included in a notification message.
9. Click Save.

Target pane Field	Value
Resource type^*	Cloud Spanner Instance
Metric	Smoothed CPU utilization
Filter	instance_id = YOUR_INSTANCE_ID
Aggregator	sum
Period	10 m
Advanced Aggregation	Aligner: mean

^* Selecting this resource and metric type is equivalent to entering the following value in the Find resource type and metric pane: spanner.googleapis.com/instance/cpu/smoothed_utilization.

Configuration pane Field	Value
Condition triggers if	Any time series violates
Condition	is above
Threshold	90%
For	10 minutes

Storage

To create an alerting policy that triggers when your storage for your Cloud Spanner instance is above a recommended threshold, use the following settings:

Steps to create an alerting policy.

To create an alerting policy, do the following:

1. In the Google Cloud Console, go to Monitoring or use the following button:
   Go to Monitoring
2. In the Monitoring navigation pane, select Alerting and then select Create Policy.
3. Click Add Condition, and then complete the Target and Configuration sections of the dialog by using the settings in the following tables. Click Add. To monitor another resource, repeat this step.
4. To advance to the notifications section, click Next.
5. (Optional) To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu and then click OK.

   If a notification channel that you want to add isn't listed, then click Manage notification channels. You are taken to the Notification channels page in a new browser tab. From this page, you can update the configured notification channels. After you have completed your updates, return to the original tab, click Refresh , and then select the notification channels to add to the alerting policy.

6. To advance to the documentation section, click Next.
7. Click Name and enter a name for the alerting policy.
8. (Optional) Click Documentation and add any information that you want included in a notification message.
9. Click Save.

Target pane Field	Value
Resource type^*	Cloud Spanner Instance
Metric	Storage used
Filter	instance_id = YOUR_INSTANCE_ID
Aggregator	sum
Period	10 m
Advanced Aggregation	Aligner: max

^* Selecting this resource and metric type is equivalent to entering the following value in the Find resource type and metric pane: spanner.googleapis.com/instance/storage/used_bytes.

Configuration pane Field	Value
Condition triggers if	Any time series violates
Condition	is above
Threshold	Set the threshold to 75% of the maximum storage per node, multiplied by the number of nodes. For the current node limits, see Cloud Spanner Quotas and limits. For a 2 TB limit per node, the recommended threshold is the number of bytes in 1.5 TB: 1649267441664 multiplied by the number of nodes in your instance.
For	10 minutes

Note: Cloud Spanner usage amounts are calculated in binary terabytes, where 1 TB is 2⁴⁰ bytes. This unit of measurement is also known as a tebibyte (TiB).

What's next

• Understand the CPU utilization and latency metrics for Cloud Spanner.
• Use the Cloud Console to get a quick view of the most important metrics for your instance.
• Learn more about Cloud Monitoring.