Monitor Cloud TPU VMs

This guide explains how to use Google Cloud Monitoring to monitor your Cloud TPU VMs. Google Cloud Monitoring automatically collects metrics and logs from your Cloud TPU and its host Compute Engine. These data can be used to monitor the health of your Cloud TPU and Compute Engine.

Metrics enable you to track a numerical quantity over time, for example, CPU utilization, network usage, or TensorCore idle duration. Logs capture events at a specific point in time. Log entries are written by your own code, Google Cloud services, third-party applications, and the Google Cloud infrastructure. You can also generate metrics from the data present in a log entry by creating a log-based metric. You can also set alert policies based on metric values or log entries.

This guide discusses Google Cloud Monitoring and shows you how to:

  • View Cloud TPU metrics
  • Set up Cloud TPU metrics alert policies
  • Query Cloud TPU logs
  • Create log-based metrics for setting up alerts and visualizing dashboards

Prerequisites

This document assumes some basic knowledge of Google Cloud Monitoring. You must have a Compute Engine VM and Cloud TPU resources created before you can begin generating and working with Google Cloud Monitoring. See the Cloud TPU Quickstart for more details.

Metrics

Google Cloud metrics are automatically generated by Compute Engine VMs and the Cloud TPU runtime. The following metrics are generated by Cloud TPU VMs:

  • memory/usage
  • network/received_bytes_count
  • network/sent_bytes_count
  • cpu/utilization
  • tpu/tensorcore/idle_duration

It can take up to 180 seconds between the time a metric value is generated and when it's displayed in the Metrics Explorer.

For more information on TPU related metrics, see TPU metrics.

Memory usage

The memory/usage metric tracks the memory used by the TPU VM in bytes. This metric is sampled every 60 seconds.

Network received bytes count

The network/received_bytes_count metric tracks the number of cumulative bytes of data the TPU VM received over the network at a point in time.

Network sent bytes count

The network/sent_bytes_count metric tracks the number of cumulative bytes the TPU VM sent over the network at a point in time.

CPU utilization

The cpu/utilization metric tracks the current CPU utilization on the TPU worker, represented as a percentage, sampled once a minute. Values are typically between 0.0 and 100.0, but might exceed 100.0.

TensorCore idle duration

The tpu/tensorcore/idle_duration metric tracks the number of seconds each TPU chip's TensorCore has been idle. This metric is available for each chip on all TPUs in use. If a TensorCore is in use, the idle duration value is reset to zero. When the TensorCore is no longer in use, the idle duration value starts to increase.

The following graph shows the tpu/tensorcore/idle_duration metric for a v2-8 TPU VM which has one worker. Each worker has four chips. In this example, all four chips have the same values for tpu/tensorcore/idle_duration, so the graphs are superimposed on each other.

image

For a complete list of metrics generated by Cloud TPU, see Google Cloud Cloud TPU metrics.

Viewing metrics

You can view metrics using the Metrics Explorer in the Google Cloud console.

In the Metrics Explorer, click SELECT A METRIC and search for TPU Worker. If Show only active resources and metrics is on, only metrics from active resources are displayed. Click TPU Worker to display all available metrics.

You can also access metrics using curl HTTP calls:

Use the Try it! button in the projects.timeSeries.query documentation to retrieve the value for a metric within the specified timeframe.

  1. Fill in the name in the following format: projects/{project-name}.
  2. Add a query to the Request body section. The following is a sample query for retrieving the idle duration metric for the specified zone for the last five minutes. fetch tpu_worker | filter zone = 'us-central2-b' | metric tpu.googleapis.com/tpu/tensorcore/idle_duration | within 5m
  3. Click Execute to make the call and see the results of the HTTP POST message.

The Monitoring Query Language reference document has more information on how to customize this query.

You can create alert policies that tell Cloud Monitoring to send an alert when a condition is met.

Creating alerts

The steps in this section show an example of how to add an alert policy for the TensorCore Idle Duration metric. Whenever this metric exceeds 24 hours, Cloud Monitoring sends an email to the registered email address.

  1. Go to the Monitoring console.
  2. In the navigation pane click Alerting.
  3. Click EDIT NOTIFICATION CHANNELS.
  4. Under Email, click ADD NEW. Type an email address, a display name, and click SAVE.
  5. Click CREATE POLICY.
  6. Click SELECT A METRIC and then select Tensorcore Idle Duration and click APPLY.
  7. Click NEXT and then Threshold.
  8. For Alert trigger, select Any time series violates.
  9. For Threshold Position, select Above threshold.
  10. For Threshold Value, type 86400000.
  11. Click NEXT.
  12. Under Notification Channels select your email notification channel and click OK.
  13. Type a name for the alert policy.
  14. Click NEXT and then CREATE POLICY.

When the TensorCore Idle Duration exceeds 24 hours, an email is sent to the email address you specified.

Logging

Log entries are written by Google Cloud services, third party services, ML frameworks or your code. You can view logs using the Logs Explorer or Logs API. For more information about Google Cloud logging, see Google Cloud Logging.

In the Logs Explorer, select a resource type:

  • Cloud TPU Worker -> Zone -> Node ID
  • Audited Resource -> Cloud TPU -> API (google.cloud.tpu.v2alpha1.Tpu.CreateNode, google.cloud.tpu.v2alpha1.Tpu.DeleteNode, google.cloud.tpu.v2alpha1.Tpu.UpdateNode)

Cloud TPU worker logs contain information about a specific Cloud TPU worker in a specific zone, for example the amount of memory available on the Cloud TPU worker (system_available_memory_GiB).

Audited Resource logs contain information about when a specific Cloud TPU API was called and who made the call. For example CreateNode, UpdateNode, and DeleteNode.

ML frameworks can generate logs to stdout and stderr. These logs are controlled by environment variables and are read by your training script.

Your code can write logs to Google Cloud Logging. For more information, see Write standard logs and Write structured logs.

To view Cloud TPU logs:

  1. Go to the Google Cloud Logs Viewer
  2. Click the Resource drop-down
  3. Click Cloud TPU Worker
  4. Select a zone
  5. Select the Cloud TPU you're interested in
  6. Click Apply. Logs are displayed in the query results

To view Audited Resource logs:

  1. Go to the Google Cloud Logs Viewer
  2. Click the Resource drop-down
  3. Click Audited Resource and then Cloud TPU
  4. Choose the Cloud TPU API that you're interested in
  5. Click Apply. Logs are displayed in the query results
  6. Choose the APIs that begin with google.cloud.tpu.v2alpha1.Tpu

Query Google Cloud Logs

When you view logs in the Google Cloud console, the page performs a default query. You can view the query by selecting the Show query toggle switch. You can modify the default query or create a new one. For more information, see Build Queries in the Logs Explorer.

Audited resource logs

Click any log entry to expand it, and you will find a field called protoPayload. Expand protoPayload, and you will see a number of subfields:

  • logName: the name of the log
  • protoPayload -> @type: the type of the log
  • resourceName: the name of your Cloud TPU
  • methodName: the name of the method called (audit logs only)
  • request -> @type: the request type
  • request -> node: details about the Cloud TPU node
  • request -> node_id: the name of the TPU
  • severity: the severity of the log

TPU Worker logs

Click any log entry to expand it, and you will find a field called jsonPayload. Expand jsonPayload and you will see a number of subfields:

  • accelerator_type: the accelerator type
  • consumer_project: the project where the Cloud TPU lives
  • evententry_timestamp: the time when the log was generated
  • system_available_memory_GiB: the available memory on the Cloud TPU worker (0 ~ 350 GiB)

Creating log-based metrics

This section describes how to create log-based metrics used for setting up monitoring dashboards and alerts. For information about programmatically creating log-based metrics, see Creating log-based metrics programmatically using the Cloud Logging REST API.

The following example uses the system_available_memory_GiB subfield to demonstrate how to create a log-based metric for monitoring Cloud TPU worker available memory.

  1. Navigate to the Logs Explorer

  2. In the query box, enter the following query to extract all log entries that have system_available_memory_GiB defined for the primary Cloud TPU worker:

    resource.type=tpu_worker
resource.labels.project_id=your-project
resource.labels.zone=your-tpu-zone
resource.labels.node_id=your-tpu-name
resource.labels.worker_id=0
logName=projects/your-project/logs/tpu.googleapis.com%2Fruntime_monitor
jsonPayload.system_available_memory_GiB:*

  3. Click Create metric to display the Metric Editor

  4. Under Metric Type, choose Distribution

  5. Type a name, optional description, and unit of measurement for your metric. Type "matrix_unit_utilization_percent" and "MXU utilization" in the Name and Description fields, respectively

  6. The filter is pre-populated with the script that you entered in the Logs Explorer

  7. Click CREATE METRIC

  8. Click Explore Metrics to view your new metric. It make take a few minutes before your metrics are displayed

Creating log-based metrics with the Cloud Logging REST API

You can also create log-based metrics through the Cloud Logging API. For more information, see Creating a distribution metric.

Creating dashboards and alerts using log-based metrics

Dashboards are useful for visualizing metrics (expect ~2 minute delay); alerts are helpful for sending notifications when errors occur. For more information, see:

Creating dashboards

To create a dashboard in Cloud Monitoring for the Tensorcore idle duration metric:

  1. Go to the Monitoring console
  2. In the navigation pane, click Dashboards
  3. Click CREATE DASHBOARD and then Add Widget
  4. Choose the chart type that you want to add. For this example, choose Line
  5. Type a title for the widget
  6. Click the Select a metric drop-down menu and type "Tensorcore idle duration" in the filter field
  7. In the list of metrics, select TPU Worker -> Tpu -> Tensorcore idle duration
  8. To filter the dashboard contents, click the Filter drop-down menu
  9. Under Resource labels, select project_id
  10. Choose a comparator and type a value in the Value field
  11. Click Apply