Stackdriver Monitoring provides visibility into the performance, uptime, and overall health of cloud-powered applications. Stackdriver collects and ingests metrics, events, and metadata from Cloud Dataproc clusters to generate insights via dashboards and charts. You can use Stackdriver to understand the performance and health of your Cloud Dataproc clusters and examine HDFS, YARN, and Cloud Dataproc job and operation metrics.
Cloud Dataproc Cluster resource metrics are automatically enabled on Cloud Dataproc clusters, and you can use Stackdriver to monitor these metrics without incurring additional costs. Cloud Dataproc metrics are retained for six (6) weeks, and are retained for this period even if the cluster associated with the metrics is deleted.
Using Stackdriver monitoring on Cloud Dataproc clusters
After creating a cluster, go to Stackdriver Monitoring to view cluster monitoring data.
When you first access Stackdriver, you are asked to create an account, and select a project. You can optionally install the Stackdriver agent on VMs in your project as an additional set-up step. You do not need to install the agent on VMs in Cloud Dataproc clusters since this step is performed for you when you create a Cloud Dataproc cluster.
After setting up Stackdriver account, the Stackdriver UI appears.
Select Resources→Metrics Explorer, then click in the "Find resource type and metric" input box to display the resource drop-down list. Select the "Cloud Dataproc Cluster" resource (or type "cloud_dataproc_cluster" in the box).
Click again in the input box, and then select a metric from the drop-down list. In the next screenshot, "YARN memory size" is selected. Hovering over the metric name displays information about the metric.
You can select filters, group by metric labels, perform aggregations, and select chart viewing options (see Stackdriver Monitoring documentation.
You can use the Stackdriver
API to capture and list metrics defined by a
Use the Try this API template on the API page to send
an API request and display the response.
Example: Here's a snapshot of a templated request and the returned
JSON response for the following Stackdriver
- name: projects/example-project-id
- filter: metric.type="dataproc.googleapis.com/cluster/hdfs/storage_capacity"
- interval.endTime: 2018-02-27T11:54:00.000-08:00
- interval.startTime: 2018-02-20T00:00:00.000-08:00
Building a custom Stackdriver monitoring dashboard
You can build a custom Stackdriver Monitoring dashboard that display charts of selected Cloud Dataproc cluster metrics.
Select Dashboards→Create Dashboard from the Stackdriver console.
An "Untitled Dashboard" opens. Click Add Chart. In the Add Chart window, select "Cloud Dataproc Cluster" as the resource type. Select one or more metrics and metric and chart properties. Confirm or type a new chart title, then Save the chart.
You can add additional charts to your dashboard. After you Save the the dashboard, its title appears in the Stackdriver Dashboards menu.
Dashboard charts can be viewed, updated, and deleted from the dashboard display page.
- Explore Stackdriver