Cloud Monitoring

Cloud Monitoring provides visibility into the performance, uptime, and overall health of cloud-powered applications. Google Cloud Observability collects and ingests metrics, events, and metadata from Dataproc Metastore services to generate insights using dashboards and charts.

Access service metrics in Monitoring

Dataproc Metastore service resource metrics are automatically enabled on Dataproc Metastore services. Use Monitoring to view these metrics.

You can access Monitoring from the Google Cloud console or by using the Monitoring API.

Console

  1. In the Google Cloud console, go to the Cloud Monitoring > Metrics explorer page:

    Go to the Metrics explorer page

  2. From the "Find resource type and metric" drop-down list, select the resource Dataproc Metastore Service.

  3. Click again in the input box, and then select a metric from the drop-down list. Hovering over the metric name displays information about the metric.

You can select filters, group by metric labels, perform aggregations, and select chart viewing options.

REST

You can use the Monitoring timeSeries.list API to capture and list metrics defined by a filter expression. Use the Try this API template on the API page to send an API request and display the response.

Dataproc Metastore service metrics in Monitoring

Metrics from Dataproc Metastore. Launch stages of these metrics: BETA GA

The "metric type" strings in this table must be prefixed with metastore.googleapis.com/. That prefix has been omitted from the entries in the table. When querying a label, use the metric.labels. prefix; for example, metric.labels.LABEL="VALUE".

Metric type Launch stage
Display name
Kind, Type, Unit
Monitored resources
Description
Labels
federation/server/request_count BETA
Federation service request count
DELTAINT64{request}
metastore.googleapis.com/Federation
The number of requests sent to the Federated Dataproc Metastore Service.
method_name: The request method name.
status: End status of the request, one of [success, client_error, server_error].
version: The Hive Metastore version of the service.
service/grpc/finished_requests_count BETA
gRPC Requests Finished Count
DELTAINT641
metastore.googleapis.com/Service
The number of requests finished by the GRPC Dataproc Metastore Service.
tier: Service tier or instance size of the service, such as ENTERPRISE.
version: The Hive Metastore version of the service.
status: The status of the request, such as `BAD_REQUEST`.
method_name: The request method name.
service/grpc/received_bytes_count BETA
gRPC Bytes Received
DELTAINT64By
metastore.googleapis.com/Service
Number of bytes received by the Dataproc Metastore Service.
tier: Service tier or instance size of the service.
version: The Hive Metastore version of the service.
service/grpc/request_latencies BETA
gRPC Request Latencies
DELTADISTRIBUTIONms
metastore.googleapis.com/Service
Distribution of gRPC request latencies for a Dataproc Metastore Service.
tier: Service tier or instance size of the service.
version: The Hive Metastore version of the service.
status: The status of the request, such as `BAD_REQUEST`.
method_name: The request method name.
service/grpc/sent_bytes_count BETA
gRPC Bytes Sent
DELTAINT64By
metastore.googleapis.com/Service
Number of bytes sent by the Dataproc Metastore Service.
tier: Service tier or instance size of the service.
version: The Hive Metastore version of the service.
service/grpc/started_requests_count BETA
gRPC Requests Started Count
DELTAINT641
metastore.googleapis.com/Service
The number of requests received and started by the GRPC Dataproc Metastore Service.
tier: Service tier or instance size of the service.
version: The Hive Metastore version of the service.
method_name: The request method name.
service/health GA
Service health
GAUGEINT641
metastore.googleapis.com/Service
The health of the Dataproc Metastore Service.
tier: Service tier or instance size of the service, such as ENTERPRISE.
version: The Hive Metastore version of the service.
release_tag: Dataproc Metastore release tag, populated by the service backend and may change over service lifetime.
service/multi_region/health GA
Multiregional service health
GAUGEINT641
metastore.googleapis.com/Service
The health of the multiregional Dataproc Metastore Service.
version: The Hive Metastore version of the service.
release_tag: Dataproc Metastore release tag, populated by the service backend and may change over service lifetime.
service_region: The constituent region that Dataproc Metastore service is hosted in.
scaling_factor: Scaling factor of the service, one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0].
service/multi_region/request_count GA
Multiregional service request count
DELTAINT64{request}
metastore.googleapis.com/Service
The number of requests sent to the multiregional Dataproc Metastore Service.
method_name: The request method name.
version: The Hive Metastore version of the service.
service_region: The constituent region that Dataproc Metastore service is hosted in.
scaling_factor: Scaling factor of the service, one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0].
service/multi_region/request_latencies GA
Multiregional service request latencies
DELTADISTRIBUTIONms
metastore.googleapis.com/Service
The latencies of the requests sent to the multiregional Dataproc Metastore Service.
method_name: The request method name.
version: The Hive Metastore version of the service.
service_region: The constituent region that Dataproc Metastore service is hosted in.
scaling_factor: Scaling factor of the service, one of [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0].
service/query_metadata/request_count BETA
Admin Interface request count
DELTAINT64{request}
metastore.googleapis.com/Service
The number of Admin Interface requests.
database_type: The type of database, one of [SQL, Spanner].
code: The gRPC status code.
service/query_metadata/request_latencies BETA
Admin Interface request latency
DELTADISTRIBUTIONms
metastore.googleapis.com/Service
The end to end request latency for Admin Interface requests.
database_type: The type of database, one of [SQL, Spanner].
code: The gRPC status code.
service/request_count GA
Request count
DELTAINT64{request}
metastore.googleapis.com/Service
The number of requests sent to the Dataproc Metastore Service.
method_name: The request method name.
version: The Hive Metastore version of the service.
service/scheduled_backups/backup_count BETA
Scheduled Backups Count
DELTAINT641
metastore.googleapis.com/Service
The number of scheduled backups currently stored for the service.
state: State of the backup, one of [SUCCEEDED, FAILED, IN_PROGRESS].
database_type: The type of database, one of [SQL, Spanner].
version: The Hive Metastore version of the service.
service/scheduled_backups/last_backup_duration BETA
Scheduled Backups - Last Backup Duration
GAUGEINT64s
metastore.googleapis.com/Service
The duration of the last successful scheduled backup in seconds. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
state: State of the backup, one of [SUCCEEDED, FAILED, IN_PROGRESS].
database_type: The type of database, one of [SQL, Spanner].
version: The Hive Metastore version of the service.
service/scheduled_backups/last_backup_size BETA
Scheduled Backups Request - Last Backup Size
GAUGEINT64By
metastore.googleapis.com/Service
The size of the last successful scheduled backup in bytes. Sampled every 60 seconds. After sampling, data is not visible for up to 120 seconds.
state: State of the backup, one of [SUCCEEDED, FAILED, IN_PROGRESS].
database_type: The type of database, one of [SQL, Spanner].
version: The Hive Metastore version of the service.

Table generated at 2024-03-20 16:34:03 UTC.

Build a custom Monitoring dashboard

You can build a custom Monitoring dashboard that displays charts of selected Dataproc Metastore service metrics.

  1. Select + CREATE DASHBOARD from the Monitoring Dashboards Overview page.

  2. Provide a name for the dashboard.

  3. In the upper-right menu, click Add Chart to open the Add Chart window.

  4. For the resource type, select Dataproc Metastore Service.

  5. Select one or more metrics and metric and chart properties, and then click Save to save the chart.

  6. Add additional charts to your dashboard if you'd like.

  7. To save the dashboard, click Save . Its title appears in the Monitoring Dashboards Overview page.

You can view, update, and delete dashboard charts from the dashboard display page.

Use Monitoring alerts

You can create a Monitoring alert that notifies you when a Dataproc Metastore service or job metric crosses a specified threshold.

Create an alert

  1. In the Google Cloud console, open Monitoring Alerting.

  2. To open the Create new alerting policy form, click + CREATE POLICY

  3. Define an alert by adding alert conditions, policy triggers, notification channels, and documentation.

  4. Select ADD CONDITION to open the alert condition form with the Metric tab selected.

  5. Fill in the fields to define an alert condition, and then click ADD.

  6. After adding the alert condition, complete the alert policy by setting notification channels, policy triggers, documentation, and the alert policy name.

View alerts

When an alert is triggered by a metric threshold condition, Monitoring creates an incident and a corresponding event. You can view incidents from the Monitoring Alerting page in the Google Cloud console. If you defined a notification mechanism in the alert policy, such as an email or SMS notification, Monitoring also sends a notification of the incident.

Whats next