Monitor and troubleshoot Dataproc Serverless workloads

You can monitor and troubleshoot Dataproc Serverless for Spark batch workloads by using the information and tools discussed in the following sections

Persistent History Server

Dataproc Serverless for Spark creates the compute resources that are needed to run a workload, runs the workload on those resources, and then deletes the resources when the workload finishes. Workload metrics and events don't persist after a workload completes. However, you can use a Persistent History Server (PHS) to retain workload application history (event logs) in Cloud Storage.

To use a PHS with a batch workload, do the following:

1. Create a Dataproc Persistent History Server (PHS).

   Serverless Spark UI Preview: This preview feature provides a Dataproc Serverless Spark UI in the Google Cloud console. This feature eliminates the need for you to create your own PHS. To request access to this feature, send an email request to [e-mail].

2. Specify your PHS when you submit a workload.

3. Use the Component Gateway to connect to the PHS to view application details, scheduler stages, task level details, and environment and executor information.

Dataproc Serverless for Spark logs

Logging is enabled by default in Dataproc Serverless for Spark, and workload logs persist after a workload finishes. Dataproc Serverless for Spark collects workload logs in Cloud Logging. You can access workload spark, agent, output and container logs under the Cloud Dataproc Batch resource in the Logs Explorer.

Dataproc Serverless for Spark batch example:

For more information, see Dataproc logs.

Workload metrics

By default, Dataproc Serverless for Spark enables the collection of available Spark metrics, unless you use Spark metrics collection properties to disable or override the collection of one or more Spark metrics.

You can view workload metrics from the Metrics Explorer or the Batch details page in the Google Cloud console.

Batch metrics

Dataproc batch resource metrics provide insight into batch resources, such as the number of batch executors. Batch metrics are prefixed with dataproc.googleapis.com/batch.

Spark metrics

Available Spark metrics include Spark driver and executor metrics, and system metrics. Available Spark metrics are prefixed with custom.googleapis.com/.

Set up metric alerts

You can create Dataproc metric alerts to receive notice of workload issues.

Create charts

You can create charts that visualize workload metrics by using the Metrics Explorer in the Google Cloud console. For example, you can create a chart to display disk:bytes_used, and then filter by batch_id.

Cloud Monitoring

Monitoring uses workload metadata and metrics to provide insights into the health and performance of Dataproc Serverless for Spark workloads. Workload metrics include Spark metrics, batch metrics, and operation metrics.

You can use Cloud Monitoring in the Google Cloud console to explore metrics, add charts, create dashboards, and create alerts.

Create dashboards

You can create a dashboard to monitor workloads using metrics from multiple projects and different Google Cloud products. For more information, see Create and manage custom dashboards.

Advanced troubleshooting (Preview)

This section covers the advanced troubleshooting preview features available in the Google Cloud console, which include Gemini-assisted troubleshooting for Dataproc Serverless. which is part of the Gemini in BigQuery offering.

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

For information about access to this release, see the access request page.

Access to preview features

To sign up for the preview release of the advanced troubleshooting features, complete and submit the Gemini in BigQuery Preview form. Once the form is approved, projects listed in the form have access to preview features.

Note: You can provide feedback on this preview release by using the Dataproc Previews: Feedback form.

Preview pricing

There is no additional charge for participation in the preview. Charges will apply to the following preview features when they become generally available (GA):

• Gemini-assisted troubleshooting for Dataproc Serverless
• Batch metrics highlights
• Job logs

Advance notice of GA charges will be sent to the email address that you provide in the preview sign-up form.

Feature requirements

• Sign-up: You must sign up for the feature.

• Permission: You must have the dataproc.batches.analyze permission.

  • If you have the predefined roles/dataproc.admin, roles/dataproc.editor or roles/dataproc.viewer role, you have the required permission. No further action is necessary.

  • If you use a custom role to access Dataproc services, the custom role must have the dataproc.batches.analyze permission. You can use the gcloud CLI to add the permission, as shown in the following command, which adds the permission at the project level:

  gcloud iam roles update CUSTOM_ROLE_ID --project=PROJECT_ID \
  --add-permissions="dataproc.batches.analyze"
  

  If you use the gcloud iam roles describe command to view granted permissions, it does not list the dataproc.batches.analyze permission.

• Enable Gemini-assisted troubleshooting for Dataproc Serverless: You enable Gemini-assisted troubleshooting for Dataproc Serverless when you submit each recurring Spark batch workload using the Google Cloud console, gcloud CLI, or the Dataproc API. Once this feature is enabled on a recurring batch workload, Dataproc stores a copy of the workload logs for 30 days, and uses the saved log data to provide Gemini-assisted troubleshooting for the workload. For information on Spark workload log content, see Dataproc Serverless for Spark logs.

Console

Perform the following steps to enable Gemini-assisted troubleshooting on each recurring Spark batch workload:

1. In the Google Cloud console, go to the Dataproc Batches page.

   Go to Dataproc Batches

2. To create a batch workload, click Create.

3. In the Container section, fill in the Cohort name, which identifies the batch as one of a series of recurring workloads. Gemini-assisted analysis is applied to the second and subsequent workloads that are submitted with this cohort name. For example, specify TPCH-Query1 as the cohort name for a scheduled workload that runs a daily TPC-H query.

4. Fill in other sections of the Create batch page as needed, then click Submit. For more information, see Submit a batch workload.

gcloud

Run the following gcloud CLI gcloud dataproc batches submit command locally in a terminal window or in Cloud Shell to enable Gemini-assisted troubleshooting on each recurring Spark batch workload:

gcloud dataproc batches submit COMMAND \
    --region=REGION \
    --cohort=COHORT \
    other arguments ...

Replace the following:

• COMMAND: the Spark workload type, such as Spark, PySpark, Spark-Sql, or Spark-R.
• REGION: the region where your workload will run.
• COHORT: the cohort name, which identifies the batch as one of a series of recurring workloads. Gemini assisted analysis is applied to the second and subsequent workloads that are submitted with this cohort name. For example, specify TPCH Query 1 as the cohort name for a scheduled workload that runs a daily TPC-H query.

API

Include the RuntimeConfig.cohort name in a batches.create request to enable Gemini-assisted troubleshooting on each recurring Spark batch workload. Gemini-assisted analysis is applied to the second and subsequent workloads submitted with this cohort name. For example, specify TPCH-Query1 as the cohort name for a scheduled workload that runs a daily TPC-H query.

Example:

...
runtimeConfig:
  cohort: TPCH-Query1
...

Gemini-assisted troubleshooting for Dataproc Serverless

The following Gemini-assisted troubleshooting preview features are available on the Batch details and Batches list pages in the Google Cloud console.

• Investigate tab: The Investigate tab on the Batch details page provides a Health Overview (Preview) section with the following Gemini-assisted troubleshooting panels:

  • What was autotuned? If you enabled autotuning on one or more workloads, this panel displays the most recent autotuning changes that were applied to running, completed, and failed workloads.

  

  • What is happening now? and What can I do about it? Click Ask Gemini to request recommendations to help fix failed workloads or improve successful but slow workloads.

  

  If you click Ask Gemini, Gemini generates a summary of any errors, anomalies, or highlights from workload logs, Spark metrics, and Spark events. Gemini can also display a list of recommended steps you can take to fix a failed workload or improve the performance of a successful, but slow workload.

  

• Gemini-assisted troubleshooting columns: As part of the preview release, the Dataproc Batches list page in the Google Cloud console includes What was Autotuned, What is happening now? and What can I do about it? columns.

  

  The Ask Gemini button is displayed and enabled only if a completed batch is in a Failed,Cancelled, or Succeeded state. If you click Ask Gemini, Gemini generates a summary of any errors, anomalies, or highlights from workload logs, Spark metrics, and Spark events. Gemini can also display a list of recommended steps you can take to fix a failed workload or improve the performance of a successful, but slow workload.

Batch metric highlights

As part of the preview release, the Batch details page in the Google Cloud console includes charts that display important batch workload metric values. The metric charts are populated with values after the batch completes.

Metrics table

The following table lists the Spark workload metrics displayed on the Batch details page in the Google Cloud console, and describes how metric values can provide insight into workload status and performance.

Metric	What does it show?
Metrics at the Executor level
Ratio of JVM GC Time to Runtime	This metric shows the ratio of JVM GC (garbage collection) time to runtime per executor. High ratios can indicate memory leaks within tasks running on particular executors or inefficient data structures, which can lead to high object churn.
Disk Bytes Spilled	This metric shows the total number of disk bytes spilled across different executors. If an executor shows high disk bytes spilled, this can indicate data skew. If the metric increases over time, this can indicate that there are stages with memory pressure or memory leaks.
Bytes Read and Written	This metric shows the bytes written versus bytes read per executor. Large discrepancies in bytes read or written can indicate scenarios where replicated joins lead to data amplification on specific executors.
Records Read and Written	This metric shows records read and written per executor. Large numbers record read with low numbers of records written can indicate a bottleneck in processing logic on specific executors, leading to records being read while waiting. Executors that consistently lag in reads and writes can indicate resource contention on those nodes or executor-specific code inefficiencies.
Ratio of Shuffle Write Time to Run Time	The metric shows the amount of time the executor spent in shuffle runtime as compared to overall runtime. If this value is high for some executors, it can indicate data skew or inefficient data serialization. You can identify stages with long shuffle write times in the Spark UI. Look for outlier tasks within those stages taking more than the average time to complete. Check whether the executors with high shuffle write times also show high disk I/O activity. More efficient serialization and additional partitioning steps might help. Very large record writes compared to record reads can indicate unintended data duplication due to inefficient joins or incorrect transformations.
Metrics at the Application level
Stages Progression	This metric shows the number of stages in failed, waiting, and running stages. A large number of failed or waiting stages can indicate data skew. Check for data partitions, and debug the reason for stage failure using the Stages tab in the Spark UI.
Batch Spark Executors	This metric shows the number of executors that might be required versus the number of executors running. A large difference between required and running executors can indicate autoscaling issues.
Metrics at the VM level
Memory Used	This metric shows the percentage of VM memory in use. If the master percentage is high, it can indicate that the driver is under memory pressure. For other VM nodes, a high percentage can indicate that the executors are running out of memory, which can lead to high disk spillage and a slower workload runtime. Use the Spark UI to analyze executors to check for high GC time and high task failures. Also debug Spark code for large dataset caching and unnecessary broadcast of variables.

Job logs

As part of the preview release, the Batch details page in the Google Cloud console lists job (batch workload) logs. The logs include warnings and errors filtered from workload output and Spark logs. You can select log Severity, add a Filter, and then click the View in Logs Explorer icon to open the selected batch logs in the Logs Explorer.

Example: Logs Explorer opens after choosing Errors from the Severity selector on the Batch details page in the Google Cloud console.

Spark UI (Preview)

If you enrolled your project in the Spark UI preview feature, you can view the Spark UI in the Google Cloud console without having to create a Dataproc PHS (Persistent History Server) cluster. The Spark UI collects Spark execution details from batch workloads. For more information, see the User Guide distributed to enrolled customers as part of the Spark UI preview release.

Note: Charges will apply to the this features if it becomes generally available (GA): Advance notice of GA charges will be sent to the email address provided in the preview sign-up form.