Dataproc driver node groups

Overview

A Dataproc NodeGroup resource is a group of Dataproc cluster nodes that execute an assigned role. This page describes the driver node group, which is a group of Compute Engine VMS that are assigned the Driver role for the purpose of running job drivers on the Dataproc cluster.

When to use driver node groups

Use driver node groups only when you need to run many concurrent jobs on a shared cluster.
Increase master node resources before using driver node groups to avoid driver node group limitations.

How driver nodes help you run concurrents job

Dataproc starts a job driver process on a Dataproc cluster master node for each job. The driver process, in turn, runs an application driver, such as spark-submit, as its child process. However, the number of concurrent jobs running on the master is limited by the resources available on master node, and since Dataproc master nodes can't be scaled, a job can fail or get throttled when master node resources are insufficient to run a job.

Driver node groups are special node groups managed by YARN, so job concurrency is not limited by master node resources. In clusters with a driver node group, application drivers run on driver nodes. Each driver node can run multiple application drivers if the node has sufficient resources.

Benefits

Using a Dataproc cluster with a driver node group lets you:

Horizontally scale job driver resources to run more concurrent jobs
Scale driver resources separately from worker resources
Obtain faster scaledown on Dataproc 2.0+ and later image clusters. On these clusters, the app master runs within a Spark driver in a driver node group (the spark.yarn.unmanagedAM.enabled is set to true by default).
Customize driver node start-up. You can add {ROLE} == 'Driver' in an initialization script to have the script perform actions for a driver node group in node selection.

Limitations

Node groups are not supported in Dataproc workflow templates.
Node group clusters cannot be stopped, restarted, or autoscaled.
The MapReduce app master runs on worker nodes. A scale down of worker nodes can be slow if you enable graceful decommissioning.
Job concurrency is affected by the dataproc:agent.process.threads.job.max cluster property. For example, with three masters and this property set to the default value of 100, maximum cluster-level job concurrency is 300.

Driver node group compared to Spark cluster mode

Feature	Spark cluster mode	Driver node group
Worker node scale down	Long-lived drivers run on the same worker nodes as short-lived containers, making scale down of workers using graceful decommission slow.	Worker nodes scale down more quickly when drivers run on node groups.
Streamed driver output	Requires searching in YARN logs to find the node where the driver was scheduled.	Driver output is streamed to Cloud Storage, and is viewable in the Google Cloud console and in the `gcloud dataproc jobs wait` command output after a job completes.

Driver node group IAM permissions

The following IAM permissions are associated the following Dataproc node group related actions.

Permission	Action
`dataproc.nodeGroups.create`	Create Dataproc node groups. If a user has `dataproc.clusters.create` in the project, this permission is granted.
`dataproc.nodeGroups.get`	Get the details of a Dataproc node group.
`dataproc.nodeGroups.update`	Resize a Dataproc node group.

Driver node group operations

You can use the gcloud CLI and Dataproc API to create, get, resize, delete, and submit a job to a Dataproc driver node group.

Create a driver node group cluster

A driver node group is associated with one Dataproc cluster. You create a node group as part of creating a Dataproc cluster. You can use the gcloud CLI or Dataproc REST API to create a Dataproc cluster with a driver node group.

gcloud

gcloud dataproc clusters create CLUSTER_NAME \
    --region=REGION \
    --driver-pool-size=SIZE \
    --driver-pool-id=NODE_GROUP_ID

Required flags:

CLUSTER_NAME: The cluster name, which must be unique within a project. The name must start with a lowercase letter, and can contain up to 51 lowercase letters, numbers, and hyphens. It cannot end with a hyphen. The name of a deleted cluster can be reused.
REGION: The region where the cluster will be located.
SIZE: The number of driver nodes in the node group. The number of nodes needed depend on job load and driver pool machine type. The number of minimum driver group nodes is equal to total memory or vCPUs required by job drivers divided by each driver pool's machine memory or vCPUs.
NODE_GROUP_ID: Optional and recommended. The ID must be unique within the cluster. Use this ID to identify the driver group in future operations, such as resizing the node group. If not specified, Dataproc generates the node group ID.

Recommended flag:

--enable-component-gateway: Add this flag to enable the Dataproc Component Gateway, which provides access to the YARN web interface. The YARN UI Application and Scheduler pages display cluster and job status, application queue memory, core capacity, and other metrics.

Additional flags: The following optional driver-pool flags can be added to the gcloud dataproc clusters create command to customize the node group.

Flag	Default value
`--driver-pool-id`	A string identifier, generated by the service if not set by the flag. This ID can be used to identify the node group when performing future node pool operations, such as resizing the node group.
`--driver-pool-machine-type`	`n1-standard-4`
`--driver-pool-accelerator`	No default. When specifying an accelerator, the GPU type is required; the number of GPUs is optional.
`--num-driver-pool-local-ssds`	No default
`--driver-pool-local-ssd-interface`	No default
`--driver-pool-boot-disk-type`	`pd-standard`
`--driver-pool-boot-disk-size`	`1000 GB`
`--driver-pool-min-cpu-platform`	`AUTOMATIC`

REST

Complete a AuxiliaryNodeGroup as part of a Dataproc API cluster.create request.

Before using any of the request data, make the following replacements:

PROJECT_ID: Required. Google Cloud project ID.
REGION: Required. Dataproc cluster region.
CLUSTER_NAME: Required. The cluster name, which must be unique within a project. The name must start with a lowercase letter, and can contain up to 51 lowercase letters, numbers, and hyphens. It cannot end with a hyphen. The name of a deleted cluster can be reused.
SIZE: Required. Number of nodes in the node group.
NODE_GROUP_ID: Optional and recommended. The ID must be unique within the cluster. Use this ID to identify the driver group in future operations, such as resizing the node group. If not specified, Dataproc generates the node group ID.

Additional options: See NodeGroup.

Set the EndpointConfig.enableHttpPortAccess property to true to enable the Dataproc Component Gateway, which provides access to the YARN web interface. The YARN UI Application and Scheduler pages display cluster and job status, application queue memory, core capacity, and other metrics.

HTTP method and URL:

POST https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/clusters

Request JSON body:

{
  "clusterName":"CLUSTER_NAME",
  "config": {
    "softwareConfig": {
      "imageVersion":""
    },
    "endpointConfig": {
      "enableHttpPortAccess": true
    },
    "auxiliaryNodeGroups": [{
        "nodeGroup":{
            "roles":["DRIVER"],
            "nodeGroupConfig": {
                "numInstances": SIZE
             }
         },
        "nodeGroupId": "NODE_GROUP_ID"
    }]
  }
}

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Note: The following command assumes that you have logged in to the gcloud CLI with your user account by running gcloud init or gcloud auth login , or by using Cloud Shell, which automatically logs you into the gcloud CLI . You can check the currently active account by running gcloud auth list.

Save the request body in a file named request.json, and execute the following command:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/clusters"

PowerShell (Windows)

Note: The following command assumes that you have logged in to the gcloud CLI with your user account by running gcloud init or gcloud auth login . You can check the currently active account by running gcloud auth list.

Save the request body in a file named request.json, and execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/clusters" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "projectId": "PROJECT_ID",
  "clusterName": "CLUSTER_NAME",
  "config": {
    ...
    "auxiliaryNodeGroups": [
      {
        "nodeGroup": {
"name": "projects/PROJECT_ID/regions/REGION/clusters/CLUSTER_NAME/nodeGroups/NODE_GROUP_ID",
          "roles": [
            "DRIVER"
          ],
          "nodeGroupConfig": {
            "numInstances": SIZE,
            "instanceNames": [
              "CLUSTER_NAME-np-q1gp",
              "CLUSTER_NAME-np-xfc0"
            ],
            "imageUri": "https://www.googleapis.com/compute/v1/projects/cloud-dataproc-ci/global/images/dataproc-2-0-deb10-...-rc01",
            "machineTypeUri": "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/zones/REGION-a/machineTypes/n1-standard-4",
            "diskConfig": {
              "bootDiskSizeGb": 1000,
              "bootDiskType": "pd-standard"
            },
            "managedGroupConfig": {
              "instanceTemplateName": "dataproc-2a8224d2-...",
              "instanceGroupManagerName": "dataproc-2a8224d2-..."
            },
            "minCpuPlatform": "AUTOMATIC",
            "preemptibility": "NON_PREEMPTIBLE"
          }
        },
        "nodeGroupId": "NODE_GROUP_ID"
      }
    ]
  },
}

Get driver node group cluster metadata

You can use the gcloud dataproc node-groups describe command or the Dataproc API to get driver node group metadata.

gcloud

gcloud dataproc node-groups describe NODE_GROUP_ID \
    --cluster=CLUSTER_NAME \
    --region=REGION

Required flags:

NODE_GROUP_ID: You can run gcloud dataproc clusters describe CLUSTER_NAME to list the node group ID.
CLUSTER_NAME: The cluster name.
REGION: The cluster region.

REST

Before using any of the request data, make the following replacements:

PROJECT_ID: Required. Google Cloud project ID.
REGION: Required. The cluster region.
CLUSTER_NAME: Required. The cluster name.
NODE_GROUP_ID: Required. You can run gcloud dataproc clusters describe CLUSTER_NAME to list the node group ID.

HTTP method and URL:

GET https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/clusters/CLUSTER_NAMEnodeGroups/Node_GROUP_ID

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Execute the following command:

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/clusters/CLUSTER_NAMEnodeGroups/Node_GROUP_ID"

PowerShell (Windows)

Execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/clusters/CLUSTER_NAMEnodeGroups/Node_GROUP_ID" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "name": "projects/PROJECT_ID/regions/REGION/clusters/CLUSTER_NAME/nodeGroups/NODE_GROUP_ID",
  "roles": [
    "DRIVER"
  ],
  "nodeGroupConfig": {
    "numInstances": 5,
    "imageUri": "https://www.googleapis.com/compute/v1/projects/cloud-dataproc-ci/global/images/dataproc-2-0-deb10-...-rc01",
    "machineTypeUri": "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/zones/REGION-a/machineTypes/n1-standard-4",
    "diskConfig": {
      "bootDiskSizeGb": 1000,
      "bootDiskType": "pd-standard"
    },
    "managedGroupConfig": {
      "instanceTemplateName": "dataproc-driver-pool-mcia3j656h2fy",
      "instanceGroupManagerName": "dataproc-driver-pool-mcia3j656h2fy"
    },
    "minCpuPlatform": "AUTOMATIC",
    "preemptibility": "NON_PREEMPTIBLE"
  }
}

Resize a driver node group

You can use the gcloud dataproc node-groups resize command or the Dataproc API to add or remove driver nodes from a cluster driver node group.

gcloud

gcloud dataproc node-groups resize NODE_GROUP_ID \
    --cluster=CLUSTER_NAME \
    --region=REGION \
    --size=SIZE

Required flags:

NODE_GROUP_ID: You can run gcloud dataproc clusters describe CLUSTER_NAME to list the node group ID.
CLUSTER_NAME: The cluster name.
REGION: The cluster region.
SIZE: Specify the new number of driver nodes in the node group.

Optional flag:

--graceful-decommission-timeout=TIMEOUT_DURATION: When scaling down a node group, you can add this flag to specify a graceful decommissioning TIMEOUT_DURATION to avoid the immediate termination of job drivers. Recommendation: Set a timeout duration that is at least equal to the duration of longest job running on the node group (recovery of failed drivers is not supported).

Example: gcloud CLI NodeGroup scale up command:

gcloud dataproc node-groups resize NODE_GROUP_ID \
    --cluster=CLUSTER_NAME \
    --region=REGION \
    --size=4

Example: gcloud CLI NodeGroup scale down command:

gcloud dataproc node-groups resize NODE_GROUP_ID \
    --cluster=CLUSTER_NAME \
    --region=REGION \
    --size=1 \
    --graceful-decommission-timeout="100s"

REST

Before using any of the request data, make the following replacements:

PROJECT_ID: Required. Google Cloud project ID.
REGION: Required. The cluster region.
NODE_GROUP_ID: Required. You can run gcloud dataproc clusters describe CLUSTER_NAME to list the node group ID.
SIZE: Required. New number of nodes in the node group.
TIMEOUT_DURATION: Optional. When scaling down a node group, you can add a gracefulDecommissionTimeout to the request body to avoid the immediate termination of job drivers. Recommendation: Set a timeout duration that is at least equal to the duration of longest job running on the node group (recovery of failed drivers is not supported).
Example:
```
{ "size": SIZE,
  "gracefulDecommissionTimeout": "TIMEOUT_DURATION"
}
   
```

HTTP method and URL:

POST https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/clusters/CLUSTER_NAME/nodeGroups/Node_GROUP_ID:resize

Request JSON body:

{
  "size": SIZE,
}

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Save the request body in a file named request.json, and execute the following command:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/clusters/CLUSTER_NAME/nodeGroups/Node_GROUP_ID:resize"

PowerShell (Windows)

Save the request body in a file named request.json, and execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/clusters/CLUSTER_NAME/nodeGroups/Node_GROUP_ID:resize" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "name": "projects/PROJECT_ID/regions/REGION/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.dataproc.v1.NodeGroupOperationMetadata",
    "nodeGroupId": "NODE_GROUP_ID",
    "clusterUuid": "CLUSTER_UUID",
    "status": {
      "state": "PENDING",
      "innerState": "PENDING",
      "stateStartTime": "2022-12-01T23:34:53.064308Z"
    },
    "operationType": "RESIZE",
    "description": "Scale "up or "down" a GCE node pool to SIZE nodes."
  }
}

Delete a driver node group cluster

When you delete a Dataproc cluster, node groups associated with the cluster are deleted.

Submit a job

You can use the gcloud dataproc jobs submit command or the Dataproc API to submit a job to a cluster with a driver node group.

gcloud

gcloud dataproc jobs submit JOB_COMMAND \
    --cluster=CLUSTER_NAME \
    --region=REGION \
    --driver-required-memory-mb=DRIVER_MEMORY \
    --driver-required-vcores=DRIVER_VCORES \
    DATAPROC_FLAGS \
    -- JOB_ARGS

Required flags:

JOB_COMMAND: Specify the job command.
CLUSTER_NAME: The cluster name.
DRIVER_MEMORY: Amount of job drivers memory in MB needed to run a job (see Yarn Memory Controls).
DRIVER_VCORES: The number of vCPUs needed to run a job.

Additional flags:

DATAPROC_FLAGS: Add any additional gcloud dataproc jobs submit flags related to the job type.
JOB_ARGS: Add any arguments (after the -- to pass to the job.

Examples: You can run the following examples from an SSH terminal session on a Dataproc driver node group cluster.

Spark job to estimate value of pi:

gcloud dataproc jobs submit spark \
    --cluster=CLUSTER_NAME \
    --region=REGION \
    --driver-required-memory-mb=2048 \
    --driver-required-vcores=2 \
    --class=org.apache.spark.examples.SparkPi \
    --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar \
    -- 1000

Spark wordcount job:

gcloud dataproc jobs submit spark \
    --cluster=CLUSTER_NAME \
    --region=REGION \
    --driver-required-memory-mb=2048 \
    --driver-required-vcores=2 \
    --class=org.apache.spark.examples.JavaWordCount \
    --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar \
    -- 'gs://apache-beam-samples/shakespeare/macbeth.txt'

PySpark job to estimate value of pi:

gcloud dataproc jobs submit pyspark \
    file:///usr/lib/spark/examples/src/main/python/pi.py \
    --cluster=CLUSTER_NAME \
    --region=REGION \
    --driver-required-memory-mb=2048 \
    --driver-required-vcores=2 \
    -- 1000

Hadoop TeraGen MapReduce job:

gcloud dataproc jobs submit hadoop \
    --cluster=CLUSTER_NAME \
    --region=REGION \
    --driver-required-memory-mb=2048 \
    --driver-required-vcores=2 \
    --jar file:///usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar \
    -- teragen 1000 \
    hdfs:///gen1/test

REST

Before using any of the request data, make the following replacements:

PROJECT_ID: Required. Google Cloud project ID.
REGION: Required. Dataproc cluster region
CLUSTER_NAME: Required. The cluster name, which must be unique within a project. The name must start with a lowercase letter, and can contain up to 51 lowercase letters, numbers, and hyphens. It cannot end with a hyphen. The name of a deleted cluster can be reused.
DRIVER_MEMORY: Required. Amount of job drivers memory in MB needed to run a job (see Yarn Memory Controls).
DRIVER_VCORES: Required. The number of vCPUs needed to run a job.

Additional fields: Add additional fields related to the job type and job arguments (the sample request includes fields needed to submit a Spark job that estimates the value of pi).

HTTP method and URL:

POST https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/jobs:submit

Request JSON body:

{
  "job": {
    "placement": {
    "clusterName": "CLUSTER_NAME",
    },
    "driverSchedulingConfig": {
      "memoryMb]": DRIVER_MEMORY,
      "vcores": DRIVER_VCORES
    },
    "sparkJob": {
      "jarFileUris": "file:///usr/lib/spark/examples/jars/spark-examples.jar",
      "args": [
        "10000"
      ],
      "mainClass": "org.apache.spark.examples.SparkPi"
    }
  }
}

To send your request, expand one of these options:

curl (Linux, macOS, or Cloud Shell)

Save the request body in a file named request.json, and execute the following command:

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/jobs:submit"

PowerShell (Windows)

Save the request body in a file named request.json, and execute the following command:

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://dataproc.googleapis.com/v1/projects/PROJECT_ID/regions/REGION/jobs:submit" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "reference": {
    "projectId": "PROJECT_ID",
    "jobId": "job-id"
  },
  "placement": {
    "clusterName": "CLUSTER_NAME",
    "clusterUuid": "cluster-Uuid"
  },
  "sparkJob": {
    "mainClass": "org.apache.spark.examples.SparkPi",
    "args": [
      "1000"
    ],
    "jarFileUris": [
      "file:///usr/lib/spark/examples/jars/spark-examples.jar"
    ]
  },
  "status": {
    "state": "PENDING",
    "stateStartTime": "start-time"
  },
  "jobUuid": "job-Uuid"
}

View job logs

To view job status and help debug job issues, you can view driver logs using the gcloud CLI or the Google Cloud console.

gcloud

Job driver logs are streamed to the gcloud CLI output or Google Cloud console during job execution. Driver logs persist in a the Dataproc cluster staging bucket in Cloud Storage.

Run the following gcloud CLI command to list the location of driver logs in Cloud Storage:

gcloud dataproc jobs describe JOB_ID \
    --region=REGION

The Cloud Storage location of driver logs is listed as the driverOutputResourceUri in the command output in the following format:

driverOutputResourceUri: gs://CLUSTER_STAGING_BUCKET/google-cloud-dataproc-metainfo/CLUSTER_UUID/jobs/JOB_ID

Console

To view node group cluster logs:

Enable Logging.
You can use the following Logs Explorer query format to find logs:
```
resource.type="cloud_dataproc_cluster"
resource.labels.project_id="PROJECT_ID"
resource.labels.cluster_name="CLUSTER_NAME"
log_name="projects/PROJECT_ID/logs/LOG_TYPE>"
```
Replace the following;
- PROJECT_ID: Google Cloud project ID.
- CLUSTER_NAME: The cluster name.
- LOG_TYPE:
  - Yarn user logs: yarn-userlogs
  - Yarn resource manager logs: hadoop-yarn-resourcemanager
  - Yarn node manager logs: hadoop-yarn-nodemanager

Monitor metrics

Dataproc node group job drivers run in a dataproc-driverpool-driver-queue child queue under a dataproc-driverpool partition.

Driver node group metrics

The following table lists the associated node group driver metrics, which are collected by default for driver node groups.

Driver node group metric	Description
`yarn:ResourceManager:DriverPoolsQueueMetrics:AvailableMB`	The amount available memory in Mebibytes in `dataproc-driverpool-driver-queue` under the `dataproc-driverpool` partition.
`yarn:ResourceManager:DriverPoolsQueueMetrics:PendingContainers`	The number of pending (queued) containers in `dataproc-driverpool-driver-queue` under the `dataproc-driverpool` partition.

Child queue metrics

The following table lists the child queue metrics. The metrics are collected by default for driver node groups, and can be enabled for collection on any Dataproc clusters.

Child queue metric	Description
`yarn:ResourceManager:ChildQueueMetrics:AvailableMB`	The amount of the available memory in Mebibytes in this queue under the default partition.
`yarn:ResourceManager:ChildQueueMetrics:PendingContainers`	Number of pending (queued) containers in this queue under the default partition.
`yarn:ResourceManager:ChildQueueMetrics:running_0`	The number of jobs with a runtime between `0` and `60` minutes in this queue under all partitions.
`yarn:ResourceManager:ChildQueueMetrics:running_60`	The number of jobs with a runtime between `60` and `300` minutes in this queue under all partitions.
`yarn:ResourceManager:ChildQueueMetrics:running_300`	The number of jobs with a runtime between `300` and `1440` minutes in this queue under all partitions.
`yarn:ResourceManager:ChildQueueMetrics:running_1440`	The number of jobs with a runtime greater than `1440` minutes in this queue under all partitions.
`yarn:ResourceManager:ChildQueueMetrics:AppsSubmitted`	Number of applications submitted to this queue under all partitions.

To view YARN ChildQueueMetrics and DriverPoolsQueueMetrics in the Google Cloud console:

Select VM Instance → Custom resources in the Metrics Explorer.

Node group job driver debugging

This section provides driver node group conditions and errors with recommendations to fix the condition or error.

Conditions

Condition: yarn:ResourceManager:DriverPoolsQueueMetrics:AvailableMB is nearing 0. This indicates that cluster driver pools queue are running out of memory.

Recommendation:: Scale up the size of the driver pool.
Condition: yarn:ResourceManager:DriverPoolsQueueMetrics:PendingContainers is larger than 0. This can indicate that cluster driver pools queue are running out of memory and YARN is queuing jobs.

Recommendation:: Scale up the size of the driver pool.

Errors

Error: Cluster <var>CLUSTER_NAME</var> requires driver scheduling config to run SPARK job because it contains a node pool with role DRIVER. Positive values are required for all driver scheduling config values.

Recommendation: Set driver-required-memory-mb and driver-required-vcores with positive numbers.
Error: Container exited with a non-zero exit code 137.

Recommendation: Increase driver-required-memory-mb to job memory usage.