Cloud Dataproc Quotas

Cloud Dataproc has API quota limits that are enforced at the project level. The quotas reset every sixty seconds (one-minute).

The following table lists the specific and default per-project Cloud Dataproc API quota types, quota limits, and the methods to which they apply.

Quota Type Limit Applicable API Methods
ClusterOperationRequestsPerMinutePerProject 100 CreateCluster, DeleteCluster, UpdateCluster, DiagnoseCluster
JobOperationRequestsPerMinutePerProject 200 SubmitJob, UpdateJob, CancelJob, DeleteJob
GetJobRequestsPerMinutePerProject 5000 GetJob
WorkflowOperationRequestsPerMinutePerProject 100 CreateWorkflowTemplate, InstantiateWorkflowTemplate, InstantiateInlineWorkflowTemplate, UpdateWorkflowTemplate, DeleteWorkflowTemplate
DefaultRequestsPerMinutePerProject 2000 All other operations (primarily Get operations)

Other Google Cloud Platform quotas

Cloud Dataproc clusters utilize other GCP products. These products have project-level quotas, which include quotas that apply to Cloud Dataproc use. Some services are required to use Cloud Dataproc, such as Compute Engine and Cloud Storage. Other services, such as BigQuery and Cloud Bigtable, can optionally be used with Cloud Dataproc.

Required services

The following services, which enforce quota limits, are automatically (required to be) used to create Cloud Dataproc clusters.

Compute Engine

Cloud Dataproc clusters utilize Compute Engine virtual machines. The Compute Engine quotas are split into regional and global limits. These limits apply to clusters you create. For example, to create a cluster with one n1-standard-4 master node and two n1-standard-4 worker nodes, you will use 12 virtual CPUs (4 * 3). This cluster usage will count against the regional quota limit of 24 virtual CPUs.

Default clusters

When you create a Cloud Dataproc cluster with default settings, the following Compute Engine resources are used.

Item Resources used
Virtual CPUs 12
Virtual Machine (VM) Instances 3
Persistent disk 1500 GB

Stackdriver Logging

Cloud Dataproc saves driver output and cluster logs in Logging. The Logging quota applies to your Cloud Dataproc clusters.

Optional services

The following services, which have quota limits, can optionally be used with Cloud Dataproc clusters.

BigQuery

When reading or writing data into BigQuery, the BigQuery quota applies.

Bigtable

When reading or writing data into Bigtable, the Bigtable quota applies.

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataproc Documentation