Dataproc Quotas

Dataproc has API quota limits that are enforced at the project and region level. The quotas reset every sixty seconds (one-minute).

Handling Dataproc Quota Errors: If you exceed a Dataproc quota limit, a RESOURCE_EXHAUSTED (HTTP code 429) is generated, and the corresponding Dataproc API request will fail. However, since your project's Dataproc quota is refreshed every sixty seconds, you can retry your request after one minute has elapsed following the failure.

The following table lists the specific and default per-project Dataproc API quota types, quota limits, and the methods to which they apply.

Quota Type	Limit	Applicable API Methods
AutoscalingOperationRequestsPerMinutePerProjectPerRegion	400	CreateAutoscalingPolicy, GetAutoscalingPolicy, ListAutoscalingPolicies, UpdateAutoscalingPolicy, DeleteAutoscalingPolicy
ClusterOperationRequestsPerMinutePerProjectPerRegion	200	CreateCluster, DeleteCluster, UpdateCluster, DiagnoseCluster
GetJobRequestsPerMinutePerProjectPerRegion	7500	GetJob
JobOperationRequestsPerMinutePerProjectPerRegion	400	SubmitJob, UpdateJob, CancelJob, DeleteJob
WorkflowOperationRequestsPerMinutePerProjectPerRegion	400	CreateWorkflowTemplate, InstantiateWorkflowTemplate, InstantiateInlineWorkflowTemplate, UpdateWorkflowTemplate, DeleteWorkflowTemplate
DefaultRequestsPerMinutePerProjectPerRegion	7500	All other operations (primarily Get operations)

Increasing Resource Quota Limits: Open the Google Cloud IAM & admin→Quotas page, and select the resources you want to modify. Then click Edit Quotas at the top of the page to start the quota increase process. If the resources you are trying to increase aren't displayed on the page and the current filtering is "Quotas with usage," change the filtering to "All quotas" by toggling the "Quota type" dropdown.

Other Google Cloud quotas

Dataproc clusters utilize other Google Cloud products. These products have project-level quotas, which include quotas that apply to Dataproc use. Some services are required to use Dataproc, such as Compute Engine and Cloud Storage. Other services, such as BigQuery and Cloud Bigtable, can optionally be used with Dataproc.

Required services

The following services, which enforce quota limits, are automatically (required to be) used to create Dataproc clusters.

Compute Engine

Dataproc clusters utilize Compute Engine virtual machines. The Compute Engine quotas are split into regional and global limits. These limits apply to clusters you create. For example, to create a cluster with one n1-standard-4 master node and two n1-standard-4 worker nodes, you will use 12 virtual CPUs (4 * 3). This cluster usage will count against the regional quota limit of 24 virtual CPUs.

Free trial quotas: If you use the Compute Engine free trial, your project will have a limit of 8 CPUs. To increase this limit, you must enable billing on the project.

Default clusters

When you create a Dataproc cluster with default settings, the following Compute Engine resources are used.

Item	Resources used
Virtual CPUs	12
Virtual Machine (VM) Instances	3
Persistent disk	1500 GB

Cloud Logging

Dataproc saves driver output and cluster logs in Cloud Logging. The Cloud Logging quota applies to your Dataproc clusters.

Optional services

The following services, which have quota limits, can optionally be used with Dataproc clusters.

BigQuery

When reading or writing data into BigQuery, the BigQuery quota applies.

Bigtable

When reading or writing data into Bigtable, the Bigtable quota applies.