The Cloud Dataflow managed service has the following quota limits:
- Each user may make up to 3000000 requests per minute.
- Each Cloud Dataflow job may use a maximum of 1000 Compute Engine instances.
- Each Google Cloud Platform project may run 25 concurrent Cloud Dataflow jobs.
- Each organization may run 125 concurrent Cloud Dataflow jobs.
- Each user may make up to 15000 monitoring requests per minute.
If you would like to run more than 25 concurrent Cloud Dataflow jobs for your project or more than 125 concurrent Cloud Dataflow jobs for your organization, please contact Google Cloud Platform Support and we will increase the limit to a value that better suits your needs.
In addition, the Cloud Dataflow service exercises various components of the GCP, such as BigQuery, Cloud Storage, Cloud Pub/Sub, and Compute Engine. These (and other GCP services) employ quotas to cap the maximum number of resources you may use within a project. When you use Cloud Dataflow, you may need to adjust your quota settings for these services.
Compute Engine quotas
When you run your pipeline on the Cloud Dataflow service, Cloud Dataflow spins up Compute Engine instances to run your pipeline code.
- CPUs: The default machine types for Cloud Dataflow are
n1-standard-1for batch and
n1-standard-4for streaming. Compute Engine calculates the number of CPUs by summing each instance’s total CPU count. For example, running 10
n1-standard-4instances counts as 40 CPUs. See Compute Engine machine types for a mapping of machine types to CPU count.
- In-Use IP Addresses: The number of in-use IP addresses in your project must be sufficient to accommodate the desired number of instances. To use 10 Compute Engine instances, you'll need 10 in-use IP addresses.
- Persistent Disk: Cloud Dataflow attaches Persistent Disks to each instance. The default disk size is 250GB for batch and 420GB for streaming. For 10 instances, by default you'll need 2,500 GB of Persistent Disk for a batch job.
- Managed Instance Groups: Cloud Dataflow deploys your
Compute Engine instances as a Managed Instance Group. You'll need
to ensure you have the following related quota available:
- One Instance Group per Cloud Dataflow job
- One Managed Instance Group per Cloud Dataflow job
- One Instance Template per Cloud Dataflow job
maxNumWorkersthat fits within your trial limit.
Depending on which sources and sinks you are using, you may also need additional quota.
- Cloud Pub/Sub: If you are using Cloud Pub/Sub, you may need additional quota. When planning for quota, note that processing 1 message from Cloud Pub/Sub involves 3 operations. If you use custom timestamps, you should double your expected number of operations, since Cloud Dataflow will create a separate subscription to track custom timestamps.
- BigQuery: If you are using the streaming API for BigQuery, quota limits and other restrictions apply.
This section describes practical production limits for Cloud Dataflow.
Note: These values are subject to change, and apply to the latest versions of the Cloud Dataflow SDKs. Pipelines compiled with earlier versions of the Cloud Dataflow SDKs may encounter stricter runtime limits.
|Maximum number of workers per pipeline.||1,000|
|Maximum size for a job creation request. Pipeline descriptions with a lot of steps and very verbose names may hit this limit.||10 MB|
|Maximum number of side input shards.||20,000|