Quotas & limits


The Dataflow managed service has the following quota limits:

  • Each user can make up to 3,000,000 requests per minute.
  • Each Dataflow job can use a maximum of 1,000 Compute Engine instances.
  • Each Google Cloud project can run at least 25 concurrent Dataflow jobs.
  • Each Dataflow worker has a maximum limit of logs that it can output in a time interval. See logging documentation for the exact limit.
  • If you opt-in to organization level quotas, each organization can run at least 125 concurrent Dataflow jobs.
  • Each user can make up to 15,000 monitoring requests per minute.
  • Each Google Cloud project gets the following shuffle slots in each region:
    • asia-east1: 48 slots
    • asia-northeast1: 24 slots
    • asia-south1: 48 slots
    • asia-southeast1: 32 slots
    • europe-west1: 160 slots
    • europe-west3: 24 slots
    • europe-west4: 128 slots
    • northamerica-northeast1: 32 slots
    • us-central1: 160 slots
    • us-east1: 160 slots
    • us-east4: 32 slots
    • us-west1: 96 slots
    • us-west2: 24 slots
    • others: 16 slots
    160 slots are sufficient to shuffle approximately 100 TB of data concurrently.
  • Each Google Cloud project gets 600 GB per minute per cloud region of Streaming Engine throughput to send data between Compute Engine instances and Streaming Engine.

You can check your current usage of Dataflow-specific quota:

  1. In the Google Cloud Console, go to the APIs & services.
    Go to API & Services
  2. Click Dashboard.
  3. Click Dataflow API.
  4. Click Quotas.
    For example, to check your current Shuffle slots quota usage, find the Shuffle slots chart on the Quotas page.
    Shuffle slots on Quotas page.

The Dataflow service exercises various components of the Google Cloud, such as BigQuery, Cloud Storage, Pub/Sub, and Compute Engine. These (and other Google Cloud services) employ quotas to cap the maximum number of resources you can use within a project. When you use Dataflow, you might need to adjust your quota settings for these services.

Compute Engine quotas

When you run your pipeline on the Dataflow service, Dataflow creates Compute Engine instances to run your pipeline code.

Compute Engine quota is specified per region. Review your project's Compute Engine quota and request the following adjustments if needed:

  • CPUs: The default machine types for Dataflow are n1-standard-1 for batch, n1-standard-2 for jobs that use Streaming Engine, and n1-standard-4 for jobs that do not use Streaming Engine. FlexRS uses n1-standard-2 machines by default. During the beta release, FlexRS uses 90% preemptible VMs and 10% regular VMs. Compute Engine calculates the number of CPUs by summing each instance’s total CPU count. For example, running 10 n1-standard-4 instances counts as 40 CPUs. See Compute Engine machine types for a mapping of machine types to CPU count.
  • In-Use IP Addresses: The number of in-use IP addresses in your project must be sufficient to accommodate the desired number of instances. To use 10 Compute Engine instances, you'll need 10 in-use IP addresses.
  • Persistent Disk: Dataflow attaches Persistent Disk to each instance.
    • The default disk size is 250 GB for batch and 400 GB for streaming pipelines. For 10 instances, by default you need 2,500 GB of Persistent Disk for a batch job.
    • The default disk size is 25 GB for Dataflow Shuffle batch pipelines.
    • The default disk size is 30 GB for Streaming Engine streaming pipelines.
  • Managed Instance Groups: Dataflow deploys your Compute Engine instances as a Managed Instance Group. You'll need to ensure you have the following related quota available:
    • One Instance Group per Dataflow job
    • One Managed Instance Group per Dataflow job
    • One Instance Template per Dataflow job

Additional quotas

Depending on which sources and sinks you are using, you might also need additional quota.

  1. Pub/Sub: If you are using Pub/Sub, you might need additional quota. When planning for quota, note that processing 1 message from Pub/Sub involves 3 operations. If you use custom timestamps, you should double your expected number of operations, since Dataflow will create a separate subscription to track custom timestamps.
  2. BigQuery: If you are using the streaming API for BigQuery, quota limits and other restrictions apply.

Dataflow Prime

Quotas and limits are the same for Dataflow and Dataflow Prime. If you have quotas for Dataflow, then you don't need additional quota to run your jobs using Dataflow Prime.


This section describes practical production limits for Dataflow.

Limit Amount
Maximum number of workers per pipeline. 1,000
Maximum size for a job creation request. Pipeline descriptions with a lot of steps and very verbose names may reach this limit. 10 MB
Maximum number of side input shards. 20,000
Maximum size for a single element value in Streaming Engine. 100 MB
Maximum number of log entries in a given time period, per worker. 15,000 messages every 30 seconds
Maximum number of custom metrics per Dataflow job 100
Streaming Engine Limits Amount
Maximum bytes for Pub/Sub messages. 7 MB
Maximum size of a large key. Keys over 64 KB cause decreased performance. 2 MB
Maximum length for state tags used by TagValue and TagBag. 64 KB