This document lists the quotas and limits that apply to Dataform.
A quota restricts how much of a shared Google Cloud resource your Google Cloud project can use, including hardware, software, and network components. Therefore, quotas are a part of a system that does the following:
- Monitors your use or consumption of Google Cloud products and services.
- Restricts your consumption of those resources, for reasons that include ensuring fairness and reducing spikes in usage.
- Maintains configurations that automatically enforce prescribed restrictions.
- Provides a means to request or make changes to the quota.
In most cases, when a quota is exceeded, the system immediately blocks access to the relevant Google resource, and the task that you're trying to perform fails. In most cases, quotas apply to each Google Cloud project and are shared across all applications and IP addresses that use that Google Cloud project.
There are also limits on Dataform resources. These limits are unrelated to the quota system. Limits cannot be changed unless otherwise stated.
Quota replenishment
Daily quotas are replenished at regular intervals throughout the day, reflecting their intent to guide rate limiting behaviors. Intermittent refresh is also done to avoid long disruptions when quota is exhausted. More quota is typically made available within minutes rather than globally replenished once daily.
Request a quota increase
To increase or decrease most quotas, use the Google Cloud console. For more information, see Request a higher quota.
Cap quota usage
To learn how you can limit usage of a particular resource by specifying a smaller quota than the default, see Cap usage.
Required permissions
To view and update your Dataform quotas in the Google Cloud console, you need the same permissions as for any Google Cloud quota. For more information, see Quota permissions.
Troubleshooting
For information about troubleshooting errors related to Dataform quotas and limits, see Troubleshooting.
Dataform quotas
Dataform has the following API quotas:
| Quotas | Limit |
|---|---|
| Total requests per project, per region | 6000/minute/project/region |
| Compilation requests per project, per region | 120/minute/project/region |
| File access requests per project, per region | 120/minute/project/region |
| Package installation requests per project, per region | 120/minute/project/region |
| Workflow invocation requests per project, per region | 60/minute/project/region |
| Workflow actions per execution | 5000 |
Quotas for services used by Dataform
Dataform uses other Google Cloud services. These services have project-level quotas, which apply when you use Dataform.
Dataform uses the following Google Cloud services that have their own quota limits:
Dataform compilation resources limits
Dataform enforces the following usage limits on compilation resources:
| Usage limit | Value |
|---|---|
| Maximum number of actions per repository compilation | 5000 |
| Maximum number of dependencies per action in the compiled graph | 50 |
| Maximum total serialized data size of the compiled graph of actions | 20 MB |
| Maximum size of NPM dependencies | 200 MB |
Additionally, Dataform imposes limits on CPU and memory use during compilation. If these limits are exceeded when you compile your code, the compilation fails and Dataform throws a compilation error.
To estimate the compilation resources usage of your repository, see Estimate compilation resources usage of your repository.
Your repository might exceed the usage limits for the following reasons:
- An infinite loop bug in the repository code.
- A memory leak bug in the repository code.
- Large repository size, approximately more than 1000 SQL workflow nodes.
- Large number of concurrent queries.
For more information about repository size in Dataform, see
Overview of repository size.