Quotas and limits

This document lists the quotas and limits that apply to Cloud Composer.

A quota restricts how much of a shared Google Cloud resource your Google Cloud project can use, including hardware, software, and network components. Therefore, quotas are a part of a system that does the following:

  • Monitors your use or consumption of Google Cloud products and services.
  • Restricts your consumption of those resources, for reasons that include ensuring fairness and reducing spikes in usage.
  • Maintains configurations that automatically enforce prescribed restrictions.
  • Provides a means to request or make changes to the quota.

In most cases, when a quota is exceeded, the system immediately blocks access to the relevant Google resource, and the task that you're trying to perform fails. In most cases, quotas apply to each Google Cloud project and are shared across all applications and IP addresses that use that Google Cloud project.

To increase or decrease most quotas, use the Google Cloud console. For more information, see Request a higher quota.

There are also limits on Cloud Composer resources. These limits are unrelated to the quota system. Limits cannot be changed unless otherwise stated.

Cloud Composer quotas

Quotas in this section apply only to Cloud Composer API and tools that utilize Cloud Composer API:

  • Cloud Composer interface in Google Cloud console
  • gcloud composer and gcloud beta composer commands
  • Cloud Composer REST API
  • Cloud Composer RPC API
  • Terraform, for operations with Cloud Composer environments

Quotas in this section do not apply to any services that you use in your Airflow DAGs. Such services use their own quotas.

Cloud Composer has the following API quotas:

Quota name Limit
Read requests per project 1,000 quota units per minute
Write requests per project 25,000 quota units per day
Write requests per project 1,500 quota units per minute
Save snapshot requests per project 5,000 quota units per day
Save snapshot requests per project 250 quota units per minute
Save snapshot requests per project per environment 2,600 quota units per day
Load snapshot requests per project 2,500 quota units per day
Load snapshot requests per project 150 quota per minute
Load snapshot requests per project per environment 700 quota units per day

Cloud Composer API calls have the following costs, in quota units:

Operation Cost in quota units Request type
All operations 1 Read
environments.create 100 Write
environments.patch 100 Write
environments.delete 100 Write
environments.databaseFailover 100 Write
environments.restartWebServer 100 Write
environments.checkUpgrade 100 Write
environments.executeAirflowCommand 25 Write
environments.stopAirflowCommand 25 Write
environments.saveSnapshot 50 Save snapshot
environments.loadSnapshot 50 Load snapshot

Quota calculation examples

  • One environments.create request consumes 100 quota units from the Write quotas.

    There are two such quotas for write requests:

    • Write requests per project per day
    • Write requests per project per minute

    This operation consumes 100 quota units from each quota.

    If you execute an environments.restartWebServer request afterwards, another 100 quota units are consumed from the same quotas, because environments.restartWebServer shares quotas with the environments.create request.

  • One environments.saveSnapshot request consumes 50 quota units from three quotas:

    • Save snapshot requests per project per day
    • Save snapshot requests per project per minute
    • Save snapshot requests per project per environment per day

    These three quotas limit the maximum number of environments.saveSnapshot requests. Each does it in a different way.

    The Save snapshot requests per project per day quota limit is 2500 quota units. You can execute up to 50 environments.saveSnapshot requests in your project every day.

    The Save snapshot requests per project per minute quota limit is 150 quota units. In one minute, you can execute only up to three environments.saveSnapshot requests in your project.

    Finally, the Save snapshot requests per project per environment per day quota limit is 750 quota units. You can execute up to 15 environments.saveSnapshot requests every day for a single environment. If all quota units are consumed for a specific environment, you can still run environments.saveSnapshot requests for other environments in your project.

Quotas for other services

Cloud Composer uses other Google Cloud services. These services have project-level quotas, which apply when you use Cloud Composer.

For example, quotas for Cloud Storage apply to all buckets associated with environments in your project. As another example, environment's clusters use Google Kubernetes Engine, so quotas for GKE apply to all clusters associated with environments in your project.

Quotas for services used by Cloud Composer

The following services are used by Cloud Composer. These services have their own quota limits:

Quotas for optional services

You can use Airflow operators with Google Cloud services. Any service that you use in a DAG is subject to that service's quotas.