Quotas and limits

This document lists the quotas and system limits that apply to Cloud Composer. Quotas specify the amount of a countable, shared resource that you can use, and they are defined by Google Cloud services such as Cloud Composer. System limits are fixed values that cannot be changed.

Google Cloud uses quotas to help ensure fairness and reduce spikes in resource use and availability. A quota restricts how much of a Google Cloud resource your Google Cloud project can use. Quotas apply to a range of resource types, including hardware, software, and network components. For example, quotas can restrict the number of API calls to a service, the number of load balancers used concurrently by your project, or the number of projects that you can create. Quotas protect the community of Google Cloud users by preventing the overloading of services. Quotas also help you to manage your own Google Cloud resources.

The Cloud Quotas system does the following:

  • Monitors your consumption of Google Cloud products and services
  • Restricts your consumption of those resources
  • Provides a way to request changes to the quota value

In most cases, when you attempt to consume more of a resource than its quota allows, the system blocks access to the resource, and the task that you're trying to perform fails.

Quotas generally apply at the Google Cloud project level. Your use of a resource in one project doesn't affect your available quota in another project. Within a Google Cloud project, quotas are shared across all applications and IP addresses.

To adjust most quotas, use the Google Cloud console. For more information, see Request a quota adjustment.

There are also system limits on Cloud Composer resources. System limits can't be changed.

Cloud Composer quotas

Quotas in this section apply only to Cloud Composer API and tools that utilize Cloud Composer API:

  • Cloud Composer interface in Google Cloud console
  • gcloud composer and gcloud beta composer commands
  • Cloud Composer REST API
  • Cloud Composer RPC API
  • Terraform, for operations with Cloud Composer environments

Quotas in this section do not apply to any services that you use in your Airflow DAGs. Such services use their own quotas.

Cloud Composer has the following API quotas:

Quota name Limit
Read requests per project 1,000 quota units per minute
Write requests per project 25,000 quota units per day
Write requests per project 1,500 quota units per minute
Save snapshot requests per project 5,000 quota units per day
Save snapshot requests per project 250 quota units per minute
Save snapshot requests per project per environment 2,600 quota units per day
Load snapshot requests per project 2,500 quota units per day
Load snapshot requests per project 150 quota per minute
Load snapshot requests per project per environment 700 quota units per day

Cloud Composer API calls have the following costs, in quota units:

Operation Cost in quota units Request type
All operations 1 Read
environments.create 100 Write
environments.patch 100 Write
environments.delete 100 Write
environments.databaseFailover 100 Write
environments.restartWebServer 100 Write
environments.checkUpgrade 100 Write
environments.executeAirflowCommand 25 Write
environments.stopAirflowCommand 25 Write
environments.saveSnapshot 50 Save snapshot
environments.loadSnapshot 50 Load snapshot

Quota calculation examples

  • One environments.create request consumes 100 quota units from the Write quotas.

    There are two such quotas for write requests:

    • Write requests per project per day
    • Write requests per project per minute

    This operation consumes 100 quota units from each quota.

    If you execute an environments.restartWebServer request afterwards, another 100 quota units are consumed from the same quotas, because environments.restartWebServer shares quotas with the environments.create request.

  • One environments.saveSnapshot request consumes 50 quota units from three quotas:

    • Save snapshot requests per project per day
    • Save snapshot requests per project per minute
    • Save snapshot requests per project per environment per day

    These three quotas limit the maximum number of environments.saveSnapshot requests. Each does it in a different way.

    The Save snapshot requests per project per day quota limit is 2500 quota units. You can execute up to 50 environments.saveSnapshot requests in your project every day.

    The Save snapshot requests per project per minute quota limit is 150 quota units. In one minute, you can execute only up to three environments.saveSnapshot requests in your project.

    Finally, the Save snapshot requests per project per environment per day quota limit is 750 quota units. You can execute up to 15 environments.saveSnapshot requests every day for a single environment. If all quota units are consumed for a specific environment, you can still run environments.saveSnapshot requests for other environments in your project.

Quotas for other services

Cloud Composer uses other Google Cloud services. These services have project-level quotas, which apply when you use Cloud Composer.

For example, quotas for Cloud Storage apply to all buckets associated with environments in your project. As another example, environment's clusters use Google Kubernetes Engine, so quotas for GKE apply to all clusters associated with environments in your project.

Quotas for services used by Cloud Composer

The following services are used by Cloud Composer. These services have their own quota limits:

Quotas for optional services

You can use Airflow operators with Google Cloud services. Any service that you use in a DAG is subject to that service's quotas.