Cloud Composer Control Plane Resource Quotas

Cloud Composer has the following API quota limits.

Quota name Limit
Write requests per project 1,500/minute, 25,000/day
Read requests per project 1,000/minute

Most API calls cost 1 unit to perform, but some are more expensive. The following APIs cost 100 units to perform: environments.create, environments.update, and environments.delete.

The following tools are built on top of the Cloud Composer API, and use your Cloud Composer API quota:

Cloud Composer utilizes other Google Cloud products. These products have project-level quotas, which include quotas that apply to Cloud Composer use. Some services are required to use Cloud Composer, such as Google Compute Engine and Google Cloud Storage. Other services can optionally be used with Cloud Composer when setting connectors in Airflow.

Required services

The following services, which enforce quota limits, are automatically (required to be) used to create Cloud Composer environments.

  • Deployment Manager
  • Kubernetes Engine (which relies on Compute Engine)
  • Cloud Storage
  • Cloud Pub/Sub
  • Cloud Logging
  • Cloud Monitoring
  • Cloud Build (if using PyPi dependencies)
  • Google Container Registry
  • Google Artifact Registry
  • Identity and Access Management
  • Virtual Private Cloud

Deployment Manager

Deployment Manager manages other Google Cloud resources needed by Cloud Composer. Each gcloud composer create, update or delete call translates to one Deployment Manager write query and multiple read queries where the Deployment Manager quota applies.

Kubernetes Engine

Each Cloud Composer environment creates one GKE cluster that is used to run Airflow (e.g. Scheduler) and Composer System workloads (e.g. Composer Agent).

Kubernetes services uses Compute Engine Virtual Machines for GKE nodes. The cluster affects the Kubernetes Engine quota while the number of nodes affects the Compute Engine quota.

Cloud Pub/Sub

Cloud Pub/Sub is used for communication between Composer Service and the Airflow deployment we create. Each gcloud composer environments create or update call creates two Cloud Pub/Sub topics and two Cloud Pub/Sub subscriptions. The Cloud Pub/Sub quota applies.

Cloud Logging and Monitoring

Cloud Logging is used for Cloud Composer logs and streaming Airflow logs. The Logging quota applies.

Cloud Monitoring enables you to collect and visualize your Airflow metrics in a custom dashboard. The Monitoring quota applies.

Cloud Build

Cloud Build allows you to build custom docker images when installing PyPi packages. The Cloud Build quotas apply.

Identity and Access Management

Identity and Access Management is used to check permissions of Cloud Composer users and service accounts. For example, only users who are registered in IAM of your project and have proper permissions are able to access the Airflow UI. The IAM quotas apply.

You can read more about access and permission controls in Access Control.

Virtual Private Cloud

Cloud Composer relies on Google VPC networking service that's why VPC's quotas apply (e.g. quota for VPC peerings withing VPC network).

Optional services

You can use Airflow operators with Google Cloud services. Any service used in a Cloud Composer environment is subject to that service's quota.