Cloud Composer Resource Quotas

Cloud Composer has the following API quota limits.

Quota metric Limit
API calls per project 1000 requests/min
API calls per user per project 300 requests/min

The following tools are built on top of the Cloud Composer API, and use your Cloud Composer API quota:

Cloud Composer utilizes other Google Cloud Platform products. These products have project-level quotas, which include quotas that apply to Cloud Composer use. Some services are required to use Cloud Composer, such as Google Compute Engine and Google Cloud Storage. Other services can optionally be used with Cloud Composer when setting connectors in Airflow.

Required services

The following services, which enforce quota limits, are automatically (required to be) used to create Cloud Composer environments.

  • Deployment Manager
  • Kubernetes Engine (which relies on Compute Engine)
  • Cloud Storage
  • Cloud Pub/Sub
  • Stackdriver Logging
  • Cloud Build (if using PyPi dependencies)

Deployment Manager

Deployment Manager manages other Google Cloud Platform resources needed by Cloud Composer. Each gcloud composer create, update or delete call translates to one Deployment Manager write query and multiple read queries where the Deployment Manager quota applies.

Kubernetes Engine

Airflow scheduler and workers run inside a Kubernetes Engine cluster. Each Cloud Composer environment creates one cluster. An environment by default has three Compute Engine VMs, though this number is configurable during or after creation. The cluster effects the Kubernetes Engine quota while the number of nodes effects the Compute Engine quota.

Cloud Pub/Sub

Cloud Pub/Sub is used for communication between Composer Service and the Airflow deployment we create. Each gcloud composer environments create or update call creates two Cloud Pub/Sub topics and two Cloud Pub/Sub subscriptions. The Cloud Pub/Sub quota applies.

Stackdriver Logging

Stackdriver Logging is used for Cloud Composer logs and streaming Airflow logs. The Logging quota applies.

Cloud Build

Cloud Build allows you to build custom docker images when installing PyPi packages. The Cloud Build quotas applies.

Optional services

You can use Airflow operators with Google Cloud Platform services. Any service used in a Cloud Composer environment is subject to that service's quota.

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Composer