About environment scaling

Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3

Note: This page is not yet revised for Cloud Composer 3 and displays content for Cloud Composer 2.

This page describes how environment scaling works in Cloud Composer 2.

Other pages about scaling:

• For a guide about selecting optimal scale and performance parameters for your environment, see Optimize environment performance and costs.
• For information about scaling your environments, see Scale environments.

Autoscaling environments

Cloud Composer 2 environments automatically scale in response to the demands of your executed DAGs and tasks:

• If your environment experiences a heavy load, Cloud Composer automatically increases the number of workers in your environment.
• If your environment does not use some of its workers, these workers are removed to save environment resources and costs.
• You can set the minimum and maximum number of workers for your environment. Cloud Composer automatically scales your environment within the set limits. You can adjust these limits at any time.

The number of workers is adjusted based on the Scaling Factor Target metric. This metric is calculated based on:

• Current number of workers
• Number of Celery tasks in the Celery queue, that are not assigned to a worker
• Number of idle workers
• celery.worker_concurrency Airflow configuration option

Cloud Composer autoscaling uses three different autoscalers provided by GKE:

• Horizontal Pod Autoscaler (HPA)
• Cluster Autoscaler (CA)
• Node auto-provisioning (NAP)

Cloud Composer configures these autoscalers in the environment's cluster. This automatically scales the number of nodes in the cluster, the machine type and the number of workers.

Scale and performance parameters

In addition to autoscaling, you can control the scale and performance parameters of your environment by adjusting the CPU, memory, and disk limits for schedulers, web server, and workers. By doing so you can scale your environment vertically, in addition to the horizontal scaling provided by the autoscaling feature. You can adjust the scale and performance parameters of Airflow schedulers, web server, and workers at any time.

The environment size performance parameter of your environment controls the performance parameters of the managed Cloud Composer infrastructure that includes the Airflow database. Consider selecting a larger environment size if you want to run a large number of DAGs and tasks with higher infrastructure performance. For example, larger environment's size increases the amount of Airflow task log entries that your environment can process with minimal delay.

Note: Environment size is different from the environment presets. Environment presets, which you can select when you create an environment, determine all limits, scale, and performance parameters of your environment, including the environment size. Environment size determines only the performance parameters of the managed Cloud Composer infrastructure of your environment.

Multiple schedulers

Airflow 2 can use more than one Airflow scheduler at the same time. This Airflow feature is also known as the HA scheduler. In Cloud Composer 2, you can set the number of schedulers for your environment and adjust it at any time. Cloud Composer does not automatically scale the number of schedulers in your environment.

For more information about configuring the number of schedulers for your environment, see Scale environments.

Database disk space

Disk space for the Airflow database automatically increases to accommodate the demand.

What's next

• Scale environments
• Cloud Composer pricing
• Create environments
• Environment architecture