Cloud Composer 1 | Cloud Composer 2
This page describes how to scale Cloud Composer environments in Cloud Composer 1.
For information about how environment scaling works, see Environment scaling.
Scale vertically and horizontally
In Cloud Composer 1, you don't define specific CPU and memory resources for Cloud Composer and Airflow components such as workers and schedulers. Instead, you specify the number and type of machines for nodes in your environment's cluster.
Options for horizontal scaling:
Options for vertical scaling:
Adjust the number of nodes
You can change the number of nodes in your environment.
This number corresponds to the number of Airflow workers in your environment. In addition to running Airflow workers, your environment nodes also run Airflow schedulers and other environment components.
Console
Go to the Environments page in the Google Cloud console:
Select your environment.
Go to the Environment configuration tab.
In the Worker nodes > Node count item, click Edit.
In the Worker nodes configuration dialog, in the Node count field, specify the number of nodes in your environment.
Click Save.
gcloud
The --node-count
argument controls the number of nodes in your environment:
gcloud composer environments update ENVIRONMENT_NAME \
--location LOCATION \
--zone NODE_ZONE \
--node-count NODE_COUNT
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.NODE_COUNT
with the number of nodes. The minimum number of nodes is3
.NODE_ZONE
with the Compute Engine zone for your environment VMs.
Example:
gcloud composer environments update example-environment \
--location us-central1 \
--zone us-central1-a \
--node-count 6
API
Create an
environments.patch
API request.In this request:
In the
updateMask
parameter, specify theconfig.nodeCount
mask.In the request body, specify the number of nodes for your environment.
"config": {
"nodeCount": NODE_COUNT
}
Replace:
NODE_COUNT
with the number of nodes. The minimum number of nodes is3
.
Example:
// PATCH https://composer.googleapis.com/v1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.nodeCount
"config": {
"nodeCount": 6
}
Terraform
The node_count
field in the node_config
block specifies the number of nodes in your environment.
resource "google_composer_environment" "example" {
config {
node_config {
node_count = NODE_COUNT
}
}
Replace:
NODE_COUNT
with the number of nodes.
Example:
resource "google_composer_environment" "example" {
name = "example-environment"
region = "us-central1"
config {
node_config {
node_count = 4
}
}
Adjust the number of schedulers
Your environment can run more than one Airflow scheduler at the same time. Use multiple schedulers to distribute load between several scheduler instances for better performance and reliability.
You can specify a number of schedulers up to the number of nodes in your environment.Increasing the number of schedulers does not always improve Airflow performance. For example, having only one scheduler might provide better performance than having two. This might happen when the extra scheduler is not utilized, and thus consumes resources of your environment without contributing to overall performance. The actual scheduler performance depends on the number of Airflow workers, the number of DAGs and tasks that run in your environment, and the configuration of both Airflow and the environment.
We recommend starting with two schedulers and then monitoring the performance of your environment. If you change the number of schedulers, you can always scale your environment back to the original number of schedulers.
For more information about configuring multiple schedulers, see Airflow documentation.
Console
Go to the Environments page in the Google Cloud console:
Select your environment.
Go to the Environment configuration tab.
In the Resources > Number of schedulers item, click Edit.
In the Scheduler configuration dialog, in the Number of schedulers field, specify the number of schedulers for your environment.
Click Save.
gcloud
Run the following Google Cloud CLI command:
gcloud composer environments update ENVIRONMENT_NAME \
--location LOCATION \
--scheduler-count SCHEDULER_COUNT
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.SCHEDULER_COUNT
with the number of schedulers.
Example:
gcloud composer environments update example-environment \
--location us-central1 \
--scheduler-count 2
API
Create an
environments.patch
API request.In this request:
In the
updateMask
parameter, specify theconfig.softwareConfig.schedulerCount
mask.In the request body, specify the number of nodes for your environment.
{
"config": {
"softwareConfig": {
"schedulerCount": SCHEDULER_COUNT
}
}
Replace:
SCHEDULER_COUNT
with the number of schedulers.
Example:
// PATCH https://composer.googleapis.com/v1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.softwareConfig.schedulerCount
{
"config": {
"softwareConfig": {
"schedulerCount": 2
}
}
Terraform
The scheduler_count
field in the software_config
block specifies the
number of schedulers in your environment.
This field is available only in Cloud Composer 1 environments that use Airflow 2.
resource "google_composer_environment" "example" {
config {
software_config {
scheduler_count = SCHEDULER_COUNT
}
}
}
Replace:
SCHEDULER_COUNT
with the number of schedulers.
Example:
resource "google_composer_environment" "example" {
name = "example-environment"
region = "us-central1"
config {
software_config {
scheduler_count = 2
}
}
}
Adjust the machine type of the Cloud SQL instance
You can change the machine type of the Cloud SQL instance that stores the Airflow database of your environment.
Console
Go to the Environments page in the Google Cloud console:
Select your environment.
Go to the Environment configuration tab.
In the Resources > Cloud SQL machine type item, click Edit.
In the Cloud SQL configuration dialog, in the Cloud SQL machine type drop-down list, select the machine type for the Cloud SQL instance of your environment.
Click Save.
gcloud
The --cloud-sql-machine-type
arguments controls the machine type of
the Cloud SQL instance in your environment.
Run the following Google Cloud CLI command:
gcloud composer environments update ENVIRONMENT_NAME \
--location LOCATION \
--cloud-sql-machine-type SQL_MACHINE_TYPE
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.SQL_MACHINE_TYPE
with the machine type for the Cloud SQL instance.
Example:
gcloud composer environments update example-environment \
--location us-central1 \
--cloud-sql-machine-type db-n1-standard-2
API
Create an
environments.patch
API request.In this request:
In the
updateMask
parameter, specify theconfig.databaseConfig.machineType
mask.In the request body, specify the machine type for the Cloud SQL instance.
{
"config": {
"databaseConfig": {
"machineType": "SQL_MACHINE_TYPE"
}
}
}
Replace:
SQL_MACHINE_TYPE
with the machine type for the Cloud SQL instance.
Example:
// PATCH https://composer.googleapis.com/v1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.databaseConfig.machineType
{
"config": {
"databaseConfig": {
"machineType": "db-n1-standard-2"
}
}
}
Terraform
The machine_type
field in the database_config
block specifies the
machine type for the Cloud SQL instance.
resource "google_composer_environment" "example" {
config {
database_config {
machine_type = "SQL_MACHINE_TYPE"
}
}
}
Replace:
SQL_MACHINE_TYPE
with the machine type for the Cloud SQL instance.
Example:
resource "google_composer_environment" "example" {
name = "example-environment"
region = "us-central1"
config {
database_config {
machine_type = "db-n1-standard-2"
}
}
Adjust the web server machine type
You can change the machine type for the Airflow web server of your environment.
Console
Go to the Environments page in the Google Cloud console:
Select your environment.
Go to the Environment configuration tab.
In the Resources > Web server machine type item, click Edit.
In the Web server configuration dialog, in the Web server machine type drop-down list, select the machine type for the Airflow web server.
Click Save.
gcloud
The --web-server-machine-type
arguments controls the machine type of
the Airflow web server instance in your environment.
Run the following Google Cloud CLI command:
gcloud composer environments update ENVIRONMENT_NAME \
--location LOCATION \
--web-server-machine-type WS_MACHINE_TYPE
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.WS_MACHINE_TYPE
with the machine type for the Airflow web server instance.
Example:
gcloud composer environments update example-environment \
--location us-central1 \
--web-server-machine-type composer-n1-webserver-2
API
Create an
environments.patch
API request.In this request:
In the
updateMask
parameter, specify theconfig.webServerConfig.machineType
mask.In the request body, specify the machine type for the web server.
{
"config": {
"webServerConfig": {
"machineType": "WS_MACHINE_TYPE"
}
}
}
Replace:
WS_MACHINE_TYPE
with the machine type for the Airflow web server instance.
Example:
// PATCH https://composer.googleapis.com/v1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.webServerConfig.machineType
{
"config": {
"webServerConfig": {
"machineType": "composer-n1-webserver-2"
}
}
}
Terraform
The machine_type
field in the web_server_config
block specifies the machine type for the Airflow web server instance.
resource "google_composer_environment" "example" {
config {
web_server_config {
machine_type = "WS_MACHINE_TYPE"
}
}
}
Replace:
WS_MACHINE_TYPE
with the machine type for the Airflow web server instance.
Example:
resource "google_composer_environment" "example" {
name = "example-environment"
region = "us-central1"
config {
web_server_config {
machine_type = "composer-n1-webserver-2"
}
}
What's next
- Environment scaling and performance
- Cloud Composer pricing
- Update environments
- Environment architecture