This page explains how to update, delete, and view the details of a Cloud Composer environment. It also explains how to update and view an environment's Apache Airflow configurations.
Before you begin
The following permissions are required to update or delete environments:
composer.environments.update
to update environmentscomposer.environments.delete
to delete environmentsFor more information, see Cloud Composer Access Control.
Most
gcloud composer
commands require a location. You can specify the location by using the--location
flag or by setting the default location.Some Airflow parameters are preconfigured for Cloud Composer, and you cannot change them. You can update the following Cloud Composer environment configurations: node count, Airflow config overrides, environment variables, labels, and PyPI packages.
Some environment updates rebuild Cloud Composer images, such as package installations and version upgrades. Cloud Composer garbage collection can remove older images.
Updates to Airflow configurations take effect on all Airflow instances (worker, server, scheduler) approximately 5 minutes after you submit the update request.
You can update the machine type for the App Engine VM running the Airflow web server and for the Cloud SQL instance running the Airflow database. To do so, use
gcloud composer
, the Cloud Console, or the Composer API.
Limitations
Cloud Composer doesn't accept concurrent environment update or delete requests. You must wait for an environment update or delete request to complete before issuing another environment update or delete request.
Updating environments
To update Cloud Composer environments, including Airflow configurations:
Console
Open the Environments page in the Google Cloud.
Click the environment Name to modify.
Select the tab corresponding to the property you want to update.
Click Edit.
After making updates, click Submit.
gcloud
To update Airflow configurations, run the gcloud composer environments update
command with the --update-airflow-configs
flag:
gcloud composer environments update ENVIRONMENT_NAME \ --location LOCATION \ --update-airflow-configs=KEY=VALUE,KEY=VALUE,...
where:
ENVIRONMENT_NAME
is the name of the environment.LOCATION
is the Compute Engine region where the environment is located.KEY=VALUE
is the configuration section and the property name separated by a hyphen, such ascore-print_stats_interval
, and its corresponding value.
For example:
gcloud composer environments update test-environment \ --location us-central1 \ --update-airflow-configs=core-load_example=True,webserver-dag_orientation=TB
See the 'gcloud composer environments update' reference page for additional examples.
API
To update a Cloud Composer environment, including Airflow
configurations, using the
Cloud Composer REST API,
issue an environments.patch
API request.
Restoring the default Airflow configuration
To restore all default Airflow configuration values:
gcloud composer environments update ENVIRONMENT_NAME \ --location LOCATION \ --clear-airflow-configs
To restore one or more default Airflow configuration values:
gcloud composer environments update ENVIRONMENT_NAME \ --location LOCATION \ --remove-airflow-configs=core-load_example,webserver-dag_orientation
where:
ENVIRONMENT_NAME
is the name of the environment.LOCATION
is the Compute Engine region where the environment is located.
For example:
gcloud composer environments update test-environment \ --location us-central1 \ --update-airflow-configs=core-load_example=True,webserver-dag_orientation=TB
The command terminates when the operation is finished. To avoid waiting,
use the --async
flag.
Viewing environment details
To view Cloud Composer environment details, including Airflow configurations:
Console
- Open the Environments page in the Google Cloud Console.
- Click the Name of the environment.
- Select the appropriate property tab.
gcloud
gcloud composer environments describe ENVIRONMENT_NAME \ --location LOCATION --format="get(PROPERTY)"
where:
ENVIRONMENT_NAME
is the name of the environment.LOCATION
is the Compute Engine region where the environment is located.--format
is an option to specify an environment property, such asconfig.airflowUri
for the Airflow web server URL.
For example:
gcloud composer environments describe test-environment
--location us-central1 --format="get(config.airflowUri)"# Returns https://a6b3z6e3nnZZZnnd3a-tp.appspot.com
API
To view Airflow configurations using
the Cloud Composer REST API,
make an environments.get
API request, filling in the environment ID.
Upgrading the machine type for GKE nodes
You can manually upgrade the machine type for your environment's GKE
cluster by deleting the existing default-pool
and creating a new default-pool
with the desired machine type.
To upgrade the machine type:
- In the Cloud Console, click the name of the environment that you want to upgrade.
- To view the default node pool:
- On the Environment details page, click the view cluster details link.
- On the Clusters page in the Node Pools section, click default-pool.
- Note all the information for default-pool on the Node pool details page. You will use this information to create a new default node pool for your environment.
- To delete default-pool:
- On the Node pool details page, click the back arrow to return to the Clusters page for your environment.
- In the Node Pools section, click the trash can to the right of default-pool.
- To confirm, click Delete.
To create the new default-pool:
- At the top of the Clusters page, click Add node pool.
- For Name, enter default-pool. You must name the node pool default-pool or workflows will fail.
- Enter the Size and Nodes settings.
- For Access scopes, select Allow full access to all Cloud APIs.
- Click Save.
If you notice workloads are distributed unevenly, scale down the airflow-worker deployment to zero and scale up again.
Deleting an environment
To delete a Cloud Composer environment:
Console
- Open the Environments page in the Google Cloud Console.
- Check the box next to the environment to delete.
- Click Delete.
gcloud
Run the following command to delete a Cloud Composer environment:
gcloud composer environments delete ENVIRONMENT_NAME \ --location LOCATION
where:
ENVIRONMENT_NAME
is the name of the environment.LOCATION
is the Compute Engine region where the environment is located.
Run gcloud composer environments describe --help
for
command parameter information.
API
To delete an environment using the Cloud Composer REST API,
make an environments.delete
API request, filling in the environment ID.