Updating and deleting environments

This page explains how to update, delete, and view the details of a Cloud Composer environment. It also explains how to update and view an environment's Apache Airflow configurations.

Caution: Do not upgrade the GKE clusters for your Cloud Composer environments to newer versions. Use the GKE version pre-configured during environment creation and rely on auto-upgrades of the GKE service.

Before you begin

• You must have a role that can trigger environment update operations. In addition, the service account of the environment must have a role that has enough permissions to perform update operations. For more information, see Access control.

• Most gcloud composer commands require a location. You can specify the location by using the --location flag or by setting the default location.

• Some Airflow parameters are preconfigured for Cloud Composer, and you cannot change them.

• Some environment updates rebuild Cloud Composer images, such as package installations and version upgrades. Cloud Composer garbage collection can remove older images.

• Updates to Airflow configurations take effect on all Airflow instances (worker, server, scheduler) approximately 5 minutes after you submit the update request.

• You can update the machine type for the App Engine VM running the Airflow web server and for the Cloud SQL instance running the Airflow database. To do so, use gcloud composer, the Cloud Console, or the Composer API.

Limitations

Cloud Composer doesn't accept concurrent environment update or delete requests. You must wait for an environment update or delete request to complete before issuing another environment update or delete request.

Updating environments

To update Cloud Composer environments, including Airflow configurations:

Console

1. Open the Environments page in the Google Cloud.

   Open the Environments page

2. Click the environment Name to modify.

3. Select the tab corresponding to the property you want to update.

4. Click Edit.

5. After making updates, click Submit.

gcloud

To update Airflow configurations, run the gcloud composer environments update command with the --update-airflow-configs flag:

gcloud composer environments update ENVIRONMENT_NAME \
    --location LOCATION \
    --update-airflow-configs=KEY=VALUE,KEY=VALUE,... 

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.
• KEY=VALUE is the configuration section and the property name separated by a hyphen, such as core-print_stats_interval, and its corresponding value.

For example:

gcloud composer environments update test-environment \
    --location us-central1 \
    --update-airflow-configs=core-load_example=True,webserver-dag_orientation=TB 

See the 'gcloud composer environments update' reference page for additional examples.

API

To update a Cloud Composer environment, including Airflow configurations, using the Cloud Composer REST API, issue an environments.patch API request.

Terraform

Warning: If you attempt to change a configuration that cannot be updated, Terraform will delete your environment and create a new one with the new value you specify with that change. Run terraform plan before terraform apply to see if Terraform will replace your environment based on your configuration instead of updating it.

To update an environment using Terraform, modify Cloud Composer environment configurations in your Terraform configuration, run terraform plan to see the actions you have proposed, and if you are satisfied, run terraform apply.

Note: To utilize this resource block, the service account that Terraform uses must have a role with the composer.environments.create permission enabled. For more information about using Terraform to configure a Cloud Composer environment, refer to the Terraform documentation.

resource "google_composer_environment" "example-resource" {
    name   = "ENVIRONMENT_NAME"
    region = "LOCATION"
    software_config = {
      airflow_config_overrides = {
        KEY = "VALUE"
      }
    }
}

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the Compute Engine region where the environment is located.
• KEY and VALUE with the Airflow configuration property name and its corresponding value. For example, core-print_stats_interval, and its corresponding value.

Usage of additional optional parameters is defined in the Terraform Argument Reference

For example, to override an Airflow configuration option:

resource "google_composer_environment" "example-environment" {
    name   = "example-environment"
    region = "us-central1"
    software_config = {
      airflow_config_overrides = {
        core-load_example = "True"
      }
    }
}

Restoring the default Airflow configuration

To restore all default Airflow configuration values:

gcloud composer environments update ENVIRONMENT_NAME \
    --location LOCATION \
    --clear-airflow-configs

To restore one or more default Airflow configuration values:

gcloud composer environments update ENVIRONMENT_NAME \
    --location LOCATION \
    --remove-airflow-configs=core-load_example,webserver-dag_orientation

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.

For example:

gcloud composer environments update test-environment \
    --location us-central1 \
    --update-airflow-configs=core-load_example=True,webserver-dag_orientation=TB

The command terminates when the operation is finished. To avoid waiting, use the --async flag.

Viewing environment details

To view Cloud Composer environment details, including Airflow configurations:

Console

1. Open the Environments page in the Google Cloud Console.

   Open the Environments page

2. Click the Name of the environment.
3. Select the appropriate property tab.

gcloud

gcloud composer environments describe ENVIRONMENT_NAME \
    --location LOCATION
    --format="get(PROPERTY)"

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.
• --format is an option to specify an environment property, such as config.airflowUri for the Airflow web server URL.

For example:

gcloud composer environments describe test-environment 

  --location us-central1 --format="get(config.airflowUri)"


# Returns https://a6b3z6e3nnZZZnnd3a-tp.appspot.com 

API

To view Airflow configurations using the Cloud Composer REST API, make an environments.get API request, filling in the environment ID.

Terraform

To view details about your Cloud Composer environment, run

terraform state show google_composer_environment.TERRAFORM_RESOURCE_NAME

where:

• TERRAFORM_RESOURCE_NAME is the name of your Cloud Composer Terraform resource

Note: The name of your Terraform resource may be different than the name of your Cloud Composer environment.

Upgrading the machine type for GKE nodes

You can manually upgrade the machine type for your environment's GKE cluster by deleting the existing default-pool and creating a new default-pool with the desired machine type.

• We recommend you specify the ideal machine type for the type of computing that will occur in your Cloud Composer environment when you create an environment. If you are running jobs that perform heavyweight computation, first try the GKEPodOperator.
• Running workflows might be interrupted during upgrade. Before upgrade, consider pausing all DAGs and waiting for in-progress tasks to finish.
• The Environment details page for Cloud Composer does not reflect the machine type for a manual upgrade.

To upgrade the machine type:

1. In the Cloud Console, click the name of the environment that you want to upgrade.
2. To view the default node pool:
   1. On the Environment details page, click the view cluster details link.
   2. On the Clusters page in the Node Pools section, click default-pool.
   3. Note all the information for default-pool on the Node pool details page. You will use this information to create a new default node pool for your environment.
3. To delete default-pool:
   1. On the Node pool details page, click the back arrow to return to the Clusters page for your environment.
   2. In the Node Pools section, click the trash can to the right of default-pool.
   3. To confirm, click Delete.

4. To create the new default-pool:

   Note: Use the same service account that runs the Cloud Composer environment.
   1. At the top of the Clusters page, click Add node pool.
   2. For Name, enter default-pool. You must name the node pool default-pool or workflows will fail.
   3. Enter the Size and Nodes settings.
   4. (Only for default Compute Engine service accounts) For Access scopes, select Allow full access to all Cloud APIs.
   5. Click Save.

5. If you notice workloads are distributed unevenly, scale down the airflow-worker deployment to zero and scale up again.

Deleting an environment

To delete a Cloud Composer environment:

Console

1. Open the Environments page in the Google Cloud Console.

   Open the Environments page

2. Check the box next to the environment to delete.
3. Click Delete.

gcloud

Run the following command to delete a Cloud Composer environment:

gcloud composer environments delete ENVIRONMENT_NAME \
    --location LOCATION   

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.

Run gcloud composer environments describe --help for command parameter information.

API

To delete an environment using the Cloud Composer REST API, make an environments.delete API request, filling in the environment ID.

Terraform

To delete an environment using Terraform, remove the Cloud Composer configuration from your overall resource configuration and run terraform apply.

Deleting your environment does not delete the following data in your customer project:

• Cloud Storage bucket for your environment
• Google Cloud's operations suite logs
• 2 GB persistent disk used by the Redis queue

To avoid incurring charges to your Google Cloud account, export your data and delete the Cloud Storage bucket, as needed.

Note: Do not delete the GKE cluster of your environment manually. This happens automatically during the environment deletion. If you delete the cluster before the environment, then attempting to delete your environment results in an error. To fix this problem, see Troubleshooting.

What's next

• Installing Python dependencies