Updating and deleting environments

This page explains how to update, delete, and view the details of a Cloud Composer environment. It also explains how to update and view an environment's Apache Airflow configurations.

Before you begin

• The following permissions are required to update or delete environments:

  • composer.environments.update to update environments

  • composer.environments.delete to delete environments

    For more information, see Cloud Composer Access Control.

• Most gcloud composer commands require a location. You can specify the location by using the --location flag or by setting the default location.

• Some Airflow parameters are preconfigured for Cloud Composer, and you cannot change them. You can update the following Cloud Composer environment configurations: node count, Airflow config overrides, environment variables, labels, and PyPI packages.

• Some environment updates rebuild Cloud Composer images, such as package installations and version upgrades. Cloud Composer garbage collection can remove older images.

• Updates to Airflow configurations take effect on all Airflow instances (worker, server, scheduler) approximately 5 minutes after you submit the update request.

Limitations

Cloud Composer doesn't accept concurrent environment update or delete requests. You must wait for an environment update or delete request to complete before issuing another environment update or delete request.

Updating environments

To update Cloud Composer environments, including Airflow configurations:

Console

1. Open the Environments page in the Google Cloud Platform.

   Open the Environments page

2. Click the environment Name to modify.

3. Select the tab corresponding to the property you want to update.

4. Click Edit.

5. After making updates, click Submit.

gcloud

To update Airflow configurations, run the gcloud composer environments update command with the --update-airflow-configs flag:

gcloud composer environments update ENVIRONMENT_NAME \
    --location LOCATION \
    --update-airflow-configs=KEY=VALUE,KEY=VALUE,... 

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.
• KEY=VALUE is the configuration section and the property name separated by a hyphen, such as core-print_stats_interval, and its corresponding value.

For example:

gcloud composer environments update test-environment \
    --location us-central1 \
    --update-airflow-configs=core-load_example=True,webserver-dag_orientation=TB 

See the 'gcloud composer environments update' reference page for additional examples.

API

To update a Cloud Composer environment, including Airflow configurations, using the Cloud Composer REST API, issue an environments.patch API request.

Restoring the default Airflow configuration

To restore all default Airflow configuration values:

gcloud composer environments update ENVIRONMENT_NAME \
    --location LOCATION \
    --clear-airflow-configs

To restore one or more default Airflow configuration values:

gcloud composer environments update ENVIRONMENT_NAME \
    --location LOCATION \
    --remove-airflow-configs=core-load_example,webserver-dag_orientation 

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.

For example:

gcloud composer environments update test-environment \
    --location us-central1 \
    --update-airflow-configs=core-load_example=True,webserver-dag_orientation=TB 

The command terminates when the operation is finished. To avoid waiting, use the --async flag.

Viewing environment details

To view Cloud Composer environment details, including Airflow configurations:

Console

1. Open the Environments page in the GCP Console.

   Open the Environments page

2. Click the Name of the environment.
3. Select the appropriate property tab.

gcloud

gcloud composer environments describe ENVIRONMENT_NAME \
    --location LOCATION
    --format="get(PROPERTY)"

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.
• --format is an option to specify an environment property, such as config.airflowUri for the Airflow web server URL.

For example:

gcloud composer environments describe test-environment 

  --location us-central1 --format="get(config.airflowUri)"


# Returns https://a6b3z6e3nnZZZnnd3a-tp.appspot.com 

API

To view Airflow configurations using the Cloud Composer REST API, make an environments.get API request, filling in the environment ID.

Upgrading the machine type

You can manually upgrade the machine type for your environment's GKE cluster by creating a new managed instance group template and updating the instances in your environment's GKE cluster.

Note: If you are running jobs that perform heavyweight computation, consider trying the GKEPodOperator first.

To upgrade the machine type:

1. In the GCP Console, click the name of the environment that you want to upgrade.
2. To find the current instance group template:
   1. On the Environment details page, click the GKE cluster ID link.
   2. On the Clusters page in the Node Pools section, make a note of the Instance groups name and then click its link.
   3. On the Instance Groups page under Members, click the Template link. The Instance templates page displays.

      

3. To create a new instance group template:
   1. On the Instance templates page, click Create similar.
   2. On the Create an instance template page, specify a name for the new template.
   3. Under Machine type, choose the machine type and click Create. The new template shows in the list on the Instance templates page.
4. To update the instances in your environment's cluster:
   1. On the Compute Engine > Instance groups page, click the name of the instance group you noted earlier.
   2. Click Rolling update.
   3. Under Template, select the new template you created.
   4. Click Update.

Deleting an environment

To delete a Cloud Composer environment:

Console

1. Open the Environments page in the GCP Console.

   Open the Environments page

2. Check the box next to the environment to delete.
3. Click Delete.

gcloud

Run the following command to delete a Cloud Composer environment:

gcloud composer environments delete ENVIRONMENT_NAME \
    --location LOCATION   

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.

Run gcloud composer environments describe --help for command parameter information.

API

To delete an environment using the Cloud Composer REST API, make an environments.delete API request, filling in the environment ID.

Note: Deleting your environment does not delete the following data in your customer project: the Cloud Storage bucket for your environment, Stackdriver logs, and Cloud Pub/Sub topics. To avoid incurring changes to your GCP account, export your data and delete the Cloud Storage bucket, as needed.

What's next

• Installing Python Dependencies