Upgrade environments

Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3

Preview — Cloud Composer 3

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This page describes how to upgrade your environment to a new Airflow version.

About upgrade operations

In Cloud Composer 3, you don't manage the Cloud Composer version of your environment:

• Cloud Composer automatically upgrades the infrastructure components of your environment. These components are related to Cloud Composer functionality and do not change how Airflow works or how your Airflow DAGs are executed.
• Cloud Composer doesn't automatically upgrade Airflow version and build, Airflow components, or components closely related to Airflow workloads. They are not changed when the infrastructure components upgrade automatically.
• You can manually upgrade to a new Airflow version or build.

For example, you can use the same version and build of Airflow for several months, without performing any upgrades, and your environment still receives latest Cloud Composer updates, fixes, and improvements of the environment's infrastructure components. When later you decide to move to a new version or build of Airflow, you upgrade the Airflow version in your environment.

About Airflow version upgrade

Your environment changes the version or build of Airflow in the following way:

• You control the Airflow version (and build) of your environment. You can perform the Airflow version upgrade operation when you choose to switch your environment to a different version of Airflow. For example, this might happen if the current version and build of Airflow are no longer supported.

• Cloud Composer re-deploys your environment's Airflow components using the specified Airflow version and build.

• Cloud Composer applies Airflow configuration changes, such as custom PyPI packages or Airflow configuration option overrides, if your environment had them before the upgrade.

• Cloud Composer updates the Airflow airflow_db connection to point to the new Cloud SQL database.

Changing the Airflow version doesn't change how you connect to the resources in your environment, such as the URL of your environment's bucket, or Airflow web server.

About automatic infrastructure upgrade operations

Cloud Composer periodically runs automatic infrastructure upgrade operations:

• Automatic infrastructure upgrade operations run periodically during maintenance windows specified for the environment.

• It is not possible to disable automatic infrastructure upgrades in Cloud Composer 3. You can control the time periods when automatic upgrades can run by specifying custom maintenance windows for your environment.

• This operation is visible in Google Cloud console and environment's logs, like any other long-running operation. While the operation is running, you can't start other operations at the environment level (but you can still run Airflow DAGs).

• In some cases, automatic infrastructure upgrades can restart Airflow components. During such restarts, Airflow workers are gracefully terminated with a grace period of 24 hours. If you have tasks that need more than 24 hours to complete, consider using deferrable operators. An upgrade might result in a short period of unavailability for environment's infrastructure components, such as environment monitoring.

Limitations of upgrade operations

Upgrade operations have the following limitations:

• You can't downgrade to an earlier version or build of Airflow.

• You can't upgrade your environment if the Airflow database contains more than 16 GB of data. During an upgrade, a warning is shown if the Airflow database size is more than 16 GB. In this case, perform the database maintenance to reduce the database size.

• If you use the XCom mechanism to transfer files, make sure that you use it according to Airflow's guidelines. Transferring big files or a large number of files using XCom impacts Airflow database's performance and can lead to failures when loading snapshots or upgrading your environment. Consider using alternatives such as Cloud Storage to transfer large volumes of data.

Before you begin

Caution:

• Make sure that you know the differences between the current version of Airflow and the version that you are upgrading to. The new version can include different versions of preinstalled packages, patches from Cloud Composer, or changes in Airflow functionality, provided by the Airflow open source project. You can check the versions of provider packages that are included in a specific version.

• Make sure that your DAGs are compatible with the new Airflow version and new versions of all provider packages (including the Airflow provider package for Google Cloud and Google services). Incompatible changes can cause your DAGs to no longer work.

• Unlike automatic infrastructure upgrades, when an environment upgrades to a new Airflow version or build, then all running tasks are terminated. After an upgrade is completed, Airflow schedules these tasks for a retry, depending on the way you configure retries for your DAGs.

• We recommend to create a new snapshot of the environment to be able to re-create the environment in case this is needed.

• Your account must have a role that can trigger environment upgrade operations. In addition, the service account of the environment must have a role that has enough permissions to perform upgrade operations. For more information, see Access control.

Check that your environment is up to date

Cloud Composer displays warnings when your environment's Airflow build approaches its end of support date. You can use these warnings to always keep your environment supported.

Figure 1. A deprecation message is displayed on the Environment details page

Cloud Composer keeps track of the Airflow version and build that your environment is based on. When it approaches the end of support date, you can see a warning in the list of environments and on the Environment details page.

Note: You can always upgrade your environment, even if the environment's Airflow version and build are past their support period.

To check if your environment is up to date:

Console

1. In Google Cloud console, go to the Environments page.

   Go to Environments

2. In the list of environments, click the name of your environment. The Environment details page opens.

3. Go to the Environment configuration tab.

4. In the Image version field, one of the following messages is displayed:

   • Newest available version. Your environment image is fully supported.

   • New version available. Your environment image is fully supported and you can upgrade it to a later version.

   • Support for this image version ends in... Your environment image approaches the end of the full support period.

   • This version is not supported as of... Your environment is past the full support period.

gcloud

This functionality is not available through Google Cloud CLI. You can View available upgrades instead, which shows new versions that are available.

API

This functionality is not available through API. You can View available upgrades instead, which shows new versions that are available.

View available upgrades

To view Cloud Composer versions that you can upgrade to:

Console

1. In Google Cloud console, go to the Environments page.

   Go to Environments

2. In the list of environments, click the name of your environment. The Environment details page opens.

3. Go to the Environment configuration tab and click Upgrade Image Version.

4. For the list of available versions, click the Cloud Composer Image version drop-down menu.

gcloud

gcloud composer environments list-upgrades \
  ENVIRONMENT_NAME \
  --location LOCATION

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the region where the environment is located.

Example:

gcloud composer environments list-upgrades example-environment \
  --location us-central1

API

You can view available versions for a location. To do so, construct an imageVersions.list API request.

For example:

// GET https://composer.googleapis.com/v1/projects/example-project/
// locations/us-central1/imageVersions

Check for PyPI package conflicts before upgrading

Preview — Upgrade check

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

You can check if PyPI packages installed in your environment have any conflicts with preinstalled packages in the new Airflow version or build.

A successful check means that there are no conflicts in PyPI package dependencies between the current and the specified version. However, an upgrade operation might still not be successful because of other reasons.

Console

To run an upgrade check for your environment:

1. In Google Cloud console, go to the Environments page.

   Go to Environments

2. In the list of environments, click the name of your environment. The Environment details page opens.

3. Go to the Environment configuration tab, locate the Image version entry, and click Upgrade.

4. In the Environment version upgrade dialog, in the New version drop-down list, select an Airflow version or build that you want to upgrade to.

5. In the PyPI packages compatibility section, click Check for conflicts.

6. Wait until the check is complete. If there are PyPI package dependency conflicts, then the displayed error messages contain details about conflicting packages and package versions.

gcloud

To run an upgrade check for your environment, run the environments check-upgrade command with the Airflow version or build. that you want to upgrade to.

gcloud composer environments check-upgrade \
  ENVIRONMENT_NAME \
  --location LOCATION \
  --airflow-version VERSION

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the region where the environment is located.
• VERSION with the new Airflow version and build that you want to upgrade to, in the airflow-x.y.z-build.t format. You can also use all version aliases.

Example:

gcloud composer environments check-upgrade example-environment \
  --location us-central1 \
  --airflow-version airflow-2.9.3-build.5

Example output:

Waiting for [projects/example-project/locations/us-central1/environments/
example-environment] to be checked for PyPI package conflicts when upgrading
to composer-3-airflow-2.9.3-build.5. Operation [projects/example-project/locations/
us-central1/operations/04d0e8b2-...]...done.
...

Response:
'@type': type.googleapis.com/
google.cloud.orchestration.airflow.service.v1.CheckUpgradeResponse
buildLogUri: https://console.cloud.google.com/cloud-build/builds/79738aa7-...
containsPypiModulesConflict: CONFLICT
pypiConflictBuildLogExtract: |-
The Cloud Build image build failed: Build failed; check build logs for
details. Full log can be found at https://console.cloud.google.com/
cloud-build/builds/79738aa7-...
Error details: tensorboard 2.2.2 has requirement
setuptools>=41.0.0, but you have setuptools 40.3.0.

As an alternative, you can run an upgrade check asynchronously. Use the --async argument to make an asynchronous call, then check the result with the gcloud composer operations describe command.

API

Construct an environments.checkUpgrade API request.

Specify the image version in the imageVersion field:

{
  "imageVersion": "VERSION"
}

Replace VERSION with the new version that you want to upgrade to, in the composer-3-airflow-x.y.z-build.t format.

Upgrade your environment

Caution: Pause all DAGs and wait for in-progress tasks to finish before upgrading.

To upgrade your environment to a new version or build of Airflow:

Console

1. In Google Cloud console, go to the Environments page.

   Go to Environments

2. In the list of environments, click the name of your environment. The Environment details page opens.

3. Go to the Environment configuration tab.

4. Locate the Image version item and click Upgrade.

5. From the Image version drop-down menu, select an Airflow version or build that you want to upgrade to.

6. Click Upgrade.

gcloud

gcloud composer environments update \
  ENVIRONMENT_NAME \
  --location LOCATION \
  --airflow-version VERSION

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the region where the environment is located.
• VERSION with the new Airflow version and build that you want to upgrade to, in the airflow-x.y.z-build.t format. You can also use all version aliases.

For example:

gcloud composer environments update
  example-environment \
  --location us-central1 \
  --airflow-version airflow-2.9.3-build.5

API

1. Construct an environments.patch API request.

2. In this request:

   1. In the updateMask parameter, specify the config.softwareConfig.imageVersion mask.

   2. In the request body, in the imageVersion field, specify a new version that you want to upgrade to.

For example:

// PATCH https://composer.googleapis.com/v1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.softwareConfig.imageVersion

  {
    "config": {
      "softwareConfig": {
        "imageVersion": "composer-3-airflow-2.9.3-build.5"
      }
    }
  }

Terraform

The image_version field in the config.software_config block controls the Airflow version and build of your environment. In this field, specify a new Airflow version and build.

  resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {
    software_config {
      image_version = "VERSION"
    }
  }
}

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the region where the environment is located. the new Airflow version and build that you want to upgrade to, in the airflow-x.y.z-build.t format. You can also use all version aliases.

Example:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {
    software_config {
      image_version = "airflow-2.9.3-build.5"
    }
  }
}

What's next

• List of versions
• Cloud Composer versioning
• Update environments