Upgrading environments

Do not upgrade the GKE clusters for your Cloud Composer environments to newer versions unless stated otherwise in the Cloud Composer release notes or the Known Issues page. Use the GKE version pre-configured during environment creation and rely on auto-upgrades of the GKE service.

Preview

This feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

This page describes how to upgrade the Airflow version or Cloud Composer version that your environment runs.

During upgrade, Cloud Composer:

• Re-deploys the Airflow scheduler and worker pods in a new Kubernetes namespace. After the upgrade is completed, Airflow will use a new Cloud SQL database; the database name matches the Kubernetes namespace. The DAG run history is preserved.

  Do not deploy workloads with KubernetesPodOperator into the new Kubernetes namespace.

• Updates the Airflow airflow_db connection to point to the new Cloud SQL database.

These changes affect how you access pods and connect to the Cloud SQL database.

• To access pods in the GKE cluster after upgrade, you need to use namespace-aware kubectl commands. For example, to list pods in the cluster, use kubectl get pods -A. To execute a command on a pod, use kubectl exec -n <NAMESPACE> ....
• If you use Airflow connections and workloads that reference the SQL proxy directly, use the default namespace as part of the hostname: airflow-sqlproxy-service.default, not airflow-sqlproxy-service.

Upgrading Cloud Composer, does not change how you connect to the resources in your environment, such as Google Kubernetes Engine node VM IP addresses, Cloud SQL instance IP address, Cloud Storage bucket, or Airflow webserver domain name.

Before you begin

• Upgrading environments is currently in Preview. Use this feature with caution in production environments.

• You must have a role that can trigger environment upgrade operations. In addition, the service account of the environment must have a role that has enough permissions to perform upgrade operations. For more information, see Access control.

• Pause all DAGs and wait for in-progress tasks to finish before upgrading.

• You can upgrade the Cloud Composer, the Airflow version, or both at the same time.

• The Cloud Composer-Airflow combination that you are upgrading to must be a released version.

  • For available upgrades, see Viewing available upgrades. To get the latest features and fixes, consider upgrading to the most recent Cloud Composer release.
  • For the list of PyPI packages and customizations in a supported version, see the Cloud Composer version list.

    Before you upgrade, ensure that you know the differences between the current versions of Airflow and Cloud Composer and the versions that you're upgrading to. Incompatible changes can cause DAGs to break.

Limitations

• You cannot downgrade to an earlier version of Cloud Composer or Airflow.
• Cloud Composer supports manually transferring DAGs and configuration from your Airflow 1.10.* environments to Airflow 2 environments. It is not possible to do in-place upgrades to Airflow 2 for environments with Airflow 1.10.*.
• You can only upgrade to the latest Cloud Composer version within the same major version, such as composer-1.12.4-airflow-1.10.10 to composer-1.13.0-airflow-1.10.10. Upgrading from composer-1.4.0-airflow-1.10.0 to composer-2.0.0-airflow-1.10.0 is not permitted because the Cloud Composer major version changes from 1 to 2.
• The image version that you upgrade to must support your environment's current Python version.
• You cannot upgrade your environment if the Airflow database contains more than 16 GB of data. During an upgrade, a warning is shown if the Airflow database size is more than 16 GB. In this case, perform the database maintenance to reduce the database size.

• You cannot upgrade from Cloud Composer 1 to Cloud Composer 2

  If you want to start using Cloud Composer 2 then you need to create new Cloud Composer 2 environments.

  • If you want to migrate from Airflow 1 to Airflow 2, see the migration guide, which provides instructions for transferring history of DAG and tasks executions from Cloud Composer 1 with Airflow 1 to Cloud Composer 2 with Airflow 2 environments.

Deprecation messages

Cloud Composer displays warnings when your environment image approaches its end of support date. You can use these warnings to always keep your environments in the full support period.

Cloud Composer keeps track of the Cloud Composer image version that your environment is based on. When the image approaches its end of support date, you can see a warning in the list of environments and on the Environment details page.

To check if your environment image is up to date:

1. Open the Environments page in the Google Cloud Console.

   Open the Environments page

2. Click the name of the environment to view its details.

3. Under Environment configuration, find the Image version field.

4. In the Image version field, one of the following messages is displayed:

   • Newest available version. Your environment image is fully supported.

   • New version available. Your environment image is fully supported and you can upgrade it to a later version.

   • Support for this image version ends in... Your environment image approaches the end of the full support period.

   • This version is not supported as of... Your environment is past the full support period.

Note: You can always upgrade your environment to a later version, even if the environment is past the full support period.

Viewing available upgrades

To view Cloud Composer versions that you can upgrade to:

Console

1. Open the Environments page in the Google Cloud.

   Open the Environments page

2. Click the environment Name.

3. In the Environment Configuration tab, click Upgrade Image Version.

4. For available versions, click the Cloud Composer Image version drop-down menu.

gcloud

gcloud beta composer environments list-upgrades ENVIRONMENT_NAME \
    --location LOCATION 

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.

For example:

gcloud beta composer environments list-upgrades test-environment \
    --location us-central1
┌─────────────────────────────────────────────────────────────────────────────┐
│                              AVAILABLE UPGRADES                             │
├──────────────────────────────┬──────────────────┬───────────────────────────┤
│        IMAGE VERSION         │ COMPOSER DEFAULT │ SUPPORTED PYTHON VERSIONS │
├──────────────────────────────┼──────────────────┼───────────────────────────┤
│ composer-1.4.0-airflow-1.9.0 │ True             │ 2,3                       │
└──────────────────────────────┴──────────────────┴───────────────────────────┘

API

To view available versions using the Cloud Composer REST API, construct an imageVersions.list API request and provide the project and location in the form projects/{projectId}/locations/{locationId}.

For example:

GET https://composer.googleapis.com/v1/projects/test-project-id/locations/us-central1/imageVersions

{
  "imageVersions": [
    {
      "imageVersionId": "composer-1.4.2-airflow-1.10.0",
      "supportedPythonVersions": [
        "2",
        "3"
      ]
    },
    {
      "imageVersionId": "composer-1.4.2-airflow-1.9.0",
      "isDefault": true,
      "supportedPythonVersions": [
        "2",
        "3"
      ]
    }
  ]
} 

Checking for PyPI package conflicts before upgrading

Preview — Upgrade check

This feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

You can check if PyPI packages installed in your environment have any conflicts with preinstalled packages in the new Cloud Composer image.

A successful check means that there is no PyPI package dependency conflicts between the current and the specified version. Note that an upgrade operation might still not be successful because of other reasons.

Console

To run an upgrade check for your environment:

1. In the Google Cloud Console, go to the Environments page.

   Go to Environments

2. Select your environment.

3. Go to the Environment configuration tab, locate the Image version entry, and click Upgrade.

4. In the Environment version upgrade dialog, in the New version drop-down list, select a Cloud Composer version that you want to upgrade to.

5. In the PyPI packages compatibility section, click Check for conflicts.

6. Wait until the check is complete. If there are PyPI package dependency conflicts, then the displayed error messages contain details about conflicting packages and package versions.

gcloud

To run an upgrade check for your environment, run the environments check-upgrade command with the Cloud Composer image version that you want to upgrade to.

gcloud beta composer environments check-upgrade ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version COMPOSER_IMAGE

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the Compute Engine region where the environment is located.
• COMPOSER_IMAGE with the Cloud Composer image version that you want to upgrade to.

Example:

gcloud beta composer environments check-upgrade example-environment \
  --location us-central1 \
  --image-version composer-1.16.6-airflow-1.10.15

Example output:

Waiting for [projects/example-project/locations/us-central1/environments/
example-environment] to be checked for PyPI package conflicts when upgrading
to composer-1.16.6-airflow-
1.10.15. Operation [projects/example-project/locations/us-central1/operations
/04d0e8b2-...]...done.
...

Response:
'@type': type.googleapis.com/
google.cloud.orchestration.airflow.service.v1beta1.CheckUpgradeResponse
buildLogUri: https://console.cloud.google.com/cloud-build/builds/79738aa7-...
containsPypiModulesConflict: CONFLICT
pypiConflictBuildLogExtract: |-
The Cloud Build image build failed: Build failed; check build logs for
details. Full log can be found at https://console.cloud.google.com/
cloud-build/builds/79738aa7-...
Error details: tensorboard 2.2.2 has requirement
setuptools>=41.0.0, but you have setuptools 40.3.0.

As an alternative, you can run an upgrade check asynchronously. Use the --async argument to make an asynchronous call, then check the result with the gcloud composer operations describe command.

API

Construct an environments.checkUpgrade API request.

Specify the image version in the imageVersion field:

{
  "imageVersion": "COMPOSER_IMAGE"
}

Replace COMPOSER_IMAGE with the Cloud Composer image version that you want to upgrade to.

Upgrading the Cloud Composer version

To upgrade the version of Cloud Composer that your environment runs:

Console

1. Open the Environments page in the Google Cloud.

   Open the Environments page

2. Click the environment Name to modify.

3. In the Environment Configuration tab, click Upgrade Image Version.

4. Click the Cloud Composer Image version drop-down menu and select a version.

5. Click Submit.

gcloud

gcloud beta composer environments update ENVIRONMENT_NAME \
    --location LOCATION --image-version VERSION

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.
• VERSION is the Cloud Composer version and Airflow version to use for your environment in the form composer-a.b.c-airflow-x.y.z or composer-a.b.c-airflow-x.y. If you do not specify the Airflow patch, the highest available patch version for the given major and minor version is used.

For example:

gcloud beta composer environments update test-environment \
    --location us-central1 --image-version composer-latest-airflow-1.10.1 

API

To upgrade using the Cloud Composer REST API, construct an environments.patch API request. Provide the version in the form composer-a.b.c-airflow-x.y.z.

For example:

PATCH https://composer.googleapis.com/v1beta1/projects/test-project/locations/us-central1/environments/test-environment?updateMask=config.software_config.image_version

The request body includes the imageVersion:

{
  "config": {
    "softwareConfig": {
      "imageVersion": "composer-1.6.0-airflow-1.10.1"
    }
  }
}

Upgrading the Airflow version

Note: For switching from Airflow 1.10.* to Airflow 2, see Migrate environments to Airflow 2.

Note: Please, get familiarized with the Before you begin section before proceeding. When your environment is running the latest Cloud Composer version, you can use the Cloud SDK to upgrade only the Airflow version, for example to upgrade from composer-1.15.2-airflow-1.10.14 to composer-1.15.2-airflow-1.10.15.

gcloud beta composer environments update ENVIRONMENT_NAME \
--location LOCATION --airflow-version VERSION

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.

• VERSION is the Airflow version to use for your environment in the form x.y.z or x.y.

  Note: If you do not specify the Airflow patch version, the latest available patch version for the specified major and minor Airflow version is used.

For example:

gcloud beta composer environments update test-environment \
--location us-central1 --airflow-version=1.10.1