Cloud Composer Versioning

This page describes how Cloud Composer versioning works. Versioning enables you to select a Cloud Composer image version when you create an environment.

To run Apache Airflow smoothly, Cloud Composer builds Docker images that bundle vanilla Airflow releases with other common binaries and Python libraries. The images include Airflow modifications that are specific to Cloud Composer and unsuitable for the upstream Airflow codebase. The preinstalled packages along with Cloud Composer-specific Airflow changes are referred to collectively as composer-addon. Periodically, Cloud Composer releases images that include new Airflow releases and composer-addon improvements.

Airflow version support

Airflow follows the semantical software versioning schema. Cloud Composer supports the last two stable minor Airflow releases and the latest two patch versions for those minor releases. For example, if the latest Airflow release is X.Y.Z, then Cloud Composer supports the following releases:

• X.Y.Z and X.Y.Z-1: The latest two patch versions for the minor release X.Y.
• X.Y-1 and X.Y-2: The last two stable minor releases. We skip the patch number and assume that the patch is the last patch number for that particular minor release.

Version schema

On top of vanilla Airflow releases, the Cloud Composer image includes some common binaries and libraries, such as open-jdk, mysql, and google-cloud-storage. The following Cloud Composer image versioning schema captures the composer-addon and Airflow versions:

composer-a.b.c-airflow-x.y.z

Where

• composer-a.b.c is the version of Cloud Composer-managed customizations
• airflow-x.y.z is the vanilla Airflow release

Version flags and aliases

Currently, Cloud Composer supports Cloud Composer image versions that are pinned to the latest composer-addon version. When you create an Cloud Composer environment, you can use the --image-version flag to specify the environment version in the form composer-a.b.c-airflow-x.y.z. Alternatively, you can use the --airflow-version flag to specify the Airflow version in the form airflow-x.y.z, which creates the environment using the latest composer-addon version and specified Airflow version.

Cloud Composer also supports using the following version aliases:

• composer-latest-airflow-x.y.z specifies the Cloud Composer image version with the latest supported composer-addon version.
• composer-a.b.c-airflow-x.y specifies a Cloud Composer image version that automatically expands to composer-a.b.c-airflow-x.y.z where z is the latest patch number.
• composer-latest-airflow-x.y specifies a Cloud Composer image version with the latest supported composer-addon version that automatically expands to airflow-x.y.z where z is the latest patch number.

Creating a Cloud Composer environment version with composer-latest-airflow-x.y.z does not provide an automatic version upgrade. Instead, Cloud Composer resolves the alias composer-latest-airflow-x.y.z to composer-a.b.c-airflow-x.y.z at creation time.

Version selection

When you create a new environment, you can select the Cloud Composer version to use in the environment. If you do not select a version, the default Cloud Composer is used.

Note: You can also select the Python version during environment creation.

To view available Cloud Composer versions:

Console

When creating a new environment, click the Cloud Composer Image version drop-down menu.

gcloud

gcloud beta composer environments list-upgrades ENVIRONMENT_NAME \
    --location LOCATION 

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.

For example:

gcloud beta composer environments list-upgrades test-environment \
    --location us-central1
┌─────────────────────────────────────────────────────────────────────────────┐
│                              AVAILABLE UPGRADES                             │
├──────────────────────────────┬──────────────────┬───────────────────────────┤
│        IMAGE VERSION         │ COMPOSER DEFAULT │ SUPPORTED PYTHON VERSIONS │
├──────────────────────────────┼──────────────────┼───────────────────────────┤
│ composer-1.4.0-airflow-1.9.0 │ True             │ 2,3                       │
└──────────────────────────────┴──────────────────┴───────────────────────────┘

API

To view available versions using the Cloud Composer REST API, construct an imageVersions.list API request and provide the project and location in the form projects/{projectId}/locations/{locationId}.

For example:

GET https://composer.googleapis.com/v1/projects/test-project-id/locations/us-central1/imageVersions

{
  "imageVersions": [
    {
      "imageVersionId": "composer-1.4.2-airflow-1.10.0",
      "supportedPythonVersions": [
        "2",
        "3"
      ]
    },
    {
      "imageVersionId": "composer-1.4.2-airflow-1.9.0",
      "isDefault": true,
      "supportedPythonVersions": [
        "2",
        "3"
      ]
    }
  ]
} 

Version deprecation and support

Cloud Composer versions are supported for a specific period of time after the version release. During this time period, Cloud Composer environments that use these versions are supported. After the support window closes, the Cloud Composer is deprecated and environments running deprecated versions are no longer supported. Cloud Composer version support is defined as follows:

• 0-12 months from the release date: Cloud Composer environments running the version are fully supported.
• 12-18 months from the release date: Cloud Composer environments running the version are unsupported except to notify customers about security issues.
• 18+ months from the release date: Cloud Composer environments running the version are unsupported and entirely user managed.