Cloud Composer versioning overview

This page describes Cloud Composer versioning and lists differences between major Cloud Composer versions.

Major versions of Cloud Composer

Cloud Composer has two major versions:

  • Cloud Composer 2. This version has autoscaling environments.
  • Cloud Composer 1. This version has manual scaling.

The following table lists major differences between Cloud Composer 1 and Cloud Composer 2.

Cloud Composer 1 Cloud Composer 2
Cloud Composer image versions composer-1.x.x composer-2.x.x
Airflow versions Airflow 1.10.* and Airflow 2 Airflow 2
Python versions 3.8.12 3.8.12
Environment's cluster Standard mode VPC-native or Routes-based Google Kubernetes Engine cluster Autopilot mode VPC-native Google Kubernetes Engine cluster
Horizontal scaling Can adjust the number of nodes in the environment's cluster. This changes the number of Airflow workers.
Can adjust the number of Airflow schedulers.
Automatically scaling number of Airflow workers, based on demand. Can set and change upper and lower limits for the number of workers.
Can adjust the number of Airflow schedulers.
Vertical scaling Can set machine types for cluster nodes, Airflow web server and database when creating an environment. Can change machine types for Airflow web server and database. Can set and change workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, and database.
Pricing model Cloud Composer 1 pricing model Cloud Composer 2 pricing model
Access control Cloud Composer roles Cloud Composer roles. Uses Workload Identity and requires additional permissions for the Cloud Composer Service Agent account.
Access through workforce identity federation Not supported Supported
Environment architecture Cloud Composer 1 environment architecture Cloud Composer 2 environment architecture
Private IP networking VPC peerings VPC peerings or Private Service Connect
Terraform support Can create and update Cloud Composer 1 environments. Can create and update Cloud Composer 2 environments.
CMEK Supported Supported
Web server access controls Supported Supported
Restarting web server Supported Supported
Web server plugins Supported in Airflow 1 environments with disabled DAG serialization. Supported
Installing custom plugins Supported Supported
Maintenance operations All tasks can be impacted. Tasks that take less than 55 minutes to execute are not impacted.
Asynchronous DAG loading Supported in Airflow 1 Not supported
DAG serialization Always enabled in Airflow 2. Can be disabled in Airflow 1. Always enabled in Airflow 2.
Support for Network Tags Yes Yes
Cluster nodes with GPUs Yes No
Storing Snapshots Supported for Airflow 2.x and 1.10.15 versions Supported
Loading Snapshots Not supported Supported
Scheduled Snapshots Not supported Supported
Custom environment's bucket Not supported Supported

Cloud Composer images

To run Apache Airflow, Cloud Composer builds Docker images that bundle Airflow releases with other common binaries and Python libraries.

Cloud Composer images include Airflow modifications that are specific to Cloud Composer and unsuitable for the upstream Airflow codebase. The preinstalled packages along with Cloud Composer-specific Airflow changes are referred to collectively as Cloud Composer versions.

Google periodically releases new Cloud Composer images that include new Airflow releases and Cloud Composer improvements.

Cloud Composer 1 version support

New Cloud Composer 1 versions are released until the end of March 2023. All released Cloud Composer 1 versions follow the version deprecation policy and are supported according to it.

Some new Cloud Composer features might be supported only in Cloud Composer 2. You can check which features are supported only in Cloud Composer 2 in the table that lists major differences between versions.

Cloud Composer version deprecation and support

Google supports Cloud Composer versions for a period of time after the version release. During this period, Cloud Composer environments that use these versions are fully supported.

Cloud Composer version support is defined as follows:

  • 0-12 months from the release date: Cloud Composer environments running these versions are fully supported.
  • 12-18 months from the release date: Cloud Composer environments running these versions are unsupported except to notify customers about security issues.

  • 18+ months from the release date: Cloud Composer environments running these versions are unsupported and entirely user managed.

Airflow version support

Airflow follows the semantical software versioning schema. Each version of Airflow has a major, minor, and patch version.

For a list of Airflow versions supported in each Cloud Composer version, see the list of Cloud Composer versions.

Airflow 2 version support

  • In each release, Cloud Composer supports two minor versions of Airflow 2.

    For example, Cloud Composer 1.17.0 supports Airflow 2.0.* and Airflow 2.1.*.

  • For each minor version of Airflow 2, Cloud Composer supports one patch version.

    For example, for Airflow 2.1.*, Cloud Composer supports Airflow 2.1.2.

  • When a new patch version of Airflow 2 becomes available in Cloud Composer, the corresponding release has two patch versions of Airflow 2 available for the same minor version of Airflow. The previous patch version is then removed in one of the next releases.

    For example, Cloud Composer 1.17.4 supports both Airflow 2.1.2 and Airflow 2.1.4.

  • Cloud Composer can skip some Airflow patch versions between releases of Cloud Composer. In addition, if there are functional, quality, or performance issues with a specific patch version of Airflow, Cloud Composer can also skip this version.

    For example, Airflow 2.1.3 is skipped and Airflow 2.1.4 is available.

  • A Cloud Composer release running a stable Airflow version can include Airflow updates that are backported from a later Airflow version.

Airflow 1 version support

Only Cloud Composer 1 supports Airflow 1.10.15, which is the latest released version of Airflow 1.

Cloud Composer version schema

Cloud Composer image versioning schema captures Cloud Composer and Airflow versions:

composer-a.b.c-airflow-x.y.z

Where

  • composer-a.b.c is the version of Cloud Composer-managed customizations
  • airflow-x.y.z is the Airflow release

Version aliases

Cloud Composer 2 supports the following version aliases:

Alias Cloud Composer version Airflow version
composer-2-airflow-2 Latest version of Cloud Composer 2 Latest version of Airflow 2
composer-2-airflow-x.y Latest version of Cloud Composer 2 Latest version of Airflow within x.y major and minor versions
composer-2-airflow-x.y.z Latest version of Cloud Composer 2 Specified version of Airflow
composer-2.b.c-airflow-x.y Specified version of Cloud Composer 2 Latest version of Airflow available within x.y major and minor versions

Cloud Composer 1 supports the following version aliases:

Alias Cloud Composer version Airflow version
composer-1-airflow-2 Latest version of Cloud Composer 1 Latest version of Airflow 2
composer-1-airflow-1 Latest version of Cloud Composer 1 Latest version of Airflow 1
composer-1-airflow-x.y Latest version of Cloud Composer 1 Latest version of Airflow within x.y major and minor versions
composer-1-airflow-x.y.z Latest version of Cloud Composer 1 Specified version of Airflow
composer-1.b.c-airflow-x.y Specified version of Cloud Composer 1 Latest version of Airflow available within x.y major and minor versions.

The following version aliases point to Cloud Composer 2. You can still use them, but consider switching to aliases for your version of Cloud Composer.

  • composer-latest-airflow-x.y is equal to composer-2-airflow-x.y.
  • composer-latest-airflow-x.y.z is equal to composer-2-airflow-x.y.z.

Version arguments in Google Cloud CLI

When you create or upgrade a Cloud Composer environment using Google Cloud CLI, you can specify Cloud Composer and Airflow versions:

  • --image-version argument specifies Cloud Composer and Airflow versions:

    • composer-a.b.c-airflow-x.y.z format creates an environment with the specified versions of Cloud Composer and Airflow.
    • Version aliases first resolve to the composer-a.b.c-airflow-x.y.z format and then create an environment with the specified versions of Cloud Composer and Airflow.
  • --airflow-version specifies the Airflow version in two formats:

    • The x.y.z format is for the latest version of Cloud Composer 2 and the specified version of Airflow. This format is equal to the composer-2-airflow-x.y.z version alias.

    • The x.y format is for the latest version of Cloud Composer 2 and the latest version of Airflow available within x.y major and minor versions. This format is equal to the composer-2-airflow-x.y version alias.

What's next