Cloud Composer versioning overview

Cloud Composer 1 | Cloud Composer 2

Preview — Cloud Composer 2

This feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

This page describes Cloud Composer versioning and lists differences between major Cloud Composer versions.

• Cloud Composer has two major versions: Cloud Composer 1 and Cloud Composer 2.
• Cloud Composer environments are based on Cloud Composer images, which are versioned.
• You can select a Cloud Composer image version when you create or upgrade your environment.

Major versions of Cloud Composer

Cloud Composer has two major versions:

• Cloud Composer 2. This version has autoscaling environments.
• Cloud Composer 1. This version has manual scaling.

The following table lists major differences between Cloud Composer 1 and Cloud Composer 2.

	Cloud Composer 1	Cloud Composer 2
Cloud Composer image versions	composer-1.x.x	composer-2.x.x
Airflow versions	Airflow 1.10.* and Airflow 2	Airflow 2
Python versions	3.8.6, 3.6.10, 2.7.17	3.8.6
Environment's cluster	Standard mode VPC-native or Routes-based Google Kubernetes Engine cluster	Autopilot mode VPC-native Google Kubernetes Engine cluster
Horizontal scaling	Can adjust the number of nodes in the environment's cluster. This changes the number of Airflow workers. Can adjust the number of Airflow schedulers.	Automatically scaling number of Airflow workers, based on demand. Can set and change upper and lower limits for the number of workers. Can adjust the number of Airflow schedulers.
Vertical scaling	Can set machine types for cluster nodes, Airflow web server and database when creating an environment. Can change machine types for Airflow web server and database.	Can set and change workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, and database.
Pricing model	Cloud Composer 1 pricing model	Cloud Composer 2 pricing model
Access control	Cloud Composer roles	Cloud Composer roles. Uses Workload Identity and requires additional permissions for the Cloud Composer Service Agent account.
Environment architecture	Cloud Composer 1 environment architecture	Cloud Composer 2 environment architecture
Terraform support	Can create and update Cloud Composer 1 environments.	Can create and update Cloud Composer 2 environments. In Preview, workloads configuration cannot be managed using Terraform. The default workloads configuration is used for environments created with Terraform.
CMEK	Supported	Not supported in Preview
Web server access controls	Supported	Not supported in Preview
Restarting web server	Supported	Not supported in Preview
Web server plugins	Supported in Airflow 1 environments with disabled DAG serialization.	Supported
Installing custom plugins	Supported	Supported
Asynchronous DAG loading	Supported in Airflow 1	Not supported
DAG serialization	Always enabled in Airflow 2. Can be disabled in Airflow 1.	Always enabled in Airflow 2.

Cloud Composer images

To run Apache Airflow, Cloud Composer builds Docker images that bundle Airflow releases with other common binaries and Python libraries.

Cloud Composer images include Airflow modifications that are specific to Cloud Composer and unsuitable for the upstream Airflow codebase. The preinstalled packages along with Cloud Composer-specific Airflow changes are referred to collectively as Cloud Composer versions.

Google periodically releases new Cloud Composer images that include new Airflow releases and Cloud Composer improvements.

Cloud Composer version deprecation and support

Google supports Cloud Composer versions for a period of time after the version release. During this period, Cloud Composer environments that use these versions are fully supported.

Cloud Composer version support is defined as follows:

• 0-12 months from the release date: Cloud Composer environments running these versions are fully supported.
• 12-18 months from the release date: Cloud Composer environments running these versions are unsupported except to notify customers about security issues.
• 18+ months from the release date: Cloud Composer environments running these versions are unsupported and entirely user managed.

Note: You can use Cloud Composer versions after the date they are no longer supported. For example, if your environment is based on a Cloud Composer version that is past the support period, the environment still works and you can keep using it. In this case, we recommend you to upgrade your environment to a supported version.

Airflow version support

Airflow follows the semantical software versioning schema. Each version of Airflow has a major, minor, and patch version.

For a list of Airflow versions supported in each Cloud Composer version, see the list of Cloud Composer versions.

Airflow 1 version support

• Cloud Composer supports Airflow 1.10.15, which is the latest released version of Airflow 1.

Airflow 2 version support

• In each release, Cloud Composer supports two minor versions of Airflow 2.

  For example, Cloud Composer 1.17.0 supports Airflow 2.0.* and Airflow 2.1.*.

• For each minor version of Airflow 2, Cloud Composer supports one patch version.

  For example, for Airflow 2.1.*, Cloud Composer supports Airflow 2.1.2.

• When a new patch version of Airflow 2 becomes available in Cloud Composer, the corresponding release has two patch versions of Airflow 2 available for the same minor version of Airflow. The previous patch version is then removed in one of the next releases.

  For example, Cloud Composer 1.17.4 supports both Airflow 2.1.2 and Airflow 2.1.4.

• Cloud Composer can skip some Airflow patch versions between releases of Cloud Composer. In addition, if there are functional, quality, or performance issues with a specific patch version of Airflow, Cloud Composer can also skip this version.

  For example, Airflow 2.1.3 is skipped and Airflow 2.1.4 is available.

• A Cloud Composer release running a stable Airflow version can include Airflow updates that are backported from a later Airflow version.

Cloud Composer version schema

Cloud Composer image versioning schema captures Cloud Composer and Airflow versions:

composer-a.b.c-airflow-x.y.z

Where

• composer-a.b.c is the version of Cloud Composer-managed customizations
• airflow-x.y.z is the Airflow release

Version flags and aliases

Cloud Composer supports Cloud Composer image versions that are pinned to the latest Cloud Composer version.

When you create an Cloud Composer environment, you can use these flags to specify the environment version:

• --image-version, to specify the environment version in the form composer-a.b.c-airflow-x.y.z.
• --airflow-version, to specify the Airflow version in the form x.y.z, which creates the environment using the latest Cloud Composer version and specified Airflow version.

Cloud Composer also supports using the following version aliases:

• composer-latest-airflow-x.y.z specifies an image version with the latest Cloud Composer version.
• composer-a.b.c-airflow-x.y specifies an image version that automatically expands to composer-a.b.c-airflow-x.y.z where z is the latest patch number.
• composer-latest-airflow-x.y specifies an image version with the latest Cloud Composer version that automatically expands to airflow-x.y.z where z is the latest patch number.

Note: Creating a Cloud Composer environment version with composer-latest-airflow-x.y.z does not provide automatic version upgrades. Instead, Cloud Composer resolves the version alias to composer-a.b.c-airflow-x.y.z at creation time.

What's next

• Cloud Composer images
• Create an environment
• Upgrade an existing Cloud Composer environment