This page describes Cloud Composer versioning and lists differences between major Cloud Composer versions.
Major versions of Cloud Composer
Cloud Composer has the following major versions:
- Cloud Composer 1. Has manual environment scaling, the infrastructure is deployed to your projects and networks.
- Cloud Composer 2. In this version, the environment's cluster scales automatically to meet the resource demands.
- Cloud Composer 3. This version has simplified network setup and hides infrastructure components including the environment's cluster and environment's dependencies on other services.
Comparison of Cloud Composer versions
The following table lists major differences between Cloud Composer 1, Cloud Composer 2, and Cloud Composer 3.
Feature | Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3 |
---|---|---|---|
Image version | composer-1.b.c-airflow-x.y.z |
composer-2.b.c-airflow-x.y.z |
composer-3-airflow-x.y.z-build.t |
Airflow versions | Airflow 1.10.* and Airflow 2 | Airflow 2 | Airflow 2 |
Python versions | 3.8.12 | 3.11.5 and 3.8.12 in earlier versions | 3.11.5 |
Environment's cluster | Standard mode VPC-native or Routes-based Google Kubernetes Engine cluster | Autopilot mode VPC-native Google Kubernetes Engine cluster | Environment's cluster is not deployed into your project. |
Horizontal scaling | Can adjust the number of nodes in the environment's cluster. This changes the number of Airflow workers. Can adjust the number of Airflow schedulers. |
Automatically scaling number of Airflow workers, based on demand. Can set and change upper and lower limits for the number of workers. Can adjust the number of Airflow schedulers and triggerers |
Automatically scaling number of Airflow workers, based on demand. Can set and change upper and lower limits for the number of workers. Can adjust the number of Airflow schedulers, triggerers, and DAG processors |
Vertical scaling | Can set machine types for cluster nodes, Airflow web server and database when creating an environment. Can change machine types for Airflow web server and database. | Can set and change the workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, and the environment size. | Can set and change the workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, DAG processors, and the environment size. |
Pricing model | Cloud Composer 1 pricing model | Cloud Composer 2 pricing model | Cloud Composer 3 pricing model |
Access control | Cloud Composer 1 roles | Cloud Composer roles. Uses Workload Identity Federation for GKE and requires additional permissions for the Cloud Composer Service Agent account. | Cloud Composer roles. Uses Workload Identity Federation for GKE. Extra permissions are not required |
Access through Workforce Identity Federation | Not supported | Supported | Supported |
Highly Resilient environments | Not supported | Supported | Not available in Public Preview |
Private IP networking | VPC peerings | Private Service Connect | Simplified network setup. Can be changed between Public and Private IP in an existing environment. |
VPC Service Controls | Supported | Supported | Not available in Public Preview |
Airflow executors | Celery Executor | Celery Executor | CeleryKubernetes Executor |
Airflow triggerer (deferrable operators) | Not supported | Supported | Supported |
CMEK | Supported | Supported | Not available in Public Preview |
Web server access controls | Supported | Supported | Supported |
Restarting web server | Supported | Supported | Supported |
Web server plugins | Supported in Airflow 1 environments with disabled DAG serialization. | Supported | Supported, plugins can be disabled and enabled on demand. |
Installing custom plugins | Supported | Supported | Supported |
Maintenance operations | All tasks can be impacted. | Tasks that take less than 55 minutes to execute are not impacted. | Tasks that take less than 55 minutes to execute are not impacted. |
Storing Snapshots | Supported for Airflow 2.x and 1.10.15 versions | Supported | Supported |
Loading Snapshots | Not supported | Supported | Supported |
Scheduled Snapshots | Not supported | Supported | Not available in Public Preview |
Custom environment's bucket | Not supported | Supported | Supported |
Saving Airflow task logs only in Cloud Logging | Not supported | Supported | Supported |
Data lineage integration | Not supported | Supported | Supported |
Cloud Composer images
To run Apache Airflow, Cloud Composer builds Docker images that bundle Airflow releases with other common binaries and Python libraries.
Cloud Composer images include Airflow modifications that are specific to Cloud Composer and unsuitable for the upstream Airflow codebase. Each Cloud Composer image contains preinstalled packages along with Cloud Composer-specific Airflow changes.
Google periodically releases new Cloud Composer images:
In Cloud Composer 3, images include new builds of supported Airflow versions. You can select an Airflow version and a build that you want to use. Your environment gets Cloud Composer improvements automatically.
In Cloud Composer 2 and Cloud Composer 1, images include Cloud Composer improvements and new builds of supported Airflow versions. To get Cloud Composer improvements, you upgrade your environment to a later version of Cloud Composer.
Version deprecation and support
Composer 3
Google supports an Airflow version for a period of time after the initial release:
If the minor version of Airflow is the latest one available in Cloud Composer 3, all Airflow builds with this minor version are fully supported. The exact end of support date is determined when a new minor version of Airflow becomes available in Cloud Composer 3.
When a new minor version of Airflow becomes available in Cloud Composer 3, all builds with the previous minor Airflow version are supported for 12 months, starting from that date. If there are several builds that contain the same minor Airflow version, all of them have the same end of full support date.
After a minor version is no longer fully supported, environments running this version are also unsupported and entirely user-managed. You can still upgrade such an environment to a fully supported Airflow version.
You can create environments with all Airflow builds that are fully supported.
You can still use an environment with an Airflow build that is no longer supported. The environment still works, you can keep using it, and you can still upgrade it to a fully supported version.
Composer 2
Google supports Cloud Composer 2 versions for a period of time after the version release. During this period, Cloud Composer 2 environments that use these versions are fully supported.
Cloud Composer 2 version support is defined as follows:
0-12 months from the release date: Cloud Composer 2 environments running these versions are fully supported.
12+ months from the release date: Cloud Composer 2 environments running these versions are unsupported.
You can use Cloud Composer versions after the date they are no longer supported. For example, if your environment is based on a Cloud Composer version that is past the support period, the environment still works and you can keep using it. In this case, we recommend you to upgrade your environment to a supported version.
Composer 1
New Cloud Composer 1 versions were released until the end of March 2023. All released Cloud Composer 1 versions follow the same version deprecation policy as Cloud Composer 2 and are supported according to it.
On March 25, 2024, Cloud Composer 1 entered its post-maintenance mode. Google will not release any further updates to Cloud Composer 1, including new versions of Airflow, bugfixes, and security updates. We recommend migration to Cloud Composer 2.
Airflow version support
Airflow follows the semantical software versioning schema. Each version of Airflow has a major, minor, and patch version.
Composer 3
In Cloud Composer 3, new Airflow versions and builds become available as the time goes.
Your environment can function with all previous Airflow versions and builds, and receives updates to its infrastructure components automatically. Each version and build is supported as described in Version deprecation and support, and you can choose to upgrade your Airflow version to a later version or build.
Composer 2
For a list of Airflow versions supported in each Cloud Composer version, see the list of Cloud Composer versions.
In each release, Cloud Composer 2 supports two minor versions of Airflow 2.
For example, Cloud Composer 2.4.0 supports Airflow 2.4.* and Airflow 2.5.*.
For each minor version of Airflow 2, Cloud Composer supports one patch version.
For example, for Airflow 2.4.*, Cloud Composer supports Airflow 2.4.3.
When a new patch version of Airflow 2 becomes available in Cloud Composer, the corresponding release has two patch versions of Airflow 2 available for the same minor version of Airflow. The previous patch version is then removed in one of the next releases.
For example, Cloud Composer 2.3.4 supports both Airflow 2.5.1 and Airflow 2.5.3.
Cloud Composer can skip some Airflow patch versions between releases of Cloud Composer. In addition, if there are functional, quality, or performance issues with a specific patch version of Airflow, Cloud Composer can also skip this version.
For example, Airflow 2.1.3 is skipped and Airflow 2.1.4 is available.
A Cloud Composer release running a stable Airflow version can include Airflow updates that are backported from a later Airflow version.
Composer 1
The latest version of Airflow supported in Cloud Composer 1 is Airflow 2.4.3.
Only Cloud Composer 1 supports Airflow 1.10.15, which is the latest released version of Airflow 1.
Cloud Composer version schema
Composer 3
Cloud Composer 3 versioning schema captures Airflow versions and builds:
composer-3-airflow-x.y.z-build.t
Where:
composer-3
is the version of Cloud Composer.airflow-x.y.z.build.t
is the Airflow version and build. Every next release of the same Airflow version in Cloud Composer 3 has a higher build number. The build number is defined by Cloud Composer and not by the Airflow open source project.
Composer 2
Cloud Composer 2 image versioning schema captures Cloud Composer and Airflow versions:
composer-2.b.c-airflow-x.y.z
Where:
composer-2.b.c
is the version of Cloud Composer-managed customizationsairflow-x.y.z
is the Airflow release
Composer 1
Cloud Composer 1 image versioning schema captures Cloud Composer and Airflow versions:
composer-1.b.c-airflow-x.y.z
Where:
composer-1.b.c
is the version of Cloud Composer-managed customizationsairflow-x.y.z
is the Airflow release
Version aliases
Version aliases work in the following way:
Cloud Composer resolves the version alias to the full format when it creates your environment:
- All version aliases become
composer-3-airflow-x.y.z-build.t
in Cloud Composer 3. - All version aliases become
composer-a.b.c-airflow-x.y.z
in Cloud Composer 2 and Cloud Composer 1.
- All version aliases become
In Cloud Composer 3, using a version alias, such as
composer-3-airflow-x.y
does not automatically upgrade the Airflow version and build of your environment. The environment stays on the same Airflow build and version until you upgrade it. The environment still receives automatic infrastructure upgrades, as usual.In Cloud Composer 2 and Cloud Composer 1, using a version alias, such as
composer-a-airflow-x.y.z
does not provide automatic upgrades for an environment. The environment stays on the same Cloud Composer and Airflow version until you upgrade it.
Cloud Composer 3 version aliases
Cloud Composer 3 supports the following version aliases:
Alias | Airflow version |
---|---|
composer-3-airflow-2 |
Latest version and build of Airflow 2 |
composer-3-airflow-x.y |
Latest version and build of Airflow within x.y major and minor versions |
composer-3-airflow-x.y.z |
Latest build of the specified version of Airflow |
Cloud Composer 2 version aliases
Cloud Composer 2 supports the following version aliases:
Alias | Cloud Composer version | Airflow version |
---|---|---|
composer-2-airflow-2 |
Latest version of Cloud Composer 2 | Latest version of Airflow 2 |
composer-2-airflow-x.y |
Latest version of Cloud Composer 2 | Latest version of Airflow within x.y major and minor versions |
composer-2-airflow-x.y.z |
Latest version of Cloud Composer 2 | Specified version of Airflow |
composer-2.b.c-airflow-x.y |
Specified version of Cloud Composer 2 | Latest version of Airflow available within x.y major and minor versions |
Cloud Composer 1 version aliases
Cloud Composer 1 supports the following version aliases:
Alias | Cloud Composer version | Airflow version |
---|---|---|
composer-1-airflow-2 |
Latest version of Cloud Composer 1 | Latest version of Airflow 2 |
composer-1-airflow-1 |
Latest version of Cloud Composer 1 | Latest version of Airflow 1 |
composer-1-airflow-x.y |
Latest version of Cloud Composer 1 | Latest version of Airflow within x.y major and minor versions |
composer-1-airflow-x.y.z |
Latest version of Cloud Composer 1 | Specified version of Airflow |
composer-1.b.c-airflow-x.y |
Specified version of Cloud Composer 1 | Latest version of Airflow available within x.y major and minor versions. |
The following version aliases point to Cloud Composer 2. You can still use them, but consider switching to aliases for your version of Cloud Composer.
composer-latest-airflow-x.y
is equal tocomposer-2-airflow-x.y
.composer-latest-airflow-x.y.z
is equal tocomposer-2-airflow-x.y.z
.
Version arguments in Google Cloud CLI
When you create or upgrade a Cloud Composer environment using Google Cloud CLI, you can specify Cloud Composer and Airflow versions:
Composer 3
--image-version
argument specifies the Airflow version and build in thecomposer-3-airflow-x.y.z-build.t
format. You can use version aliases, which are resolved by Google Cloud CLI into the full format.--airflow-version
specifies the Airflow version in the following formats:The
x.y.z
format is for the latest build of the specified version of Airflow. This format is equal to thecomposer-3-airflow-x.y.z
version alias.The
x.y
format is for the latest build of Airflow available withinx.y
major and minor versions. This format is equal to thecomposer-3-airflow-x.y
version alias.x.y.z-build.t
is the specified version and build of Airflow. This format is equal tocomposer-3-airflow-x.y.z-build.t
.
Composer 2
--image-version
argument specifies Cloud Composer 2 and Airflow versions:composer-a.b.c-airflow-x.y.z
format creates an environment with the specified versions of Cloud Composer and Airflow.- Version aliases first resolve to
the
composer-a.b.c-airflow-x.y.z
format and then create an environment with the specified versions of Cloud Composer 2 and Airflow.
--airflow-version
specifies the Airflow version in the following formats:The
x.y.z
format is for the latest version of Cloud Composer 2 and the specified version of Airflow. This format is equal to thecomposer-2-airflow-x.y.z
version alias.The
x.y
format is for the latest version of Cloud Composer 2 and the latest version of Airflow available withinx.y
major and minor versions. This format is equal to thecomposer-2-airflow-x.y
version alias.
Composer 1
--image-version
argument specifies Cloud Composer 1 and Airflow versions:composer-a.b.c-airflow-x.y.z
format creates an environment with the specified versions of Cloud Composer and Airflow.- Version aliases first resolve to
the
composer-a.b.c-airflow-x.y.z
format and then create an environment with the specified versions of Cloud Composer 1 and Airflow.
--airflow-version
specifies the Airflow version in the following formats:The
x.y.z
format is for the latest version of Cloud Composer 1 and the specified version of Airflow. This format is equal to thecomposer-1-airflow-x.y.z
version alias.The
x.y
format is for the latest version of Cloud Composer 1 and the latest version of Airflow available withinx.y
major and minor versions. This format is equal to thecomposer-1-airflow-x.y
version alias.