This page describes how to upgrade the Airflow version or Cloud Composer version that your environment runs.
During upgrade, Cloud Composer:
Re-deploys the Airflow scheduler and worker pods in a new Kubernetes namespace. After the upgrade is completed, Airflow will use a new MySQL database; the database name matches the Kubernetes namespace. The DAG run history is preserved.
Updates the Airflow
airflow_db
connection to point to the new Cloud SQL database.
These changes affect how you access pods and connect to the Cloud SQL database.
- To access pods in the GKE cluster after upgrade, you need to
use namespace-aware
kubectl
commands. For example, to list pods in the cluster, usekubectl get pods -A
. To execute a command on a pod, usekubectl exec -n <NAMESPACE> ...
. - If you use Airflow connections and workloads that reference the SQL proxy directly,
use the default namespace as part of the hostname:
airflow-sqlproxy-service.default
, notairflow-sqlproxy-service
.
Upgrading Cloud Composer, does not change how you connect to the resources in your environment, such as Google Kubernetes Engine node VM IP addresses, Cloud SQL instance IP address, Cloud Storage bucket, or Airflow webserver domain name.
Before you begin
- Upgrading environments is currently in Preview. Use this feature with caution in production environments.
- The following role is required to upgrade:
roles/editor
orroles/composer.admin
- Pause all DAGs and wait for in-progress tasks to finish before upgrading.
- You can upgrade the Cloud Composer, the Airflow version, or both at the same time.
- The Cloud Composer-Airflow combination that you are upgrading to
must be a released version.
- For available upgrades, see Viewing available upgrades. To get the latest features and fixes, consider upgrading to the most recent Cloud Composer release.
- For the list of PyPI packages and customizations in a supported version,
see the Cloud Composer version list.
Before you upgrade, ensure that you know the differences between the current versions of Airflow and Cloud Composer and the versions that you're upgrading to. Incompatible changes can cause DAGs to break.
Airflow Database Maintenance
Airflow database is collecting more and more data with time going.
You can use this maintenance DAG to prune the content of your database. This DAG is going to remove old entries from DagRun, TaskInstance, Log, XCom, Job DB and SlaMiss tables.
Database maintenance DAG
You can also remove entries related to DAGs that are not longer needed as it was described in "Deleting a DAG" section.
Limitations
- You cannot downgrade to an earlier version of Cloud Composer or Airflow.
- You can only upgrade to the latest Cloud Composer version within the same
major version, such as
composer-1.12.4-airflow-1.10.10
tocomposer-1.13.0-airflow-1.10.10
. Upgrading fromcomposer-1.4.0-airflow-1.10.0
tocomposer-2.0.0-airflow-1.10.0
is not permitted because the Cloud Composer major version changes from1
to2
. - The image version that you upgrade to must support your environment's current Python version.
- Upgrades cannot be performed if the Airflow database contains more than 16GB of data. During the upgrade, you will be shown a warning if it is the case and you will need to perform database maintenance as it was described in Airflow Database Maintenance section
Viewing available upgrades
To view Cloud Composer versions that you can upgrade to:
Console
Open the Environments page in the Google Cloud.
Click the environment Name.
In the Environment Configuration tab, click Upgrade Image Version.
For available versions, click the Cloud Composer Image version drop-down menu.
gcloud
gcloud beta composer environments list-upgrades ENVIRONMENT_NAME \ --location LOCATION
where:
ENVIRONMENT_NAME
is the name of the environment.LOCATION
is the Compute Engine region where the environment is located.
For example:
gcloud beta composer environments list-upgrades test-environment \ --location us-central1 ┌─────────────────────────────────────────────────────────────────────────────┐ │ AVAILABLE UPGRADES │ ├──────────────────────────────┬──────────────────┬───────────────────────────┤ │ IMAGE VERSION │ COMPOSER DEFAULT │ SUPPORTED PYTHON VERSIONS │ ├──────────────────────────────┼──────────────────┼───────────────────────────┤ │ composer-1.4.0-airflow-1.9.0 │ True │ 2,3 │ └──────────────────────────────┴──────────────────┴───────────────────────────┘
API
To view available versions using the
Cloud Composer REST API,
construct an imageVersions.list
API request and provide the project and location in the form projects/{projectId}/locations/{locationId}
.
For example:
GET https://composer.googleapis.com/v1/projects/test-project-id/locations/us-central1/imageVersions { "imageVersions": [ { "imageVersionId": "composer-1.4.2-airflow-1.10.0", "supportedPythonVersions": [ "2", "3" ] }, { "imageVersionId": "composer-1.4.2-airflow-1.9.0", "isDefault": true, "supportedPythonVersions": [ "2", "3" ] } ] }
Upgrading the Cloud Composer version
To upgrade the version of Cloud Composer that your environment runs:
Console
Open the Environments page in the Google Cloud.
Click the environment Name to modify.
In the Environment Configuration tab, click Upgrade Image Version.
Click the Cloud Composer Image version drop-down menu and select a version.
Click Submit.
gcloud
gcloud beta composer environments update ENVIRONMENT_NAME \ --location LOCATION --image-version VERSION
where:
ENVIRONMENT_NAME
is the name of the environment.LOCATION
is the Compute Engine region where the environment is located.VERSION
is the Cloud Composer version and Airflow version to use for your environment in the formcomposer-a.b.c-airflow-x.y.z
orcomposer-a.b.c-airflow-x.y
. If you do not specify the Airflow patch, the highest available patch version for the given major and minor version is used.
For example:
gcloud beta composer environments update test-environment \ --location us-central1 --image-version composer-latest-airflow-1.10.1
API
To upgrade using the Cloud Composer REST API,
construct an environments.patch
API request. Provide the version in the form composer-a.b.c-airflow-x.y.z
.
For example:
PATCH https://composer.googleapis.com/v1beta1/projects/test-project/locations/us-central1/environments/test-environment?updateMask=config.software_config.image_version
The request body includes the imageVersion
:
{ "config": { "softwareConfig": { "imageVersion": "composer-1.6.0-airflow-1.10.1" } } }
Upgrading the Airflow version
When your environment is running the latest Cloud Composer version,
you can use the Cloud SDK to upgrade only the Airflow version, for example
to upgrade from composer-1.6.1-airflow-1.9.0
to composer-1.6.1-airflow-1.10.0
.
gcloud beta composer environments update ENVIRONMENT_NAME \ --location LOCATION --airflow-version VERSION
where:
ENVIRONMENT_NAME
is the name of the environment.LOCATION
is the Compute Engine region where the environment is located.VERSION
is the Airflow version to use for your environment in the formx.y.z
orx.y
.
For example:
gcloud beta composer environments update test-environment \ --location us-central1 --airflow-version=1.10.1