Cloud Composer 1 | Cloud Composer 2
This page explains what Airflow backport provider packages are and how you can use them in your DAGs.
Backport provider packages are Airflow 2 versions of operators, transfers, sensors, hooks, and secrets that are packaged as PyPI modules.
To simplify things, this document focuses just on operators. You can use all other content from a backport package in the same way as shown for operators.
Backport provider packages solve an important problem: you can get new features and security updates for operators, transfers, sensors, and hooks without upgrading your Airflow environment to a later version. For example, the provider package for Google is available in Airflow 2. You can import operators from the backported version of this package in your Airflow 1.10.* environment.
Before Apache Airflow introduced the concept of backport provider packages, operators were an integral part of Airflow. To get new versions of operators, you needed a newer version of Airflow. This required upgrading your Cloud Composer environment to a later version. Airflow 2 moves away from this model and introduces provider packages. A provider package for Airflow 2 is a PyPI module that contains operators, transfers, sensors, hooks, and secrets for a specific provider (for example, Google). If you use Airflow 1.10.*, you can benefit from this change and use backported versions of provider packages.
Use preinstalled backport packages
Some backport packages are already installed in Cloud Composer images. You don't need to install these backport packages to your environment. Just make sure that you import operators from a backport package in your DAG code as described later on this page.
To check which backport packages are available in your environment, see the list of packages in the Cloud Composer image for your environment.
Each preinstalled backport package in your environment has a specific version. If you want to use a different version, update your environment and specify the required version. We do not recommend to downgrade preinstalled backport packages. Install an earlier version only if you discover an issue with a later version. You cannot uninstall preinstalled backport packages, only change the installed versions.
Install and upgrade backport packages
To install or upgrade a backport package:
Install or upgrade the package like any other PyPI package.
If required, install cross provider package dependencies. These are extra dependencies that you might need to use all the features of the backport package.
For example, to use
apache-airflow-backport-providers-google, you need the
apache-airflow-backport-providers-googleand specify the
[salesforce]extra in the Extras and version field.
Import operators from backport provider packages in DAGs
To see a list of contents in a backport package, go to the backport package
page on PyPI.org. For example, the page for
lists operators, transfers, sensors, hooks, and secrets for this package.
Backport provider packages introduce new and moved operators and other content. There is a difference between these two types of content, as explained later.
Import new operators
New operators are those that don't exist in Airflow 1.10.*. If you try to import such an operator without its backport package, you get an import error.
To use new operators from a backport package, import them from their
airflow.providers.* package, as described on the page for this
backport package on PyPI.org.
The following example imports new operators from the
from airflow.providers.google.cloud.operators.bigquery_dts import ( BigQueryCreateDataTransferOperator, BigQueryDeleteDataTransferConfigOperator, )
Import moved operators
Moved operators are those that already exist in Airflow 1.10.*. After you install a backport operator package, you can import two different versions of an operator. One version is bundled with Airflow, and another one is a moved operator. To use a moved operator, import it by using a new import path.
To use moved operators from a backport package, import them from a
airflow.contrib.* package, as described on the page for this
backport package on PyPI.org.
The following example imports moved operators from the
from airflow.contrib.operators.bigquery_operator import ( BigQueryCreateEmptyDatasetOperator, BigQueryOperator, )
- Writing DAGs
- Installing PyPI packages from PyPI.org
- List of preinstalled packages in Cloud Composer images