Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3
Apache Airflow has a REST API interface that you can use to perform tasks such as getting information about DAG runs and tasks, updating DAGs, getting Airflow configuration, adding and deleting connections, and listing users.
For an example of using Airflow REST API with Cloud Run functions, see Triggering DAGs with Cloud Functions.
Airflow REST API versions
The following Airflow REST API versions are available in Cloud Composer 1:
- Airflow 1 uses the experimental REST API.
Airflow 2 uses the stable REST API. The experimental REST API is deprecated by Airflow.
You can still use the experimental REST API in Airflow 2 if you enable it through an Airflow configuration override, as described further.
Before you begin
Enable the Cloud Composer API.
Enable the stable Airflow REST API
Airflow 2
The stable REST API is already enabled by default in Airflow 2.
Cloud Composer uses its own API authentication backend, which is integrated with Identity-Aware Proxy.Authorization works in the standard way provided by Airflow. When a new user
authorizes through the API, the user's account gets the Op
role by default.
You can enable or disable the stable REST API, or change the default user role by overriding the following Airflow configuration options:
Section | Key | Value | Notes |
---|---|---|---|
api
|
(Airflow 2.2.5 and earlier) auth_backend (Airflow 2.3.0 and later) auth_backends
|
airflow.composer.api.backend.composer_auth
|
To disable the stable REST API, change to
airflow.api.auth.backend.deny_all |
api
|
composer_auth_user_registration_role
|
Op
|
You can specify any other role. |
Airflow 1
The stable REST API is not available in Airflow 1. You can use the experimental REST API instead.
Enable the experimental Airflow REST API
Airflow 2
By default, the API authentication feature is disabled in the experimental API. The Airflow web server denies all requests that you make.
To enable the API authentication feature and the Airflow 2 experimental API, override the following Airflow configuration option:
Section | Key | Value | Notes |
---|---|---|---|
api
|
(Airflow 2.2.5 and earlier) auth_backend (Airflow 2.3.0 and later) auth_backends
|
airflow.api.auth.backend.default
|
The default is airflow.composer.api.backend.composer_auth . |
api
|
enable_experimental_api
|
True
|
The default is False . |
Airflow 1
By default, the API authentication feature is disabled in Airflow 1.10.11 and later versions. The Airflow web server denies all requests that you make. You use requests to trigger DAGs, so enable this feature.
To enable the API authentication feature in Airflow 1, override the following Airflow configuration option:
Section | Key | Value | Notes |
---|---|---|---|
api |
auth_backend |
airflow.api.auth.backend.default |
The default is airflow.api.auth.backend.deny_all |
After you set this configuration option to
airflow.api.auth.backend.default
, the Airflow web server accepts all API
requests without authentication. Even though the Airflow web server itself
does not require authentication, it is still protected by Identity-Aware Proxy which
provides its own authentication layer.
Allow API calls to Airflow REST API using Webserver Access Control
Depending on the method used to call Airflow REST API, the caller method can be using either IPv4 or IPv6 address. Remember to unblock IP traffic to Airflow REST API using Webserver Access Control.
Use the default configuration option which is All IP addresses have access (default)
if you are not sure from which IP addresses your calls to Airflow REST API
will be sent.
Make calls to Airflow REST API
Get the client_id of the IAM proxy
To make a request to the Airflow REST API endpoint, the function requires the client ID of the IAM proxy that protects the Airflow web server.
Cloud Composer does not provide this information directly. Instead, make an unauthenticated request to the Airflow web server and capture the client ID from the redirect URL:
cURL
curl -v AIRFLOW_URL 2>&1 >/dev/null | grep -o "client_id\=[A-Za-z0-9-]*\.apps\.googleusercontent\.com"
Replace AIRFLOW_URL
with the URL of the Airflow web interface.
In the output, search for the string following client_id
. For example:
client_id=836436932391-16q2c5f5dcsfnel77va9bvf4j280t35c.apps.googleusercontent.com
Python
Save the following code in a file called get_client_id.py
. Fill in your
values for project_id
, location
, and composer_environment
, then run
the code in Cloud Shell or your local environment.
Call Airflow REST API using client_id
Make the following replacements:
- Replace the value of the
client_id
variable with theclient_id
value obtained in the previous step. - Replace the value of the
webserver_id
variable with your tenant project ID, which is a part of the Airflow web interface URL before.appspot.com
. You obtained the Airflow web interface URL on a previous step. Specify the Airflow REST API version that you use:
- If you use the stable Airflow REST API, set the
USE_EXPERIMENTAL_API
variable toFalse
. - If you use the experimental Airflow REST API, no changes are needed. The
USE_EXPERIMENTAL_API
variable is already set toTrue
.
- If you use the stable Airflow REST API, set the
Access Airflow REST API using a service account
The Airflow database limits the length of the email field to 64 characters. Service accounts sometimes have email addresses that are longer than 64 characters. It is not possible to create Airflow users for such service accounts in the usual way. If there is no Airflow user for such a service account, then accessing the Airflow REST API results in HTTP errors 401 and 403.
As a workaround, you can preregister an Airflow user for a service account. To
do so, use accounts.google.com:NUMERIC_USER_ID
as the username, and any
unique string as the email.
To get
NUMERIC_USER_ID
for a service account, run:gcloud iam service-accounts describe \ SA_NAME@PROJECT_ID.iam.gserviceaccount.com \ --format="value(oauth2ClientId)"
Replace:
SA_NAME
with the service account name.PROJECT_ID
with the Project ID.
Create an Airflow user with the
Op
role for the service account:Airflow UI
Go to Admin > Users and click Create. Your Airflow user must have the
Admin
role to open this page.Specify
accounts.google.com:NUMERIC_USER_ID
as the user name. ReplaceNUMERIC_USER_ID
with the user ID obtained on the previous step.Specify a unique identifier as the email. You can use any unique string.
Specify the role for the user. For example,
Op
.Make sure that Is Active? checkbox is selected.
Specify the first and the last name for the user. You can use any string.
Click Save.
gcloud
In Airflow 2, run the following Airflow CLI command:
gcloud composer environments run ENVIRONMENT_NAME \ --location LOCATION \ users create -- \ -u accounts.google.com:NUMERIC_USER_ID \ -e UNIQUE_ID \ -f UNIQUE_ID \ -l - -r Op --use-random-password
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.NUMERIC_USER_ID
with the user ID obtained in the previous step.UNIQUE_ID
with the identifier for the Airflow user. You can use any unique string.
After you create an Airflow user for a service account, a caller authenticated as the service account is recognized as a pre-registered user, and is logged into Airflow.
Scaling Airflow REST API component
Airflow REST API and Airflow UI endpoints are run within the component, i.e. Airflow Webserver. In case, you use REST API intensively, consider increasing CPU and memory parameters to adjust Airflow Webserver resources to the expected load.