Invoke Airflow REST API impersonating service account

Problem

While invoking the Airflow REST API, the connection fails with the following 401 Error.

Environment

  • Cloud Composer version 2.x.x 
  • Airflow version 2.x.x

Solution

You need a Cloud Composer environment up and running, if you do not have one then please follow our public documentation to create a Cloud Composer environment.

Step 1: Enable the stable Airflow REST API 
  1. The stable REST API is already enabled by default in Airflow 2. But if you are using Airflow 1 then please follow the steps mentioned in our public documentation to enable or disable the stable API.
Step 2: Create a Service Account for this process
  1. Navigate to Admin console and search for Service Account
  2. Create a new service account.
  3. Grant serviceAccountOpenIdTokenCreator role to this SA.
Step 3: After creating a SA, create a pre-registered Airflow User for your SA.
  1. Follow Step 1 and Step 2 from our public documentation.
  2. After you create an Airflow user for a service account, a caller authenticated as the service account is recognized as a pre-registered user, and is logged into Airflow.
Step 4: Activate your Cloud Shell to test our access with a GET request test
  1.  You will impersonate the SA into your cloud shell by running the following command: 
    gcloud config set auth/impersonate_service_account YOUR_SERVICE_ACCOUNT
  2. cURL your webserver URL to fetch the DAGs list: 
    curl WEB_SERVER_URL/api/v1/dags -H "Authorization: Bearer $(gcloud auth print-access-token)"
  3. To get your WEB_SERVER_URL, Navigate to the Composer interface in the Admin console and and copy the link address of the OPEN AIRFLOW UI button on the top bar.  
  4. You will get all your DAGs list as an output. Which means now you have accessed your Airflow data using a REST API call by impersonating a SA.
  5. You can unset your SA and get back to your user account: 
    gcloud config unset auth/impersonate_service_account