Orchestrate Airflow DAGs

Preview

This product is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products are available "as is" and might have limited support. For more information, see the launch stage descriptions.

To request support or provide feedback for this feature, email [e-mail].

This document describes how to orchestrate Airflow directed acyclic graphs (DAGs) from Cloud Composer 3 (Preview) on the Orchestration page in BigQuery, including how to trigger DAGs manually, and how to view the history and logs of past DAG runs.

About Airflow DAGs in BigQuery

The Orchestration page in BigQuery provides tools to orchestrate Airflow DAGs that run in your Cloud Composer 3 environments.

Airflow DAGs that you orchestrate in BigQuery are executed in one or more Cloud Composer environments in your project. The Orchestration page in BigQuery combines information for all Airflow DAGs in your project.

During a DAG run, Airflow schedules and executes individual tasks that make up a DAG in a sequence defined by the DAG. On the Orchestration page in BigQuery, you can view statuses of past DAG runs, explore detailed logs of all DAG runs and all tasks from these DAG runs, and view details about DAGs.

Note: You can't manage Cloud Composer environments in BigQuery. To manage environments, for example, to create an environment, install dependencies for your DAG files, upload, delete, or change individual DAGs, you use Cloud Composer.

To learn more about Airflow's core concepts such as Airflow DAGs, DAG runs, tasks, or operators, see the Core Concepts page in the Airflow documentation.

To learn more about Cloud Composer environments, see the Cloud Composer 3 overview page in the Cloud Composer documentation.

Before you begin

1. Enable the Cloud Composer API.

   Enable the API

2. Make sure that your Google Cloud project has at least one Cloud Composer 3 environment, with at least one already uploaded DAG file:
• To get started with Airflow DAGs, follow the instructions in the Run an Apache Airflow DAG in Cloud Composer 3 guide. As a part of this guide, you create a Cloud Composer 3 environment with the default configuration, upload a DAG to it, and check that Airflow runs it.
• For detailed instructions to upload an Airflow DAG to a Cloud Composer 3 environment, see Add and update DAGs.
• For detailed instructions to create a Cloud Composer 3 environment, see Create Cloud Composer environments.

Required permissions

To get the permissions that you need to orchestrate Airflow DAGs, ask your administrator to grant you the following IAM roles on the project:

• To view Airflow DAGs and their details: Environment and Storage Object Viewer (roles/composer.environmentAndStorageObjectViewer)
• To trigger and pause Airflow DAGs: Environment and Storage Object User (roles/composer.environmentAndStorageObjectUser)

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to orchestrate Airflow DAGs. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to orchestrate Airflow DAGs:

• To view Airflow DAGs and their details: composers.dags.list, composer.environments.list
• To trigger and pause Airflow DAGs: composers.dags.list, composer.environments.list, composer.dags.execute

You might also be able to get these permissions with custom roles or other predefined roles.

For more information about Cloud Composer 3 IAM, see Access control with IAM in Cloud Composer documentation.

Manually trigger a DAG

When you manually trigger an Airflow DAG, Airflow runs the DAG once, independently from the schedule specified for the DAG.

To manually trigger a selected Airflow DAG, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Do either of the following:

   • Click the name of the selected DAG, and then on the DAG details page, click Trigger DAG.

   • In the row that contains the selected DAG, click View actions in the Actions column, and then click Trigger DAG.

View DAG run logs and details

To view details of a selected Airflow DAG, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Click the name of the selected DAG.

3. On the DAG details page, select the Details tab.

4. To view past DAG runs, select the Runs tab.

   1. Optional: The Runs tab displays DAG runs from the last 10 days by default. To filter DAG runs by a different time range, in the 10 days drop-down menu, select a time range, and then click OK.

   2. Optional: To display additional columns with DAG run details in the list of all DAG runs, click Column display options, and then select columns and click OK.

   3. To view details and logs for a selected DAG run, select a DAG run.

5. To view a visualization of the DAG with task dependencies, select the Diagram tab.

   1. To view task details, select a task on the diagram.

6. To view the source code of the DAG, select the Code tab.

7. Optional: To refresh the displayed data, click Refresh.

View all DAGs

To view Airflow DAGs from all Cloud Composer 3 environments in your Google Cloud project, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Optional: To display additional columns with DAG details, click Column display options, and then select columns and click OK.

Pause a DAG

To pause a selected Airflow DAG, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Do either of the following:

   • Click the name of the selected DAG, and then on the DAG details page, click Pause DAG.

   • In the row that contains the selected DAG, click View actions in the Actions column, and then click Pause DAG.

Troubleshooting

For instructions to troubleshoot Airflow DAGs, see Troubleshooting Airflow DAGs in Cloud Composer documentation.

What's next

• Learn more about writing Airflow DAGs.
• Learn more about Airflow in Cloud Composer 3.