Testing DAGs (workflows)

Before deploying DAGs to production, you can execute Airflow CLI sub-commands to parse DAG code in the same context under which the DAG is executed.

Note: Because Apache Airflow does not provide strong DAG isolation, we recommend that you maintain separate production and test environments to prevent DAG interference.

Testing during DAG creation

You can run a single task instance locally and view the log output. Viewing the output enables you to check for syntax and task errors. Testing locally does not check dependencies or communicate status to the database.

We recommend that you put the DAGs in a data/test folder in your test environment.

Checking for PyPI package errors

Because PyPI dependencies might cause conflicts with dependencies that Airflow requires, we recommend that you install your desired Python packages locally in an Airflow worker container and test the package.

1. Determine the Cloud Composer environment's GKE cluster.

2. Connect to the GKE cluster.

3. View and choose an Airflow worker pod.

   kubectl get pods

   Look for a pod with a name like airflow-worker-1a2b3c-x0yz.

4. Connect to a remote shell in an Airflow worker container.

   kubectl exec -it airflow-worker-1a2b3c-x0yz -c airflow-worker -- /bin/bash

   While connected to the remote shell, your command prompt shows the name of the Airflow worker pod, such as airflow-worker-1a2b3c-x0yz:.

5. For the version of Python running in your environment, install the Python package in the Airflow worker container, such as:

   sudo python2 -m pip install "[PACKAGE]"

6. Test for compatibility in the Airflow worker container.

   • Check for syntax errors.

     airflow list_dags

   • Render the template.

     airflow test --dry_run [DAG_ID]

   • Check for task errors.

     airflow test [DAG_ID] [TASK_ID] [EXECUTION_DATE]

     Note: To avoid distributing tasks to the worker nodes that don't have the PyPI package you're testing, do not use the trigger_dag command.

7. Uninstall the Python package from the Airflow worker container, such as:

   sudo python2 -m pip uninstall "[PACKAGE]"

Checking for syntax errors

1. In the Cloud Storage bucket for your environment, create a test directory.

2. To check for syntax errors, enter the following gcloud command:

   gcloud composer environments run ENVIRONMENT_NAME \
    --location LOCATION \
    list_dags -- -sd /home/airflow/gcs/data/test

   where:

   • ENVIRONMENT_NAME is the name of the environment.
   • LOCATION is the Compute Engine region where the environment is located.

   For example:

   gcloud composer environments run \
    test-environment --location us-central1 \
    list_dags -- -sd /home/airflow/gcs/data/test

Checking for task errors

To check for task-specific errors, enter the following gcloud command:

gcloud composer environments run ENVIRONMENT_NAME \
  --location LOCATION \
  test -- -sd /home/airflow/gcs/data/test DAG_ID \
  TASK_ID DAG_EXECUTION_DATE

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.
• DAG_ID is the ID of the DAG.
• TASK_ID is the ID of the task.
• DAG_EXECUTION_DATE is the execution date of the DAG. This date is used for templating purposes. Regardless of the date you specify here, the DAG runs immediately.

For example:

gcloud composer environments run test-environment --location us-central1 \
        -- -sd /home/airflow/gcs/data/test-dags hello_world print_date 2018-09-03

Updating and testing a deployed DAG

To test updates to your DAGs in your test environment:

1. Copy the deployed DAG that you want to update to data/test.
2. Update the DAG.
3. Test the DAG.
   1. Check for syntax errors.
   2. Check for task-specific errors.
4. Make sure the DAG runs successfully.
5. Turn off the DAG in your test environment.
   1. Go to the Airflow UI > DAGs page.
   2. If the DAG you're modifying runs constantly, turn off the DAG.
   3. To expedite outstanding tasks, click the task and Mark Success.
6. Deploy the DAG to your production environment.
   1. Turn off the DAG in your production environment.
   2. Upload the updated DAG to the dags/ folder in your production environment.

FAQs for testing workflows

How do I isolate DAG runs in my production and test environments?

For example, Airflow has a global repository of source code in the dags/ folder that all DAG runs share. I want to update source code in production or test without interfering with running DAGS by changing the code mid way.

Airflow does not provide strong DAG isolation. We recommend that you maintain separate production and test Cloud Composer environments to prevent your test DAGs from interfering with your production DAGs.

How do I avoid DAG interference when I run integration tests from different GitHub branches?

Use unique task names to prevent interference. For example, you can prefix your task IDs with the branch name.

What is a best practice for integration testing with Airflow?

We recommend that you use a dedicated environment for integration testing with Airflow. One way to signal the DAG run success is to write into a file in a Cloud Storage folder and check the content in your own integration test cases.

How do I collaborate efficiently with other DAG contributors?

Each contributor can have a subdirectory in the data/ or dags/folder for development. DAGs added to subdirectories won’t be picked up automatically by the scheduler or webserver.

DAG contributors can create manual DAG runs by using the gcloud composer environments run command and the trigger_dag and test sub-commands with the --subdir flag to specify the contributor's development directory.

For example:

gcloud composer environments run test-environment-name \
         trigger_dag -- dag-id --subdir /home/airflow/gcs/dags/alice_dev

How do I keep my deployment and production environments in sync?

• To manage access:
  • For authentication, use service accounts.
  • For access control, use Cloud Identity and Access Management and Cloud Composer roles and permissions.
• To deploy from development to production:
  • Ensure consistent configuration, such as environment variables and PyPI packages.
  • Ensure consistent DAG arguments. To avoid hard coding, we recommend that you use Airflow macros and variables, for example:

    gcloud composer environments run test-environment-name \
                 variables -- --set DATA_ENDPOINT_KEY DATA_ENDPOINT_VALUE

What's next

• Troubleshooting DAGs
• Adding and Updating DAGs