Apache Airflow includes a command-line interface (CLI) that you can use to manage Airflow environments. The CLI is useful for tasks such as managing workflows, changing the Airflow environment, and obtaining log information.
Cloud Composer simplifies Airflow CLI commands with the Google Cloud SDK.
Once installed, you can use the
gcloud command line tool to run
gcloud composer environments run command to execute
Airflow CLI sub-commands.
Before you begin
The following permissions are required to access the
gcloud command-line tool (Airflow CLI)
in the Cloud Composer environment:
For more information, see Cloud Composer Access Control.
Running Airflow CLI commands
You can run Airflow CLI commands on a Cloud Composer environment by using the following command:
gcloud composer environments run ENVIRONMENT_NAME \ --location LOCATION SUBCOMMAND
gcloud composer commands require a location. You can specify the location by
--location flag or by setting the default location.
For example, to trigger a DAG named
sample_quickstart with the ID
5077 in your Cloud Composer environment:
gcloud composer environments run test-environment \ --location us-central1 trigger_dag -- sample_quickstart \ --run_id=5077
For example, to check for syntax errors in DAGs in a
gcloud composer environments run test-environment \ --location us-central1 \ list_dags -- -sd /home/airflow/gcs/data/test
Running commands on a private IP environment
To run Airflow CLI commands on a private IP environment, you must run them on a machine that can access the GKE cluster's master endpoint. Your options may vary depending on your private cluster configuration.
If public endpoint access is disabled in your cluster, you must run Airflow commands from a VM in the same VPC network. Create a VM in your VPC network to enable this path.
If public endpoint access and master authorized networks are enabled, you can also run Airflow commands from a machine with a public IP address that has been added to master authorized networks. To enable access from your machine, adjust the configuration of the GKE cluster:
Find the name of the GKE cluster that runs your Composer environment by using this command:
gcloud beta composer environments describe ENVIRONMENT_NAME \ --location LOCATION \ --format="value(config.gkeCluster)"
Find the public IP of the machine where you want to run Airflow CLI commands on your environment. If you're using Cloud Shell, use
digto find the external IP address of your Cloud Shell:
dig +short myip.opendns.com @resolver1.opendns.com
Add the external address of your machine to your cluster's list of master authorized networks:
gcloud container clusters update GKE_CLUSTER \ --enable-master-authorized-networks \ --master-authorized-networks EXISTING_AUTH_NETS,SHELL_IP/32
EXISTING_AUTH_NETS is your existing list of master authorized networks. You can find your master authorized networks by running the following command:
gcloud container clusters describe GKE_CLUSTER \ --format "flattened(masterAuthorizedNetworksConfig.cidrBlocks)"
SHELL_IP is the external IP address of your Cloud Shell.
You should now be able to run Airflow CLI commands on your private IP environment.
Supported Airflow commands
Cloud Composer supports the following commands for environments running Airflow 1.9.0 and newer:
Airflow 1.10.1 adds support for the following command:
Airflow 1.10.2 adds support for the following commands: