Airflow command-line interface

Apache Airflow has a command-line interface (CLI) that you can use to perform tasks such as triggering and managing DAGs, getting information about DAG runs and tasks, adding and deleting connections and users.

About CLI syntax versions

Airflow provides two different syntaxes for CLI commands:

Before you begin

You must have enough permissions to use the gcloud command-line tool with Cloud Composer. For more information, see access control.

Run Airflow CLI commands

To run Airflow CLI commands in your environments, use gcloud:

gcloud composer environments run ENVIRONMENT_NAME \
    --location LOCATION \
    SUBCOMMAND \
    -- SUBCOMMAND_ARGUMENTS

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the Compute Engine region where the environment is located.
  • SUBCOMMAND with one of the supported Airflow CLI commands.
  • SUBCOMMAND_ARGUMENTS with arguments for the Airflow CLI command.

Subcommand arguments separator

Separate the arguments for the specified Airflow CLI command with --:

Airflow 1.10 CLI

For Airflow 2.0 CLI syntax:

  • Specify CLI commands as a subcommand.
  • Specify any arguments after a -- separator.
gcloud composer composer environments run example-environment \
    list_dags -- --report

Airflow 2.0 CLI

For Airflow 2.0 CLI syntax:

  • Specify compound CLI commands as a subcommand.
  • Specify any arguments for compound commands as subcommand arguments, after a -- separator.
gcloud beta composer composer environments run example-environment \
    dags list -- --output=json

Default location

Most gcloud composer commands require a location. You can specify the location with the --location flag, or by setting the default location.

Example

For example, to trigger a DAG named sample_quickstart with the ID 5077 in your Cloud Composer environment:

Airflow 1.10 CLI

gcloud composer environments run example-environment \
    --location us-central1 trigger_dag -- sample_quickstart \
    --run_id=5077

Airflow 2.0 CLI

gcloud beta composer composer environments run example-environment \
    --location us-central1 dags trigger -- sample_quickstart \
    --run-id=5077

Running commands on a private IP environment

To run Airflow CLI commands on a private IP environment, run them on a machine that can access the GKE cluster's master endpoint. Your options may vary depending on your private cluster configuration.

If public endpoint access is disabled in your cluster, you must run Airflow commands from a VM in the same VPC network. Create a VM in your VPC network to enable this path.

If public endpoint access and master authorized networks are enabled, you can also run Airflow commands from a machine with a public IP address that is added to master authorized networks. To enable access from your machine, adjust the configuration of the GKE cluster:

  1. Find the name of the GKE cluster that runs your Cloud Composer environment:

    gcloud composer environments describe \
    ENVIRONMENT_NAME \
    --location LOCATION \
    --format="value(config.gkeCluster)"

    Replace:

    • ENVIRONMENT_NAME with the name of the environment.
    • LOCATION with the Compute Engine region where the environment is located.

  2. Find the public IP of the machine where you want to run Airflow CLI commands on your environment. If you're using Cloud Shell, use dig to find the external IP address of your Cloud Shell:

    dig +short myip.opendns.com @resolver1.opendns.com

  3. Add the external address of your machine to your cluster's list of master authorized networks:

    gcloud container clusters update GKE_CLUSTER \
    --enable-master-authorized-networks \
    --master-authorized-networks EXISTING_AUTH_NETS,EXTERNAL_IP/32

    Replace:

    • EXISTING_AUTH_NETS with your existing list of master authorized networks. You can find your master authorized networks by running the following command:

      gcloud container clusters describe GKE_CLUSTER \
    --format "flattened(masterAuthorizedNetworksConfig.cidrBlocks[])"

    • EXTERNAL_IP with the external IP address of your Cloud Shell or the public IP of the machine where you want to run Airflow CLI commands.

  4. Run Airflow CLI commands on your private IP environment.

Supported Airflow CLI commands

Cloud Composer supports Airflow CLI commands based on the version of Airflow used in your environment: