Cloud Composer 1 | Cloud Composer 2
Apache Airflow has a command-line interface (CLI) that you can use to perform tasks such as triggering and managing DAGs, getting information about DAG runs and tasks, adding and deleting connections and users.
About CLI syntax versions
Airflow in Cloud Composer 2 uses Airflow 2 CLI syntax
Supported Airflow CLI commands
For a full list of supported Airflow CLI commands, see
gcloud composer environments run
reference.
Before you begin
You must have enough permissions to use the gcloud
command-line
tool with Cloud Composer. For more information, see
access control.
Run Airflow CLI commands
To run Airflow CLI commands in your environments, use gcloud
:
gcloud composer environments run ENVIRONMENT_NAME \
--location LOCATION \
SUBCOMMAND \
-- SUBCOMMAND_ARGUMENTS
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.SUBCOMMAND
with one of the supported Airflow CLI commands.SUBCOMMAND_ARGUMENTS
with arguments for the Airflow CLI command.
Subcommand arguments separator
Separate the arguments for the specified Airflow CLI command with --
:
Airflow 2
For Airflow 2 CLI syntax:
- Specify compound CLI commands as a subcommand.
- Specify any arguments for compound commands as subcommand arguments,
after a
--
separator.
gcloud composer environments run example-environment \
dags list -- --output=json
Airflow 1
Cloud Composer 2 supports only Airflow 2.
Default location
Most gcloud composer
commands require a location. You can specify the
location with the --location
flag, or by
setting the default location.
Example
For example, to trigger a DAG named sample_quickstart
with the ID 5077
in
your Cloud Composer environment:
Airflow 2
gcloud composer environments run example-environment \
--location us-central1 dags trigger -- sample_quickstart \
--run-id=5077
Airflow 1
Cloud Composer 2 supports only Airflow 2.
Running commands on a private IP environment
To run Airflow CLI commands on a Private IP environment, run them on a machine that can access the GKE cluster's control plane endpoint. Your options may vary depending on your private cluster configuration.
If public endpoint access is disabled in your environment's cluster, you must run Airflow commands from a VM in the same VPC network. Create a VM in your VPC network to enable this path.
If public endpoint access is enabled in your environment's cluster, you can also run Airflow commands from a machine with a public IP address that is added to authorized networks. To enable access from your machine, add the external address of your machine to your environment's list of authorized networks.
Troubleshooting
No connectivity to the cluster control plane
When running gcloud composer environments run
or kubectl
commands, you might encounter the following error:
Get "https://<IP Address>/api?timeout=32s": dial tcp <IP Address>:443: i/o timeout"
Symptom: This error message indicates that there is no network connectivity from a computer where you run these commands.
Solution: Follow the guidelines presented in the
Running commands on a private IP environment
section or use the instructions available in the
kubectl
command times out section.