Unable to run command in BashOperator

Problem

When you execute the gcloud composer environments run command in BashOperator toward a private IP Cloud Composer environment, you get the following error:

Unable to connect to the server: dial tcp ENDPOINT:443: i/o timeout

Environment

• Private IP Cloud Composer environments

Solution

1. Run the Airflow CLI directly through the airflow command instead of the gcloud composer environments run command. See the following example:
   • Before:

     gcloud composer environments run myenv trigger_dag -- some_dag --run_id=foo

   • After:

     airflow trigger_dag some_dag --run_id=foo

Cause

The gcloud composer environments run command in BashOperator uses a temporary configuration (kubeconfig). Hence, it tries to access the Google Kubernetes Engine cluster's public endpoint rather than its private endpoint and ends with failing access to the public endpoint. Note that if you run the gcloud command from the Google Kubernetes Engine cluster directly (not inside of BashOperator), it would work without error.