Problem
When you try to create a Private IP Composer environment, operation fails with the following error in console:
Your environment could not complete its creation process because it could not successfully initialize the Airflow database. This can happen when the GKE cluster is unable to reach the SQL database over the network.
Environment
- New Private IP environments
- Error in airflow-sqlproxy logs in your project
I [DATETIME]Z [ALERT] 013/121825 (30) : backend 'sql_backend' has no server available!
- Cloud SQL instance in the tenant project is running
Solution
- Allow egress traffic on port 3306 and 3307 (MySQL, healthcheck) in the server IP range.
Cause
In newer private IP environments, the tenant project has two VM instances in Google App Engine each running a CloudSQL proxy. Within your project, the Google Kubernetes Engine cluster there is a HAProxy running in the airflow-sqlproxy pod to load balance connections between these CloudSQL proxies. Therefore, ports tcp/3306,3307 (MySQL, healthcheck) in the server IP range must be accessible from the Google Kubernetes Engine cluster.