Problem
The Cloud Composer Database Health metrics in the monitoring dashboard constantly show a red status.
The airflow-monitoring pod shows errors when trying to connect to a database: every minute there is an Access Denied error in Cloud SQL logs, that indicates the Airflow monitoring is trying to check if the connection works every minute.
Environment
- Cloud Composer
Solution
A red Database health timeline shows a poor connection to Composer Cloud SQL instance, and the liveness probe on the sql-proxy pod is continuously failing.
Check if the SQL password is correct
If you are using the same password in both a scheduler or worker and the SQL proxy:- Use ssh to connect to any worker, or scheduler, and use MySQL to connect to host airflow-sqlproxy-service.default as a root user using a password (steps are mentioned in points below).
- Use ssh to connect to an active airflow-sqlproxy pod and perform the following checks:
mysql -uroot -p"${SQL_PASSWORD}" -hairflow-sqlproxy-service.default
mysql -uroot -p"${SQL_PASSWORD}" -h127.0.0.1MySQL is preinstalled on the worker/scheduler pods, on the SQL proxy the customer may need to run:
apk update && apk add mysql-client
On any of the above pods, they should be able to use SQL password from the environment variable (${SQL_PASSWORD}) if they are using the same in both a scheduler/worker and the SQL proxy.
- If so, update the k8s YAML config with the correct password and restart the SQL proxy pod.
- Connect using ssh to airflow-monitoring and try to connect to the database with the right SQL password