Problem
Your Cloud Composer tasks remain in Unknown state and Running tasks never finish.
Environment
- Cloud Composer < 2.0
- Cloud Tasks
Solution
Adjust the Airflow configurations, following the sizing guide. This sizing guide can be found at the bottom of this article, along with more background information.
Note: You must update the environment after overriding the airflow configurations.
Cause
There is a ramp up in terms of the number of tasks queued/scheduled/completed, while you only ran 18 jobs at once. The airflow worker workload also has a very high usage of memory while the actual node (Google Kubernetes Engine node) is underutilized. This means their airflow configurations need to be adjusted to optimize performance. This would reduce the number of tasks being queued and allow the tasks to get scheduled and executed.