Composer restart airflow-worker by scaling the Deployment to zero

Problem

In Composer 1 airflow-worker pods can sometimes become stuck or unresponsive, but we do not want to rebuild the entire environment using a new PyPI package or environment variable.

Environment

  • Composer version 1

Solution

  1. Scale down the replicas for the airflow-worker pods to zero.
  2. Wait for pods to be deleted.
  3. Scale replicas back to desired number.

Cause

The airflow-worker can sometimes enter an unrecoverable state due to issues on the Google Kubernetes Engine node that they are scheduled on.