Cloud Composer 1 is in the post-maintenance mode. Google does not release any further updates to Cloud Composer 1, including new versions of Airflow, bugfixes, and security updates. We recommend planning migration to Cloud Composer 2.

Clean up the Airflow database
Stay organized with collections Save and categorize content based on your preferences.

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page explains how to maintain the Airflow database in your environment.

Automatic database cleanup with a database retention policy

You can configure automatic database cleanup for your Cloud Composer 3 environment by setting a database retention policy. After you set up this policy, records older than a certain period are automatically removed from the Airflow database daily. For more information, see Configure database retention policy.

Deprecated cleanup procedures

Before the database retention policy became available in Cloud Composer, we recommended a different approach for automating the database cleanup, through a database cleanup DAG. This approach is obsolete in Cloud Composer 3. This DAG does redundant work and you can reduce the resource consumption by removing it and replacing it with a database retention policy.

Limits for database size

As the time goes, the Airflow database of your environment stores more and more data. This data includes information and logs related to past DAG runs, tasks, and other Airflow operations.

If the Airflow database size is more than 20 GB, then you can't upgrade your environment to a later version.
If the Airflow database size is more than 20 GB, it is not possible to create snapshots.

Maintain database performance

Airflow database performance issues can lead to overall DAG execution issues. Observe Database CPU and memory usage statistics. If CPU and memory utilization approaches the limits, then the database is overloaded and requires scaling. The amount of resources available to the Airflow database is controlled by the environment size property of your environment. To scale the database up change the environment size to a larger tier. Increasing the environment size increases the costs of your environment.
If you use the XCom mechanism to transfer files, make sure that you use it according to Airflow's guidelines. Transferring big files or a large number of files using XCom impacts Airflow database's performance and can lead to failures when loading snapshots or upgrading your environment. Consider using alternatives such as Cloud Storage to transfer large volumes of data.

Remove entries for unused DAGs

You can remove database entries for unused DAGs by removing DAGs from the Airflow UI.