Enabling and disabling DAG serialization

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

Follow the instructions on this page to enable or disable DAG serialization.

In a Cloud Composer environment, both the Airflow scheduler and the Airflow web server continuously process DAGs. You can improve the reliability and performance of the Airflow web server by enabling DAG serialization.

Enabling DAG serialization forces the scheduler to process DAG files before they are sent to the web server. The web server does not process DAGs. Instead, it reads the serialized DAGs from the Airflow database. Serializing DAGs in this way reduces the CPU and memory usage of the web server, especially when processing a large number of DAGs.

Before you begin

Note: Starting from Cloud Composer version 1.15.0, DAG serialization is enabled by default. You can enable and disable DAG serialization in new and existing environments by overriding Airflow configuration options, as described in the following sections.

DAG serialization affects the following Cloud Composer features:

• DAG serialization cannot be enabled at the same time as asynchronous DAG loading.
• Enabling DAG serialization disables all Airflow web server plugins for Cloud Composer. This does not impact scheduler or worker plugins, including Airflow operators, sensors, and so on.

Enabling DAG serialization

You can enable DAG serialization for an existing environment or when creating a new environment.

Note: Do not apply these settings to environments running Airflow 2.x. The DAG serialization and storing DAG code in Airflow Metadata Database is by default turned on in Airflow 2.x and you cannot switch it on or off.

To enable DAG serialization, override the following Airflow configuration options:

Section	Key	Value
core	store_serialized_dags	True
core	store_dag_code	True
core	min_serialized_dag_update_interval	30
scheduler	dag_dir_list_interval	30

The min_serialized_dag_update_interval Airflow configuration option controls how frequently serialized DAGs are updated in the database. The dag_dir_list_interval option controls how frequently removed DAGs are deleted from the Airflow database. A high update frequency can negatively impact performance. We recommend setting min_serialized_dag_update_interval and dag_dir_list_interval to 30 seconds.

Disabling DAG serialization

You can enable DAG serialization for an existing environment or when creating a new environment.

To disable DAG serialization, override the following Airflow configuration options:

Section	Key	Value
core	store_serialized_dags	False
core	store_dag_code	False