Blocked Airflow Configurations

This page lists the Apache Airflow configuration values associated with a Cloud Composer environment image version that are blocked and can't be overridden.

Section and Configuration Notes
api-* The API config section is blocked.
celery-* Cloud Composer configures Airflow to use Celery executor. In composer-0.5.x-airflow-1.9.0, the Celery config section is blocked.

In composer-1.0.0-airflow-1.9.0, the following celery properties are blocked: celery-celery_app_name, celery-worker_log_server_port, celery-broker_url, celery-celery_result_backend, celery-default_queue, celery-celery_config_options.

In composer-1.3.0-airflow-1.10.0, the following celery properties are blocked: celery-celery_app_name, celery-worker_log_server_port, celery-broker_url, celery-celery_result_backend, celery-result_backend, celery-default_queue, celery-celery_config_options.

In composer-1.4.2-airflow-1.10.0, the following celery properties are blocked: celery-celery_app_name, celery-worker_log_server_port, celery-broker_url, celery-celery_result_backend, celery-result_backend, celery-default_queue.

Worker pods might require a restart for celery-related configurations to take effect.

cli-* The CLI config section is blocked.
core-airflow_home
core-base_log_folder
core-dag_concurrency DAG concurrency automatically scales to the number of nodes. Since composer-1.1.1-airflow-1.9.0, this property is updatable.
core-dags_folder To allow Cloud Composer to upload DAGs, the dags folder must remain in the Google Cloud Storage bucket.
core-donot_pickle Pickling poses security risks and has been disabled.
core-enable_xcom_pickling Pickling poses security risks and has been disabled.
core-encrypt_s3_logs Cannot be changed. Required for Cloud Storage logging.
core-executor Cloud Composer configures Airflow to use the Celery executor.
core-fernet_key Not supported yet in Cloud Composer as it may cause Airflow database encryption readback error.
core-logging_config_class Overriding has no effect. The logs folder must remain in the Cloud Storage bucket.
core-max_active_runs_per_dag Max active runs per DAG automatically scales with the number of nodes. Since composer-1.1.1-airflow-1.9.0, this property is updatable.
core-parallelism Parallelism automatically scales to the number of nodes. Since composer-1.1.1-airflow-1.9.0, this property is updatable.
core-plugins_folder To be able to upload plugins through Cloud Composer, the dags folder must remain in the Cloud Storage bucket.
core-remote_base_log_folder The logs folder must remain in the Cloud Storage bucket.
core-remote_log_conn_id Cannot be changed. Required for Cloud Storage logging.
core-sql_alchemy_conn Changing sql alchemy settings can break the connection to the database.
core-sql_alchemy_pool_recycle Changing sql alchemy settings can break the connection to the database.
core-sql_alchemy_pool_size Changing sql alchemy settings can break the connection to the database.
core-task_log_reader
smtp-smtp_password
scheduler-authenticate Cloud Composer configures the Airflow scheduler. The following scheduler properties are updatable: catchup_by_default, dag_dir_list_interval, max_tis_per_query, min_file_process_interval, scheduler_zombie_task_threshold, and all properties starting with statsd (statsd_*).
scheduler-child_process_log_directory
scheduler-job_heartbeat_sec
scheduler-max_threads Since composer-1.1.1-airflow-1.9.0, this property is updatable.
scheduler-print_stats_interval
scheduler-run_duration
scheduler-scheduler_heartbeat_sec
webserver-access_logfile Cloud Composer configures the Airflow webserver. The following webserver properties are updatable: dag_default_view, dag_orientation, demo_mode, expose_config, hide_paused_dags_by_default, and page_size.
webserver-authenticate
webserver-base_url
webserver-error_logfile
webserver-filter_by_owner
webserver-log_fetch_timeout_sec
webserver-owner_mode
webserver-secret_key
webserver-web_server_host
webserver-web_server_port
webserver-web_server_ssl_cert
webserver-web_server_ssl_key
webserver-web_server_worker_timeout
webserver-worker_refresh_batch_size
webserver-worker_refresh_interval Since composer-1.7.1-airflow-1.10.2, this property is updatable. The recommended value is 3600 when webserver-async_dagbag_loader is True.
webserver-workers Since composer-1.7.2-airflow-1.10.2, this property is updatable. If your environment contains approximately 1000 DAG files or more, the recommended value is 1.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Composer