Blocked Airflow Configurations

This page lists the Apache Airflow configuration values associated with a Cloud Composer environment image version that are blocked and can't be overridden.

Section and Configuration Notes
api-* The API config section is blocked.
celery-* Cloud Composer configures Airflow to use Celery executor. In composer-0.5.x-airflow-1.9.0, the Celery config section is blocked.

In composer-1.0.0-airflow-1.9.0, the following celery properties are blocked: celery-celery_app_name, celery-worker_log_server_port, celery-broker_url, celery-celery_result_backend, celery-default_queue, celery-celery_config_options.

In composer-1.3.0-airflow-1.10.0, the following celery properties are blocked: celery-celery_app_name, celery-worker_log_server_port, celery-broker_url, celery-celery_result_backend, celery-result_backend, celery-default_queue, celery-celery_config_options.

In composer-1.4.2-airflow-1.10.0, the following celery properties are blocked: celery-celery_app_name, celery-worker_log_server_port, celery-broker_url, celery-celery_result_backend, celery-result_backend, celery-default_queue.

Worker pods might require a restart for celery-related configurations to take effect.

cli-* The CLI config section is blocked.
core-airflow_home
core-base_log_folder
core-dag_concurrency DAG concurrency automatically scales to the number of nodes. Since composer-1.1.1-airflow-1.9.0, this property is updatable.
core-dags_folder To allow Cloud Composer to upload DAGs, the dags folder must remain in the Google Cloud Storage bucket.
core-donot_pickle Pickling poses security risks and has been disabled.
core-enable_xcom_pickling Pickling poses security risks and has been disabled.
core-encrypt_s3_logs Cannot be changed. Required for Cloud Storage logging.
core-executor Cloud Composer configures Airflow to use the Celery executor.
core-fernet_key Not supported yet in Cloud Composer as it may cause Airflow database encryption readback error.
core-logging_config_class Overriding has no effect. The logs folder must remain in the Cloud Storage bucket.
core-max_active_runs_per_dag Max active runs per DAG automatically scales with the number of nodes. Since composer-1.1.1-airflow-1.9.0, this property is updatable.
core-parallelism Parallelism automatically scales to the number of nodes. Since composer-1.1.1-airflow-1.9.0, this property is updatable.
core-plugins_folder To be able to upload plugins through Cloud Composer, the dags folder must remain in the Cloud Storage bucket.
core-remote_base_log_folder The logs folder must remain in the Cloud Storage bucket.
core-remote_log_conn_id Cannot be changed. Required for Cloud Storage logging.
core-sql_alchemy_conn Changing sql alchemy settings can break the connection to the database.
core-sql_alchemy_pool_recycle Changing sql alchemy settings can break the connection to the database.
core-sql_alchemy_pool_size Changing sql alchemy settings can break the connection to the database.
core-task_log_reader
smtp-smtp_password
scheduler-authenticate Cloud Composer configures the Airflow scheduler. The following scheduler properties are updatable: catchup_by_default, dag_dir_list_interval, max_tis_per_query, min_file_process_interval, scheduler_zombie_task_threshold, and all properties starting with statsd (statsd_*).
scheduler-child_process_log_directory
scheduler-job_heartbeat_sec
scheduler-max_threads Since composer-1.1.1-airflow-1.9.0, this property is updatable.
scheduler-print_stats_interval
scheduler-run_duration
scheduler-scheduler_heartbeat_sec
webserver-access_logfile Cloud Composer configures the Airflow webserver. The following webserver properties are updatable: dag_default_view, dag_orientation, demo_mode, expose_config, hide_paused_dags_by_default, and page_size.
webserver-authenticate
webserver-base_url
webserver-error_logfile
webserver-filter_by_owner
webserver-log_fetch_timeout_sec
webserver-owner_mode
webserver-secret_key
webserver-web_server_host
webserver-web_server_port
webserver-web_server_ssl_cert
webserver-web_server_ssl_key
webserver-web_server_worker_timeout
webserver-worker_refresh_batch_size
webserver-worker_refresh_interval Since composer-1.7.1-airflow-1.10.2, this property is updatable. The recommended value is 3600 when webserver-async_dagbag_loader is True.
webserver-workers Since composer-1.7.2-airflow-1.10.2, this property is updatable. If your environment contains approximately 1000 DAG files or more, the recommended value is 1.

What's next

Var denne siden nyttig? Si fra hva du synes:

Send tilbakemelding om ...