Overriding Airflow configurations

Follow the instructions on this page to override Airflow configuration options for new and existing environments.

When you create or update an environment, you can override Apache Airflow configuration options with different values. By doing so you can adjust the Airflow instance to your needs and requirements.

Overriding configurations for new environments

Console

  1. Open the Create Environment page.

    Open the Create Environment page

  2. Under Airflow configuration overrides, click Add Airflow configuration override.

  3. Enter the Section, Key, and new Value for the configuration.

For example:

Section Key Value
webserver dag_orientation RL

gcloud

To override Airflow configuration options when you create an environment:

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --airflow-configs=KEY=VALUE,KEY=VALUE,...

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the Compute Engine region where the environment is located.
  • KEY with the configuration section and the option name separated by a hyphen, for example, core-print_stats_interval.
  • VALUE with the corresponding value for an option.

For example:

gcloud composer environments create test-environment \
    --location us-central1 \
    --airflow-configs=core-load_example=True,webserver-dag_orientation=TB 

The command terminates when the operation is finished. To avoid waiting, use the --async flag. For additional examples, see the gcloud composer environments create reference page.

API

To override Airflow properties during the creation of the Cloud Composer environment with the Cloud Composer REST API:

  1. Construct the environments.create request.
  2. In this request, fill in the Environment resource's optional airflowConfigOverrides field.

Overriding configurations for existing environments

Console

  1. Open the Environments page.

    Open the Environments page

  2. In the list of environments, click your environment. The Environment details page opens.
  3. Navigate to Airflow configuration overrides tab.
  4. Click Edit.
  5. Enter the Section, Key, and new Value for the configuration.

For example:

Section Key Value
webserver dag_orientation RL

gcloud

To override Airflow configuration options for an existing environment:

gcloud composer environments update ENVIRONMENT_NAME \
  --location LOCATION \
  --update-airflow-configs=KEY=VALUE,KEY=VALUE,...

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the Compute Engine region where the environment is located.
  • KEY with the configuration section and the option name separated by a hyphen, for example, core-print_stats_interval.
  • VALUE with the corresponding value for an option.

For example:

gcloud composer environments update test-environment \
    --location us-central1 \
    --update-airflow-configs=core-load_example=True,webserver-dag_orientation=TB 

The command terminates when the operation is finished. To avoid waiting, use the --async flag. For additional examples, see the gcloud composer environments update reference page.

API

To override Airflow properties for an existing Cloud Composer environment with the Cloud Composer REST API:

  1. Construct the environments.patch request.
  2. In this request:

    • To replace all existing Airflow configuration overrides with the specified options, in the updateMask parameter, specify the config.softwareConfig.airflowConfigOverrides mask.
    • To override specific Airflow configuration options, in the updateMask parameter, specify the config.softwareConfig.airflowConfigOverrides. section-name mask.

What's next