Setting Environment Variables

This page describes the requirements for setting environment variables in Cloud Composer.

When you create or update an environment, you can add environment variables, which Cloud Composer provides to the Apache Airflow scheduler, worker, and webserver processes.

For example, Cloud Composer uses the Apache Airflow sendgrid module for email notifications. To configure email notifications, you need to set the SENDGRID_API_KEY and SENDGRID_MAIL_FROM environment variables.

Environment variables are persistent. Once you specify an environment variable, Airflow keeps using it until you remove the variable from your environment.

Requirements

Environment variables must match the regular expression [a-zA-Z_][a-zA-Z0-9_]*.

Environment variables must not specify Apache Airflow configuration overrides. This means the variables cannot use the AIRFLOW__{SECTION}__{KEY} format or match the regular expression AIRFLOW__[A-Z0-9_]+__[A-Z0-9_]+.

Reserved names

The following names are reserved and cannot be used:

  • AIRFLOW_HOME
  • C_FORCE_ROOT
  • COMPOSER_ENVIRONMENT
  • COMPOSER_GKE_NAME
  • COMPOSER_GKE_ZONE
  • COMPOSER_LOCATION
  • COMPOSER_PYTHON_VERSION
  • CONTAINER_NAME
  • DAGS_FOLDER
  • GCP_PROJECT
  • GCS_BUCKET
  • GKE_CLUSTER_NAME
  • GOOGLE_APPLICATION_CREDENTIALS
  • SQL_DATABASE
  • SQL_INSTANCE
  • SQL_PASSWORD
  • SQL_PROJECT
  • SQL_REGION
  • SQL_USER

Before you begin

  • The following permissions are required to set environment variables:

    • composer.environments.create to set environment variables during environment creation

    • composer.environments.update to update environment variables

      For more information, see Cloud Composer Access Control.

  • Most gcloud composer commands require a location. You can specify the location by using the --location flag or by setting the default location.

Adding and updating environment variables

To set environment variables:

Console

To add an environment variable:

  1. Open the Create Environment page in the Google Cloud Console.

    Open the Create Environment page

  2. Under Node configuration, click Add environment variable.

  3. Enter the NAME and VALUE for the environment variables.

gcloud 

gcloud composer environments update ENVIRONMENT_NAME \
  --location LOCATION \
  --update-env-variables=KEY=VALUE,KEY=VALUE...

where:

  • ENVIRONMENT_NAME is the name of the environment.
  • LOCATION is the Compute Engine region where the environment is located.
  • KEY=VALUE specifies the environment variables to create or update.

For example: 

gcloud composer environments update test-environment 

   --location us-central1 

   --update-env-variables=SENDGRID_MAIL_FROM=updated-email@domain.com

API

To use the Cloud Composer REST API, issue an environments.patch.

Removing environment variables

To remove environment variables:

Console

  1. Open the Environments page.

    Open the Environments page

  2. On the List Environments page, click the name of the environment to view its details.
  3. Under Environment Variables, click Edit
  4. To the right of the variable value, click the trash can icon.

gcloud

To remove all environment variables: 

gcloud composer environments update ENVIRONMENT_NAME 

  --location LOCATION 

  --clear-env-variables

where:

  • ENVIRONMENT_NAME is the name of the environment
  • LOCATION is the Compute Engine region where the environment is located.

For example: 

gcloud composer environments update test-environment 

   --location us-central1 

   --clear-env-variables

To remove a list of environment variables: 

gcloud composer environments update ENVIRONMENT_NAME 

  --location LOCATION 

  --remove-env-variables=NAME,NAME,...
where:

  • ENVIRONMENT_NAME is the name of the environment,
  • LOCATION is the Compute Engine region where the environment is located.
  • NAME is the environment variable to remove.

API

To use the Cloud Composer REST API, issue an environments.patch.