Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3
This page explains how to set environment variables in your Cloud Composer environment.
Cloud Composer uses the environment variables to alter the runtime behavior of Airflow or your DAGs.
Environment variables are persistent. Once you specify an environment variable, Airflow keeps using it until you remove the variable from your environment.
For example, Cloud Composer
uses the Apache Airflow Sendgrid module for
email notifications. To configure email notifications,
you need to set the SENDGRID_API_KEY
and SENDGRID_MAIL_FROM
environment
variables.
Name format
Environment variables must match the regular expression
[a-zA-Z_][a-zA-Z0-9_]*
.
Airflow configuration options
It is not possible to specify variables that change Apache Airflow configuration options.
The variables cannot use the AIRFLOW__{SECTION}__{KEY}
format or match the
regular expression AIRFLOW__[A-Z0-9_]+__[A-Z0-9_]+
. For example, it is not
possible to set AIRFLOW__WEBSERVER__DAG_DEFAULT_VIEW
.
Instead, you can override Airflow configuration options.
Reserved variables
The following variable names are reserved by Cloud Composer. It is not possible to create variables with these names.
AIRFLOW_DATABASE_VERSION
AIRFLOW_HOME
AIRFLOW_SRC_DIR
AIRFLOW_WEBSERVER
AUTO_GKE
CLOUDSDK_METRICS_ENVIRONMENT
CLOUD_LOGGING_ONLY
COMPOSER_ENVIRONMENT
COMPOSER_GKE_LOCATION
COMPOSER_GKE_NAME
COMPOSER_GKE_ZONE
COMPOSER_LOCATION
COMPOSER_OPERATION_UUID
COMPOSER_PYTHON_VERSION
COMPOSER_VERSION
CONTAINER_NAME
C_FORCE_ROOT
DAGS_FOLDER
GCP_PROJECT
GCP_TENANT_PROJECT
GCSFUSE_EXTRACTED
GCS_BUCKET
GKE_CLUSTER_NAME
GKE_IN_TENANT
GOOGLE_APPLICATION_CREDENTIALS
MAJOR_VERSION
MINOR_VERSION
PATH
PIP_DISABLE_PIP_VERSION_CHECK
PORT
PROJECT_ID
PYTHONPYCACHEPREFIX
PYTHONWARNINGS
SQL_DATABASE
SQL_HOST
SQL_INSTANCE
SQL_PASSWORD
SQL_PROJECT
SQL_REGION
SQL_USER
Set environment variables for new environments
You can specify environment variables when you create an environment. For more information, see Create environments.
Set environment variables for existing environments
Console
To set environment variables for an existing environment:
In Google Cloud console, go to the Environments page.
In the list of environments, click the name of your environment. The Environment details page opens.
Go to the Environment variables tab and click Edit.
Add, change, or delete environment variables for your environment:
- In the Name field, specify the variable name.
- In the Value field, specify the variable value.
- To add an extra variable, click Add environment variable.
- To delete a variable, hold the pointer over a variable, then click Delete item.
gcloud
Following arguments control environment variables:
--update-env-variables
specifies a comma-separated list of environment variables to create or update.--remove-env-variables
specifies a comma-separated list of environment variables to delete.--clear-env-variables
deletes all existing environment variables.
gcloud composer environments update \
ENVIRONMENT_NAME \
--location LOCATION \
--update-env-variables=NAME=VALUE
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.NAME
with the name of the variable that you want to create or update.VALUE
with the value for the variable.- To add more than one variable, separate the definitions with commas:
NAME=VALUE,NAME=VALUE,...
.
Example:
gcloud composer environments update \
example-environment \
--location us-central1 \
--update-env-variables=EXAMPLE_VARIABLE=True,ANOTHER_VARIABLE=test
API
Construct an environments.patch
API request.
In this request:
In the
updateMask
parameter, specify theconfig.softwareConfig.envVariables
mask to replace all existing variables with the specified variables. Variables that you do not specify are deleted.In the request body, specify variables and their values:
{ "config": { "softwareConfig": { "envVariables": { "VAR_NAME": "VAR_VALUE" } } } }
Replace:
VAR_NAME
with the name of the environment variable.VAR_VALUE
with the value of the environment variable.- To add more than one variable, add extra entries for variables
to
envVariables
.
Example:
// PATCH https://composer.googleapis.com/v1beta1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.softwareConfig.envVariables
{
"config": {
"softwareConfig": {
"envVariables": {
"EXAMPLE_VARIABLE": "True",
"ANOTHER_VARIABLE": "test"
}
}
}
}
Terraform
The env_variables
block in the software_config
block specifies
environment variables.
resource "google_composer_environment" "example" {
provider = google-beta
name = "ENVIRONMENT_NAME"
region = "LOCATION"
config {
software_config {
env_variables = {
VAR_NAME = "VAR_VALUE"
}
}
}
}
Replace:
ENVIRONMENT_NAME
with the name of the environment.LOCATION
with the region where the environment is located.VAR_NAME
with the name of the environment variable.VAR_VALUE
with the value of the environment variable.- To add more than one variable, add extra entries for variables
to
env_variables
.
Example:
resource "google_composer_environment" "example" {
provider = google-beta
name = "example-environment"
region = "us-central1"
config {
software_config {
env_variables = {
EXAMPLE_VARIABLE = "True"
ANOTHER_VARIABLE = "test"
}
}
}
}