This page explains how to create a Cloud Composer environment and override default Airflow environment settings during the creation process.
A Cloud Composer environment runs the Apache Airflow software. When creating a new environment in a Google Cloud Platform (GCP) project, you can specify several parameters, such as the Compute Engine machine type or the number of nodes in the cluster.
Before you begin
- You must have the appropriate Cloud Identity and Access Management (Cloud IAM)
permissions to create Cloud Composer environments:
environments.createFor more information, see Cloud Composer Access Control.
- After you create a new environment, it takes approximately 25 minutes to deploy the Airflow web interface and complete the environment creation process.
- When using
gcloudcommand-line tool to create an environment, you must either configure the Cloud Composer location before running the
gcloudcommand, or you must use the
--locationflag with the
Creating a new environment
To create a new environment:
- Open the Create Environment page in the Google Cloud Platform Console.
- Enter a name for your environment. The name must start with a lowercase letter followed by up to 62 lowercase letters, numbers, or hyphens, and cannot end with a hyphen.
Under Node configuration, specify the settings for the Kubernetes Engine cluster. If you do not specify a setting, the default is used.
Setting Description Node count The number of Kubernetes Engine nodes used to run the environment. The default is 3 nodes. Location (Required) The Compute Engine region where the environment is created. Zone The Compute Engine zone where the virtual machine instances that run Apache Airflow are created. A random zone within the location is selected if unspecified. Machine type The Compute Engine machine type used for cluster instances. The machine type determines the number of CPUs and the amount of memory for your environment. Cloud Composer supports Compute Engine standard machine types.
The default machine type is n1-standard-1.
Disk size The disk size in GB used for the node VM instances. The minimum size is 20 GB. The default size is 100 GB. OAuth Scopes The set of Google API scopes made available on all node VM instances. The default is https://www.googleapis.com/auth/cloud-platform. Network ID The Virtual Private Cloud network ID that is used for machine communications. The network ID is required to specify a subnetwork. Shared Virtual Private Cloud is not currently supported.
The default network is used if unspecified.
Subnetwork ID The Compute Engine subnetwork ID that is used for machine communications. Service account The Google Cloud Platform service account to be used by the node VM instances. You cannot change the service account after the environment is created. The default Compute Engine service account is used if unspecified. Tags The list of instance tags applied to all the node VM instances. Tags are used to identify valid sources or targets for network firewalls. Each tag within the list must comply with RFC 1035.
(Optional) To change or override the default values in the Airflow configuration file (
airflow.cfg), click Add Airflow configuration property.
(Optional) To configure environment variables, click Add environment variable.
(Optional) To add a label, click Add labels. Label keys and label values can only contain letters, numbers, dashes, and underscores. Label keys must start with a letter or number.
To create a Cloud Composer environment, enter the following commands, replacing the VARIABLES with appropriate values:
gcloud beta composer environments create ENVIRONMENT_NAME \ --location LOCATION \ OTHER_ARGUMENTS ...
For example, to create an environment in the
us-central1 region that uses
n1-standard-2 machine type, enter the following command:
gcloud beta composer environments create test_environment \ --location us-central1 \ --zone us-central1-f \ --machine-type n1-standard-2 \ --labels env=beta \
gcloud beta composer environments create --help for
command parameter information, or see
Environment Configuration Parameters.
Configuring email notifications
To receive notifications, configure your environment variables to send email through the SendGrid email service.
If you haven't already, sign up with SendGrid via the Google Cloud Platform Console and create an API key. As a Google Cloud Platform developer, you can start with 12,000 free emails per month.
Open the Create Environment page in the Google Cloud Platform Console.
Under Node configuration, click Add environment variable.
Enter the following environment variables:
The address marked as the email "from" address. For example:
Your SendGrid API key.
Overriding Airflow configurations when creating an environment
During the Cloud Composer environment creation process, you can override Apache Airflow configurations.
- Open the Create Environment page.
Under Airflow configuration overrides, click Add Airflow configuration override.
Enter the Section, Key, and new Value for the configuration.
When you create an environment with the
gcloud command-line tool, you can use
Follow the flag with a comma-delimited list of
section names, property names, and values to override. Format the
list as follows:
gcloud beta composer environments create ENVIRONMENT_NAME \ --location LOCATION \ --airflow-configs=core-load_example=True,webserver-dag_orientation=TB
The command terminates when the operation is finished. To avoid
waiting, use the
To override Airflow properties during the creation of the Cloud Composer environment with the
Cloud Composer REST API,
fill in the
field when constructing the