Creating Environments

This page explains how to create a Cloud Composer environment and override default Airflow environment settings during the creation process.

A Cloud Composer environment runs the Apache Airflow software. When creating a new environment in a Google Cloud Platform (GCP) project, you can specify several parameters, such as the Compute Engine machine type or the number of nodes in the cluster.

Before you begin

  • You must have the appropriate Cloud Identity and Access Management (Cloud IAM) permissions to create Cloud Composer environments: environments.create For more information, see Cloud Composer Access Control.
  • After you create a new environment, it takes approximately 25 minutes to deploy the Airflow web interface and complete the environment creation process.
  • When using gcloud command-line tool to create an environment, you must either configure the Cloud Composer location before running the gcloud command, or you must use the --location flag with the gcloud command.

Creating a new environment

To create a new environment:

Console

  1. Open the Create Environment page in the Google Cloud Platform Console.

    Open the Create Environment page

  2. Enter a name for your environment.
    The name must start with a lowercase letter followed by up to 62 lowercase letters, numbers, or hyphens, and cannot end with a hyphen.
  3. Under Node configuration, specify the settings for the Kubernetes Engine cluster. If you do not specify a setting, the default is used.

    SettingDescription
    Node countThe number of Kubernetes Engine nodes used to run the environment. The default is 3 nodes.
    Location(Required) The Compute Engine region where the environment is created.
    ZoneThe Compute Engine zone where the virtual machine instances that run Apache Airflow are created. A random zone within the location is selected if unspecified.
    Machine typeThe Compute Engine machine type used for cluster instances. The machine type determines the number of CPUs and the amount of memory for your environment. Cloud Composer supports Compute Engine standard machine types.

    The default machine type is n1-standard-1.

    Disk sizeThe disk size in GB used for the node VM instances. The minimum size is 20 GB. The default size is 100 GB.
    OAuth ScopesThe set of Google API scopes made available on all node VM instances. The default is https://www.googleapis.com/auth/cloud-platform.
    Network IDThe Virtual Private Cloud network ID that is used for machine communications. The network ID is required to specify a subnetwork. Shared Virtual Private Cloud is not currently supported.

    The default network is used if unspecified.

    Subnetwork IDThe Compute Engine subnetwork ID that is used for machine communications.
    Service accountThe Google Cloud Platform service account to be used by the node VM instances. You cannot change the service account after the environment is created. The default Compute Engine service account is used if unspecified.
    TagsThe list of instance tags applied to all the node VM instances. Tags are used to identify valid sources or targets for network firewalls. Each tag within the list must comply with RFC 1035.
  4. (Optional) To change or override the default values in the Airflow configuration file (airflow.cfg), click Add Airflow configuration property.

  5. (Optional) To configure environment variables, click Add environment variable.

  6. (Optional) To add a label, click Add labels.
    Label keys and label values can only contain letters, numbers, dashes, and underscores. Label keys must start with a letter or number.

  7. Click Create.

gcloud

To create a Cloud Composer environment, enter the following commands, replacing the VARIABLES with appropriate values:

gcloud beta composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    OTHER_ARGUMENTS ... 

For example, to create an environment in the us-central1 region that uses the n1-standard-2 machine type, enter the following command:

gcloud beta composer environments create test_environment \
    --location us-central1 \
    --zone us-central1-f \
    --machine-type n1-standard-2 \
    --labels env=beta \ 

Use gcloud beta composer environments create --help for command parameter information, or see Environment Configuration Parameters.

API

To create a new Cloud Composer environment with the Cloud Composer REST API, construct an environments.create API request, filling in the Environment resource with your configuration information.

Configuring email notifications

To receive notifications, configure your environment variables to send email through the SendGrid email service.

  1. If you haven't already, sign up with SendGrid via the Google Cloud Platform Console and create an API key. As a Google Cloud Platform developer, you can start with 12,000 free emails per month.

  2. Open the Create Environment page in the Google Cloud Platform Console.

    Open the Create Environment page

  3. Under Node configuration, click Add environment variable.

  4. Enter the following environment variables:

    Name Value
    SENDGRID_MAIL_FROM The address marked as the email "from" address. For example: test-email@
    SENDGRID_API_KEY Your SendGrid API key.

Overriding Airflow configurations when creating an environment

During the Cloud Composer environment creation process, you can override Apache Airflow configurations.

Console

  1. Open the Create Environment page.

    Open the Create Environment page

  2. Under Airflow configuration overrides, click Add Airflow configuration override.

  3. Enter the Section, Key, and new Value for the configuration.

gcloud

When you create an environment with the gcloud command-line tool, you can use the --airflow-configs flag.

Follow the flag with a comma-delimited list of section names, property names, and values to override. Format the list as follows: section-property_name=value.

gcloud beta composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --airflow-configs=core-load_example=True,webserver-dag_orientation=TB

The command terminates when the operation is finished. To avoid waiting, use the --async flag.

API

To override Airflow properties during the creation of the Cloud Composer environment with the Cloud Composer REST API, fill in the Environment resource's optional airflowConfigOverrides field when constructing the environments.create request.

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Composer