Creating environments

This page explains how to create a Cloud Composer environment.

A Cloud Composer environment runs the Apache Airflow software. When creating a new environment in a Google Cloud (GCP) project, you can specify several parameters, such as the Compute Engine machine type or the number of nodes in the cluster.

Before you begin

Access control

  • During the environment creation, you specify a service account that runs the environment's GKE nodes. By default, Cloud Composer environments run using the Google-managed default Compute Engine service account. We recommend you to run your environments with a user-managed service account that has a role specific for Cloud Composer service accounts.

  • If your user-managed service account must access other resources in your Google Cloud project during task execution, you can grant the service account the required roles. As an alternative, you can provide the relevant credentials as an Airflow connection, then reference the connection in the operator.

  • Google-managed service accounts are displayed in your project's IAM policy and GCP Console. For example, service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com is the name of a special Google-managed service account, called a service agent, used by the Cloud Composer service to manage (create, update, and delete) Cloud Composer environments in your project.

  • If you want a Cloud Composer environment in one project to run as a user-managed service account in a different project, you must configure the user-managed service account to work across projects.

    As part of this process, you must grant the Service Account Token Creator role to the following Google-managed service accounts on your user-managed service account:

    • Compute Engine default service account (PROJECT_NUMBER-compute@developer.gserviceaccount.com)
    • Cloud Composer Service Agent (service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com)
  • To deploy Cloud Composer environments inside a security perimeter, see Configuring VPC Service Controls. When used with Cloud Composer, VPC Service Controls have several known limitations.

Creating a new environment

To create a Cloud Composer environment:

Console

  1. Open the Create Environment page in the Google Cloud Console.

    Open the Create Environment page

  2. Enter a name for your environment.

    The name must start with a lowercase letter followed by up to 62 lowercase letters, numbers, or hyphens, and cannot end with a hyphen. The environment name is used to create subcomponents for the environment, so you must provide a name that is also valid as a Cloud Storage bucket name. See Bucket naming guidelines for a list of restrictions.

  3. Under Node configuration, specify the settings for nodes in the Google Kubernetes Engine cluster. If you do not specify a setting, the default is used.

    Setting Description
    Node count The number of Google Kubernetes Engine nodes used to run the environment. The default is 3 nodes. Once you specify the number of nodes, it stays fixed until you update your environment.
    Location (Required) The Compute Engine region where the environment is created.
    Zone suffix The Compute Engine zone where the virtual machine instances that run Apache Airflow are created. A random zone within the location is selected if unspecified.
    Machine type The Compute Engine machine type used for cluster instances. The machine type determines the number of CPUs and the amount of memory for your environment. The default machine type is n1-standard-1.
    Disk size The disk size in GB used for the node VM instances. The minimum size is 20 GB. The default size is 100 GB.
    OAuth Scopes The set of Google API scopes made available on all node VM instances. The default is https://www.googleapis.com/auth/cloud-platform and must be included in the list of specified scopes.
    Service account The service account that runs the environment's GKE nodes. The default Compute Engine service account is used if unspecified.
    Tags The list of instance tags applied to all the node VM instances. Tags are used to identify valid sources or targets for network firewalls. Each tag within the list must comply with RFC 1035.
    Image version The Cloud Composer version to use for your environment (includes Cloud Composer and Airflow version). For default version information, see Versions list.
    Python version The Python version to use for your environment. Supported versions are Python 2 and Python 3. The default version is 3.
  4. Under Cloud SQL configuration, specify the settings for Cloud SQL instance running the Airflow database. If you do not specify a setting, the default is used.

    Setting Description
    Cloud SQL machine type The machine type for the Cloud SQL instance running the Airflow database. The machine type determines the number of CPUs and the amount of memory for your environment.
  5. Under Network configuration, specify the network settings for the Google Kubernetes Engine cluster. If you do not specify a setting, the default is used.

    Setting Description
    Enable VPC-native (using alias IP) Creates a VPC-native GKE cluster with alias IPs for your environment. The default is a routes-based GKE cluster. Required for a private IP Cloud Composer environment.
    Network The Virtual Private Cloud network that is used for machine communications. The network is required to specify a subnetwork. The default network is used if unspecified. Shared VPC requires a host project.
    Subnetwork The Virtual Private Cloud subnetwork that is used for machine communications. If your network uses a custom-mode network, the subnetwork is required.
    Pod IP Address Allocation The secondary range to allocate IP addresses for pods in the GKE cluster. If unspecified, a new secondary range is created. This setting is permanent.
    Service IP Address Allocation The secondary range to reserve space for Cloud Composer services. If unspecified, a new secondary range is created. This setting is permanent.
    Private IP environment Enables a private IP Cloud Composer environment. Disabled by default.
    Access GKE master using its external IP address Enables public access to the GKE cluster master. Requires Private IP environment.
    GKE Master IP range The private RFC 1918 range for the master's VPC. If unspecified, uses the default value 172.16.0.0/28. Required for Private IP environment.

    Ensure that secondary ranges are large enough to accommodate the cluster's size and anticipated growth. For example, the network prefixes of the secondary ranges for a 3-node Cloud Composer environment should be no longer than:

    • Pods: /22
    • Services: /27

    See Creating a VPC-native cluster for guidelines on configuring secondary ranges for Pods and Services.

  6. Under Web server configuration, specify the IP ranges that can access the Airflow web server for your environment and a machine type for the Airflow web server.

    Setting Description
    Allow access from all IP addresses (default) All IP ranges can access the Airflow web server.
    Allow access only from specific IP addresses Only specific IP ranges can access the web server. To add a new range, click Add IP range. To remove a range, click the trash button for that row. To deny all IP ranges, delete all rows.
    Web server machine type The machine type for the Compute Engine instance that runs the Airflow web server. The machine type determines the number of CPUs and the amount of memory for your environment.
  7. (Preview) Under Maintenance windows, you can set custom time windows for Cloud Composer to perform environment maintenance. Your environment may be temporarily unavailable during these windows, so choose hours (e.g. weekend or off-peak) when you are less likely to run workflows. Your maintenance windows must encompass at least 12 hours per week in total. This can also be set after environment creation from the Environment configuration tab of the Environment details page.

  8. (Optional) To override the default values in the Airflow configuration file (airflow.cfg), click Add Airflow configuration property.

  9. (Optional) To configure environment variables, click Add environment variable. See Environment Variables for requirements.

  10. (Optional) To add a label, click Add labels.

    Label keys and label values can only contain letters, numbers, dashes, and underscores. Label keys must start with a letter or number.

  11. Click Create.

gcloud

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    OTHER_ARGUMENTS

The following parameters are required:

  • ENVIRONMENT_NAME is the name of the environment. Must match the pattern: ^[a-z](?:[-0-9a-z]{0,61}[0-9a-z])?$. The environment name is used to create subcomponents for the environment, so you must provide a name that is also valid as a Cloud Storage bucket name. See Bucket naming guidelines for a list of restrictions.
  • LOCATION is the Compute Engine region where the environment is located. Ensure that the location you specify is one where Composer is available.

The following parameters are optional:

  • airflow-configs is a list of SECTION_NAME-PROPERTY_NAME=VALUE Airflow configuration overrides. The section name and property name must be separated by a hyphen.
  • cloud-sql-machine-type is a machine type for the Cloud SQL instance running the Airflow database. The machine type determines the number of CPUs and the amount of memory for your environment. The default machine type is db-n1-standard-2. Possible values for this parameter are: db-n1-standard-2, db-n1-standard-4, db-n1-standard-8, and db-n1-standard-16.
  • disk-size is the disk size in GB used for the node VMs. The minimum size is 20 GB. The default disk size is 100 GB.
  • env-variables is a list of NAME=VALUE environment variables that are set on the Airflow scheduler, worker, and web server processes.
  • enable-private-environment enables a private IP Cloud Composer environment.
    • master-ipv4-cidr is the private RFC 1918 range for the master's VPC. Required when enable-private-environment is true.
  • enable-private-endpoint enables public access to the GKE cluster master. Requires enable-private-environment.
  • enable-ip-alias enables VPC Native using alias IP addresses. Required when enable-private environment is true or to configure secondary ranges for pods and services:
    • cluster-secondary-range-name or cluster-ipv4-cidr configures the configure the secondary range for pods.
    • services-secondary-range-name orservices-ipv4-cidr configures the secondary range for services.
    • Preview: max-pods-per-node configures the maximum pods per node in the GKE cluster allocated during environment creation. Lowering this value reduces IP address consumption by the Cloud Composer Kubernetes cluster. For more information, see Optimizing IP address allocation. This value can only be set during environment creation, and only if the environment is VPC-Native. The range of possible values is 8-110, and the default is 32. While in Preview, this parameter requires you to use the gcloud beta composer command.
  • image-version is the composer-addon version and Airflow version to use for your environment in the form composer-a.b.c-airflow-x.y.z. For version alias and default version information, see Cloud Composer Versioning.
  • labels are user-specified labels that are attached to the environment and its resources.
  • machine-type is the Compute Engine machine type. The machine type determines the number of CPUs and the amount of memory for your environment. The default machine type is n1-standard-1.
  • network is the Virtual Private Cloud network used for machine communications.
    • The network is required to specify a subnetwork. The default network is used if unspecified.
    • When using Shared VPC, the network's relative resource name be provided using the format projects/HOST_PROJECT_ID/global/networks/NETWORK_ID. For Shared VPC subnetwork requirements, see subnetwork below.
  • node-count is the number of GKE nodes used to run the environment. The default node count is 3. Once you specify the number of nodes, it stays fixed until you update your environment.
  • oauth-scopes is the set of Google API scopes made available on all of the node VMs. The default OAuth scope is https://www.googleapis.com/auth/cloud-platform and must be included in the list of scopes if specified.
  • python-version is the Python version to use for your environment. Supported versions are Python 2 and Python 3. The default version is 2.
  • subnetwork is the Compute Engine subnetwork to which the environment is connected.
    • If your network uses a custom-mode network, the subnetwork is required.
    • When creating a Shared VPC environment using gcloud, you must use the secondary IP ranges composer-pods and composer-services. You can specify different secondary range names by using the Cloud Composer API. The subnetwork name must also be specified as a relative resource name using the format projects/HOST_PROJECT_ID/regions/REGION_ID/subnetworks/SUBNET_ID.
  • service-account is the service account that runs the environment's GKE nodes. The default Compute Engine service account is used if unspecified.
  • tags is the list of instance tags applied to all the node VMs. Tags are used to identify valid sources or targets for network firewalls. Each tag within the list must comply with RFC 1035.
  • web-server-machine-type is a machine type for the Compute Engine instance that runs the Airflow web server. The machine type determines the number of CPUs and the amount of memory for your environment. The default machine type is composer-n1-webserver-2. Possible values for this parameter are: composer-n1-webserver-2, composer-n1-webserver-4, and composer-n1-webserver-8.

The following example creates an environment running the latest supported Cloud Composer image version in the us-central1 region that uses the n1-standard-2 machine type with a beta environment label:

gcloud beta composer environments create test-environment \
    --location us-central1 \
    --zone us-central1-f \
    --machine-type n1-standard-2 \
    --image-version composer-latest-airflow-x.y.z \
    --labels env=beta  

The following Shared VPC example creates an environment in the host project. The environment is in the us-central1 region and uses the n1-standard-2 machine type with a beta environment label:

gcloud beta composer environments create host-project-environment \
    --network vpc-network-name --subnetwork vpc-subnetwork-name \
    --location us-central1 \
    --zone us-central1-f \
    --machine-type n1-standard-2 \
    --labels env=beta  

Preview: Custom maintenance windows

You can set custom time windows for Cloud Composer to perform environment maintenance. Your environment may be temporarily unavailable during these windows, so choose hours (e.g. weekend or off-peak) when you are less likely to run workflows. Your maintenance windows must encompass at least 12 hours per week in total. Use the following optional parameters:

  • maintenance-window-start sets the start time of a custom maintenance window.
  • maintenance-window-end sets the end time of a custom maintenance window.
  • maintenance-window-recurrence sets the days for maintenance window recurrence.

For example:

gcloud beta composer environments create test-environment \
    --location us-central1 \
    --zone us-central1-f \
    --machine-type n1-standard-2 \
    --image-version composer-latest-airflow-x.y.z \
    --labels env=beta \
    --maintenance-window-start='2019-08-01T01:00:00Z' \
    --maintenance-window-end='2019-08-01T07:00:00Z' \
    --maintenance-window-recurrence='FREQ=WEEKLY;BYDAY=SA,SU'
  

This creates an environment with a maintenance window between 01:00 and 07:00 (UTC) every Sunday and Saturday. Days of the week (Sunday through Saturday) are represented as follow: SU, MO, TU, WE, TH, FR, SA. Using FREQ=DAILY will set the maintenance window to recur every day.

API

To create a new Cloud Composer environment with the Cloud Composer REST API, construct an environments.create API request, filling in the Environment resource with your configuration information.

Terraform

To configure this environment using Terraform, add the following resource block to your Terraform configuration and run terraform apply.

resource "google_composer_environment" "example-resource" {
  name   = "ENVIRONMENT_NAME"
  region = "LOCATION"
}

The following parameters are required:

  • name, where ENVIRONMENT_NAME is the name of the environment. Must match the pattern: ^[a-z](?:[-0-9a-z]{0,61}[0-9a-z])?$. The environment name is used to create subcomponents for the environment, so you must provide a name that is also valid as a Cloud Storage bucket name. See Bucket naming guidelines for a list of restrictions.
  • region, where LOCATION is the Compute Engine region where the environment is located. Ensure that the location you specify is one where Composer is available.

Usage of additional optional parameters is defined in the Terraform Argument Reference

The following example creates an environment running the latest supported Cloud Composer image version in the us-central1 region that uses the n1-standard-2 machine type with a beta environment label. To configure this environment using Terraform, add the following resource block to your Terraform configuration and run terraform apply:

resource "google_composer_environment" "example-resource" {
  name   = "example-environment"
  region = "us-central1"

  config {
    node_config {
      zone = "us-central1-f"
      machine_type = "n1-standard-2"
    }
    software_config {
      image_version = "composer-latest-airflow-x.y.z"
    }
  }
  labels = {"env": "beta"}
}

The following Shared VPC example creates an environment in the host project. The environment is in the us-central1 region and uses the n1-standard-2 machine type with a beta environment label. To configure this environment using Terraform, add the following resource block to your Terraform configuration and run terraform apply:

resource "google_composer_environment" "example-resource" {
  name   = "host-project-environment"
  region = "us-central1"

  config {
    node_config {
      zone = "us-central1-f"
      machine_type = "n1-standard-2"
      network = "vpc-network-name"
      subnetwork = "vpc-subnetwork-name"
    }
    software_config {
      image_version = "composer-latest-airflow-x.y.z"
    }
  }
  labels = {"env": "beta"}
}

What's next