Create environments with Terraform

Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3

This page is a companion to the main page about creating environments. It demonstrates how to set up a Cloud Composer environment and a user-managed service account for this environment in an existing Google Cloud project with Terraform. You can use this page as a start, then add more configuration parameters for your environment, as needed.

Before you begin

• This guide assumes that you have a Google Cloud project with configured billing:

  • You can use an existing project.
  • You can create a new project using Google Cloud console, Google Cloud CLI, API, or a Python client library.
  • You can create and manage your project using Terraform. For more information, see Terraform documentation for the google_project resource.

• Install gcloud CLI.

Authenticate with Google Cloud

To authenticate with Google Cloud, run:

gcloud auth application-default login

For more information about this command, see gcloud auth application-default.

Configure the Google provider in Terraform

Specify your existing project ID and a default region for resources. Your Cloud Composer environment uses this region.

Note: The google-beta provider supports Cloud Composer features that are in Preview. If you want to use only GA features of Cloud Composer, use the google provider instead of google-beta.

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

Enable the Cloud Composer API

Enable the Cloud Composer API in your project:

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  // This parameter prevents automatic disabling
  // of the API when the resource is destroyed.
  // We recommend to disable the API only after all environments are deleted.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

Create a custom service account in your project

By default, Cloud Composer environments use the default Compute Engine account. This guide demonstrates another approach, by creating a new service account, which has all required permissions to run a Cloud Composer environment.

Note: This service account might need additional permissions to access other resources in your project. For example, if your DAGs transfer data into BigQuery, this account might need permissions or roles specific to BigQuery.

Define a custom service account with the following roles and permissions. For more information about permissions for environment's service accounts, see Access control with IAM.

resource "google_service_account" "custom_service_account" {
  provider = google-beta
  account_id   = "custom-service-account"
  display_name = "Example Custom Service Account"
}

resource "google_project_iam_member" "custom_service_account" {
  provider = google-beta
  project  = "example-project"
  member   = format("serviceAccount:%s", google_service_account.custom_service_account.email)
  // Role for Public IP environments
  role     = "roles/composer.worker"
}

Create an environment

Create your environment using Terraform.

The example demonstrates how to create an environment that uses a custom service account. You can add more parameters that define other configuration parameters of your environment, such as custom scale and performance parameters, or additional PyPI packages.

For more information about other parameters, see Create environments.

Caution: If you use version aliases in the image_version parameter, then your environment might upgrade to a later version when you run terraform apply. To avoid this, use a specific version in this parameter.

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    software_config {
      image_version = "composer-1.20.12-airflow-1.10.15"
    }

    node_config {
      service_account = google_service_account.custom_service_account.email
    }

  }
}

Full Terraform script (default parameters)

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    // Add your environment configuration here

    software_config {
      image_version = "composer-1.20.12-airflow-1.10.15"
    }

  }
}

Full Terraform script (custom service account)

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

resource "google_service_account" "custom_service_account" {
  provider = google-beta
  account_id   = "custom-service-account"
  display_name = "Example Custom Service Account"
}

resource "google_project_iam_member" "custom_service_account" {
  provider = google-beta
  project  = "example-project"
  member   = format("serviceAccount:%s", google_service_account.custom_service_account.email)
  // Role for Public IP environments
  role     = "roles/composer.worker"
}

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    software_config {
      image_version = "composer-1.20.12-airflow-1.10.15"
    }

    node_config {
      service_account = google_service_account.custom_service_account.email
    }

  }
}

What's next

See other documentation pages for information about configuring your environment with Terraform. For example:

• Create environments
• Override Airflow configuration options
• Set environment variables
• Install Python dependencies
• Scale environments