Create environments with Terraform

Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3

This page is a companion to the main page about creating environments. It demonstrates how to set up a Cloud Composer environment and a user-managed service account for this environment in an existing Google Cloud project with Terraform. You can use this page as a start, then add more configuration parameters for your environment, as needed.

Before you begin

  • This guide assumes that you have a Google Cloud project with configured billing:

    • You can use an existing project.
    • You can create a new project using Google Cloud console, Google Cloud CLI, API, or a Python client library.
    • You can create and manage your project using Terraform. For more information, see Terraform documentation for the google_project resource.

  • Install gcloud CLI.

Authenticate with Google Cloud

To authenticate with Google Cloud, run:

gcloud auth application-default login

For more information about this command, see gcloud auth application-default.

Configure the Google provider in Terraform

Specify your existing project ID and a default region for resources. Your Cloud Composer environment uses this region.

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

Enable the Cloud Composer API

Enable the Cloud Composer API in your project:

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  // This parameter prevents automatic disabling
  // of the API when the resource is destroyed.
  // We recommend to disable the API only after all environments are deleted.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

Create a custom service account in your project

By default, Cloud Composer environments use the default Compute Engine account. This guide demonstrates another approach, by creating a new service account, which has all required permissions to run a Cloud Composer environment.

Define a custom service account with the following roles and permissions. For more information about permissions for environment's service accounts, see Access control with IAM.

resource "google_service_account" "custom_service_account" {
  provider = google-beta
  account_id   = "custom-service-account"
  display_name = "Example Custom Service Account"
}

resource "google_project_iam_member" "custom_service_account" {
  provider = google-beta
  project  = "example-project"
  member   = format("serviceAccount:%s", google_service_account.custom_service_account.email)
  // Role for Public IP environments
  role     = "roles/composer.worker"
}

Grant required permissions to Cloud Composer service account

Add a new role binding to your environment's service account allow policy.

You add Cloud Composer Service Agent account as a new principal on your environment's service account and grant the Cloud Composer v2 API Service Agent Extension role to it.

If you are not using Terraform to define your project's allow policy, do not use the following example. Instead, add this binding using other methods.

resource "google_service_account_iam_member" "custom_service_account" {
  provider = google-beta
  service_account_id = google_service_account.custom_service_account.name
  role = "roles/composer.ServiceAgentV2Ext"
  member = "serviceAccount:service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com"
}

Create an environment

Create your environment using Terraform.

The example demonstrates how to create an environment that uses a custom service account. You can add more parameters that define other configuration parameters of your environment, such as custom scale and performance parameters, or additional PyPI packages.

For more information about other parameters, see Create environments.

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    software_config {
      image_version = "composer-2.9.11-airflow-2.9.3"
    }

    node_config {
      service_account = google_service_account.custom_service_account.email
    }

  }
}

Full Terraform script (default parameters)

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

resource "google_project_iam_member" "service_agent_role" {
  provider = google-beta
  project  = "example-project"
  member   = "serviceAccount:service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com"
  role     = "roles/composer.ServiceAgentV2Ext"
}

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    // Add your environment configuration here

    software_config {
      image_version = "composer-2.9.11-airflow-2.9.3"
    }

  }
}

Full Terraform script (custom service account)

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

resource "google_service_account" "custom_service_account" {
  provider = google-beta
  account_id   = "custom-service-account"
  display_name = "Example Custom Service Account"
}

resource "google_project_iam_member" "custom_service_account" {
  provider = google-beta
  project  = "example-project"
  member   = format("serviceAccount:%s", google_service_account.custom_service_account.email)
  // Role for Public IP environments
  role     = "roles/composer.worker"
}

resource "google_service_account_iam_member" "custom_service_account" {
  provider = google-beta
  service_account_id = google_service_account.custom_service_account.name
  role = "roles/composer.ServiceAgentV2Ext"
  member = "serviceAccount:service-PROJECT_NUMBER@cloudcomposer-accounts.iam.gserviceaccount.com"
}

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    software_config {
      image_version = "composer-2.9.11-airflow-2.9.3"
    }

    node_config {
      service_account = google_service_account.custom_service_account.email
    }

  }
}

What's next

See other documentation pages for information about configuring your environment with Terraform. For example: