Create environments with Terraform

Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3

This page is a companion to the main page about creating environments. It demonstrates how to set up a Cloud Composer environment and a user-managed service account for this environment in an existing Google Cloud project with Terraform. You can use this page as a start, then add more configuration parameters for your environment, as needed.

Before you begin

• This guide assumes that you have a Google Cloud project with configured billing:

  • You can use an existing project.
  • You can create a new project using Google Cloud console, Google Cloud CLI, API, or a Python client library.
  • You can create and manage your project using Terraform. For more information, see Terraform documentation for the google_project resource.

• Install gcloud CLI.

Authenticate with Google Cloud

To authenticate with Google Cloud, run:

gcloud auth application-default login

For more information about this command, see gcloud auth application-default.

Configure the Google provider in Terraform

Specify your existing project ID and a default region for resources. Your Cloud Composer environment uses this region.

Note: The google-beta provider supports Cloud Composer features that are in Preview. If you want to use only GA features of Cloud Composer, use the google provider instead of google-beta.

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

Enable the Cloud Composer API

Enable the Cloud Composer API in your project:

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  // This parameter prevents automatic disabling
  // of the API when the resource is destroyed.
  // We recommend to disable the API only after all environments are deleted.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

Create an environment's service account in your project

This guide demonstrates how to create an environment's service account that has all required permissions to run a Cloud Composer environment.

We strongly recommend to set up a user-managed service account for your Cloud Composer environments that has only permissions required to run your environment and operations in your DAGs, as described in this guide.

Although we recommend against using this approach, if you do not specify an environment's service account, then your Cloud Composer environment uses the default Compute Engine service account.

Upcoming deprecation: In the future releases of Cloud Composer, new Cloud Composer environments will no longer use the Compute Engine service account by default.

Warning: Your environment's service account can have too broad permissions on your project. Because your environment runs DAGs on behalf of your environment's service account, users who can add and modify DAGs in your environment's bucket or modify Artifact Registry repositories in your project can run their code on behalf of the environment's service account and exercise all permissions of this account. Make sure that you are familiar with security considerations for environment's service accounts and understand how this account interacts with permissions and roles that you grant to individual users in your project.

The service account of your environment might need additional permissions to access other resources in your project. For example, if your DAGs transfer data into BigQuery, this account might need permissions or roles specific to BigQuery.

Define a custom service account with the following roles and permissions:

resource "google_service_account" "custom_service_account" {
  provider = google-beta
  account_id   = "custom-service-account"
  display_name = "Example Custom Service Account"
}

resource "google_project_iam_member" "custom_service_account" {
  provider = google-beta
  project  = "example-project"
  member   = format("serviceAccount:%s", google_service_account.custom_service_account.email)
  // Role for Public IP environments
  role     = "roles/composer.worker"
}

Grant required permissions to Cloud Composer service account

Note: For more information about how Cloud Composer uses this role, see Grant roles to Cloud Composer Service Agent account.

Add a new role binding to your environment's service account allow policy.

You add Cloud Composer Service Agent account as a new principal on your environment's service account and grant the Cloud Composer v2 API Service Agent Extension role to it.

If you are not using Terraform to define your project's allow policy, do not use the following example. Instead, add this binding using other methods.

resource "google_service_account_iam_member" "custom_service_account" {
  provider = google-beta
  service_account_id = google_service_account.custom_service_account.name
  role = "roles/composer.ServiceAgentV2Ext"
  member = "serviceAccount:[e-mail]"
}

Create an environment

Create your environment using Terraform.

The example demonstrates how to create an environment that uses a custom service account. You can add more parameters that define other configuration parameters of your environment, such as custom scale and performance parameters, or additional PyPI packages.

For more information about other parameters, see Create environments.

Caution: If you use version aliases in the image_version parameter, then your environment might upgrade to a later version when you run terraform apply. To avoid this, use a specific version in this parameter.

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    software_config {
      image_version = "composer-2.9.11-airflow-2.9.3"
    }

    node_config {
      service_account = google_service_account.custom_service_account.email
    }

  }
}

Full Terraform script

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

resource "google_service_account" "custom_service_account" {
  provider = google-beta
  account_id   = "custom-service-account"
  display_name = "Example Custom Service Account"
}

resource "google_project_iam_member" "custom_service_account" {
  provider = google-beta
  project  = "example-project"
  member   = format("serviceAccount:%s", google_service_account.custom_service_account.email)
  // Role for Public IP environments
  role     = "roles/composer.worker"
}

resource "google_service_account_iam_member" "custom_service_account" {
  provider = google-beta
  service_account_id = google_service_account.custom_service_account.name
  role = "roles/composer.ServiceAgentV2Ext"
  member = "serviceAccount:[e-mail]"
}

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    software_config {
      image_version = "composer-2.9.11-airflow-2.9.3"
    }

    node_config {
      service_account = google_service_account.custom_service_account.email
    }

  }
}

What's next

See other documentation pages for information about configuring your environment with Terraform. For example:

• Create environments
• Override Airflow configuration options
• Set environment variables
• Install Python dependencies
• Scale environments