使用 Terraform 创建环境

Cloud Composer 1 |Cloud Composer 2 |Cloud Composer 3

本页是介绍创建环境的主页面的补充内容。该教程演示了如何使用 Terraform 在现有 Google Cloud 项目中为 Cloud Composer 环境设置此环境以及用户管理的服务账号。您可以先从此页面开始，然后再添加更多内容 根据需要配置环境的配置参数。

准备工作

• 本指南假定您有一个 Google Cloud 项目， 配置的结算信息：

  • 您可以使用现有项目。
  • 您可以使用 Google Cloud 控制台、Google Cloud CLI、API 或 Python 客户端库创建新项目。
  • 您可以使用 Terraform 创建和管理项目。如需了解详情，请参阅 google_project 资源的 Terraform 文档。

• 安装 gcloud CLI。

向 Google Cloud 进行身份验证

如需使用 Google Cloud 进行身份验证，请运行以下命令：

gcloud auth application-default login

如需详细了解此命令，请参阅 gcloud auth application-default。

在 Terraform 中配置 Google 提供程序

指定现有的项目 ID 和资源的默认区域。您的 Cloud Composer 环境使用此区域。

注意：google-beta 提供程序支持处于预览版阶段的 Cloud Composer 功能。如果您只想使用 Cloud Composer 的 GA 功能，请使用 google 提供程序，而不是 google-beta。

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

启用 Cloud Composer API

在项目中启用 Cloud Composer API：

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  // This parameter prevents automatic disabling
  // of the API when the resource is destroyed.
  // We recommend to disable the API only after all environments are deleted.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

在项目中创建自定义服务账号

默认情况下，Cloud Composer 环境使用默认 Compute Engine 账号。本指南演示了另一种方法，即创建一个新服务账号，该账号拥有运行 Cloud Composer 环境所需的所有权限。

注意 ：此服务账号可能需要额外的权限才能访问 项目中的资源。例如，如果您的 DAG 将数据转移到 BigQuery，则此账号可能需要 BigQuery 专有的权限或角色。

定义具有以下角色和权限的自定义服务账号。如需详细了解环境的服务账号的权限，请参阅使用 IAM 进行访问权限控制。

resource "google_service_account" "custom_service_account" {
  provider = google-beta
  account_id   = "custom-service-account"
  display_name = "Example Custom Service Account"
}

resource "google_project_iam_member" "custom_service_account" {
  provider = google-beta
  project  = "example-project"
  member   = format("serviceAccount:%s", google_service_account.custom_service_account.email)
  // Role for Public IP environments
  role     = "roles/composer.worker"
}

创建环境

使用 Terraform 创建环境。

此示例演示了如何创建使用自定义 服务账号。您可以添加更多参数来定义环境的其他配置参数，例如自定义规模和性能参数或其他 PyPI 软件包。

如需详细了解其他参数，请参阅 创建环境。

注意 ：如果您在 image_version 参数，则您的环境可能会升级到 terraform apply 时的版本。为避免出现这种情况，请在此参数中使用特定版本。

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    software_config {
      image_version = "composer-1.20.12-airflow-1.10.15"
    }

    node_config {
      service_account = google_service_account.custom_service_account.email
    }

  }
}

完整 Terraform 脚本（默认参数）

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    // Add your environment configuration here

    software_config {
      image_version = "composer-1.20.12-airflow-1.10.15"
    }

  }
}

完整 Terraform 脚本（自定义服务账号）

provider "google-beta" {
  project = "example-project"
  region  = "us-central1"
}

resource "google_project_service" "composer_api" {
  provider = google-beta
  project = "example-project"
  service = "composer.googleapis.com"
  // Disabling Cloud Composer API might irreversibly break all other
  // environments in your project.
  disable_on_destroy = false
  // this flag is introduced in 5.39.0 version of Terraform. If set to true it will
  //prevent you from disabling composer_api through Terraform if any environment was
  //there in the last 30 days
  check_if_service_has_usage_on_destroy = true
}

resource "google_service_account" "custom_service_account" {
  provider = google-beta
  account_id   = "custom-service-account"
  display_name = "Example Custom Service Account"
}

resource "google_project_iam_member" "custom_service_account" {
  provider = google-beta
  project  = "example-project"
  member   = format("serviceAccount:%s", google_service_account.custom_service_account.email)
  // Role for Public IP environments
  role     = "roles/composer.worker"
}

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"

  config {

    software_config {
      image_version = "composer-1.20.12-airflow-1.10.15"
    }

    node_config {
      service_account = google_service_account.custom_service_account.email
    }

  }
}

后续步骤

请参阅其他文档页面，了解如何配置 环境。例如：

• 创建环境
• 替换 Airflow 配置选项
• 设置环境变量
• 安装 Python 依赖项
• 扩缩环境