使用 Terraform 创建集群并部署工作负载

Kubernetes 集群可为应用提供计算、存储、网络和其他服务，类似于一个虚拟数据中心。在 Kubernetes 中运行的应用及其关联的服务称为工作负载。

本教程可让您快速查看正在运行的 Google Kubernetes Engine 集群和示例工作负载，所有这些都使用 Terraform 进行设置。然后，您可以在 Google Cloud 控制台中探索工作负载，然后继续学习更深入的学习路线，或开始规划和创建自己的可用于生产用途的集群。本教程假定您已熟悉 Terraform。

如果您希望在 Google Cloud 控制台中设置示例集群和工作负载，请参阅在 Google Cloud 控制台中创建集群。

准备工作

请按照以下步骤启用 Kubernetes Engine API：

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. Install the Google Cloud CLI.

  3. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  4. To initialize the gcloud CLI, run the following command: 

    gcloud init

  5. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  6. Make sure that billing is enabled for your Google Cloud project.

  7. Enable the GKE API: 

    gcloud services enable container.googleapis.com

  8. Install the Google Cloud CLI.

  9. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  10. To initialize the gcloud CLI, run the following command: 

    gcloud init

  11. Create or select a Google Cloud project.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  12. Make sure that billing is enabled for your Google Cloud project.

  13. Enable the GKE API: 

    gcloud services enable container.googleapis.com

  14. Grant roles to your user account. Run the following command once for each of the following IAM roles: roles/container.admin, roles/compute.networkAdmin, roles/iam.serviceAccountUser 

    gcloud projects add-iam-policy-binding PROJECT_ID --member="user:USER_IDENTIFIER" --role=ROLE
    • Replace PROJECT_ID with your project ID.

    • Replace USER_IDENTIFIER with the identifier for your user account. For example, user:myemail@example.com.

    • Replace ROLE with each individual role.

准备环境

在本教程中，您将使用 Cloud Shell 来管理Google Cloud上托管的资源。Cloud Shell 预安装了本教程所需的软件，包括 Terraform、kubectl 和 Google Cloud CLI。

  1. 通过点击 Cloud Shell 激活图标（激活 Cloud Shell）激活 Shell 按钮，从 Google Cloud 控制台启动 Cloud Shell 会话。此操作会在 Google Cloud 控制台的底部窗格中启动会话。

    与此虚拟机关联的服务凭据会自动获取，因此您无需设置或下载服务账号密钥。

  2. 在运行命令之前，请在 gcloud CLI 中使用以下命令设置默认项目：

    gcloud config set project PROJECT_ID

    请将 PROJECT_ID 替换为您的项目 ID。

  3. 克隆 GitHub 代码库：

    git clone https://github.com/terraform-google-modules/terraform-docs-samples.git --single-branch

  4. 切换到工作目录：

    cd terraform-docs-samples/gke/quickstart/autopilot

查看 Terraform 文件

Google Cloud 提供程序是一款插件，可让您使用 Terraform 管理和预配 Google Cloud 资源。它充当 Terraform 配置与 Google Cloud API 之间的桥梁，可让您以声明方式定义基础设施资源，例如虚拟机和网络。

本教程的集群和示例应用在两个使用 Google Cloud 和 Kubernetes 提供程序的 Terraform 文件中指定。

  1. 查看 cluster.tf 文件：

    cat cluster.tf

    输出类似于以下内容

    resource "google_compute_network" "default" {
  name = "example-network"

  auto_create_subnetworks  = false
  enable_ula_internal_ipv6 = true
}

resource "google_compute_subnetwork" "default" {
  name = "example-subnetwork"

  ip_cidr_range = "10.0.0.0/16"
  region        = "us-central1"

  stack_type       = "IPV4_IPV6"
  ipv6_access_type = "INTERNAL" # Change to "EXTERNAL" if creating an external loadbalancer

  network = google_compute_network.default.id
  secondary_ip_range {
    range_name    = "services-range"
    ip_cidr_range = "192.168.0.0/24"
  }

  secondary_ip_range {
    range_name    = "pod-ranges"
    ip_cidr_range = "192.168.1.0/24"
  }
}

resource "google_container_cluster" "default" {
  name = "example-autopilot-cluster"

  location                 = "us-central1"
  enable_autopilot         = true
  enable_l4_ilb_subsetting = true

  network    = google_compute_network.default.id
  subnetwork = google_compute_subnetwork.default.id

  ip_allocation_policy {
    stack_type                    = "IPV4_IPV6"
    services_secondary_range_name = google_compute_subnetwork.default.secondary_ip_range[0].range_name
    cluster_secondary_range_name  = google_compute_subnetwork.default.secondary_ip_range[1].range_name
  }

  # Set `deletion_protection` to `true` will ensure that one cannot
  # accidentally delete this instance by use of Terraform.
  deletion_protection = false
}

    此文件描述了以下资源：

    • google_compute_network：启用了内部 IPv6 的 VPC 网络。
    • google_compute_subnetwork：双栈子网。
    • google_container_cluster：位于 us-central1 的双栈 Autopilot 集群。deletion_protection 设置用于控制是否可以使用 Terraform 删除此集群。如果您将 deletion_protection 字段中的值设置为 false，Terraform 便可以删除集群。如需了解详情，请参阅 google_container_cluster 参考文档。

  2. 查看 app.tf 文件：

    cat app.tf

    输出类似于以下内容：

    data "google_client_config" "default" {}

provider "kubernetes" {
  host                   = "https://${google_container_cluster.default.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(google_container_cluster.default.master_auth[0].cluster_ca_certificate)

  ignore_annotations = [
    "^autopilot\\.gke\\.io\\/.*",
    "^cloud\\.google\\.com\\/.*"
  ]
}

resource "kubernetes_deployment_v1" "default" {
  metadata {
    name = "example-hello-app-deployment"
  }

  spec {
    selector {
      match_labels = {
        app = "hello-app"
      }
    }

    template {
      metadata {
        labels = {
          app = "hello-app"
        }
      }

      spec {
        container {
          image = "us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0"
          name  = "hello-app-container"

          port {
            container_port = 8080
            name           = "hello-app-svc"
          }

          security_context {
            allow_privilege_escalation = false
            privileged                 = false
            read_only_root_filesystem  = false

            capabilities {
              add  = []
              drop = ["NET_RAW"]
            }
          }

          liveness_probe {
            http_get {
              path = "/"
              port = "hello-app-svc"

              http_header {
                name  = "X-Custom-Header"
                value = "Awesome"
              }
            }

            initial_delay_seconds = 3
            period_seconds        = 3
          }
        }

        security_context {
          run_as_non_root = true

          seccomp_profile {
            type = "RuntimeDefault"
          }
        }

        # Toleration is currently required to prevent perpetual diff:
        # https://github.com/hashicorp/terraform-provider-kubernetes/pull/2380
        toleration {
          effect   = "NoSchedule"
          key      = "kubernetes.io/arch"
          operator = "Equal"
          value    = "amd64"
        }
      }
    }
  }
}

resource "kubernetes_service_v1" "default" {
  metadata {
    name = "example-hello-app-loadbalancer"
    annotations = {
      "networking.gke.io/load-balancer-type" = "Internal" # Remove to create an external loadbalancer
    }
  }

  spec {
    selector = {
      app = kubernetes_deployment_v1.default.spec[0].selector[0].match_labels.app
    }

    ip_family_policy = "RequireDualStack"

    port {
      port        = 80
      target_port = kubernetes_deployment_v1.default.spec[0].template[0].spec[0].container[0].port[0].name
    }

    type = "LoadBalancer"
  }

  depends_on = [time_sleep.wait_service_cleanup]
}

# Provide time for Service cleanup
resource "time_sleep" "wait_service_cleanup" {
  depends_on = [google_container_cluster.default]

  destroy_duration = "180s"
}

    此文件描述了以下资源：

（可选）向互联网公开应用

示例的 Terraform 文件描述了具有内部 IP 地址的应用，只能通过与示例应用相同的 Virtual Private Cloud (VPC) 访问该应用。如果您想通过互联网（例如，通过笔记本电脑）访问正在运行的演示版应用的网页界面，请在创建集群之前修改 Terraform 文件以创建公共 IP 地址。您可以直接在 Cloud Shell 中使用文本编辑器，也可以使用 Cloud Shell Editor 来执行此操作。

如需向互联网公开演示版应用，请执行以下操作：

  1. 在 cluster.tf 中，将 ipv6_access_type 从 INTERNAL 更改为 EXTERNAL。

    ipv6_access_type = "EXTERNAL"

  2. 在 app.tf 中，通过移除 networking.gke.io/load-balancer-type 注解来配置外部负载均衡器。

     annotations = {
   "networking.gke.io/load-balancer-type" = "Internal" # Remove this line
 }

创建集群并部署应用

  1. 在 Cloud Shell 中，运行以下命令以验证 Terraform 可用：

    terraform

    输出应类似如下所示：

    Usage: terraform [global options] <subcommand> [args]

The available commands for execution are listed below.
The primary workflow commands are given first, followed by
less common or more advanced commands.

Main commands:
  init          Prepare your working directory for other commands
  validate      Check whether the configuration is valid
  plan          Show changes required by the current configuration
  apply         Create or update infrastructure
  destroy       Destroy previously-created infrastructure

  2. 初始化 Terraform：

    terraform init

  3. 规划 Terraform 配置：

    terraform plan

  4. 应用 Terraform 配置

    terraform apply

    出现提示时，输入 yes 以确认操作。此命令可能需要几分钟才能完成。输出类似于以下内容：

    Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

验证集群正常运行

请执行以下操作，以确认集群运行正常：

  1. 前往 Google Cloud 控制台中的工作负载页面：

    转到“工作负载”

  2. 点击 example-hello-app-deployment 工作负载。系统会显示 Pod 详情页面。此页面显示有关 Pod 的信息，例如注解、Pod 上运行的容器、公开 Pod 的服务，以及 CPU、内存和磁盘用量等指标。

  3. 前往 Google Cloud 控制台中的服务 Ingress 页面：

    打开“Service 和 Ingress”

  4. 点击 example-hello-app-loadbalancer LoadBalancer Service。系统会显示 Service 详情页面。此页面显示有关 Service 的信息，例如与 Service 关联的 Pod 以及 Service 使用的端口。

  5. 在外部端点部分中，点击 IPv4 link 或 IPv6 link 以在浏览器中查看 Service。输出类似于以下内容：

    Hello, world!
Version: 2.0.0
Hostname: example-hello-app-deployment-5df979c4fb-kdwgr

清理

为避免因本页面中使用的资源导致您的 Google Cloud 账号产生费用，请删除包含这些资源的 Google Cloud 项目。

如果您打算学习其他教程或进一步探索示例，请等到完成后再执行此清理步骤。

  • 在 Cloud Shell 中，运行以下命令以删除 Terraform 资源：

    terraform destroy --auto-approve

排查清理错误

如果您看到类似于 The network resource 'projects/PROJECT_ID/global/networks/example-network' is already being used by 'projects/PROJECT_ID/global/firewalls/example-network-yqjlfql57iydmsuzd4ot6n5v' 的错误消息，请执行以下操作：

  1. 删除防火墙规则：

    gcloud compute firewall-rules list --filter="NETWORK:example-network" --format="table[no-heading](name)" | xargs gcloud --quiet compute firewall-rules delete

  2. 重新运行 Terraform 命令：

    terraform destroy --auto-approve

后续步骤