Anthos
Deploy Anthos on GKE with Terraform Part 3: Enabling Cloud Resources Provisioning
In the previous two parts of the series (1, 2) we discussed how new features in Terraform Provider for GCP make it easier for platform administrators to extend their Terraform automation to add Anthos Config Management (ACM) features to their GKE clusters. Using familiar Terraform resource syntax, you can add google_gke_hub_feature
and google_gke_hub_feature_membership
resource with configmanagement
section to enable Config Sync for GitOps integration and policy_controller
section for policy validation.
resource "google_gke_hub_feature" "configmanagement_acm_feature" {
name = "configmanagement"
location = "global"
provider = google-beta
}
resource "google_gke_hub_feature_membership" "feature_member" {
provider = google-beta
location = "global"
feature = "configmanagement"
membership = google_gke_hub_membership.membership.membership_id
configmanagement {
version = "1.8.0"
config_sync {
source_format = "unstructured"
git {
sync_repo = var.sync_repo
sync_branch = var.sync_branch
policy_dir = var.policy_dir
secret_type = "none"
}
}
policy_controller {
enabled = true
template_library_installed = true
referential_rules_enabled = true
}
}
}
So far the cluster in our example was only used to host the configuration consisting of Kubernetes native resources - containerized Wordpress application powered by in-cluster MySQL database. We are getting all the advantages of Kubernetes: continuous reconciliation and drift correction, eventual consistency, order independence and idempotence. We are also deriving benefits from GitOps approach using the repo as the source of truth, enabling reviewable and version-controlled workflow.
Now let’s take it even further. We’ll demonstrate how the same model can be expanded to create and manage not just native Kubernetes resources (Kubernetes service accounts, pods, deployments) but also GCP cloud resources - Cloud databases, storage buckets, VM instances and many other GCP resources. Since Config Connector was launched in 2020, many Kuberentes shops have embraced its convenient way of managing GCP resources. Now that we have enabled Terraform support for Anthos features, combined with a Terraform configuration option to install Config Connector on the cluster, the full GitOps workflow and Kubernetes lifecycle spanning native and cloud resources can be enabled during cluster creation.
In our example, we are enabling Config Connector using the config_connector
setting in the gke
Terraform module. We also use the workload-identity
module to create a GCP service account that will be used to make the changes to K8s resources and bind it to Kubernetes Service Account (cnrm-controller-manager
in cnrm-system namespace
). You can choose the appropriate permissions to GCP service account - in our examples we are giving it the owner
role for simplicity.
module "gke" {
source = "terraform-google-modules/kubernetes-engine/google//modules/beta-public-cluster"
...
config_connector = true
}
module "wi" {
source = "terraform-google-modules/kubernetes-engine/google//modules/workload-identity"
version = "~> 16.0.1"
gcp_sa_name = "cnrmsa"
cluster_name = module.gke.name
name = "cnrm-controller-manager"
location = var.zone
use_existing_k8s_sa = true
annotate_k8s_sa = false
namespace = "cnrm-system"
project_id = module.enabled_google_apis.project_id
roles = ["roles/owner"]
}
Let’s review what changed in this part in the repo that is synchronized with our cluster via Config Sync. In the first part, we added a collection of configs, all native Kubernetes objects. These configs provisioned an in-cluster Wordpress application with an in-cluster MySQL database. In the second part, we added a set of rules used by PolicyController to audit our cluster. In this part, let’s start by setting up the Config Connector to create a Cloud SQL database and other GCP resources. First, we are going to add a config representing an instance of the Config Connector addon. While the addon is enabled on the cluster, this config instance is required to activate it. It specifies the settings, such as mode
(cluster or namespace) and GCP service account, linking it to the cnrmsa account that we created above using the workload-identity
module.
apiVersion: core.cnrm.cloud.google.com/v1beta1
kind: ConfigConnector
metadata:
name: configconnector.core.cnrm.cloud.google.com
spec:
googleServiceAccount: "cnrmsa@to-be-specified.iam.gserviceaccount.com"
mode: cluster
And now we can create GCP resources in addition to in-cluster native resources directly in our K8s configuration. We are configuring an SQLDatabase and SQLInstance resources:
apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLDatabase
metadata:
name: wordpress
namespace: wp
spec:
charset: utf8
instanceRef:
name: wp-db
---
apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLInstance
metadata:
name: wp-db
namespace: wp
spec:
databaseVersion: MYSQL_5_7
region: us-central1
settings:
tier: db-f1-micro
as well as SQLUser, IAMPolicy, IAMPolicyMember (see complete example here). Overall, more than 130 resources are now supported with Config Connector covering many of the most popular GCP configuration patterns.
You will notice that this configuration is expanded and parameterized for our specific project. How did we specify the parameter values? While many tools can be used, including Helm and Kustomize and some of them used together, we recommend Kpt that fully embraces the principles of configuration-as-data. In this example we used set-project-id kpt function to specify project id on the config-root directory before submitting the change to Git.
kpt fn eval --include-meta-resources --image gcr.io/kpt-fn/set-project-id:v0.1 ./config-root -- "project-id=$PROJECT_ID"
kpt fn render ./config-root
This repo provides a complete example of provisioning a cluster that is synchronized with a repo that contains a WordPress configuration powered, this time, by GCP MySQL database.
This was the third and the final article of the three part series that showcased Terraform support for ACM features and how it simplifies cluster provisioning for platform administrators.