Managing GKE clusters with Config Controller

Preview

This product is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA products may have limited support, and changes to pre-GA products may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

This tutorial shows how to use the GKE cluster blueprint to provision a Google Kubernetes Engine (GKE) cluster with Config Controller. Follow along if you are a GKE cluster operator and want to declaratively manage your cluster configuration.

Config Controller is a hosted service to provision and orchestrate Anthos and Google Cloud resources. It offers an API endpoint that can provision, actuate, and orchestrate Google Cloud resources as part of Anthos Config Management.

KRM blueprints are a way to package resources that are commonly used together while codifying best practices that can be rolled out across your organization.

The GKE cluster blueprint is a KRM blueprint that includes all the resources you need to manage a GKE cluster on top of an existing Google Cloud network. You can instantiate the blueprint multiple times to set up multiple clusters.

Objectives

• Configure a GKE cluster declaratively.
• Apply the configuration using Config Controller.

Costs

This tutorial uses the following billable components of Google Cloud:

• Compute Engine
• Google Kubernetes Engine

For a full list of resources included in the GKE cluster blueprint, see the Resources section of the GKE package, and its sub-packages.

To generate a cost estimate based on your projected usage, use the pricing calculator.

Note: The GKE cluster blueprint is configured with defaults that are intended to align with security and production readiness best practices. For this reason, the default configuration can cost more than default clusters created with other methods. Please review the configuration carefully before applying.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Requirements

• You must have a Google Workspace or Cloud Identity subscription.

  Learn how to set up Cloud Identity.

  Learn how to set up Google Workspace.

• Your organization must have a verified domain.

  Learn how to verify your domain with Cloud Identity.

  Learn how to verify your domain with Google Workspace.

• You must have a group named gke-security-groups in your organization.

  For more information about this security group, see Groups for GKE.

  Ways to create a group:

  • Google Workspace Admin Console
  • Groups app (requires Groups for Business)
  • Groups in Cloud Console (requires Groups for Business)

  Learn how to turn Groups for Business on or off.

• You must have a Config Controller instance.

  Learn how to set up Config Controller.

  Note: This tutorial uses kpt to apply resources. GitOps is not required.

• You must have a Google Cloud project. This is the project your GKE cluster will be created in. You can use the project your Config Controller cluster is in or a different one.

  Learn how to create and manage projects

• You must have billing enabled for your Google Cloud project.

  Learn how to confirm that billing is enabled for your project.

• You must have a Config Controller namespace configured to manage resources in your project.

  By default, Config Controller comes with the config-control namespace pre-configured for use with Config Connector, and the instructions to set up Config Controller explain how to authorize Config Connector to manage your project.

  However, you can also use another namespace similarly configured. By convention, these project namespaces are usually named after the project ID of the project they manage.

  Learn how to configure Config Connector to watch your namespace.

Note: You do not need an Anthos entitlement to follow this tutorial.

Before you begin

1. In the Cloud Console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

You run all commands in this tutorial from Cloud Shell.

Set up the environment

In Cloud Shell, run the following commands:

1. Install kubectl, the primary command-line interface for Kubernetes:

   gcloud components install kubectl
   

2. Install kpt, the primary command-line interface for KRM blueprints:

   gcloud components install kpt
   

   Note: This tutorial requires kpt v1+.

3. Configure kubectl and kpt to connect with Config Controller:

   gcloud alpha anthos config controller get-credentials CONFIG_CONTROLLER_NAME \
       --location COMPUTE_REGION \
       --project CONFIG_CONTROLLER_PROJECT_ID
   

   Replace the following:

   • CONFIG_CONTROLLER_NAME: the name of your Config Controller cluster.

   • COMPUTE_REGION: the region of your Config Controller cluster (for example, us-central1).

   • CONFIG_CONTROLLER_PROJECT_ID: the project ID of your Config Controller cluster.

   Note: While in preview, Config Controller is only available in the us-central1 region.

4. Enable the Resource Manager API:

   gcloud services enable cloudresourcemanager.googleapis.com \
       --project PROJECT_ID
   

   Replace PROJECT_ID with the ID of your project.

   Note: The Resource Manager API is used by Config Connector to manage enablement of other service APIs.

5. Install the ResourceGroup CRD, if not already installed:

   kpt live install-resource-group
   

6. Verify that Config Connector is configured and healthy in the project namespace:

   kubectl get ConfigConnectorContext -n PROJECT_NAMESPACE \
       -o "custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,HEALTHY:.status.healthy"
   

   Replace PROJECT_NAMESPACE with the namespace you want to use for managing project resources (for example, config-control).

   Example output:

   NAMESPACE        NAME                                                HEALTHY
   config-control   configconnectorcontext.core.cnrm.cloud.google.com   true
   

Configure a GKE cluster

To configure a GKE cluster with the GKE cluster blueprint, you run the following commands.

1. Fetch the GKE cluster blueprint with kpt, from within the desired working directory:

   kpt pkg get \
       https://github.com/GoogleCloudPlatform/blueprints.git/catalog/gke@v0.3.0 \
       CLUSTER_NAME
   

   Replace CLUSTER_NAME with the desired name to use for the GKE cluster (for example, hello-cluster).

2. Move into the cluster directory:

   cd ./CLUSTER_NAME/
   

3. Configure the package by modifying the setters.yaml file:

   cat > setters.yaml << EOF
   apiVersion: v1
   kind: ConfigMap
   metadata: # kpt-merge: /setters
     name: setters
   data:
     # The cluster name
     cluster-name: CLUSTER_NAME
     # The environment (set as a label on the cluster)
     environment: dev
     # The compute location (region or zone)
     location: us-central1
     # The project in which to manage cluster resources
     platform-project-id: PROJECT_ID
     # The namespace in which to manage cluster resources
     platform-namespace: PROJECT_NAMESPACE
     # The name of the VPC in which to create a dedicated subnet
     network-name: default
     # The project that the VPC is in
     network-project-id: PROJECT_ID
     # The namespace in which to manage network resources
     networking-namespace: PROJECT_NAMESPACE
     # The private IP range for masters to use when peering to the VPC
     master-ip-range: 192.168.0.0/28
     # The private IP range for nodes to use, allocated to the dedicated subnet
     node-ip-range: 10.4.0.0/22
     # The private IP range for pods to use, allocated to the dedicated subnet
     pod-ip-range: 10.5.0.0/16
     # The private IP range for services to use, allocated to the dedicated subnet
     service-ip-range: 10.6.0.0/16
     # The namespace in which to manage service enablement resources
     projects-namespace: PROJECT_NAMESPACE
     # The group in which to manage the list of groups that can be used for RBAC.
     # Must be named exactly 'gke-security-groups'.
     security-group: gke-security-groups@YOUR_DOMAIN
   EOF
   

   Replace the following:

   • PROJECT_ID: the ID of your project.

     For this tutorial, the cluster and network are deployed to the same project.

   • PROJECT_NAMESPACE: the namespace to use for managing project resources (for example, config-control).

     For this tutorial, the cluster, network, and service enablement are managed in the same namespace.

   • YOUR_DOMAIN: the domain used by your Groups (for example, example.com).

     Note: The security-group value must be a group email, not a service account or user email. The group must be named exactly gke-security-groups.

   All other data fields can be reconfigured as desired.

   The defaults provided should work in an otherwise empty project with the default network.

4. Render the setter values into the templated resources:

   kpt fn render
   

   Example output:

   Package "example/cluster":
   [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
   [PASS] "gcr.io/kpt-fn/apply-setters:v0.1"
     Results:
       [INFO] set field value to "example-us-west4" in file "cluster/cluster.yaml" in field "metadata.name"
       [INFO] set field value to "config-control" in file "cluster/cluster.yaml" in field "metadata.namespace"
       [INFO] set field value to "dev" in file "cluster/cluster.yaml" in field "metadata.labels.gke.io/environment"
       [INFO] set field value to "platform-project-id" in file "cluster/cluster.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
       ...(10 line(s) truncated, use '--truncate-output=false' to disable)
   
   Package "example/nodepools/primary":
   [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
   [PASS] "gcr.io/kpt-fn/apply-setters:v0.1"
     Results:
       [INFO] set field value to "gke-example-us-east4-primary" in file "nodepools/primary/node-iam.yaml" in field "metadata.name"
       [INFO] set field value to "config-control" in file "nodepools/primary/node-iam.yaml" in field "metadata.namespace"
       [INFO] set field value to "platform-project-id" in file "nodepools/primary/node-iam.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
       [INFO] set field value to "gke-example-us-east4-primary" in file "nodepools/primary/node-iam.yaml" in field "spec.displayName"
       ...(23 line(s) truncated, use '--truncate-output=false' to disable)
   
   Package "example/subnet":
   [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
   [PASS] "gcr.io/kpt-fn/apply-setters:v0.1"
     Results:
       [INFO] set field value to "platform-project-id-example-us-west4" in file "subnet/subnet.yaml" in field "metadata.name"
       [INFO] set field value to "networking" in file "subnet/subnet.yaml" in field "metadata.namespace"
       [INFO] set field value to "network-project-id" in file "subnet/subnet.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
       [INFO] set field value to "platform-project-id-example-us-west4" in file "subnet/subnet.yaml" in field "spec.description"
       ...(5 line(s) truncated, use '--truncate-output=false' to disable)
   
   Package "example":
   [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
   [PASS] "gcr.io/kpt-fn/apply-setters:v0.1"
     Results:
       [INFO] set field value to "example" in file "cluster/cluster.yaml" in field "metadata.name"
       [INFO] set field value to "config-control" in file "cluster/cluster.yaml" in field "metadata.namespace"
       [INFO] set field value to "dev" in file "cluster/cluster.yaml" in field "metadata.labels.gke.io/environment"
       [INFO] set field value to "example-project-1234" in file "cluster/cluster.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
       ...(44 line(s) truncated, use '--truncate-output=false' to disable)
   
   Successfully executed 4 function(s) in 4 package(s).
   

Apply config changes

Local changes in the previous steps do not affect the cloud until applied.

To apply your configuration changes, you run the following commands.

1. Initialize the working directory with kpt, which creates a resource to track changes:

   kpt live init --namespace PROJECT_NAMESPACE
   

   Replace PROJECT_NAMESPACE with the namespace used for managing project resources (for example, config-control).

2. Preview the resources that will be created:

   kpt live apply --dry-run
   

   All resources should say "created (dry-run)".

   Example output:

   service.serviceusage.cnrm.cloud.google.com/example-project-1234-example-container created (dry-run)
   computesubnetwork.compute.cnrm.cloud.google.com/example-project-1234-example created (dry-run)
   containercluster.container.cnrm.cloud.google.com/example created (dry-run)
   containernodepool.container.cnrm.cloud.google.com/example-primary created (dry-run)
   iampolicymember.iam.cnrm.cloud.google.com/artifactreader-gke-example-primary created (dry-run)
   iampolicymember.iam.cnrm.cloud.google.com/logwriter-gke-example-primary created (dry-run)
   iampolicymember.iam.cnrm.cloud.google.com/metricwriter-gke-example-primary created (dry-run)
   iamserviceaccount.iam.cnrm.cloud.google.com/gke-example-primary created (dry-run)
   8 resource(s) applied. 8 created, 0 unchanged, 0 configured, 0 failed (dry-run)
   0 resource(s) pruned, 0 skipped, 0 failed (dry-run)
   

3. Apply the resources with kpt:

   kpt live apply
   

   All resources should say "created".

   Example output:

   service.serviceusage.cnrm.cloud.google.com/example-project-1234-example-container created
   computesubnetwork.compute.cnrm.cloud.google.com/example-project-1234-example created
   containercluster.container.cnrm.cloud.google.com/example created
   containernodepool.container.cnrm.cloud.google.com/example-primary created
   iampolicymember.iam.cnrm.cloud.google.com/artifactreader-gke-example-primary created
   iampolicymember.iam.cnrm.cloud.google.com/logwriter-gke-example-primary created
   iampolicymember.iam.cnrm.cloud.google.com/metricwriter-gke-example-primary created
   iamserviceaccount.iam.cnrm.cloud.google.com/gke-example-primary created
   8 resource(s) applied. 8 created, 0 unchanged, 0 configured, 0 failed
   0 resource(s) pruned, 0 skipped, 0 failed
   

Verify success

To verify that your changes are applied and the resources they specify are provisioned, you run the following commands.

1. Wait until the resources are ready:

   kpt live status --output table --poll-until current
   

   This command will poll until all the resources have a status of Current and a condition of Ready.

   Use ctrl-c to interrupt, if needed.

   Example output:

   NAMESPACE   RESOURCE                                  STATUS      CONDITIONS      AGE     MESSAGE
   config-con  ComputeSubnetwork/example-project-1234-e  Current     Ready           41m     Resource is Ready
   config-con  ContainerCluster/example                  Current     Ready           41m     Resource is Ready
   config-con  ContainerNodePool/example-primary         Current     Ready           41m     Resource is Ready
   config-con  IAMPolicyMember/artifactreader-gke-examp  Current     Ready           41m     Resource is Ready
   config-con  IAMPolicyMember/logwriter-gke-example-pr  Current     Ready           41m     Resource is Ready
   config-con  IAMPolicyMember/metricwriter-gke-example  Current     Ready           41m     Resource is Ready
   config-con  IAMServiceAccount/gke-example-primary     Current     Ready           41m     Resource is Ready
   config-con  Service/example-project-1234-example-con  Current     Ready           41m     Resource is Ready
   

   Note: The resources in this blueprint can take over 40 minutes to provision.

2. In the case of error, use the default event output to see full error messages:

   kpt live status
   

Frequently asked questions

How do I change an immutable field?

First, delete the resource with kubectl, then re-apply with kubectl or kpt live apply.

Why does kpt live status say the condition of my resource is <NONE>?

This usually indicates that Config Connector has not been configured for the project namespace.

Learn how to configure Config Connector to watch your namespace.

I modified setters.yaml and re-applied. Why does the output say that my resources are unchanged?

This can happen if you try to apply your changes without rendering first.

Rendering executes the pipeline of functions specified in the Kptfile(s). One of those functions is apply-setters, which uses setters.yaml as input.

Run kpt fn render to execute the function pipeline.

The cluster is ready now, why isn't the node pool being created yet?

Config Connector uses retry loops with exponential backoff when applying changes to Google Cloud resources. If the resource cannot be successfully updated for long enough, the maximum retry backoff of 15 minutes can be reached. This can delay applying changes to other resources when there is a chain of dependencies.

Since cluster provisioning can take over 15 minutes, and it depends on subnet provisioning, by the time the cluster is ready, the node pool might be waiting up to 15 minutes to retry.

The node pool should eventually be provisioned, but if you want to force a retry manually, you can delete and reapply the affected resource.

Cleaning up

If you decide to stop using Config Controller, you should first clean up all resources created with Config Controller and then delete Config Controller itself.

1. Delete the resources with kpt, from within the working directory:

   kpt live destroy
   

2. Wait until all resources are deleted:

   until [ -z "$(kubectl get -R -f . --ignore-not-found | tee /dev/fd/2)" ]; \
   do sleep 1; done
   

   This command will poll until all the resources have a status of Deleted.

   Use ctrl-c to interrupt, if needed.

   Note: The resources in this blueprint can take over 5 minutes to delete.

What's next

• Learn more about blueprints.
• Explore the Anthos Config Management tutorials.
• Browse the KRM blueprint catalog.
• Browse the Terraform blueprint catalog.