Managing GKE clusters with Config Controller

This tutorial shows how to use the GKE cluster blueprint to provision a Google Kubernetes Engine (GKE) cluster with Config Controller. Follow along if you are a GKE cluster operator and want to declaratively manage your cluster configuration.

Config Controller is a hosted service to provision and orchestrate Anthos and Google Cloud resources. It offers an API endpoint that can provision, actuate, and orchestrate Google Cloud resources as part of Anthos Config Management.

KRM blueprints are a way to package resources that are commonly used together while codifying best practices that can be rolled out across your organization.

The GKE cluster blueprint is a KRM blueprint that includes all the resources you need to manage a GKE cluster on top of an existing Google Cloud network. You can instantiate the blueprint multiple times to set up multiple clusters.

Objectives

  • Configure a GKE cluster declaratively.
  • Apply the configuration using Config Controller.

Costs

This tutorial uses the following billable components of Google Cloud:

For a full list of resources included in the GKE cluster blueprint, see the Resources section of the GKE package, and its sub-packages.

To generate a cost estimate based on your projected usage, use the pricing calculator.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Requirements

Before you begin

  1. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

  2. You run all commands in this tutorial from Cloud Shell.

Set up the environment

In Cloud Shell, run the following commands:

  1. Install kubectl, the primary command-line interface for Kubernetes:

    gcloud components install kubectl
    
  2. Install kpt, the primary command-line interface for KRM blueprints:

    gcloud components install kpt
    
  3. Configure kubectl and kpt to connect with Config Controller:

    gcloud anthos config controller get-credentials CONFIG_CONTROLLER_NAME \
        --location COMPUTE_REGION \
        --project CONFIG_CONTROLLER_PROJECT_ID
    

    Replace the following:

    • CONFIG_CONTROLLER_NAME: the name of your Config Controller cluster.

    • COMPUTE_REGION: the region of your Config Controller cluster (for example, us-central1).

    • CONFIG_CONTROLLER_PROJECT_ID: the project ID of your Config Controller cluster.

  4. Enable the Resource Manager API:

    gcloud services enable cloudresourcemanager.googleapis.com \
        --project PROJECT_ID
    

    Replace PROJECT_ID with the ID of your project.

  5. Install the ResourceGroup CRD, if not already installed:

    kpt live install-resource-group
    
  6. Verify that Config Connector is configured and healthy in the project namespace:

    kubectl get ConfigConnectorContext -n PROJECT_NAMESPACE \
        -o "custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,HEALTHY:.status.healthy"
    

    Replace PROJECT_NAMESPACE with the namespace you want to use for managing project resources (for example, config-control).

    Example output:

    NAMESPACE        NAME                                                HEALTHY
    config-control   configconnectorcontext.core.cnrm.cloud.google.com   true
    

Configure a GKE cluster

To configure a GKE cluster with the GKE cluster blueprint, you run the following commands.

  1. Fetch the GKE cluster blueprint with kpt, from within the desired working directory:

    kpt pkg get \
        https://github.com/GoogleCloudPlatform/blueprints.git/catalog/gke@v0.3.0 \
        CLUSTER_NAME
    

    Replace CLUSTER_NAME with the desired name to use for the GKE cluster (for example, hello-cluster).

  2. Move into the cluster directory:

    cd ./CLUSTER_NAME/
    
  3. Configure the package by modifying the setters.yaml file:

    cat > setters.yaml << EOF
    apiVersion: v1
    kind: ConfigMap
    metadata: # kpt-merge: /setters
      name: setters
    data:
      # The cluster name
      cluster-name: CLUSTER_NAME
      # The environment (set as a label on the cluster)
      environment: dev
      # The compute location (region or zone)
      location: us-central1
      # The project in which to manage cluster resources
      platform-project-id: PROJECT_ID
      # The namespace in which to manage cluster resources
      platform-namespace: PROJECT_NAMESPACE
      # The name of the VPC in which to create a dedicated subnet
      network-name: default
      # The project that the VPC is in
      network-project-id: PROJECT_ID
      # The namespace in which to manage network resources
      networking-namespace: PROJECT_NAMESPACE
      # The private IP range for masters to use when peering to the VPC
      master-ip-range: 192.168.0.0/28
      # The private IP range for nodes to use, allocated to the dedicated subnet
      node-ip-range: 10.4.0.0/22
      # The private IP range for pods to use, allocated to the dedicated subnet
      pod-ip-range: 10.5.0.0/16
      # The private IP range for services to use, allocated to the dedicated subnet
      service-ip-range: 10.6.0.0/16
      # The namespace in which to manage service enablement resources
      projects-namespace: PROJECT_NAMESPACE
      # The group in which to manage the list of groups that can be used for RBAC.
      # Must be named exactly 'gke-security-groups'.
      security-group: gke-security-groups@YOUR_DOMAIN
    EOF
    

    Replace the following:

    • PROJECT_ID: the ID of your project.

      For this tutorial, the cluster and network are deployed to the same project.

    • PROJECT_NAMESPACE: the namespace to use for managing project resources (for example, config-control).

      For this tutorial, the cluster, network, and service enablement are managed in the same namespace.

    • YOUR_DOMAIN: the domain used by your Groups (for example, example.com).

    All other data fields can be reconfigured as desired.

    The defaults provided should work in an otherwise empty project with the default network.

  4. Render the setter values into the templated resources:

    kpt fn render
    

    Example output:

    Package "example/cluster":
    [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
    [PASS] "gcr.io/kpt-fn/apply-setters:v0.1"
      Results:
        [INFO] set field value to "example-us-west4" in file "cluster/cluster.yaml" in field "metadata.name"
        [INFO] set field value to "config-control" in file "cluster/cluster.yaml" in field "metadata.namespace"
        [INFO] set field value to "dev" in file "cluster/cluster.yaml" in field "metadata.labels.gke.io/environment"
        [INFO] set field value to "platform-project-id" in file "cluster/cluster.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
        ...(10 line(s) truncated, use '--truncate-output=false' to disable)
    
    Package "example/nodepools/primary":
    [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
    [PASS] "gcr.io/kpt-fn/apply-setters:v0.1"
      Results:
        [INFO] set field value to "gke-example-us-east4-primary" in file "nodepools/primary/node-iam.yaml" in field "metadata.name"
        [INFO] set field value to "config-control" in file "nodepools/primary/node-iam.yaml" in field "metadata.namespace"
        [INFO] set field value to "platform-project-id" in file "nodepools/primary/node-iam.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
        [INFO] set field value to "gke-example-us-east4-primary" in file "nodepools/primary/node-iam.yaml" in field "spec.displayName"
        ...(23 line(s) truncated, use '--truncate-output=false' to disable)
    
    Package "example/subnet":
    [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
    [PASS] "gcr.io/kpt-fn/apply-setters:v0.1"
      Results:
        [INFO] set field value to "platform-project-id-example-us-west4" in file "subnet/subnet.yaml" in field "metadata.name"
        [INFO] set field value to "networking" in file "subnet/subnet.yaml" in field "metadata.namespace"
        [INFO] set field value to "network-project-id" in file "subnet/subnet.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
        [INFO] set field value to "platform-project-id-example-us-west4" in file "subnet/subnet.yaml" in field "spec.description"
        ...(5 line(s) truncated, use '--truncate-output=false' to disable)
    
    Package "example":
    [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
    [PASS] "gcr.io/kpt-fn/apply-setters:v0.1"
      Results:
        [INFO] set field value to "example" in file "cluster/cluster.yaml" in field "metadata.name"
        [INFO] set field value to "config-control" in file "cluster/cluster.yaml" in field "metadata.namespace"
        [INFO] set field value to "dev" in file "cluster/cluster.yaml" in field "metadata.labels.gke.io/environment"
        [INFO] set field value to "example-project-1234" in file "cluster/cluster.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
        ...(44 line(s) truncated, use '--truncate-output=false' to disable)
    
    Successfully executed 4 function(s) in 4 package(s).
    

Apply config changes

Local changes in the previous steps do not affect the cloud until applied.

To apply your configuration changes, you run the following commands.

  1. Initialize the working directory with kpt, which creates a resource to track changes:

    kpt live init --namespace PROJECT_NAMESPACE
    

    Replace PROJECT_NAMESPACE with the namespace used for managing project resources (for example, config-control).

  2. Preview the resources that will be created:

    kpt live apply --dry-run
    

    All resources should say "created (dry-run)".

    Example output:

    service.serviceusage.cnrm.cloud.google.com/example-project-1234-example-container created (dry-run)
    computesubnetwork.compute.cnrm.cloud.google.com/example-project-1234-example created (dry-run)
    containercluster.container.cnrm.cloud.google.com/example created (dry-run)
    containernodepool.container.cnrm.cloud.google.com/example-primary created (dry-run)
    iampolicymember.iam.cnrm.cloud.google.com/artifactreader-gke-example-primary created (dry-run)
    iampolicymember.iam.cnrm.cloud.google.com/logwriter-gke-example-primary created (dry-run)
    iampolicymember.iam.cnrm.cloud.google.com/metricwriter-gke-example-primary created (dry-run)
    iamserviceaccount.iam.cnrm.cloud.google.com/gke-example-primary created (dry-run)
    8 resource(s) applied. 8 created, 0 unchanged, 0 configured, 0 failed (dry-run)
    0 resource(s) pruned, 0 skipped, 0 failed (dry-run)
    
  3. Apply the resources with kpt:

    kpt live apply
    

    All resources should say "created".

    Example output:

    service.serviceusage.cnrm.cloud.google.com/example-project-1234-example-container created
    computesubnetwork.compute.cnrm.cloud.google.com/example-project-1234-example created
    containercluster.container.cnrm.cloud.google.com/example created
    containernodepool.container.cnrm.cloud.google.com/example-primary created
    iampolicymember.iam.cnrm.cloud.google.com/artifactreader-gke-example-primary created
    iampolicymember.iam.cnrm.cloud.google.com/logwriter-gke-example-primary created
    iampolicymember.iam.cnrm.cloud.google.com/metricwriter-gke-example-primary created
    iamserviceaccount.iam.cnrm.cloud.google.com/gke-example-primary created
    8 resource(s) applied. 8 created, 0 unchanged, 0 configured, 0 failed
    0 resource(s) pruned, 0 skipped, 0 failed
    

Verify success

To verify that your changes are applied and the resources they specify are provisioned, you run the following commands.

  1. Wait until the resources are ready:

    kpt live status --output table --poll-until current
    

    This command will poll until all the resources have a status of Current and a condition of Ready.

    Use ctrl-c to interrupt, if needed.

    Example output:

    NAMESPACE   RESOURCE                                  STATUS      CONDITIONS      AGE     MESSAGE
    config-con  ComputeSubnetwork/example-project-1234-e  Current     Ready           41m     Resource is Ready
    config-con  ContainerCluster/example                  Current     Ready           41m     Resource is Ready
    config-con  ContainerNodePool/example-primary         Current     Ready           41m     Resource is Ready
    config-con  IAMPolicyMember/artifactreader-gke-examp  Current     Ready           41m     Resource is Ready
    config-con  IAMPolicyMember/logwriter-gke-example-pr  Current     Ready           41m     Resource is Ready
    config-con  IAMPolicyMember/metricwriter-gke-example  Current     Ready           41m     Resource is Ready
    config-con  IAMServiceAccount/gke-example-primary     Current     Ready           41m     Resource is Ready
    config-con  Service/example-project-1234-example-con  Current     Ready           41m     Resource is Ready
    
  2. In the case of error, use the default event output to see full error messages:

    kpt live status
    

Frequently asked questions

Cleaning up

If you decide to stop using Config Controller, you should first clean up all resources created with Config Controller and then delete Config Controller itself.

  1. Delete the resources with kpt, from within the working directory:

    kpt live destroy
    
  2. Wait until all resources are deleted:

    until [ -z "$(kubectl get -R -f . --ignore-not-found | tee /dev/fd/2)" ]; \
    do sleep 1; done
    

    This command will poll until all the resources have a status of Deleted.

    Use ctrl-c to interrupt, if needed.

What's next