Stay organized with collections Save and categorize content based on your preferences.

Managing GKE clusters with Config Controller and KRM blueprints

This tutorial shows how to compose the KRM blueprints with Config Controller to provision a Google Kubernetes Engine (GKE) cluster and the required networking infrastructure such as a Virtual Private Cloud (VPC) and a subnet to host the GKE cluster, and named IP ranges for pods and services. Follow along if you are a GKE cluster operator and want to declaratively manage your cluster configuration and the networking infrastructure.

Config Controller is a hosted service to provision and orchestrate Anthos and Google Cloud resources. It offers an API endpoint that can provision, actuate, and orchestrate Google Cloud resources as part of Anthos Config Management.

KRM blueprints are a way to package resources that are commonly used together while codifying best practices that can be rolled out across your organization.

The GKE cluster blueprint is a KRM blueprint that includes all the resources you need to manage a GKE cluster on top of an existing Google Cloud VPC, subnet and IP ranges. You can instantiate the blueprint multiple times to set up multiple clusters.

The Networking blueprints are a set of KRM blueprints that help you create the necessary networking components such as a VPC, subnets and alias IP ranges required to create a GKE cluster. You can instantiate these blueprints multiple times to set up multiple subnets and alias IP ranges as required for multiple clusters.

Objectives

  • Declaratively create networking infrastructure required to host a GKE cluster.
  • Configure a GKE cluster declaratively within this networking infrastructure.
  • Apply the configuration using Config Controller.

Costs

This tutorial uses the following billable components of Google Cloud:

For a full list of resources included in the GKE cluster blueprint, see the Resources section of the GKE package, and its sub-packages.

To generate a cost estimate based on your projected usage, use the pricing calculator.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Requirements

Before you begin

  1. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

    2. You run all commands in this tutorial from Cloud Shell.

Set up the environment

In Cloud Shell, run the following commands:

  1. Install kubectl, the primary command-line interface for Kubernetes:

    gcloud components install kubectl

  2. Install kpt, the primary command-line interface for KRM blueprints:

    gcloud components install kpt

  3. Configure kubectl and kpt to connect with Config Controller:

    gcloud anthos config controller get-credentials CONFIG_CONTROLLER_NAME \
    --location COMPUTE_REGION \
    --project CONFIG_CONTROLLER_PROJECT_ID

    Replace the following:

    • CONFIG_CONTROLLER_NAME: the name of your Config Controller cluster.

    • COMPUTE_REGION: the region of your Config Controller cluster (for example, us-central1).

    • CONFIG_CONTROLLER_PROJECT_ID: the project ID of your Config Controller cluster.

  4. Enable the Resource Manager API:

    gcloud services enable cloudresourcemanager.googleapis.com \
    --project PROJECT_ID

    Replace PROJECT_ID with the ID of your project.

  5. Verify that Config Connector is configured and healthy in the project namespace:

    kubectl get ConfigConnectorContext -n PROJECT_NAMESPACE \
    -o "custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,HEALTHY:.status.healthy"

    Replace PROJECT_NAMESPACE with the namespace you want to use for managing project resources (for example, config-control).

    Example output:

    NAMESPACE        NAME                                                HEALTHY
config-control   configconnectorcontext.core.cnrm.cloud.google.com   true

Configure a VPC, a subnet, and alias IP ranges for the cluster

Configure the VPC

To set up and configure the VPC with the Networking blueprints, you run the following commands.

  1. Fetch the Networking blueprints with kpt, from within the desired working directory:

    kpt pkg get \
  https://github.com/GoogleCloudPlatform/blueprints.git/catalog/networking/network/vpc@networking-blueprint-v0.4.0 \
  VPC_NAME

    Replace VPC_NAME with the desired name to use for the VPC (for example, my-vpc).

  2. Move in to the newly created directory:

    cd VPC_NAME

  3. Configure the package by modifying the setters.yaml file:

    Update the following fields in the file. Make sure to provide your own project ID:

      namespace: PROJECT_NAMESPACE
  network-name: VPC_NAME
  project-id: PROJECT_ID

    Replace the following:

    • PROJECT_ID: the ID of your project.

      For this tutorial, the cluster and network are deployed to the same project.

    • PROJECT_NAMESPACE: the namespace to use for managing project resources (for example, config-control).

      For this tutorial, the cluster, network, and service enablement are managed in the same namespace.

  4. Add a new field called prefix at the end of the file. Make sure you have the indentation right:

      prefix: NAT-PREFIX

    Replace NAT-PREFIX with a prefix (for example, nat).

    This is used as a prefix for the NAT name when setting up the VPC.

    Your file would look something like the following:

    apiVersion: v1
kind: ConfigMap
metadata: # kpt-merge: /setters
  name: setters
data:
  namespace: PROJECT_NAMESPACE
  network-name: VPC_NAME
  project-id: PROJECT_ID
  prefix: NAT-PREFIX

  5. Use the kpt set-namespace function to change the namespace in the blueprint as follows:

      kpt fn eval --image set-namespace:v0.1 -- namespace=PROJECT_NAMESPACE

    Replace PROJECT_NAMESPACE with the namespace used for managing project resources (for example, config-control).

Configure the subnet

  1. Fetch the subnetwork blueprint with kpt, from within the VPC_NAME directory:

    kpt pkg get \
  https://github.com/GoogleCloudPlatform/blueprints.git/catalog/networking/network/subnet@networking-blueprint-v0.4.0 \
  SUBNET_NAME

    Replace SUBNET_NAME with the namespace used for managing project resources (for example, gke-subnet).

  2. Move into the subnet directory:

    cd SUBNET_NAME

  3. Edit the subnet.yaml file and add the following snippet to the end of the file, under the spec section. This defines the two named ranges which will be used to allocate the IP addresses for the GKE cluster Pods and Services:

      secondaryIpRange:
  - ipCidrRange: 172.17.0.0/16
    rangeName: pods
  - ipCidrRange: 172.18.0.0/16
    rangeName: services

    Your subnet.yaml file would look something like the following:

    apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeSubnetwork
metadata: # kpt-merge: networking/network-name-subnetwork
  name: network-name-subnetwork # kpt-set: ${prefix}${network-name}-subnetwork
  namespace: networking # kpt-set: ${namespace}
  annotations:
    cnrm.cloud.google.com/project-id: project-id # kpt-set: ${project-id}
    cnrm.cloud.google.com/blueprint: cnrm/landing-zone:networking/v0.4.0
spec:
  description: Subnetwork
  ipCidrRange: 10.2.0.0/16 # kpt-set: ${ip-cidr-range}
  logConfig:
    metadata: INCLUDE_ALL_METADATA
    aggregationInterval: INTERVAL_10_MIN
    flowSampling: 0.5
  networkRef:
    name: network-name # kpt-set: ${network-name}
  privateIpGoogleAccess: false
  region: us-central1 # kpt-set: ${region}
  secondaryIpRange:
  - ipCidrRange: 172.17.0.0/16
    rangeName: pods
  - ipCidrRange: 172.18.0.0/16
    rangeName: services

Render the blueprint

Before the blueprint can be applied, it needs to be rendered first. This step executes the pipeline of functions, as defined in the Kptfile, on the resources in the blueprint. A typical example of a function that may be executed is apply-setters which applies the setters that you edited earlier.

  1. Now move back into the VPC_NAME directory and then use kpt to render the setter values into the templated resources:

    cd ..
kpt fn render

    The output should be similar to the following:

    [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 5.4s
  Results:
    [INFO] set field value to "ALL_SUBNETWORKS_ALL_IP_RANGES" in file "nat.yaml" in field "spec.sourceSubnetworkIpRangesToNat"
    [INFO] set field value to "10.2.0.0/16" in file "subnet.yaml" in field "spec.ipCidrRange"

Package "my-vpc":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 2.3s
  Results:
    [INFO] set field value to "00-my-vpc-router-nat" in file "nat.yaml" in field "metadata.name"
    [INFO] set field value to "config-control" in file "nat.yaml" in field "metadata.namespace"
    [INFO] set field value to "krm-playground-00" in file "nat.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
    [INFO] set field value to "00-my-vpc-router" in file "nat.yaml" in field "spec.routerRef.name"
    ...(13 line(s) truncated, use '--truncate-output=false' to disable)

Successfully executed 2 function(s) in 2 package(s).

Apply config changes

Local changes in the previous steps do not affect the cloud until applied.

To apply your configuration changes, you run the following commands.

  1. Initialize the working directory with kpt, which creates a resource to track changes:

    kpt live init --namespace PROJECT_NAMESPACE

    Replace PROJECT_NAMESPACE with the namespace used for managing project resources (for example, config-control).

    initializing Kptfile inventory info (namespace: config-control)...success

  2. Preview the resources that will be created:

    kpt live apply --dry-run

    All resources should say "created (dry-run)".

    Example output:

    computerouter.compute.cnrm.cloud.google.com/my-vpc-router created (dry-run)
computerouternat.compute.cnrm.cloud.google.com/my-vpc-router-nat created (dry-run)
computesubnetwork.compute.cnrm.cloud.google.com/my-vpc-subnetwork created (dry-run)
service.serviceusage.cnrm.cloud.google.com/proj-id-00-compute created (dry-run)
computenetwork.compute.cnrm.cloud.google.com/my-vpc created (dry-run)
5 resource(s) applied. 5 created, 0 unchanged, 0 configured, 0 failed (dry-run)

  3. Apply the resources with kpt:

    kpt live apply

    All resources should say "reconciled".

    Example output:

    computenetwork.compute.cnrm.cloud.google.com/my-vpc created
computerouter.compute.cnrm.cloud.google.com/my-vpc-router created
computerouternat.compute.cnrm.cloud.google.com/my-vpc-router-nat created
computesubnetwork.compute.cnrm.cloud.google.com/my-vpc-subnetwork created
service.serviceusage.cnrm.cloud.google.com/proj-id-00-compute created
5 resource(s) applied. 5 created, 0 unchanged, 0 configured, 0 failed

Verify networking resources are created successfully

To verify that your changes are applied and the resources they specify are provisioned, you run the following commands.

  1. Wait until the resources are ready:

    kpt live status --output table --poll-until current

    This command will poll until all the resources have a status of Current and a condition of Ready.

    Use ctrl-c to interrupt, if needed.

    Example output:

    NAMESPACE   RESOURCE                                STATUS      CONDITIONS      AGE     MESSAGE
config-con  ComputeNetwork/my-vpc                   Current     Ready           2m      Resource is Ready
config-con  ComputeRouter/my-vpc-router             Current     Ready           2m      Resource is Ready
config-con  ComputeRouterNAT/my-vpc-router-nat      Current     Ready           2m      Resource is Ready
config-con  ComputeSubnetwork/my-vpc-subnetwork     Current     Ready           2m      Resource is Ready
config-con  Service/proj-id-00-compute              Current     Ready           2m      Resource is Ready

  2. In the case of error, use the default event output to see full error messages:

    kpt live status

    It takes a few minutes for all the resources to be created and ready.

    Once the networking resources have been successfully created move up one directory to start configuring the GKE cluster.

    cd ..

Configure a GKE cluster

To configure a GKE cluster with the GKE cluster blueprint, you run the following commands.

  1. Fetch the GKE cluster blueprint with kpt, from within the desired working directory:

    kpt pkg get \
    https://github.com/GoogleCloudPlatform/blueprints.git/catalog/gke@gke-blueprint-v0.4.0 \
    CLUSTER_NAME

    Replace CLUSTER_NAME with the desired name to use for the GKE cluster (for example, hello-cluster).

  2. Move into the cluster directory:

    cd ./CLUSTER_NAME/

  3. Configure the package by modifying the setters.yaml file:

    cat > setters.yaml << EOF
apiVersion: v1
kind: ConfigMap
metadata: # kpt-merge: /setters
  name: setters
data:
  # The name of this cluster
  cluster-name: CLUSTER_NAME
  # The compute location (region for a regional cluster or zone for a zonal cluster)
  location: us-central1
  # The private IP range for masters to use when peering to the VPC
  master-ip-range: 10.254.0.0/28
  # The reference to the network
  network-ref: projects/PROJECT_ID/global/networks/VPC_NAME
  # The reference to the subnet
  subnet-ref: projects/PROJECT_ID/regions/us-central1/subnetworks/subnetwork
  # The namespace in which to manage cluster resources
  platform-namespace: PROJECT_NAMESPACE
  # The project in which to manage cluster resources
  project-id: PROJECT_ID
  # The namespace in which to manage service enablement resources
  projects-namespace: PROJECT_NAMESPACE
  # The private IP range name for Pods to use, this range must already exist
  pods-range-name: pods
  # The private IP range name for services to use, this range must already exist
  services-range-name: services
  # The group in which to manage the list of groups that can be used for RBAC.
  # Must be named exactly 'gke-security-groups'.
  security-group: gke-security-groups@YOUR_DOMAIN
EOF

    Replace the following:

    • PROJECT_ID: the ID of your project.

      For this tutorial, the cluster and network are deployed to the same project.

    • PROJECT_NAMESPACE: the namespace to use for managing project resources (for example, config-control).

      For this tutorial, the cluster, network, and service enablement are managed in the same namespace.

    • YOUR_DOMAIN: the domain used by your Google Groups (for example, example.com).

    • network-ref: the selfLink reference to the network where the cluster should be created.

      It's in the format: projects/{network-project-id}/global/networks/{vpc-name}

    • subnet-ref the selfLink reference to the subnet where the cluster should be created.

      It's in the format: projects/{network-project-id}/regions/{region}/subnetworks/{subnet-name}

    All other data fields can be reconfigured as desired.

    The defaults provided should work in an otherwise empty project with the default network.

Disabling Google Groups for RBAC

If you do not wish to configure RBAC to be able to use Google Groups for authorizing, in addition to just individuals, you can change the cluster configuration to disable Google Groups for RBAC feature. This might be needed, for example,if you do not have the gke-security-groups created and lack the permissions to create it. To learn more, see group setup

  1. To disable Google Groups for RBAC, simply edit the cluster/cluster.yaml file directly.

  2. Find the section with the authenticatorGroupsConfig field and remove the following three lines:

      # Enable Groups for GKE, to allow role binding to Google Groups.
  authenticatorGroupsConfig:
    securityGroup: gke-security-group@example.com # kpt-set: ${security-group}

  3. Save the file

    This disables the RBAC for Google Groups feature.

Render the blueprint

This step executes the pipeline of functions, as defined in the Kptfile, on the resources in the blueprint. Typically this executes apply-setters which applies the setters that you edited earlier.

  1. Render the setter values into the templated resources:

    kpt fn render

    Example output:

    Package "example/cluster":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 3.3s
  Results:
    [INFO] set field value to "example-us-west4" in file "cluster.yaml" in field "metadata.name"
    [INFO] set field value to "config-control" in file "cluster.yaml" in field "metadata.namespace"
    [INFO] set field value to "project-id" in file "cluster.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
    ...(9 line(s) truncated, use '--truncate-output=false' to disable)

Package "test-00/nodepools/primary":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 2.2s
  Results:
    [INFO] set field value to "gke-example-us-east4-primary" in file "node-iam.yaml" in field "metadata.name"
    [INFO] set field value to "config-control" in file "node-iam.yaml" in field "metadata.namespace"
    [INFO] set field value to "project-id" in file "node-iam.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
    [INFO] set field value to "gke-example-us-east4-primary" in file "node-iam.yaml" in field "spec.displayName"
    ...(23 line(s) truncated, use '--truncate-output=false' to disable)

Package "test-00":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 2.3s
  Results:
    [INFO] set field value to "test-00" in file "cluster.yaml" in field "metadata.name"
    [INFO] set field value to "config-control" in file "cluster.yaml" in field "metadata.namespace"
    [INFO] set field value to "krm-playground-00" in file "cluster.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
    ...(36 line(s) truncated, use '--truncate-output=false' to disable)

Successfully executed 3 function(s) in 3 package(s).

Apply config changes

Local changes in the previous steps do not affect the cloud until applied.

To apply your configuration changes, you run the following commands.

  1. Initialize the working directory with kpt, which creates a resource to track changes:

    kpt live init --namespace PROJECT_NAMESPACE

    Replace PROJECT_NAMESPACE with the namespace used for managing project resources (for example, config-control).

  2. Preview the resources that will be created:

    kpt live apply --dry-run

    All resources should say "created (dry-run)".

    Example output:

    service.serviceusage.cnrm.cloud.google.com/proj-id-00-test-00-container created (dry-run)
containercluster.container.cnrm.cloud.google.com/test-00 created (dry-run)
containernodepool.container.cnrm.cloud.google.com/test-00-primary created (dry-run)
iampolicymember.iam.cnrm.cloud.google.com/artifactreader-gke-test-00-primary created (dry-run)
iampolicymember.iam.cnrm.cloud.google.com/logwriter-gke-test-00-primary created (dry-run)
iampolicymember.iam.cnrm.cloud.google.com/metricwriter-gke-test-00-primary created (dry-run)
iamserviceaccount.iam.cnrm.cloud.google.com/gke-test-00-primary created (dry-run)
7 resource(s) applied. 7 created, 0 unchanged, 0 configured, 0 failed (dry-run)

  3. Apply the resources with kpt:

    kpt live apply

    All resources should say "created".

    Example output:

    iamserviceaccount.iam.cnrm.cloud.google.com/gke-test-00-primary created
service.serviceusage.cnrm.cloud.google.com/proj-id-00-test-00-container created
containercluster.container.cnrm.cloud.google.com/test-00 created
containernodepool.container.cnrm.cloud.google.com/test-00-primary created
iampolicymember.iam.cnrm.cloud.google.com/artifactreader-gke-test-00-primary created
iampolicymember.iam.cnrm.cloud.google.com/logwriter-gke-test-00-primary created
iampolicymember.iam.cnrm.cloud.google.com/metricwriter-gke-test-00-primary created
7 resource(s) applied. 7 created, 0 unchanged, 0 configured, 0 failed

Verify GKE cluster resources are created successfully

To verify that your changes are applied and the resources they specify are provisioned, you run the following commands.

  1. Wait until the resources are ready:

    kpt live status --output table --poll-until current

    This command will poll until all the resources have a status of Current and a condition of Ready.

    Use ctrl-c to interrupt, if needed.

    Example output:

    NAMESPACE   RESOURCE                                  STATUS      CONDITIONS      AGE     MESSAGE
config-con  ContainerCluster/test-00                  Current     Ready           12m     Resource is Ready
config-con  ContainerNodePool/test-00-primary         Current     Ready           12m     Resource is Ready
config-con  IAMPolicyMember/artifactreader-gke-test-  Current     Ready           12m     Resource is Ready
config-con  IAMPolicyMember/logwriter-gke-test-00-pr  Current     Ready           12m     Resource is Ready
config-con  IAMPolicyMember/metricwriter-gke-test-00  Current     Ready           12m     Resource is Ready
config-con  IAMServiceAccount/gke-test-00-primary     Current     Ready           12m     Resource is Ready
config-con  Service/proj-id-00-test-00-contai         Current     Ready           12m     Resource is Ready

  2. In the case of error, use the default event output to see full error messages:

    kpt live status

Frequently asked questions

Cleaning up

If you decide to stop using Config Controller, you should first clean up all resources created with Config Controller and then delete Config Controller itself.

  1. Delete the cluster resources first with kpt, from within the GKE cluster blueprint working directory:

    kpt live destroy

  2. Wait until all resources are deleted:

    until [ -z "$(kubectl get -R -f . --ignore-not-found | tee /dev/fd/2)" ]; \
do sleep 1; done

    This command will poll until all the resources have a status of Deleted.

    Use ctrl-c to interrupt, if needed.

  3. Delete the networking resources if you created them as part of this tutorial. Use kpt from the VPC directory you created for the blueprint:

    kpt live destroy

  4. Wait until all resources are deleted:

    until [ -z "$(kubectl get -R -f . --ignore-not-found | tee /dev/fd/2)" ]; \
do sleep 1; done

    This command will poll until all the resources have a status of Deleted.

    Use ctrl-c to interrupt, if needed.

What's next