Managing GKE clusters with Config Controller and KRM blueprints
This tutorial shows how to compose the KRM blueprints with Config Controller to provision a Google Kubernetes Engine (GKE) cluster and the required networking infrastructure such as a Virtual Private Cloud (VPC) and a subnet to host the GKE cluster, and named IP ranges for pods and services. Follow along if you are a GKE cluster operator and want to declaratively manage your cluster configuration and the networking infrastructure.
Config Controller is a hosted service to provision and orchestrate Anthos and Google Cloud resources. It offers an API endpoint that can provision, actuate, and orchestrate Google Cloud resources as part of Anthos Config Management.
KRM blueprints are a way to package resources that are commonly used together while codifying best practices that can be rolled out across your organization.
The GKE cluster blueprint is a KRM blueprint that includes all the resources you need to manage a GKE cluster on top of an existing Google Cloud VPC, subnet and IP ranges. You can instantiate the blueprint multiple times to set up multiple clusters.
The Networking blueprints are a set of KRM blueprints that help you create the necessary networking components such as a VPC, subnets and alias IP ranges required to create a GKE cluster. You can instantiate these blueprints multiple times to set up multiple subnets and alias IP ranges as required for multiple clusters.
Objectives
- Declaratively create networking infrastructure required to host a GKE cluster.
- Configure a GKE cluster declaratively within this networking infrastructure.
- Apply the configuration using Config Controller.
Costs
This tutorial uses the following billable components of Google Cloud:
For a full list of resources included in the GKE cluster blueprint, see the Resources section of the GKE package, and its sub-packages.
To generate a cost estimate based on your projected usage, use the pricing calculator.
When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.
Requirements
You must have a Config Controller instance.
You must have a Google Cloud project. This is the project your GKE cluster will be created in. You can use the project your Config Controller cluster is in or a different one.
You must have billing enabled for your Google Cloud project.
Learn how to confirm that billing is enabled for your project.
You must have a Config Controller namespace configured to manage resources in your project.
By default, Config Controller comes with the
config-control
namespace pre-configured for use with Config Connector, and the instructions to set up Config Controller explain how to authorize Config Connector to manage your project.However, you can also use another namespace similarly configured. By convention, these project namespaces are usually named after the project ID of the project they manage.
Learn how to configure Config Connector to watch your namespace.
The GKE cluster blueprint enables Google Groups for RBAC for GKE by default. This is recommended because it allows authorizing groups instead of just individuals. However, if you prefer, you can disable Google Groups for RBAC
Google Groups for RBAC requires the following:
You must have a Google Workspace or Cloud Identity subscription.
Your organization must have a verified domain.
You must have a group named
gke-security-groups
in your organization.For more information about this security group, see Google Groups for GKE.
Ways to create a group:
- Google Admin console
- Google Groups app (requires Google Groups for Business)
- Google Groups in Google Cloud console (requires Google Groups for Business)
Before you begin
-
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
You run all commands in this tutorial from Cloud Shell.
Set up the environment
In Cloud Shell, run the following commands:
Install
kubectl
, the primary command-line interface for Kubernetes:gcloud components install kubectl
Install
kpt
, the primary command-line interface for KRM blueprints:gcloud components install kpt
Configure
kubectl
andkpt
to connect with Config Controller:gcloud anthos config controller get-credentials CONFIG_CONTROLLER_NAME \ --location COMPUTE_REGION \ --project CONFIG_CONTROLLER_PROJECT_ID
Replace the following:
CONFIG_CONTROLLER_NAME
: the name of your Config Controller cluster.COMPUTE_REGION
: the region of your Config Controller cluster (for example,us-central1
).CONFIG_CONTROLLER_PROJECT_ID
: the project ID of your Config Controller cluster.
Enable the Resource Manager API:
gcloud services enable cloudresourcemanager.googleapis.com \ --project PROJECT_ID
Replace
PROJECT_ID
with the ID of your project.Verify that Config Connector is configured and healthy in the project namespace:
kubectl get ConfigConnectorContext -n PROJECT_NAMESPACE \ -o "custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,HEALTHY:.status.healthy"
Replace
PROJECT_NAMESPACE
with the namespace you want to use for managing project resources (for example,config-control
).Example output:
NAMESPACE NAME HEALTHY config-control configconnectorcontext.core.cnrm.cloud.google.com true
Configure a VPC, a subnet, and alias IP ranges for the cluster
Configure the VPC
To set up and configure the VPC with the Networking blueprints, you run the following commands.
Fetch the Networking blueprints with
kpt
, from within the desired working directory:kpt pkg get \ https://github.com/GoogleCloudPlatform/blueprints.git/catalog/networking/network/vpc@networking-blueprint-v0.4.0 \ VPC_NAME
Replace
VPC_NAME
with the desired name to use for the VPC (for example,my-vpc
).Move in to the newly created directory:
cd VPC_NAME
Configure the package by modifying the
setters.yaml
file:Update the following fields in the file. Make sure to provide your own project ID:
namespace: PROJECT_NAMESPACE network-name: VPC_NAME project-id: PROJECT_ID
Replace the following:
PROJECT_ID
: the ID of your project.For this tutorial, the cluster and network are deployed to the same project.
PROJECT_NAMESPACE
: the namespace to use for managing project resources (for example,config-control
).For this tutorial, the cluster, network, and service enablement are managed in the same namespace.
Add a new field called prefix at the end of the file. Make sure you have the indentation right:
prefix: NAT-PREFIX
Replace NAT-PREFIX with a prefix (for example,
nat
).This is used as a prefix for the NAT name when setting up the VPC.
Your file would look something like the following:
apiVersion: v1 kind: ConfigMap metadata: # kpt-merge: /setters name: setters data: namespace: PROJECT_NAMESPACE network-name: VPC_NAME project-id: PROJECT_ID prefix: NAT-PREFIX
Use the
kpt
set-namespace
function to change the namespace in the blueprint as follows:kpt fn eval --image set-namespace:v0.1 -- namespace=PROJECT_NAMESPACE
Replace
PROJECT_NAMESPACE
with the namespace used for managing project resources (for example,config-control
).
Configure the subnet
Fetch the subnetwork blueprint with
kpt
, from within theVPC_NAME
directory:kpt pkg get \ https://github.com/GoogleCloudPlatform/blueprints.git/catalog/networking/network/subnet@networking-blueprint-v0.4.0 \ SUBNET_NAME
Replace
SUBNET_NAME
with the namespace used for managing project resources (for example,gke-subnet
).Move into the subnet directory:
cd SUBNET_NAME
Edit the
subnet.yaml
file and add the following snippet to the end of the file, under thespec
section. This defines the two named ranges which will be used to allocate the IP addresses for the GKE cluster Pods and Services:secondaryIpRange: - ipCidrRange: 172.17.0.0/16 rangeName: pods - ipCidrRange: 172.18.0.0/16 rangeName: services
Your
subnet.yaml
file would look something like the following:apiVersion: compute.cnrm.cloud.google.com/v1beta1 kind: ComputeSubnetwork metadata: # kpt-merge: networking/network-name-subnetwork name: network-name-subnetwork # kpt-set: ${prefix}${network-name}-subnetwork namespace: networking # kpt-set: ${namespace} annotations: cnrm.cloud.google.com/project-id: project-id # kpt-set: ${project-id} cnrm.cloud.google.com/blueprint: cnrm/landing-zone:networking/v0.4.0 spec: description: Subnetwork ipCidrRange: 10.2.0.0/16 # kpt-set: ${ip-cidr-range} logConfig: metadata: INCLUDE_ALL_METADATA aggregationInterval: INTERVAL_10_MIN flowSampling: 0.5 networkRef: name: network-name # kpt-set: ${network-name} privateIpGoogleAccess: false region: us-central1 # kpt-set: ${region} secondaryIpRange: - ipCidrRange: 172.17.0.0/16 rangeName: pods - ipCidrRange: 172.18.0.0/16 rangeName: services
Render the blueprint
Before the blueprint can be applied, it needs to be rendered first. This step
executes the pipeline of functions, as defined in the Kptfile
, on the
resources in the blueprint. A typical example of a function that may be
executed is apply-setters
which applies the setters that you edited earlier.
Now move back into the VPC_NAME directory and then use
kpt
to render the setter values into the templated resources:cd .. kpt fn render
The output should be similar to the following:
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1" [PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 5.4s Results: [INFO] set field value to "ALL_SUBNETWORKS_ALL_IP_RANGES" in file "nat.yaml" in field "spec.sourceSubnetworkIpRangesToNat" [INFO] set field value to "10.2.0.0/16" in file "subnet.yaml" in field "spec.ipCidrRange" Package "my-vpc": [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1" [PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 2.3s Results: [INFO] set field value to "00-my-vpc-router-nat" in file "nat.yaml" in field "metadata.name" [INFO] set field value to "config-control" in file "nat.yaml" in field "metadata.namespace" [INFO] set field value to "krm-playground-00" in file "nat.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id" [INFO] set field value to "00-my-vpc-router" in file "nat.yaml" in field "spec.routerRef.name" ...(13 line(s) truncated, use '--truncate-output=false' to disable) Successfully executed 2 function(s) in 2 package(s).
Apply config changes
Local changes in the previous steps do not affect the cloud until applied.
To apply your configuration changes, you run the following commands.
Initialize the working directory with kpt, which creates a resource to track changes:
kpt live init --namespace PROJECT_NAMESPACE
Replace
PROJECT_NAMESPACE
with the namespace used for managing project resources (for example,config-control
).initializing Kptfile inventory info (namespace: config-control)...success
Preview the resources that will be created:
kpt live apply --dry-run
All resources should say "created (dry-run)".
Example output:
computerouter.compute.cnrm.cloud.google.com/my-vpc-router created (dry-run) computerouternat.compute.cnrm.cloud.google.com/my-vpc-router-nat created (dry-run) computesubnetwork.compute.cnrm.cloud.google.com/my-vpc-subnetwork created (dry-run) service.serviceusage.cnrm.cloud.google.com/proj-id-00-compute created (dry-run) computenetwork.compute.cnrm.cloud.google.com/my-vpc created (dry-run) 5 resource(s) applied. 5 created, 0 unchanged, 0 configured, 0 failed (dry-run)
Apply the resources with kpt:
kpt live apply
All resources should say "reconciled".
Example output:
computenetwork.compute.cnrm.cloud.google.com/my-vpc created computerouter.compute.cnrm.cloud.google.com/my-vpc-router created computerouternat.compute.cnrm.cloud.google.com/my-vpc-router-nat created computesubnetwork.compute.cnrm.cloud.google.com/my-vpc-subnetwork created service.serviceusage.cnrm.cloud.google.com/proj-id-00-compute created 5 resource(s) applied. 5 created, 0 unchanged, 0 configured, 0 failed
Verify networking resources are created successfully
To verify that your changes are applied and the resources they specify are provisioned, you run the following commands.
Wait until the resources are ready:
kpt live status --output table --poll-until current
This command will poll until all the resources have a status of
Current
and a condition ofReady
.Use
ctrl-c
to interrupt, if needed.Example output:
NAMESPACE RESOURCE STATUS CONDITIONS AGE MESSAGE config-con ComputeNetwork/my-vpc Current Ready 2m Resource is Ready config-con ComputeRouter/my-vpc-router Current Ready 2m Resource is Ready config-con ComputeRouterNAT/my-vpc-router-nat Current Ready 2m Resource is Ready config-con ComputeSubnetwork/my-vpc-subnetwork Current Ready 2m Resource is Ready config-con Service/proj-id-00-compute Current Ready 2m Resource is Ready
In the case of error, use the default event output to see full error messages:
kpt live status
It takes a few minutes for all the resources to be created and ready.
Once the networking resources have been successfully created move up one directory to start configuring the GKE cluster.
cd ..
Configure a GKE cluster
To configure a GKE cluster with the GKE cluster blueprint, you run the following commands.
Fetch the GKE cluster blueprint with
kpt
, from within the desired working directory:kpt pkg get \ https://github.com/GoogleCloudPlatform/blueprints.git/catalog/gke@gke-blueprint-v0.4.0 \ CLUSTER_NAME
Replace
CLUSTER_NAME
with the desired name to use for the GKE cluster (for example,hello-cluster
).Move into the cluster directory:
cd ./CLUSTER_NAME/
Configure the package by modifying the
setters.yaml
file:cat > setters.yaml << EOF apiVersion: v1 kind: ConfigMap metadata: # kpt-merge: /setters name: setters data: # The name of this cluster cluster-name: CLUSTER_NAME # The compute location (region for a regional cluster or zone for a zonal cluster) location: us-central1 # The private IP range for masters to use when peering to the VPC master-ip-range: 10.254.0.0/28 # The reference to the network network-ref: projects/PROJECT_ID/global/networks/VPC_NAME # The reference to the subnet subnet-ref: projects/PROJECT_ID/regions/us-central1/subnetworks/subnetwork # The namespace in which to manage cluster resources platform-namespace: PROJECT_NAMESPACE # The project in which to manage cluster resources project-id: PROJECT_ID # The namespace in which to manage service enablement resources projects-namespace: PROJECT_NAMESPACE # The private IP range name for Pods to use, this range must already exist pods-range-name: pods # The private IP range name for services to use, this range must already exist services-range-name: services # The group in which to manage the list of groups that can be used for RBAC. # Must be named exactly 'gke-security-groups'. security-group: gke-security-groups@YOUR_DOMAIN EOF
Replace the following:
PROJECT_ID
: the ID of your project.For this tutorial, the cluster and network are deployed to the same project.
PROJECT_NAMESPACE
: the namespace to use for managing project resources (for example,config-control
).For this tutorial, the cluster, network, and service enablement are managed in the same namespace.
YOUR_DOMAIN
: the domain used by your Google Groups (for example,example.com
).network-ref
: the selfLink reference to the network where the cluster should be created.It's in the format:
projects/{network-project-id}/global/networks/{vpc-name}
subnet-ref
the selfLink reference to the subnet where the cluster should be created.It's in the format:
projects/{network-project-id}/regions/{region}/subnetworks/{subnet-name}
All other data fields can be reconfigured as desired.
The defaults provided should work in an otherwise empty project with the default network.
Disabling Google Groups for RBAC
If you do not wish to configure RBAC to be able to use Google Groups for
authorizing, in addition to just individuals, you can change the cluster configuration to disable Google Groups for RBAC feature. This might
be needed, for example,if you do not have the gke-security-groups
created
and lack the permissions to create it. To learn more, see group setup
To disable Google Groups for RBAC, simply edit the
cluster/cluster.yaml
file directly.Find the section with the
authenticatorGroupsConfig
field and remove the following three lines:# Enable Groups for GKE, to allow role binding to Google Groups. authenticatorGroupsConfig: securityGroup: gke-security-group@example.com # kpt-set: ${security-group}
Save the file
This disables the RBAC for Google Groups feature.
Render the blueprint
This step executes the pipeline of functions, as defined in the Kptfile
, on
the resources in the blueprint. Typically this executes apply-setters
which
applies the setters that you edited earlier.
Render the setter values into the templated resources:
kpt fn render
Example output:
Package "example/cluster": [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1" [PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 3.3s Results: [INFO] set field value to "example-us-west4" in file "cluster.yaml" in field "metadata.name" [INFO] set field value to "config-control" in file "cluster.yaml" in field "metadata.namespace" [INFO] set field value to "project-id" in file "cluster.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id" ...(9 line(s) truncated, use '--truncate-output=false' to disable) Package "test-00/nodepools/primary": [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1" [PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 2.2s Results: [INFO] set field value to "gke-example-us-east4-primary" in file "node-iam.yaml" in field "metadata.name" [INFO] set field value to "config-control" in file "node-iam.yaml" in field "metadata.namespace" [INFO] set field value to "project-id" in file "node-iam.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id" [INFO] set field value to "gke-example-us-east4-primary" in file "node-iam.yaml" in field "spec.displayName" ...(23 line(s) truncated, use '--truncate-output=false' to disable) Package "test-00": [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1" [PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 2.3s Results: [INFO] set field value to "test-00" in file "cluster.yaml" in field "metadata.name" [INFO] set field value to "config-control" in file "cluster.yaml" in field "metadata.namespace" [INFO] set field value to "krm-playground-00" in file "cluster.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id" ...(36 line(s) truncated, use '--truncate-output=false' to disable) Successfully executed 3 function(s) in 3 package(s).
Apply config changes
Local changes in the previous steps do not affect the cloud until applied.
To apply your configuration changes, you run the following commands.
Initialize the working directory with kpt, which creates a resource to track changes:
kpt live init --namespace PROJECT_NAMESPACE
Replace
PROJECT_NAMESPACE
with the namespace used for managing project resources (for example,config-control
).Preview the resources that will be created:
kpt live apply --dry-run
All resources should say "created (dry-run)".
Example output:
service.serviceusage.cnrm.cloud.google.com/proj-id-00-test-00-container created (dry-run) containercluster.container.cnrm.cloud.google.com/test-00 created (dry-run) containernodepool.container.cnrm.cloud.google.com/test-00-primary created (dry-run) iampolicymember.iam.cnrm.cloud.google.com/artifactreader-gke-test-00-primary created (dry-run) iampolicymember.iam.cnrm.cloud.google.com/logwriter-gke-test-00-primary created (dry-run) iampolicymember.iam.cnrm.cloud.google.com/metricwriter-gke-test-00-primary created (dry-run) iamserviceaccount.iam.cnrm.cloud.google.com/gke-test-00-primary created (dry-run) 7 resource(s) applied. 7 created, 0 unchanged, 0 configured, 0 failed (dry-run)
Apply the resources with kpt:
kpt live apply
All resources should say "created".
Example output:
iamserviceaccount.iam.cnrm.cloud.google.com/gke-test-00-primary created service.serviceusage.cnrm.cloud.google.com/proj-id-00-test-00-container created containercluster.container.cnrm.cloud.google.com/test-00 created containernodepool.container.cnrm.cloud.google.com/test-00-primary created iampolicymember.iam.cnrm.cloud.google.com/artifactreader-gke-test-00-primary created iampolicymember.iam.cnrm.cloud.google.com/logwriter-gke-test-00-primary created iampolicymember.iam.cnrm.cloud.google.com/metricwriter-gke-test-00-primary created 7 resource(s) applied. 7 created, 0 unchanged, 0 configured, 0 failed
Verify GKE cluster resources are created successfully
To verify that your changes are applied and the resources they specify are provisioned, you run the following commands.
Wait until the resources are ready:
kpt live status --output table --poll-until current
This command will poll until all the resources have a status of
Current
and a condition ofReady
.Use
ctrl-c
to interrupt, if needed.Example output:
NAMESPACE RESOURCE STATUS CONDITIONS AGE MESSAGE config-con ContainerCluster/test-00 Current Ready 12m Resource is Ready config-con ContainerNodePool/test-00-primary Current Ready 12m Resource is Ready config-con IAMPolicyMember/artifactreader-gke-test- Current Ready 12m Resource is Ready config-con IAMPolicyMember/logwriter-gke-test-00-pr Current Ready 12m Resource is Ready config-con IAMPolicyMember/metricwriter-gke-test-00 Current Ready 12m Resource is Ready config-con IAMServiceAccount/gke-test-00-primary Current Ready 12m Resource is Ready config-con Service/proj-id-00-test-00-contai Current Ready 12m Resource is Ready
In the case of error, use the default event output to see full error messages:
kpt live status
Frequently asked questions
Cleaning up
If you decide to stop using Config Controller, you should first clean up all resources created with Config Controller and then delete Config Controller itself.
Delete the cluster resources first with
kpt
, from within the GKE cluster blueprint working directory:kpt live destroy
Wait until all resources are deleted:
until [ -z "$(kubectl get -R -f . --ignore-not-found | tee /dev/fd/2)" ]; \ do sleep 1; done
This command will poll until all the resources have a status of
Deleted
.Use
ctrl-c
to interrupt, if needed.Delete the networking resources if you created them as part of this tutorial. Use
kpt
from the VPC directory you created for the blueprint:kpt live destroy
Wait until all resources are deleted:
until [ -z "$(kubectl get -R -f . --ignore-not-found | tee /dev/fd/2)" ]; \ do sleep 1; done
This command will poll until all the resources have a status of
Deleted
.Use
ctrl-c
to interrupt, if needed.
What's next
- Learn more about blueprints.
- Browse the KRM blueprint catalog.
- Browse the Terraform blueprint catalog.
- See GKE Autopilot blueprint for details about how to create an GKE Autopilot cluster.
- See Anthos cluster blueprint for details about how to create an Anthos cluster.