Managing GKE clusters with Config Controller and KRM blueprints

This tutorial shows how to compose the KRM blueprints with Config Controller to provision a Google Kubernetes Engine (GKE) cluster and the required networking infrastructure such as a Virtual Private Cloud (VPC) and a subnet to host the GKE cluster, and named IP ranges for pods and services. Follow along if you are a GKE cluster operator and want to declaratively manage your cluster configuration and the networking infrastructure.

Config Controller is a hosted service to provision and orchestrate Anthos and Google Cloud resources. It offers an API endpoint that can provision, actuate, and orchestrate Google Cloud resources as part of Anthos Config Management.

KRM blueprints are a way to package resources that are commonly used together while codifying best practices that can be rolled out across your organization.

The GKE cluster blueprint is a KRM blueprint that includes all the resources you need to manage a GKE cluster on top of an existing Google Cloud VPC, subnet and IP ranges. You can instantiate the blueprint multiple times to set up multiple clusters.

The Networking blueprints are a set of KRM blueprints that help you create the necessary networking components such as a VPC, subnets and alias IP ranges required to create a GKE cluster. You can instantiate these blueprints multiple times to set up multiple subnets and alias IP ranges as required for multiple clusters.

Objectives

• Declaratively create networking infrastructure required to host a GKE cluster.
• Configure a GKE cluster declaratively within this networking infrastructure.
• Apply the configuration using Config Controller.

Costs

This tutorial uses the following billable components of Google Cloud:

• Compute Engine
• Google Kubernetes Engine

For a full list of resources included in the GKE cluster blueprint, see the Resources section of the GKE package, and its sub-packages.

To generate a cost estimate based on your projected usage, use the pricing calculator.

Note: The GKE cluster blueprint is configured with defaults that are intended to align with security and production readiness best practices. For this reason, the default configuration can cost more than default clusters created with other methods. Please review the configuration carefully before applying.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Requirements

• You must have a Config Controller instance.

  Learn how to set up Config Controller.

  Note: This tutorial uses kpt to apply resources. GitOps is not required.

• You must have a Google Cloud project. This is the project your GKE cluster will be created in. You can use the project your Config Controller cluster is in or a different one.

  Learn how to create and manage projects

• You must have billing enabled for your Google Cloud project.

  Learn how to confirm that billing is enabled for your project.

• You must have a Config Controller namespace configured to manage resources in your project.

  By default, Config Controller comes with the config-control namespace pre-configured for use with Config Connector, and the instructions to set up Config Controller explain how to authorize Config Connector to manage your project.

  However, you can also use another namespace similarly configured. By convention, these project namespaces are usually named after the project ID of the project they manage.

  Learn how to configure Config Connector to watch your namespace.

• The GKE cluster blueprint enables Google Groups for RBAC for GKE by default. This is recommended because it allows authorizing groups instead of just individuals. However, if you prefer, you can disable Google Groups for RBAC

  Google Groups for RBAC requires the following:

  • You must have a Google Workspace or Cloud Identity subscription.

    Learn how to set up Cloud Identity.

    Learn how to set up Google Workspace.

  • Your organization must have a verified domain.

    Learn how to verify your domain with Cloud Identity.

    Learn how to verify your domain with Google Workspace.

  • You must have a group named gke-security-groups in your organization.

    For more information about this security group, see Groups for GKE.

    Ways to create a group:

    • Google Workspace Admin Console
    • Groups app (requires Groups for Business)
    • Groups in Cloud console (requires Groups for Business)

    Learn how to turn Groups for Business on or off.

Note: You do not need an Anthos entitlement to follow this tutorial.

Before you begin

1. In the Cloud console, activate Cloud Shell.

   Activate Cloud Shell

   At the bottom of the Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

You run all commands in this tutorial from Cloud Shell.

Set up the environment

In Cloud Shell, run the following commands:

1. Install kubectl, the primary command-line interface for Kubernetes:

   gcloud components install kubectl
   

2. Install kpt, the primary command-line interface for KRM blueprints:

   gcloud components install kpt
   

   Note: This tutorial requires kpt v1+.

3. Configure kubectl and kpt to connect with Config Controller:

   gcloud anthos config controller get-credentials CONFIG_CONTROLLER_NAME \
       --location COMPUTE_REGION \
       --project CONFIG_CONTROLLER_PROJECT_ID
   

   Replace the following:

   • CONFIG_CONTROLLER_NAME: the name of your Config Controller cluster.

   • COMPUTE_REGION: the region of your Config Controller cluster (for example, us-central1).

   • CONFIG_CONTROLLER_PROJECT_ID: the project ID of your Config Controller cluster.

   Note: Currently Config Controller is only available in the us-central1, us-east1, northamerica-northeast1, europe-north1, australia-southeast1 and, asia-northeast1 regions.

4. Enable the Resource Manager API:

   gcloud services enable cloudresourcemanager.googleapis.com \
       --project PROJECT_ID
   

   Replace PROJECT_ID with the ID of your project.

   Note: The Resource Manager API is used by Config Connector to manage enablement of other service APIs.

5. Verify that Config Connector is configured and healthy in the project namespace:

   kubectl get ConfigConnectorContext -n PROJECT_NAMESPACE \
       -o "custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,HEALTHY:.status.healthy"
   

   Replace PROJECT_NAMESPACE with the namespace you want to use for managing project resources (for example, config-control).

   Example output:

   NAMESPACE        NAME                                                HEALTHY
   config-control   configconnectorcontext.core.cnrm.cloud.google.com   true
   

Configure a VPC, a subnet, and alias IP ranges for the cluster

Note: The GKE cluster blueprint requires that you provide an existing VPC, subnet and alias IP ranges for the GKE cluster you are creating. If you do not have existing infrastructure that you would like to use then follow this section to create these resources using the Networking blueprints. If you already have the required networking resources you can skip this section and go directly to configure a GKE cluster section.

Configure the VPC

To set up and configure the VPC with the Networking blueprints, you run the following commands.

1. Fetch the Networking blueprints with kpt, from within the desired working directory:

   kpt pkg get \
     https://github.com/GoogleCloudPlatform/blueprints.git/catalog/networking/network/vpc@networking-blueprint-v0.4.0 \
     VPC_NAME
   

   Replace VPC_NAME with the desired name to use for the VPC (for example, my-vpc).

2. Move in to the newly created directory:

   cd VPC_NAME
   

3. Configure the package by modifying the setters.yaml file:

   Update the following fields in the file. Make sure to provide your own project ID:

     namespace: PROJECT_NAMESPACE
     network-name: VPC_NAME
     project-id: PROJECT_ID
   

   Replace the following:

   • PROJECT_ID: the ID of your project.

     For this tutorial, the cluster and network are deployed to the same project.

   • PROJECT_NAMESPACE: the namespace to use for managing project resources (for example, config-control).

     For this tutorial, the cluster, network, and service enablement are managed in the same namespace.

4. Add a new field called prefix at the end of the file. Make sure you have the indentation right:

     prefix: NAT-PREFIX
   

   Replace NAT-PREFIX with a prefix (for example, nat).

   This is used as a prefix for the NAT name when setting up the VPC.

   Your file would look something like the following:

   apiVersion: v1
   kind: ConfigMap
   metadata: # kpt-merge: /setters
     name: setters
   data:
     namespace: PROJECT_NAMESPACE
     network-name: VPC_NAME
     project-id: PROJECT_ID
     prefix: NAT-PREFIX
   

5. Use the kpt set-namespace function to change the namespace in the blueprint as follows:

     kpt fn eval --image set-namespace:v0.1 -- namespace=PROJECT_NAMESPACE
   

   Replace PROJECT_NAMESPACE with the namespace used for managing project resources (for example, config-control).

Configure the subnet

1. Fetch the subnetwork blueprint with kpt, from within the VPC_NAME directory:

   kpt pkg get \
     https://github.com/GoogleCloudPlatform/blueprints.git/catalog/networking/network/subnet@networking-blueprint-v0.4.0 \
     SUBNET_NAME
   

   Replace SUBNET_NAME with the namespace used for managing project resources (for example, gke-subnet).

2. Move into the subnet directory:

   cd SUBNET_NAME
   

3. Edit the subnet.yaml file and add the following snippet to the end of the file, under the spec section. This defines the two named ranges which will be used to allocate the IP addresses for the GKE cluster Pods and Services:

     secondaryIpRange:
     - ipCidrRange: 172.17.0.0/16
       rangeName: pods
     - ipCidrRange: 172.18.0.0/16
       rangeName: services
   

   Your subnet.yaml file would look something like the following:

   apiVersion: compute.cnrm.cloud.google.com/v1beta1
   kind: ComputeSubnetwork
   metadata: # kpt-merge: networking/network-name-subnetwork
     name: network-name-subnetwork # kpt-set: ${prefix}${network-name}-subnetwork
     namespace: networking # kpt-set: ${namespace}
     annotations:
       cnrm.cloud.google.com/project-id: project-id # kpt-set: ${project-id}
       cnrm.cloud.google.com/blueprint: cnrm/landing-zone:networking/v0.4.0
   spec:
     description: Subnetwork
     ipCidrRange: 10.2.0.0/16 # kpt-set: ${ip-cidr-range}
     logConfig:
       metadata: INCLUDE_ALL_METADATA
       aggregationInterval: INTERVAL_10_MIN
       flowSampling: 0.5
     networkRef:
       name: network-name # kpt-set: ${network-name}
     privateIpGoogleAccess: false
     region: us-central1 # kpt-set: ${region}
     secondaryIpRange:
     - ipCidrRange: 172.17.0.0/16
       rangeName: pods
     - ipCidrRange: 172.18.0.0/16
       rangeName: services
   

Render the blueprint

Before the blueprint can be applied, it needs to be rendered first. This step executes the pipeline of functions, as defined in the Kptfile, on the resources in the blueprint. A typical example of a function that may be executed is apply-setters which applies the setters that you edited earlier.

1. Now move back into the VPC_NAME directory and then use kpt to render the setter values into the templated resources:

   cd ..
   kpt fn render
   

   The output should be similar to the following:

   [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
   [PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 5.4s
     Results:
       [INFO] set field value to "ALL_SUBNETWORKS_ALL_IP_RANGES" in file "nat.yaml" in field "spec.sourceSubnetworkIpRangesToNat"
       [INFO] set field value to "10.2.0.0/16" in file "subnet.yaml" in field "spec.ipCidrRange"
   
   Package "my-vpc":
   [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
   [PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 2.3s
     Results:
       [INFO] set field value to "00-my-vpc-router-nat" in file "nat.yaml" in field "metadata.name"
       [INFO] set field value to "config-control" in file "nat.yaml" in field "metadata.namespace"
       [INFO] set field value to "krm-playground-00" in file "nat.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
       [INFO] set field value to "00-my-vpc-router" in file "nat.yaml" in field "spec.routerRef.name"
       ...(13 line(s) truncated, use '--truncate-output=false' to disable)
   
   Successfully executed 2 function(s) in 2 package(s).
   

Apply config changes

Local changes in the previous steps do not affect the cloud until applied.

To apply your configuration changes, you run the following commands.

1. Initialize the working directory with kpt, which creates a resource to track changes:

   kpt live init --namespace PROJECT_NAMESPACE
   

   Replace PROJECT_NAMESPACE with the namespace used for managing project resources (for example, config-control).

   initializing Kptfile inventory info (namespace: config-control)...success
   

2. Preview the resources that will be created:

   kpt live apply --dry-run
   

   All resources should say "created (dry-run)".

   Example output:

   computerouter.compute.cnrm.cloud.google.com/my-vpc-router created (dry-run)
   computerouternat.compute.cnrm.cloud.google.com/my-vpc-router-nat created (dry-run)
   computesubnetwork.compute.cnrm.cloud.google.com/my-vpc-subnetwork created (dry-run)
   service.serviceusage.cnrm.cloud.google.com/proj-id-00-compute created (dry-run)
   computenetwork.compute.cnrm.cloud.google.com/my-vpc created (dry-run)
   5 resource(s) applied. 5 created, 0 unchanged, 0 configured, 0 failed (dry-run)
   

3. Apply the resources with kpt:

   kpt live apply
   

   All resources should say "reconciled".

   Example output:

   computenetwork.compute.cnrm.cloud.google.com/my-vpc created
   computerouter.compute.cnrm.cloud.google.com/my-vpc-router created
   computerouternat.compute.cnrm.cloud.google.com/my-vpc-router-nat created
   computesubnetwork.compute.cnrm.cloud.google.com/my-vpc-subnetwork created
   service.serviceusage.cnrm.cloud.google.com/proj-id-00-compute created
   5 resource(s) applied. 5 created, 0 unchanged, 0 configured, 0 failed
   

Verify networking resources are created successfully

To verify that your changes are applied and the resources they specify are provisioned, you run the following commands.

1. Wait until the resources are ready:

   kpt live status --output table --poll-until current
   

   This command will poll until all the resources have a status of Current and a condition of Ready.

   Use ctrl-c to interrupt, if needed.

   Example output:

   NAMESPACE   RESOURCE                                STATUS      CONDITIONS      AGE     MESSAGE
   config-con  ComputeNetwork/my-vpc                   Current     Ready           2m      Resource is Ready
   config-con  ComputeRouter/my-vpc-router             Current     Ready           2m      Resource is Ready
   config-con  ComputeRouterNAT/my-vpc-router-nat      Current     Ready           2m      Resource is Ready
   config-con  ComputeSubnetwork/my-vpc-subnetwork     Current     Ready           2m      Resource is Ready
   config-con  Service/proj-id-00-compute              Current     Ready           2m      Resource is Ready
   

2. In the case of error, use the default event output to see full error messages:

   kpt live status
   

   It takes a few minutes for all the resources to be created and ready.

   Once the networking resources have been successfully created move up one directory to start configuring the GKE cluster.

   cd ..
   

Configure a GKE cluster

To configure a GKE cluster with the GKE cluster blueprint, you run the following commands.

1. Fetch the GKE cluster blueprint with kpt, from within the desired working directory:

   kpt pkg get \
       https://github.com/GoogleCloudPlatform/blueprints.git/catalog/gke@gke-blueprint-v0.4.0 \
       CLUSTER_NAME
   

   Replace CLUSTER_NAME with the desired name to use for the GKE cluster (for example, hello-cluster).

2. Move into the cluster directory:

   cd ./CLUSTER_NAME/
   

3. Configure the package by modifying the setters.yaml file:

   cat > setters.yaml << EOF
   apiVersion: v1
   kind: ConfigMap
   metadata: # kpt-merge: /setters
     name: setters
   data:
     # The name of this cluster
     cluster-name: CLUSTER_NAME
     # The compute location (region for a regional cluster or zone for a zonal cluster)
     location: us-central1
     # The private IP range for masters to use when peering to the VPC
     master-ip-range: 10.254.0.0/28
     # The reference to the network
     network-ref: projects/PROJECT_ID/global/networks/VPC_NAME
     # The reference to the subnet
     subnet-ref: projects/PROJECT_ID/regions/us-central1/subnetworks/subnetwork
     # The namespace in which to manage cluster resources
     platform-namespace: PROJECT_NAMESPACE
     # The project in which to manage cluster resources
     project-id: PROJECT_ID
     # The namespace in which to manage service enablement resources
     projects-namespace: PROJECT_NAMESPACE
     # The private IP range name for Pods to use, this range must already exist
     pods-range-name: pods
     # The private IP range name for services to use, this range must already exist
     services-range-name: services
     # The group in which to manage the list of groups that can be used for RBAC.
     # Must be named exactly 'gke-security-groups'.
     security-group: gke-security-groups@YOUR_DOMAIN
   EOF
   

   Replace the following:

   • PROJECT_ID: the ID of your project.

     For this tutorial, the cluster and network are deployed to the same project.

   • PROJECT_NAMESPACE: the namespace to use for managing project resources (for example, config-control).

     For this tutorial, the cluster, network, and service enablement are managed in the same namespace.

   • YOUR_DOMAIN: the domain used by your Groups (for example, example.com).

     Note: The security-group value must be a group email, not a service account or user email. The group must be named exactly gke-security-groups. If you do not have

   • network-ref: the selfLink reference to the network where the cluster should be created.

     It's in the format: projects/{network-project-id}/global/networks/{vpc-name}

   • subnet-ref the selfLink reference to the subnet where the cluster should be created.

     It's in the format: projects/{network-project-id}/regions/{region}/subnetworks/{subnet-name}

   All other data fields can be reconfigured as desired.

   The defaults provided should work in an otherwise empty project with the default network.

Disabling Google Groups for RBAC

If you do not wish to configure RBAC to be able to use Google Groups for authorizing, in addition to just individuals, you can change the cluster configuration to disable Google Groups for RBAC feature. This might be needed, for example,if you do not have the gke-security-groups created and lack the permissions to create it. To learn more, see group setup

1. To disable Google Groups for RBAC, simply edit the cluster/cluster.yaml file directly.

2. Find the section with the authenticatorGroupsConfig field and remove the following three lines:

     # Enable Groups for GKE, to allow role binding to Google Groups.
     authenticatorGroupsConfig:
       securityGroup: [e-mail] # kpt-set: ${security-group}
   

3. Save the file

   This disables the RBAC for Google Groups feature.

Render the blueprint

This step executes the pipeline of functions, as defined in the Kptfile, on the resources in the blueprint. Typically this executes apply-setters which applies the setters that you edited earlier.

1. Render the setter values into the templated resources:

   kpt fn render
   

   Example output:

   Package "example/cluster":
   [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
   [PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 3.3s
     Results:
       [INFO] set field value to "example-us-west4" in file "cluster.yaml" in field "metadata.name"
       [INFO] set field value to "config-control" in file "cluster.yaml" in field "metadata.namespace"
       [INFO] set field value to "project-id" in file "cluster.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
       ...(9 line(s) truncated, use '--truncate-output=false' to disable)
   
   Package "test-00/nodepools/primary":
   [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
   [PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 2.2s
     Results:
       [INFO] set field value to "gke-example-us-east4-primary" in file "node-iam.yaml" in field "metadata.name"
       [INFO] set field value to "config-control" in file "node-iam.yaml" in field "metadata.namespace"
       [INFO] set field value to "project-id" in file "node-iam.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
       [INFO] set field value to "gke-example-us-east4-primary" in file "node-iam.yaml" in field "spec.displayName"
       ...(23 line(s) truncated, use '--truncate-output=false' to disable)
   
   Package "test-00":
   [RUNNING] "gcr.io/kpt-fn/apply-setters:v0.1"
   [PASS] "gcr.io/kpt-fn/apply-setters:v0.1" in 2.3s
     Results:
       [INFO] set field value to "test-00" in file "cluster.yaml" in field "metadata.name"
       [INFO] set field value to "config-control" in file "cluster.yaml" in field "metadata.namespace"
       [INFO] set field value to "krm-playground-00" in file "cluster.yaml" in field "metadata.annotations.cnrm.cloud.google.com/project-id"
       ...(36 line(s) truncated, use '--truncate-output=false' to disable)
   
   Successfully executed 3 function(s) in 3 package(s).
   

Apply config changes

Local changes in the previous steps do not affect the cloud until applied.

To apply your configuration changes, you run the following commands.

1. Initialize the working directory with kpt, which creates a resource to track changes:

   kpt live init --namespace PROJECT_NAMESPACE
   

   Replace PROJECT_NAMESPACE with the namespace used for managing project resources (for example, config-control).

2. Preview the resources that will be created:

   kpt live apply --dry-run
   

   All resources should say "created (dry-run)".

   Example output:

   service.serviceusage.cnrm.cloud.google.com/proj-id-00-test-00-container created (dry-run)
   containercluster.container.cnrm.cloud.google.com/test-00 created (dry-run)
   containernodepool.container.cnrm.cloud.google.com/test-00-primary created (dry-run)
   iampolicymember.iam.cnrm.cloud.google.com/artifactreader-gke-test-00-primary created (dry-run)
   iampolicymember.iam.cnrm.cloud.google.com/logwriter-gke-test-00-primary created (dry-run)
   iampolicymember.iam.cnrm.cloud.google.com/metricwriter-gke-test-00-primary created (dry-run)
   iamserviceaccount.iam.cnrm.cloud.google.com/gke-test-00-primary created (dry-run)
   7 resource(s) applied. 7 created, 0 unchanged, 0 configured, 0 failed (dry-run)
   

3. Apply the resources with kpt:

   kpt live apply
   

   All resources should say "created".

   Example output:

   iamserviceaccount.iam.cnrm.cloud.google.com/gke-test-00-primary created
   service.serviceusage.cnrm.cloud.google.com/proj-id-00-test-00-container created
   containercluster.container.cnrm.cloud.google.com/test-00 created
   containernodepool.container.cnrm.cloud.google.com/test-00-primary created
   iampolicymember.iam.cnrm.cloud.google.com/artifactreader-gke-test-00-primary created
   iampolicymember.iam.cnrm.cloud.google.com/logwriter-gke-test-00-primary created
   iampolicymember.iam.cnrm.cloud.google.com/metricwriter-gke-test-00-primary created
   7 resource(s) applied. 7 created, 0 unchanged, 0 configured, 0 failed
   

Verify GKE cluster resources are created successfully

To verify that your changes are applied and the resources they specify are provisioned, you run the following commands.

1. Wait until the resources are ready:

   kpt live status --output table --poll-until current
   

   This command will poll until all the resources have a status of Current and a condition of Ready.

   Use ctrl-c to interrupt, if needed.

   Example output:

   NAMESPACE   RESOURCE                                  STATUS      CONDITIONS      AGE     MESSAGE
   config-con  ContainerCluster/test-00                  Current     Ready           12m     Resource is Ready
   config-con  ContainerNodePool/test-00-primary         Current     Ready           12m     Resource is Ready
   config-con  IAMPolicyMember/artifactreader-gke-test-  Current     Ready           12m     Resource is Ready
   config-con  IAMPolicyMember/logwriter-gke-test-00-pr  Current     Ready           12m     Resource is Ready
   config-con  IAMPolicyMember/metricwriter-gke-test-00  Current     Ready           12m     Resource is Ready
   config-con  IAMServiceAccount/gke-test-00-primary     Current     Ready           12m     Resource is Ready
   config-con  Service/proj-id-00-test-00-contai         Current     Ready           12m     Resource is Ready
   

   Note: The resources in this blueprint can take over 40 minutes to provision.

2. In the case of error, use the default event output to see full error messages:

   kpt live status
   

Frequently asked questions

How do I change an immutable field?

First, delete the resource with kubectl, then re-apply with kubectl or kpt live apply.

Why does kpt live status say the condition of my resource is <NONE>?

This usually indicates that Config Connector has not been configured for the project namespace.

Learn how to configure Config Connector to watch your namespace.

I modified setters.yaml and re-applied. Why does the output say that my resources are unchanged?

This can happen if you try to apply your changes without rendering first.

Rendering executes the pipeline of functions specified in the Kptfile(s). One of those functions is apply-setters, which uses setters.yaml as input.

Run kpt fn render to execute the function pipeline.

The cluster is ready now, why isn't the node pool being created yet?

Config Connector uses retry loops with exponential backoff when applying changes to Google Cloud resources. If the resource cannot be successfully updated for long enough, the maximum retry backoff of 15 minutes can be reached. This can delay applying changes to other resources when there is a chain of dependencies.

Since cluster provisioning can take over 15 minutes, and it depends on subnet provisioning, by the time the cluster is ready, the node pool might be waiting up to 15 minutes to retry.

The node pool should eventually be provisioned, but if you want to force a retry manually, you can delete and reapply the affected resource.

Cleaning up

If you decide to stop using Config Controller, you should first clean up all resources created with Config Controller and then delete Config Controller itself.

1. Delete the cluster resources first with kpt, from within the GKE cluster blueprint working directory:

   kpt live destroy
   

2. Wait until all resources are deleted:

   until [ -z "$(kubectl get -R -f . --ignore-not-found | tee /dev/fd/2)" ]; \
   do sleep 1; done
   

   This command will poll until all the resources have a status of Deleted.

   Use ctrl-c to interrupt, if needed.

   Note: The resources in this blueprint can take over 5 minutes to delete.

3. Delete the networking resources if you created them as part of this tutorial. Use kpt from the VPC directory you created for the blueprint:

   kpt live destroy
   

4. Wait until all resources are deleted:

   until [ -z "$(kubectl get -R -f . --ignore-not-found | tee /dev/fd/2)" ]; \
   do sleep 1; done
   

   This command will poll until all the resources have a status of Deleted.

   Use ctrl-c to interrupt, if needed.

   Note: The resources in this blueprint can take a few minutes to delete.

What's next

• Learn more about blueprints.
• Browse the KRM blueprint catalog.
• Browse the Terraform blueprint catalog.
• See GKE Autopilot blueprint for details about how to create an GKE Autopilot cluster.
• See Anthos cluster blueprint for details about how to create an Anthos cluster.