Configure NAT for all GKE CIDR blocks

Stay organized with collections Save and categorize content based on your preferences.
Last reviewed 2022-12-16 UTC

This tutorial is intended for network architects who are finding it difficult to allocate RFC 1918 IPv4 addresses for GKE Pods, Nodes, and Services due to private address space exhaustion or fragmentation. The tutorial walks you through a solution that allows for a full-scale GKE installation by leveraging NAT and the strategic reuse of RFC 1918 CIDR blocks already assigned within your organization. You then use Terraform to automate the infrastructure build, and you use the Google Cloud CLI to inspect the individual components shown in the following diagram.

RFC 1918 address reuse with GKE. (for a screen-readable PDF version, click the image)
Figure 1. RFC 1918 address reuse with GKE. (For a screen-readable PDF version, click the image.)

At a high level, you set up a host project, a service project, a shared VPC, and a Compute Engine VM called an isolated-VPC gateway. You also set up the relevant static routes, firewall rules, and APIs. For a more in-depth discussion of the overall solution and its individual components, see NAT for all GKE CIDR blocks.

This tutorial assumes you are familiar with:

  • Linux sysadmin commands
  • GKE
  • Compute Engine
  • Shared VPCs
  • NAT
  • Terraform

Objectives

  • Configure a host project with a shared VPC called the routed-domain VPC.
  • Configure a service project with a VPC called the isolated VPC.
  • Configure the isolated-VPC gateway VM.
  • Configure routing across VPCs through the isolated-VPC gateway.
  • Configure NAT on the isolated-VPC gateway.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

In this section, you prepare Cloud Shell, set up your environment variables, and deploy the supporting infrastructure.

Prepare Cloud Shell

  1. In the Google Cloud console, open Cloud Shell.

    Go to Cloud Shell

    You complete most of this tutorial from the Cloud Shell terminal using HashiCorp's Terraform and the Google Cloud CLI.

  2. In Cloud Shell, clone the GitHub repository and change to the local directory:

    git clone https://github.com/GoogleCloudPlatform/terraform-gke-nat-connectivity.git kam
cd kam/allnat

    The repository contains all the files you need to complete this tutorial. For a complete description of each file, see the README.md file in the repository.

  3. Make all shell scripts executable:

    sudo chmod 755 *.sh

  4. Install Terraform.

  5. Initialize Terraform:

    terraform init

    The output is similar to the following:

    ...
Initializing provider plugins...
The following providers do not have any version constraints in configuration, so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below.

∗ provider.google: version = "~> 2.5"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.

If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
...

Set environment variables

  1. In Cloud Shell, set and verify the TF_VAR_org_id variable, replacing your-organization-name with the Google Cloud organization name you want to use in this tutorial:

    export TF_VAR_org_id=$(gcloud organizations list | \
    awk '/your-organization-name/ {print $2}')

  2. Verify that the environment variable is set correctly:

    echo $TF_VAR_org_id

    The output lists your numeric organization ID and looks similar to the following:

    ...
123123123123
...

  3. Set the remaining environment variables by running the following shell command from the repo:

    source set_variables.sh

  4. Verify that the environment variables are set correctly:

    env | grep TF_

    The output is similar to the following:

    ...
TF_VAR_isolated_net_name=isolated-vpc
TF_VAR_isolated_pname=isolated-vpc-project
TF_VAR_isolated_pid=isolated-vpc-project-id
TF_VAR_shared_net_name=routed-domain-vpc
TF_VAR_shared_pname=routed-domain-project
TF_VAR_shared_pid=routed-domain-project-id
TF_VAR_isolated_vpc_gw_host_nic_ip=10.97.0.2
TF_VAR_isolated_vpc_gw_service_nic_ip=10.32.0.2
TF_VAR_isolated_vpc_gw_service_dgw_ip=10.32.0.1
TF_VAR_org_id=123123123123
TF_VAR_region=us-west1
TF_VAR_zone=us-west1-b
TF_VAR_user_account=user@example.com
TF_VAR_node_cidr=10.32.0.0/19
TF_VAR_pod_cidr=10.0.0.0/11
TF_VAR_service_cidr=10.224.0.0/20
TF_VAR_shared_cidr=10.97.0.0/24
TF_VAR_test1_cidr=10.0.0.0/11
TF_VAR_test2_cidr=10.160.0.0/11
TF_VAR_test3_cidr=10.64.0.0/19
TF_VAR_ilb_cidr=10.32.31.0/24
TF_VAR_masquerade=true
...

  5. Update the TF_VAR_masquerade variable if it is not set.

    This document defaults to setting up the iptables masquerade option, as described in the associated concept document. If you want to set up the iptables SNAT option, you must change the TF_VAR_masquerade variable to false:

    export TF_VAR_masquerade=false

  6. Create an environment variable file:

    env | grep TF_ | sed 's/^/export /' > TF_ENV_VARS

    This command chain redirects the environment variables that you created into a file called TF_ENV_VARS. Each variable is prepended with the export command. You can use this file to reset the environment variables in case your Cloud Shell session is terminated. These variables are used by the Terraform scripts, shell scripts, and the SDK commands in this tutorial.

    If you need to reinitialize the variables, you can run the following command from the directory where the file resides:

    source TF_ENV_VARS

Deploy the supporting infrastructure

  • In Cloud Shell, deploy the Terraform supporting infrastructure:

    terraform apply

    Terraform prompts for confirmation before making any changes. Answer yes to apply either configuration.

    The terraform apply command instructs Terraform to deploy all the solution's components. To better understand how the infrastructure is declaratively defined, you can read through the Terraform manifests—that is, the files with the .tf extension.

Inspecting the supporting infrastructure

You now use the Google Cloud CLI to view and verify the infrastructure that Terraform created. Verification involves running a command to see if the resource responds and was created correctly.

Verify the projects

  1. In Cloud Shell, list the two projects:

    gcloud projects list | grep pid

    The output is similar to the following:

    ...
isolated-vpc-pid       isolated-vpc-pname        777999333962
routed-domain-pid      routed-domain-pname       777852333528
...

  2. List the API status:

    1. For the isolated-VPC project:

      gcloud services list --project=$TF_VAR_isolated_vpc_pid | grep -E "compute|container"

      The output is similar to the following:

      ...
compute.googleapis.com            Compute Engine API
container.googleapis.com          Google Kubernetes Engine API
...

    2. For the routed-domain project:

      gcloud services list --project=$TF_VAR_shared_vpc_pid | grep -E "compute"

      The output is similar to the following:

      ...
compute.googleapis.com            Compute Engine API
...

  3. List the shared VPC status:

    gcloud compute shared-vpc organizations list-host-projects $TF_VAR_org_id
gcloud compute shared-vpc get-host-project $TF_VAR_isolated_vpc_pid
gcloud compute shared-vpc list-associated-resources $TF_VAR_shared_vpc_pid

    The output is similar to the following:

    ...
NAME                 CREATION_TIMESTAMP  XPN_PROJECT_STATUS
svpc-pid--210756488
...
kind: compute#project
name: svpc-pid--210756488
...
RESOURCE_ID          RESOURCE_TYPE
ivpc-pid--695116665  PROJECT
...

Verify the networks and subnetworks

  1. In Cloud Shell, verify the networks and subnetworks for the isolated VPC:

    gcloud compute networks describe $TF_VAR_isolated_net_name \
  --project=$TF_VAR_isolated_vpc_pid

gcloud compute networks subnets describe node-cidr \
  --project=$TF_VAR_isolated_vpc_pid \
  --region=$TF_VAR_region

    The output is similar to the following:

    ...
kind: compute#network
name: isolated-vpc-net
routingConfig:
  routingMode: GLOBAL
...
subnetworks:
‐ https://www.googleapis.com/compute/v1/projects/ivpc-pid--695116665/regions/us-west1/subnetworks/node-cidr
x_gcloud_bgp_routing_mode: GLOBAL
...
gatewayAddress: 10.32.0.1
ipCidrRange: 10.32.0.0/19
kind: compute#subnetwork
name: node-cidr
...
secondaryIpRanges:
‐ ipCidrRange: 10.0.0.0/11
  rangeName: pod-cidr
...

  2. Verify the networks and subnetworks for the routed-domain VPC:

    gcloud compute networks describe $TF_VAR_shared_net_name \
    --project=$TF_VAR_shared_vpc_pid

gcloud compute networks subnets describe shared-cidr \
    --project=$TF_VAR_shared_vpc_pid \
    --region=$TF_VAR_region

gcloud compute networks subnets list $TF_VAR_shared_net_name \
    --project=$TF_VAR_shared_vpc_pid

    The output is similar to the following:

    ...
kind: compute#network
name: routed-domain-vpc-net
routingConfig:
  routingMode: GLOBAL
...
subnetworks:
‐ https://www.googleapis.com/compute/v1/projects/svpc-pid--210756488/regions/us-west1/subnetworks/shared-cidr
x_gcloud_bgp_routing_mode: GLOBAL
...
gatewayAddress: 10.97.0.1
...
ipCidrRange: 10.97.0.0/24
kind: compute#subnetwork
name: shared-cidr
...
region: https://www.googleapis.com/compute/v1/projects/svpc-pid--210756488/regions/us-west1
...
NAME            REGION         NETWORK                RANGE
...
test-10-11      us-west1       routed-domain-vpc-net  10.0.0.0/11
test-10-160-11  us-west1       routed-domain-vpc-net  10.160.0.0/11
test-10-64-19   us-west1       routed-domain-vpc-net  10.64.0.0/19
...

Verify firewall rules

  • In Cloud Shell, verify the firewall rules in both VPCs:

    gcloud compute firewall-rules list --project=$TF_VAR_isolated_vpc_pid
gcloud compute firewall-rules list --project=$TF_VAR_shared_vpc_pid

    The output is similar to the following:

    ...
NAME                  NETWORK           DIRECTION  PRIORITY  ALLOW  DENY  DISABLED
allow-rfc1918-in-fwr  isolated-vpc-net  INGRESS    1000      all          False
allow-ssh-in-fwr      isolated-vpc-net  INGRESS    1000      22           False
...
NAME                  NETWORK           DIRECTION  PRIORITY  ALLOW  DENY  DISABLED
allow-rfc1918-in-fwr  routed-domain-vpc-net  INGRESS 1000    all          False
allow-ssh-in-fwr      routed-domain-vpc-net  INGRESS 1000    22           False
...

Verify test machines

  • In Cloud Shell, verify the test machines:

    gcloud compute instances list --project=$TF_VAR_shared_vpc_pid

    The output is similar to the following:

    ...
NAME               ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
test-10-11-vm      us-west1-b  n1-standard-1               10.0.0.2     192.0.2.1       RUNNING
test-10-160-11-vm  us-west1-b  n1-standard-1               10.160.0.2   192.0.2.2       RUNNING
test-10-64-19-vm   us-west1-b  n1-standard-1               10.64.0.2    192.0.2.3       RUNNING
...

Verify the isolated-VPC gateway

  1. In Cloud Shell, verify the instance:

    gcloud compute instances describe isolated-vpc-gw \
    --project=$TF_VAR_isolated_vpc_pid \
    --zone $TF_VAR_zone

    The output is similar to the following:

    ...
metadata:
  fingerprint: e_P00DAAAAA=
  items:
  ‐ key: startup-script
    value: |+
      #! /bin/bash
      #Enable packet forwarding
      sudo sysctl -w net.ipv4.ip_forward=1
...
name: isolated-vpc-gw
networkInterfaces:
‐ accessConfigs:
  ‐ kind: compute#accessConfig
    name: external-nat
    natIP: 192.0.2.76
...
  networkIP: 10.97.0.12
...
  networkIP: 10.32.0.15
...
tags:
  fingerprint: FOrvKltVVVV=
  items:
  ‐ allow-rfc1918-in-fwr
  ‐ allow-ssh-in-fwr
  ‐ isolated-vpc-allocated-ilb-route
...

  2. Connect to the isolated-VPC gateway by using SSH:

    gcloud compute ssh isolated-vpc-gw \
    --project=$TF_VAR_isolated_vpc_pid \
    --zone $TF_VAR_zone

  3. In the instance, check for the existence of the following:

    • The routed-domain route table
    • The policy rules that direct traffic bound for the routed domain to the routed-domain route table
    • The static routes in the routed-domain route table that direct traffic to the routed domain
    sudo sysctl net.ipv4.ip_forward
cat /etc/iproute2/rt_tables | grep routed-domain
ip rule show
ip route show table routed-domain

    The output is similar to the following:

    ....
net.ipv4.ip_forward = 1
....
routed-domain
100 routed-domain
....
0:      from all lookup local
32763:  from all to 192.168.0.0/16 lookup routed-domain
32764:  from all to 172.16.0.0/12 lookup routed-domain
32765:  from all to 10.0.0.0/8 lookup routed-domain
32766:  from all lookup main
32767:  from all lookup default
....
10.0.0.0/8 via 10.97.0.1 dev eth0
10.32.31.0/24 via 10.32.0.1 dev eth1
172.16.0.0/12 via 10.97.0.1 dev eth0
192.168.0.0/16 via 10.97.0.1 dev eth0
....

  4. Verify the iptables NAT configuration:

    sudo iptables -t nat -L -v

    For the iptables masquerade configuration, the output is similar to the following:

    ...
Chain PREROUTING (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 24 packets, 1525 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   24  1525 MASQUERADE  all  --  any    eth0    anywhere             anywhere
    0     0 MASQUERADE  all  --  any    eth1    anywhere             anywhere
...

    For the iptables SNAT option, the output is similar to the following:

    ...
Chain PREROUTING (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 21 packets, 1345 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 21 packets, 1345 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 SNAT       tcp  --  any    eth1    10.160.0.0/11        anywhere             to:10.160.0.0-10.191.255.255:1024-33279
    0     0 SNAT       tcp  --  any    eth1    10.0.0.0/11          anywhere             to:10.160.0.0-10.191.255.255:33280-65535
    0     0 SNAT       tcp  --  any    eth1    10.64.0.0/19         anywhere             to:10.64.0.0-10.64.31.255:1024-33279
    0     0 SNAT       tcp  --  any    eth1    10.32.0.0/19         anywhere             to:10.64.0.0-10.64.31.255:33280-65535
    0     0 SNAT       udp  --  any    eth1    10.160.0.0/11        anywhere             to:10.160.0.0-10.191.255.255:1024-33279
    0     0 SNAT       udp  --  any    eth1    10.0.0.0/11          anywhere             to:10.160.0.0-10.191.255.255:33280-65535
    0     0 SNAT       udp  --  any    eth1    10.64.0.0/19         anywhere             to:10.64.0.0-10.64.31.255:1024-33279
    0     0 SNAT       udp  --  any    eth1    10.32.0.0/19         anywhere             to:10.64.0.0-10.64.31.255:33280-65535
    0     0 SNAT       icmp --  any    eth1    10.160.0.0/11        anywhere             to:10.160.0.0-10.191.255.255
    0     0 SNAT       icmp --  any    eth1    10.0.0.0/11          anywhere             to:10.160.0.0-10.191.255.255
    0     0 SNAT       icmp --  any    eth1    10.64.0.0/19         anywhere             to:10.64.0.0-10.64.31.255
    0     0 SNAT       icmp --  any    eth1    10.32.0.0/19         anywhere             to:10.64.0.0-10.64.31.255
...

Verify the GKE cluster

  • In Cloud Shell, verify the GKE cluster:

    gcloud container clusters list \
    --project=$TF_VAR_isolated_vpc_pid \
    --zone=$TF_VAR_zone

    The output is similar to the following:

    ...
NAME     LOCATION    MASTER_VERSION MASTER_IP   MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
cluster1 us-west1-b  1.11.8-gke.6   192.0.2.58  n1-standard-1  1.11.8-gke.6  3          RUNNING
...

Verify inter-VPC routing

  1. In Cloud Shell, verify inter-VPC routing for the shared VPC:

    gcloud compute routes list \
    --filter=name="('ivpc-allocated-ilb-route')" \
    --project=$TF_VAR_shared_vpc_pid

    The output is similar to the following:

    NAME                      NETWORK                DEST_RANGE     NEXT_HOP   PRIORITY
ivpc-allocated-ilb-route  routed-domain-vpc-net  10.32.31.0/24  10.97.0.2  1000

  2. Verify inter-VPC routing for the isolated VPC:

    gcloud compute routes list --project=$TF_VAR_isolated_vpc_pid

    The output is similar to the following:

    ...
NAME                            NETWORK           DEST_RANGE      NEXT_HOP  PRIORITY
ivpc-to-10net                   isolated-vpc-net  10.0.0.0/8      10.32.0.2 1000
ivpc-to-172-16net               isolated-vpc-net  172.16.0.0/12   10.32.0.2 1000
ivpc-to-192-168net              isolated-vpc-net  192.168.0.0/16  10.32.0.2 1000
...

Verifying the hello-world app

In this section, you get cluster credentials and verify the hello-server service.

  1. In Cloud Shell, get the cluster credentials:

    gcloud container clusters get-credentials cluster1 \
   --project=$TF_VAR_isolated_vpc_pid \
   --zone $TF_VAR_zone

    The output is similar to the following:

    ...
Fetching cluster endpoint and auth data.
kubeconfig entry generated for cluster1.
...

  2. Verify the hello-server service:

    kubectl get service hello-server

    The output is similar to the following:

    ...
NAME           TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
hello-server   LoadBalancer   10.224.7.87   <pending>     8080:30635/TCP   3m
...

Verifying the solution

In this section, you set up monitoring on the isolated-VPC gateway and test the app.

Set up monitoring on the isolated-VPC gateway

  1. In Cloud Shell, use SSH to connect to the isolated-VPC gateway:

    gcloud compute ssh isolated-vpc-gw \
    --project=$TF_VAR_isolated_vpc_pid \
    --zone=$TF_VAR_zone

  2. In the instance, run the tcpdump utility:

    sudo tcpdump -n -i eth1 host 10.32.31.1

    The output is similar to the following:

    ...
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
...

    This command starts tcpdump to capture the packets that transverse the isolated-VPC gateway. The utility only captures packets that contain the IP address of the internal load balancer that the GKE service created.

Test the app

  1. Open a new instance of Cloud Shell.

    OPEN Cloud Shell

  2. Use ssh to connect to the test-10-11-vm Compute Engine instance. Then, in the new instance, change to the Git repository directory:

    source TF_ENV_VARS
gcloud compute ssh test-10-11-vm \
   --project=$TF_VAR_shared_vpc_pid \
   --zone=$TF_VAR_zone

  3. In the new instance, use curl to connect to the app:

    curl http://10.32.31.1:8080/

    The preceding command retrieves the webpage from the app running on the GKE cluster. The output is similar to the following:

    ...
Hello, world!
Version: 1.0.0
Hostname: my-app-6597cdc789-d6phf
...

    In the original Cloud Shell terminal, when you verify the isolated-VPC gateway, the output is similar to the following:

    ...
17:45:32.674129 IP 10.32.0.2.56698 > 10.32.31.1.8080: Flags [S], seq 2205959832, win 28400, options [mss 1420,sackOK,TS val 296564 ecr 0,nop,wscale 7], length 0
17:45:32.678253 IP 10.32.31.1.8080 > 10.32.0.2.56698: Flags [S.], seq 4243326926, ack 2205959833, win 28160, options [mss 1420,sackOK,TS val 632036963 ecr 296564,nop,wscale 7], length 0
17:45:32.678982 IP 10.32.0.2.56698 > 10.32.31.1.8080: Flags [.], ack 1, win 222, options [nop,nop,TS val 296565 ecr 632036963], length 0
17:45:32.678995 IP 10.32.0.2.56698 > 10.32.31.1.8080: Flags [P.], seq 1:80, ack 1, win 222, options [nop,nop,TS val 296565 ecr 632036963], length 79: HTTP: GET / HTTP/1.1
17:45:32.679411 IP 10.32.31.1.8080 > 10.32.0.2.56698: Flags [.], ack 80, win 220, options [nop,nop,TS val 632036965 ecr 296565], length 0
17:45:32.679904 IP 10.32.31.1.8080 > 10.32.0.2.56698: Flags [P.], seq 1:181, ack 80, win 220, options [nop,nop,TS val 632036965 ecr 296565], length 180: HTTP: HTTP/1.1 200 OK
17:45:32.680032 IP 10.32.0.2.56698 > 10.32.31.1.8080: Flags [.], ack 181, win 231, options [nop,nop,TS val 296565 ecr 632036965], length 0
...

    This output is a packet capture of the curl command that you just ran.

  4. In the first Cloud Shell instance, press Control+C to stop the tcpdump listing.

  5. In that same instance, inspect the connection table on the isolated-VPC gateway:

    sudo conntrack -L | grep 10.32.31.49

    For the iptables masquerade option, the output is similar to the following:

    ...
tcp      6 118 TIME_WAIT src=10.0.0.2 dst=10.32.31.1 sport=56760 dport=8080 src=10.32.31.1 dst=10.32.0.2 sport=8080 dport=56760 [ASSURED] mark=0 use=1
...

    This command displays source NAT entries created in the connection table from packet capture in the previous step; src=10.0.0.2 has been translated to dst=10.32.0.2. Output for the iptables SNAT option will be similar.

  6. Rerun this entire testing section, substituting test-10-11-vm for test-10-160-11-vm and test-10-64-19-vm.

Clean up

Destroy the infrastructure

  1. From the first Cloud Shell terminal, close the SSH session to the isolated-VPC gateway:

    exit

  2. Destroy the Terraform configuration:

    terraform destroy

    Terraform prompts for confirmation before making the change. Answer yes to destroy the configuration and all of the tutorial's components.

    You might see the following Terraform error:

    ...
∗ google_compute_shared_vpc_host_project.host (destroy): 1 error(s) occurred:
∗ google_compute_shared_vpc_host_project.host: Error disabling Shared VPC Host "svpc-pid--23156887": googleapi: Error 400: Cannot disable project as a shared VPC host because it has active service projects., badRequest
...

    This error occurs when you attempt to destroy the shared VPC before you destroy the isolated-VPC gateway.

  3. Continue the clean up:

    gcloud compute shared-vpc associated-projects remove $TF_VAR_isolated_vpc_pid \
    --host-project=$TF_VAR_shared_vpc_pid

gcloud compute shared-vpc disable $TF_VAR_shared_vpc_pid

    You might see the following Terraform error:

    ...
∗ google_compute_network.ivpc (destroy): 1 error(s) occurred:
∗ google_compute_network.ivpc: Error waiting for Deleting Network: The network resource 'projects/ivpc-pid--1058675427/global/networks/isolated-vpc-net' is already being used by 'projects/ivpc-pid--1058675427/global/firewalls/k8s-05693142c93de80e-node-hc'
...

    This error occurs when you try to destroy the isolated-VPC network before the GKE firewall rules.

  4. Complete the clean up by removing the non-default firewall rules from the isolated VPC:

    ./k8-fwr.sh

    The output shows you which firewall rules will be removed.

  5. Review the rules and, when prompted, type yes.

  6. From the first Cloud Shell terminal, reissue the following command:

    terraform destroy

    Terraform prompts for confirmation before making the change. Answer yes to destroy the configuration.

  7. Remove the files created during this tutorial:

    rm -rf .git
rm -rf .terr*
rm*
cd ..
rm -rf kam

What's next