Remotely access a private cluster using a bastion host

---

This tutorial shows you how to access a private cluster in Google Kubernetes Engine (GKE) over the internet by using a bastion host.

You can create GKE private clusters with no client access to the public endpoint. This access option improves the cluster security by preventing all internet access to the control plane. However, disabling access to the public endpoint prevents you from interacting with your cluster remotely, unless you add the IP address of your remote client as an authorized network.

This tutorial shows you how to set up a bastion host, which is a special-purpose host machine designed to withstand attack. The bastion host uses Tinyproxy to forward client traffic to the cluster. You use Identity-Aware Proxy (IAP) to securely access the bastion host from your remote client.

Note: This tutorial provides instructions for working with this app: Tinyproxy. The instructions might not represent newer versions of the app. For more information, see the documentation: Tinyproxy.

Objectives

• Create a private cluster with no access to the public endpoint.
• Deploy a Compute Engine virtual machine (VM) to act as a bastion host in the cluster subnet.
• Use IAP to connect a remote client to the cluster over the internet.

Costs

In this document, you use the following billable components of Google Cloud:

• GKE
• Identity-Aware Proxy
• Compute Engine

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the GKE, Compute Engine, Identity-Aware Proxy APIs.

   Enable the APIs

5. Install the Google Cloud CLI.

6. To initialize the gcloud CLI, run the following command:

   gcloud init

7. Update and install gcloud components:

   gcloud components update
   gcloud components install alpha beta

8. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

9. Make sure that billing is enabled for your Google Cloud project.

10. Enable the GKE, Compute Engine, Identity-Aware Proxy APIs.

    Enable the APIs

11. Install the Google Cloud CLI.

12. To initialize the gcloud CLI, run the following command:

    gcloud init

13. Update and install gcloud components:

    gcloud components update
    gcloud components install alpha beta

Create a private cluster

Create a new private cluster with no client access to the public endpoint. Place the cluster in its own subnet. You can do this using the Google Cloud CLI or the Google Cloud console.

gcloud

Run the following command:

gcloud container clusters create-auto CLUSTER_NAME \
    --region=COMPUTE_REGION \
    --create-subnetwork=name=SUBNET_NAME \
    --enable-master-authorized-networks \
    --enable-private-nodes \
    --enable-private-endpoint

Replace the following:

• CLUSTER_NAME: the name of the new cluster.
• COMPUTE_REGION: the Compute Engine region for the cluster.
• SUBNET_NAME: the name of the new subnetwork in which you want to place the cluster.

Console

Create a Virtual Private Cloud subnetwork

1. Go to the VPC networks page in the Google Cloud console.

   Go to VPC networks

2. Click the default network.

3. In the Subnets section, click Add subnet.

4. On the Add a subnet dialog, specify the following:

   1. Name: A name for the new subnet.
   2. Region: A region for the subnet. This must be the same as the cluster region.
   3. IP address range: Specify 10.2.204.0/22 or another range that doesn't conflict with other ranges in the VPC network.
   4. For Private Google Access, select the On option.

5. Click Add.

Create a private cluster

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. Click Create.

3. Click Configure for GKE Autopilot.

4. Specify a Name and Region for the new cluster. The region must be the same as the subnet.

5. In the Networking section, select the Private cluster option.

6. Clear the Access control plane using its external IP address checkbox.

7. From the Node subnet drop-down list, select the subnet you created.

8. Optionally, configure other settings for the cluster.

9. Click Create.

You can also use a GKE Standard cluster with the --master-ipv4-cidr flag specified.

Create a bastion host VM

Create a Compute Engine VM within the private cluster internal network to act as a bastion host that can manage the cluster.

gcloud

Create a Compute Engine VM:

gcloud compute instances create INSTANCE_NAME \
    --zone=COMPUTE_ZONE \
    --machine-type=e2-micro \
    --network-interface=no-address,network-tier=PREMIUM,subnet=SUBNET_NAME

Replace the following:

• INSTANCE_NAME: the name of the VM.
• COMPUTE_ZONE: the Compute Engine zone for the VM. Place this in the same region as the cluster.

• SUBNET_NAME: the subnetwork in which you want to place the VM.

  Note: We recommend that you create the VM in the same VPC subnet as the cluster. If you create the VM in a different subnet, add the VM IP address range as an authorized network for the cluster.

Console

1. Go to the VM instances page in the Google Cloud console.

   Go to VM instances

2. Click Create instance.

3. Specify the following:

   1. Name: the name of your VM.
   2. Region and Zone: the region and zone of your VM. Use the same region as your cluster.
   3. Machine type: a machine type. Choose a small machine type, such as e2-micro.
   4. For Network interfaces, select the same VPC network and subnet as the cluster.
   5. Optionally, configure other settings for the instance.

4. Click Create.

Note: Create the VM in the same VPC subnet as the cluster. If you create the VM in a different subnet, add the VM IP address range as an authorized network for the cluster.

Note: You should automate bastion host setup and upgrades using a tool like Terraform.

Create firewall rule

To allow IAP to connect to your bastion host VM, create a firewall rule.

Deploy the proxy

Note: Some commands in this section require administrator privileges.

With the bastion host and the private cluster configured, you must deploy a proxy daemon in the host to forward traffic to the cluster control plane. For this tutorial, you install Tinyproxy.

1. Start a session into your VM:

   gcloud compute ssh INSTANCE_NAME --tunnel-through-iap --project=PROJECT_ID
   

2. Install Tinyproxy:

   sudo apt install tinyproxy
   

3. Open the Tinyproxy configuration file:

   sudo vi /etc/tinyproxy/tinyproxy.conf
   

4. In the file, do the following:

   1. Verify that the port is 8888.

   2. Search for the Allow section:

        /Allow 127
      

   3. Add the following line to the Allow section:

        Allow localhost
      

5. Save the file and restart Tinyproxy:

   sudo service tinyproxy restart
   

6. Exit the session:

   exit
   

Connect to your cluster from the remote client

After configuring Tinyproxy, you must set up the remote client with cluster credentials and specify the proxy. Do the following on the remote client:

1. Get credentials for the cluster:

   gcloud container clusters get-credentials CLUSTER_NAME \
       --region=COMPUTE_REGION \
       --project=PROJECT_ID
   

   Replace the following:

   • CLUSTER_NAME: the name of the private cluster.
   • COMPUTE_REGION: the region of the cluster.
   • PROJECT_ID: the ID of the Google Cloud project of the cluster.

2. Tunnel to the bastion host using IAP:

   gcloud compute ssh INSTANCE_NAME \
       --tunnel-through-iap \
       --project=PROJECT_ID \
       --zone=COMPUTE_ZONE \
       --ssh-flag="-4 -L8888:localhost:8888 -N -q -f"
   

3. Specify the proxy:

   export HTTPS_PROXY=localhost:8888
   kubectl get ns
   

   The output is a list of namespaces in the private cluster.

Stop listening on the remote client

If you want to revert the change on the remote client at any time, you should end the listener process on TCP port 8888. The command to do this is different depending on the client operating system.

netstat -lnpt | grep 8888 | awk '{print $7}' | grep -o '[0-9]\+' | sort -u | xargs sudo kill

Troubleshooting

Firewall restrictions in enterprise networks

If you're on an enterprise network with a strict firewall, you might not be able to complete this tutorial without requesting an exception. If you request an exception, the source IP range for the bastion host is 35.235.240.0/20 by default.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete individual resources

1. Delete the bastion host that you deployed in this tutorial:

   gcloud compute instances delete INSTANCE_NAME \
       --zone=COMPUTE_ZONE
   

2. Delete the cluster:

   gcloud container clusters delete CLUSTER_NAME \
       --region=COMPUTE_REGION
   

3. Delete the subnet:

   gcloud compute networks subnets delete SUBNET_NAME \
       --region=COMPUTE_REGION