Create a user cluster with a new installation model

Version 1.13 

• Latest version
• Version 1.16
• Earlier versions (unsupported)

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This document shows how to create a user cluster that uses a new installation model that is available as a preview feature in version 1.13 of GKE on VMware.

What is kubeception?

The term kubeception is used to convey the idea that a Kubernetes cluster is used to create and manage other Kubernetes clusters. In the context of GKE on VMware, kubeception refers to the case where the control plane for a user cluster runs on one or more nodes in an admin cluster.

Starting in version 1.13.0, you have the option of having the control plane for a user cluster run on one or more nodes in the user cluster itself. In previous versions of GKE on VMware, kubeception was the only option. That is, all user clusters had their control planes in the admin cluster.

The new installation model brings architectural consistency between admin and user clusters, and decouples the user cluster from the admin cluster. This is an advantage for scenarios like running clusters on different geographical sites.

Before you begin

You must have an admin cluster, and the admin cluster version must be 1.13.0 or later.

Plan your IP addresses

As you plan your IP addresses for your user cluster, be aware of these requirements for the new installation model:

• The control-plane nodes for your user cluster must get their IP addresses from a list of static addresses that you provide. This is the case even if your worker nodes get their addresses from a DHCP server. You need three control-plane nodes for a high availability (HA) user cluster, but only one control-plane node for a non-HA user cluster.

• The user cluster control plane node(s) must be in the same VLAN as the user cluster worker nodes. This is in contrast to the kubeception model.

• The user cluster control-plane VIP must be in the same VLAN as the user cluster worker nodes and the user cluster control-plane node(s). This is in contrast to the kubeception model.

Plan your vSphere environment

The admin cluster configuration file and user cluster configuration file both have a vCenter section where you can specify the vSphere resources that you want the admin or user cluster to use.

For a user cluster, you do not have to fill in the vCenter section, because by default, a user cluster uses the same vSphere resources as the admin cluster. But if you want your user cluster to use vSphere resources different from what you specified for the admin cluster, you can fill in fields of your choice in the vCenter section of the user cluster configuration file. For more information, see Set up your vSphere object hierarchy.

In the new installation model, the control-plane nodes for the user cluster are in the user cluster itself. So the control-plane nodes use the vSphere resources specified in the user cluster configuration file. For example, suppose you specify datacenter-1 for your admin cluster and datacenter-2 for your user cluster. The control-plane nodes for your user cluster will be in datacenter-2.

Anti-affinity groups

The admin cluster configuration file and user cluster configuration file both have an antiAffinityGroups.enabled field that you can set to true or false.

In the new installation model, the control-plane nodes for the user cluster are distributed according to the antiAffinityGroups.enabled value in the user cluster configuration file.

In contrast, with the kubeception model, the control-plane nodes for the user cluster are distributed according to the antiAffinityGroups.enabled value in the admin cluster configuration file.

Example

Here we give an example of a user cluster that has the following characteristics:

• There are three worker nodes.

• The worker nodes have static IP addresses.

• There are three control-plane nodes. That is, it is an HA cluster.

• The load balancer is MetalLB.

• The user cluster uses the same vSphere resources as the admin cluster.

The following diagram illustrates the network for the admin and user clusters:

An admin cluster and a user cluster (Click to enlarge)

Here is an example of an IP block file and a portion of a user cluster configuration file.

user-ipblock.yaml

blocks:
  - netmask: 255.255.255.0
    gateway: 172.16.21.1
    ips:
    - ip: 172.16.21.2
      hostname: worker-vm-1
    - ip: 172.16.21.3
      hostname: worker-vm-2
    - ip: 172.16.21.4
      hostname: worker-vm-3
    - ip: 172.16.21.5
      hostname: worker-vm-4

user-cluster.yaml

apiVersion: v1
kind: UserCluster
...
kubeception: false
network:
  hostConfig:
    dnsServers:
    - "203.0.113.2"
    - "198.51.100.2"
    ntpServers:
    - "216.239.35.4"
  ipMode:
    type: "static"
    ipBlockFilePath: "user-ipblock.yaml"
  controlPlaneIPBlock:
    netmask: "255.255.255.0"
    gateway: "172.16.21.1"
    ips:
    - ip: "172.16.21.6"
      hostname: "cp-vm-1"
    - ip: "172.16.21.7"
      hostname: "cp-vm-2"
    - ip: "172.16.21.8"
      hostname: "cp-vm-3"
loadBalancer:
  vips:
    controlPlaneVIP: "172.16.21.40"
    ingressVIP: "172.16.21.30"
  kind: MetalLB
  metalLB:
    addressPools:
    - name: "address-pool-1"
      addresses:
    - "172.16.21.30-172.16.21.39"
...
masterNode:
  cpus: 4
  memoryMB: 8192
  replicas: 3
nodePools:
- name: "worker-node-pool"
  enableLoadBalancer: true

These are the important points to understand in the preceding example:

• The static IP addresses for the worker nodes are specified in an IP block file. The IP block file has four addresses even though there are only three worker nodes. The extra IP address is needed during cluster upgrade, update, and auto repair.

• The static IP addresses for the three control-plane nodes are specified in the network.controlPlaneIPBlock section of the user cluster configuration file. There is no need for an extra IP address in this block.

• The masterNode.replicas field is set to 3.

• The control-plane VIP and the ingress VIP are both in the same VLAN as the worker nodes and the control-plane nodes.

• The VIPs that are set aside for Services of type LoadBalancer are specified in the loadBalancer.metalLB.addressPools section of the user cluster configuration file. These VIPs are in the same VLAN as the worker nodes and the control-plane nodes.

• The user cluster configuration file does not include a vCenter section. So the user cluster uses the same vSphere resources as the admin cluster.

Create firewall rules

In addition to the standard firewall rules, create the following firewall rules as needed:

From	Source port	To	Dst port	Protocol	Description
Admin cluster nodes	All	User cluster control-plane VIP	443	https	Allow nodes and Pods in the admin cluster to talk to the Kubernetes API server of the user cluster.
Admin cluster nodes	All	User cluster control-plane nodes	443	https	Allow nodes and Pods in the admin cluster to talk to the Kubernetes API server of the user cluster by using the IP address of a user cluster control-plane node.
Admin cluster nodes	All	User cluster vCenter Server	443	https	Allow the admin cluster to manage the lifecycle of the user cluster. Required if you specified a value for vCenter.address that is different from the vCenter address in your admin cluster configuration file.
User cluster control-plane nodes	1024 - 65535	On-premises local Docker registry	Depends on your registry	TCP/https	Required if you specified a private registry in your admin cluster configuration file.

Create a user cluster with the new model

This section shows how to create a user cluster that uses the new installation model. In this example, the user cluster uses the same instance of vCenter Server as the admin cluster.

Follow the instructions in Create a user cluster.

As you fill in your user cluster configuration file:

• Set kubeception to false.

• Do not specify a value for vCenter.address.

• Fill in the network.controlPlaneIPBlock section. If you want an HA user cluster, specify three IP addresses. Otherwise, specify one IP address.

When your user cluster is running, verify that the control plane is running on one or three nodes in the user cluster:

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG get nodes

Replace USER_CLUSTER_KUBECONFIG with the path of the user cluster kubeconfig file.

The output shows one or three control-plane nodes along with the worker nodes. For example:

cp-vm-1       Ready    control-plane,master   18m
cp-vm-2       Ready    control-plane,master   18m
cp-vm-3       Ready    control-plane,master   18m
worker-vm-1   Ready                           6m7s
worker-vm-2   Ready                           6m6s
worker-vm-3   Ready                           6m14s

Verify that the control plane for the user cluster is not running in the admin cluster:

kubectl --kubeconfig ADMIN_CLUSTER_KUBECONFIG get nodes --output wide

Replace ADMIN_CLUSTER_KUBECONFIG with the path of the admin cluster kubeconfig file.

The output shows a control-plane node for the admin cluster itself, but does not show any control-plane nodes for the user cluster. For example:

admin-vm-1   Ready    control-plane,master   82m
admin-vm-2   Ready                           71m
admin-vm-3   Ready                           71m

Upgrade a user cluster

Follow the instructions in Upgrading Anthos clusters on VMware.

Note that when you upgrade a user cluster using the new installation model, the entire user cluster, including the user control plane nodes, is upgraded.

In contrast, for user clusters deployed with the kubeception model, the control plane nodes are not upgraded during user cluster upgrade and they will not be upgraded until admin cluster upgrade.

Delete a user cluster

To delete a user cluster:

gkectl delete cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --cluster USER_CLUSTER

Replace USER_CLUSTER with the name of the user cluster.

When you delete a user cluster that uses the new installation model, the cluster's data disks do not get deleted automatically. So you need to delete the data disks manually. An HA cluster has three data disks, and a non-HA cluster has one data disk.

The data disks for the user cluster are in one of these locations:

• ADMIN_CLUSTER/USER_CLUSTER/

• ADMIN_DISK_FOLDER/ADMIN_CLUSTER/USER_CLUSTER/

Replace ADMIN_CLUSTER with the name of the admin cluster.

If you specified a folder in the vCenter.dataDisk field of the admin cluster configuration file, replace ADMIN_DISK_FOLDER with the name of the folder.

For a non-HA cluster, the data disk is named USER_CLUSTER-0-data.vmdk.

For an HA cluster, the data disks are named:

• USER_CLUSTER-0-data.vmdk.
• USER_CLUSTER-1-data.vmdk.
• USER_CLUSTER-2-data.vmdk.

You can use the vSphere client to delete the data disks.

Create a user cluster with its own vCenter Server

In certain situations, it makes sense to create a user cluster that uses its own instance of vCenter Server. That is, the admin cluster and an associated user cluster use different instances of vCenter Server.

For example, in an edge location, you might want to have a physical machine running vCenter Server and one or more physical machines running ESXi. You could then use your local instance of vCenter Server to create a vSphere object hierarchy, including data centers, clusters, resource pools, datastores, and folders.

Follow the instructions under Create a user cluster with the new model.

In addition to the steps given in that section, fill in the entire vCenter section of your user cluster configuration file. In particular, specify a value for vCenter.address that is different from the vCenter Server address you specified in the admin cluster configuration file.

Example

vCenter:
  address: "vc-edge.example"
  datacenter: "vc-edge"
  cluster: "vc-edge-workloads"
  resourcePool: "vc-edge-pool
  datastore: "vc-edge-datastore
  caCertPath: "/usr/local/google/home/me/certs/edge-cacert.pem"
  credentials:
    fileRef:
      path: "credential.yaml"
      entry: "vCenter-edge"
  folder: "edge-vm-folder"

Troubleshooting

See Troubleshooting cluster creation and upgrade.

What's next

• Connect to your cluster from the Google Cloud console.
• Deploy an application.