Create a user cluster with Controlplane V2

This document shows how to create a user cluster with Controlplane V2 enabled.

What is kubeception?

The term kubeception is used to convey the idea that a Kubernetes cluster is used to create and manage other Kubernetes clusters. In the context of GKE on VMware, kubeception refers to the case where the control plane for a user cluster runs on one or more nodes in an admin cluster.

With Controlplane V2, the control plane for a user cluster runs on one or more nodes in the user cluster itself.

The benefits of Controlplane V2 include:

  • Architectural consistency between admin and user clusters

  • Failure isolation. An admin cluster failure does not affect user clusters.

  • Operational separation. An admin cluster upgrade does not cause downtime for user clusters.

  • Deployment separation. You can put the admin and user clusters in different failure domains or geographical sites. For example, a user cluster in an edge location could be in a different geographical site from the admin cluster.

Before you begin

You must have an admin cluster, and the admin cluster version must be 1.13.0 or later.

Requirements

A user cluster that has Controlplane V2 enabled:

  • Must use the bundled MetalLB load balancer
  • Must have Dataplane V2 enabled

Plan your IP addresses

As you plan your IP addresses for your user cluster, be aware of these requirements for Controlplane V2:

  • The control-plane nodes for your user cluster must get their IP addresses from a list of static addresses that you provide. This is the case even if your worker nodes get their addresses from a DHCP server. You need three control-plane nodes for a high availability (HA) user cluster, but only one control-plane node for a non-HA user cluster.

  • The user cluster control plane node(s) must be in the same VLAN as the user cluster worker nodes. This is in contrast to the kubeception model.

  • The user cluster control-plane VIP must be in the same VLAN as the user cluster worker nodes and the user cluster control-plane node(s). This is in contrast to the kubeception model.

Plan your vSphere environment

The admin cluster configuration file and user cluster configuration file both have a vCenter section where you can specify the vSphere resources that you want the admin or user cluster to use.

For a user cluster, you do not have to fill in the vCenter section, because by default, a user cluster uses the same vSphere resources as the admin cluster. But if you want your user cluster to use vSphere resources different from what you specified for the admin cluster, you can fill in fields of your choice in the vCenter section of the user cluster configuration file. For more information, see Set up your vSphere object hierarchy.

With Controlplane V2, the control-plane nodes for the user cluster are in the user cluster itself. So the control-plane nodes use the vSphere resources specified in the user cluster configuration file. For example, suppose you specify datacenter-1 for your admin cluster and datacenter-2 for your user cluster. The control-plane nodes for your user cluster will be in datacenter-2.

Anti-affinity groups

The admin cluster configuration file and user cluster configuration file both have an antiAffinityGroups.enabled field that you can set to true or false.

With Controlplane V2, the control-plane nodes for the user cluster are distributed according to the antiAffinityGroups.enabled value in the user cluster configuration file.

In contrast, with the kubeception model, the control-plane nodes for the user cluster are distributed according to the antiAffinityGroups.enabled value in the admin cluster configuration file.

Example

Here we give an example of a user cluster that has the following characteristics:

  • There are three worker nodes.

  • The worker nodes have static IP addresses.

  • There are three control-plane nodes. That is, it is an HA cluster.

  • The load balancer is MetalLB.

  • Dataplane V2 and Controlplane V2 are enabled.

  • The user cluster uses the same vSphere resources as the admin cluster.

The following diagram illustrates the network for the admin and user clusters:

Admin cluster and user cluster
An admin cluster and a user cluster (Click to enlarge)

Here is an example of an IP block file and a portion of a user cluster configuration file.

user-ipblock.yaml

blocks:
  - netmask: 255.255.255.0
    gateway: 172.16.21.1
    ips:
    - ip: 172.16.21.2
      hostname: worker-vm-1
    - ip: 172.16.21.3
      hostname: worker-vm-2
    - ip: 172.16.21.4
      hostname: worker-vm-3
    - ip: 172.16.21.5
      hostname: worker-vm-4

user-cluster.yaml

apiVersion: v1
kind: UserCluster
...
network:
  hostConfig:
    dnsServers:
    - "203.0.113.2"
    - "198.51.100.2"
    ntpServers:
    - "216.239.35.4"
  ipMode:
    type: "static"
    ipBlockFilePath: "user-ipblock.yaml"
  controlPlaneIPBlock:
    netmask: "255.255.255.0"
    gateway: "172.16.21.1"
    ips:
    - ip: "172.16.21.6"
      hostname: "cp-vm-1"
    - ip: "172.16.21.7"
      hostname: "cp-vm-2"
    - ip: "172.16.21.8"
      hostname: "cp-vm-3"
loadBalancer:
  vips:
    controlPlaneVIP: "172.16.21.40"
    ingressVIP: "172.16.21.30"
  kind: MetalLB
  metalLB:
    addressPools:
    - name: "address-pool-1"
      addresses:
    - "172.16.21.30-172.16.21.39"
...
enableControlplaneV2: true
enableDataplaneV2: true
masterNode:
  cpus: 4
  memoryMB: 8192
  replicas: 3
nodePools:
- name: "worker-node-pool"
  enableLoadBalancer: true

These are the important points to understand in the preceding example:

  • The static IP addresses for the worker nodes are specified in an IP block file. The IP block file has four addresses even though there are only three worker nodes. The extra IP address is needed during cluster upgrade, update, and auto repair.

  • DNS and NTP servers are specified in the hostConfig section. In this example, these DNS and NTP servers are for the control-plane nodes and the worker nodes. That is because the worker nodes have static IP addresses. If the worker nodes were to get their IP addresses from a DHCP server, then these DNS and NTP servers would be only for the control-plane nodes.

  • The static IP addresses for the three control-plane nodes are specified in the network.controlPlaneIPBlock section of the user cluster configuration file. There is no need for an extra IP address in this block.

  • The masterNode.replicas field is set to 3.

  • The control-plane VIP and the ingress VIP are both in the same VLAN as the worker nodes and the control-plane nodes.

  • The VIPs that are set aside for Services of type LoadBalancer are specified in the loadBalancer.metalLB.addressPools section of the user cluster configuration file. These VIPs are in the same VLAN as the worker nodes and the control-plane nodes. The set of VIPs specified in this section must include the ingress VIP and must not include the control-plane VIP.

  • The user cluster configuration file does not include a vCenter section. So the user cluster uses the same vSphere resources as the admin cluster.

Firewall rules for Controlplane V2

In addition to the standard firewall rules, create the following firewall rules:

From

Source port

To

Dst port

Protocol

Description

Admin cluster nodes

All

User cluster control-plane VIP

443

https

Allow nodes and Pods in the admin cluster to talk to the Kubernetes API server of the user cluster.

Admin cluster nodes

All

User cluster control-plane nodes

443

https

Allow nodes and Pods in the admin cluster to talk to the Kubernetes API server of the user cluster by using the IP address of a user cluster control-plane node.

The following standard firewall rules are not required for a user cluster that has Controlplane V2 enabled:

From

Source port

To

Dst port

Protocol

Description

Admin cluster worker nodes

All

User cluster nodes

22

SSH

API server to kubelet communication over an SSH tunnel

User cluster worker nodes

All

User cluster control-plane VIP

8132

GRPC

Konnectivity connection

Create a user cluster that has Controlplane V2 enabled

This section shows how to create a user cluster that has Controlplane V2 enabled. In this example, the user cluster uses the same instance of vCenter server as the admin cluster.

Follow the instructions in Create a user cluster.

As you fill in your user cluster configuration file:

  • Set enableControlplaneV2 to true.

  • Set enableDataplaneV2 to true.

  • Set loadBalancer.kind to "MetalLB"

  • Do not specify a value for vCenter.address.

  • Fill in the network.controlPlaneIPBlock section. If you want an HA user cluster, specify three IP addresses. Otherwise, specify one IP address.

When your user cluster is running, verify that the control plane is running on one or three nodes in the user cluster:

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG get nodes

Replace USER_CLUSTER_KUBECONFIG with the path of the user cluster kubeconfig file.

The output shows one or three control-plane nodes along with the worker nodes. For example:

cp-vm-1       Ready    control-plane,master   18m
cp-vm-2       Ready    control-plane,master   18m
cp-vm-3       Ready    control-plane,master   18m
worker-vm-1   Ready                           6m7s
worker-vm-2   Ready                           6m6s
worker-vm-3   Ready                           6m14s

Verify that the control plane for the user cluster is not running in the admin cluster:

kubectl --kubeconfig ADMIN_CLUSTER_KUBECONFIG get nodes --output wide

Replace ADMIN_CLUSTER_KUBECONFIG with the path of the admin cluster kubeconfig file.

The output shows a control-plane node for the admin cluster itself, but does not show any control-plane nodes for the user cluster. For example:

admin-vm-1   Ready    control-plane,master   82m
admin-vm-2   Ready                           71m
admin-vm-3   Ready                           71m

Verify that there are no Pods running in the admin cluster in the USER_CLUSTER_NAME namespace. This is in contrast to the kubeception model where there would be several control-plane Pods running in USER_CLUSTER_NAME namespace of the admin cluster:

kubectl --kubeconfig kubeconfig get pods --namespace USER_CLUSTER_NAME

Output:

No resources found in ... namespace.

Verify that there are control-plane Pods running in the user cluster:

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG get pods\
    --namespace kube-system\
    --selector tier=control-plane

Example output:

clusterapi-controllers-5bbfb6bd9f-57gz8   3/3     Running   1
…
etcd-cp-vm-1                              1/1     Running   0
…
kube-apiserver-cp-vm-1                    1/1     Running   0
…
kube-controller-manager-cp-vm-1           1/1     Running   0
…
kube-scheduler-cp-vm-1                    1/1     Running   0

SSH connections to nodes

With Controlplane V2, to get an SSH connection to a user cluster control-plane node, use the same SSH key that you use to connect to user cluster worker nodes.

This is in contrast to a kubeception user cluster, where you would use the same SSH key that you use to connect to admin cluster worker nodes.

Upgrade a user cluster

Follow the instructions in Upgrading Anthos clusters on VMware.

Note that when you upgrade a user cluster that has Controlplane V2 enabled, the entire user cluster, including the user control plane nodes, is upgraded.

In contrast, for user clusters deployed with the kubeception model, the control plane nodes are not upgraded during user cluster upgrade, and they will not be upgraded until admin cluster upgrade.

Delete a user cluster

To delete a user cluster:

gkectl delete cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --cluster USER_CLUSTER

Replace USER_CLUSTER with the name of the user cluster.

When you delete a user cluster that has Controlplane V2 enabled, the cluster's data disks do not get deleted automatically. So you need to delete the data disks manually. An HA cluster has three data disks, and a non-HA cluster has one data disk.

The data disks for the user cluster control-plane nodes are in the datastore specified in the masterNode.vSphere.datastore field of the user cluster configuration file.

The data disks for the user cluster are in one of these locations:

  • ADMIN_CLUSTER/USER_CLUSTER/

  • ADMIN_DISK_FOLDER/ADMIN_CLUSTER/USER_CLUSTER/

Replace ADMIN_CLUSTER with the name of the admin cluster.

If you specified a folder in the vCenter.dataDisk field of the admin cluster configuration file, replace ADMIN_DISK_FOLDER with the name of the folder.

For a non-HA cluster, the data disk is named USER_CLUSTER-0-data.vmdk.

For an HA cluster, the data disks are named:

  • USER_CLUSTER-0-data.vmdk.
  • USER_CLUSTER-1-data.vmdk.
  • USER_CLUSTER-2-data.vmdk.

You can use the vSphere client to delete the data disks.

Create a user cluster with its own vCenter Server

In certain situations, it makes sense to create a user cluster that uses its own instance of vCenter Server. That is, the admin cluster and an associated user cluster use different instances of vCenter Server.

For example, in an edge location, you might want to have a physical machine running vCenter Server and one or more physical machines running ESXi. You could then use your local instance of vCenter Server to create a vSphere object hierarchy, including data centers, clusters, resource pools, datastores, and folders.

Follow the instructions under Create a user cluster that has Controlplane V2 enabled.

In addition to the steps given in that section, fill in the entire vCenter section of your user cluster configuration file. In particular, specify a value for vCenter.address that is different from the vCenter Server address you specified in the admin cluster configuration file.

Example

vCenter:
  address: "vc-edge.example"
  datacenter: "vc-edge"
  cluster: "vc-edge-workloads"
  resourcePool: "vc-edge-pool
  datastore: "vc-edge-datastore
  caCertPath: "/usr/local/google/home/me/certs/edge-cacert.pem"
  credentials:
    fileRef:
      path: "credential.yaml"
      entry: "vCenter-edge"
  folder: "edge-vm-folder"

Firewall rules

In addition to the standard firewall rules and the firewall rules required for Controlplane V2 in general, create the following firewall rule if your user cluster has its own vCenter Server:

From

Source port

To

Dst port

Protocol

Description

Admin cluster nodes

All

User cluster vCenter Server

443

https

Allow the admin cluster to manage the lifecycle of the user cluster. Required if you specified a value for vCenter.address that is different from the vCenter address in your admin cluster configuration file.

Troubleshooting

See Troubleshooting cluster creation and upgrade.

What's next