Creating an admin cluster

This page shows how to create an admin cluster for GKE on VMware.

The instructions here are complete. For a shorter introduction to creating an admin cluster, see Create an admin cluster (quickstart).

Before you begin

Create an admin workstation.

Get an SSH connection to your admin workstation

Get an SSH connection to your admin workstation.

Recall that gkeadm activated your component access service account on the admin workstation.

Do all the remaining steps in this topic on your admin workstation in the home directory.

Credentials configuration file

When you used gkeadm to create your admin workstation, you filled in a credentials configuration file named credential.yaml. This file holds the username and password for your vCenter server.

Admin cluster configuration file

When gkeadm created your admin workstation, it generated a configuration file named admin-cluster.yaml. This configuration file is for creating your admin cluster.

Filling in your configuration file

name

If you want to specify a name for your admin cluster, fill in the name field.

bundlePath

This field is already filled in for you.

vCenter

Most of the fields in this section are already filled in with values that you entered when you created your admin workstation. The exception is the dataDisk field, which you must fill in now.

network

Decide how you want your cluster nodes to get their IP addresses. The options are:

  • From a DHCP server. Set network.ipMode.type to "dhcp".

  • From a list of static IP addresses that you provide. Set network.ipMode.type to "static", and create an IP block file that provides the static IP addresses.

Provide values for the remaining fields in the network section.

Regardless of whether you rely on a DHCP server or specify a list of static IP addresses, you need enough IP addresses to satisfy the following:

  • Three nodes in the admin cluster to run the admin cluster control plane and add-ons.

  • An additional node in the admin cluster to be used temporarily during upgrades.

  • For each user cluster that you intend to create, one or three nodes in the admin cluster to run the control-plane components for the user cluster. If you want a high availability (HA) control plane for a user cluster, then you need three nodes in the admin cluster for the user cluster control plane. Otherwise, you need only one node in the admin cluster for the user cluster control plane.

For example, suppose you intend to create two user clusters: one with an HA control plane and one with a non-HA control plane. Then you would need eight IP addresses for the following nodes in the admin cluster:

  • Three nodes for the admin cluster control plane and add-ons
  • One temporary node
  • Three nodes for the HA user-cluster control plane
  • One node for the non-HA user-cluster control plane

As mentioned previously, if you want to use static IP addresses, then you need to provide an IP block file. Here is an example of an IP block file with eight hosts:

blocks:
  - netmask: 255.255.252.0
    gateway: 172.16.23.254
    ips:
    - ip: 172.16.20.10
      hostname: admin-host1
    - ip: 172.16.20.11
      hostname: admin-host2
    - ip: 172.16.20.12
      hostname: admin-host3
    - ip: 172.16.20.13
      hostname: admin-host4
    - ip: 172.16.20.14
      hostname: admin-host5
    - ip: 172.16.20.15
      hostname: admin-host6
    - ip: 172.16.20.16
      hostname: admin-host7
    - ip: 172.16.20.17
      hostname: admin-host8

loadBalancer

Set aside a VIP for the Kubernetes API server of your admin cluster. Set aside another VIP for the add-ons server. Provide your VIPs as values for loadBalancer.vips.controlPlaneVIP and loadBalancer.vips.addonsVIP.

Decide what type of load balancing you want to use. The options are:

  • Seesaw bundled load balancing. Set loadBalancer.kind to "Seesaw", and fill in the loadBalancer.seesaw section.

  • Integrated load balancing with F5 BIG-IP. Set loadBalancer.kind to "F5BigIP", and fill in the f5BigIP section.

  • Manual load balancing. Set loadBalancer.kind to "ManualLB", and fill in the manualLB section.

antiAffinityGroups

Set antiAffinityGroups.enabled to true or false according to your preference.

adminMaster

Fill in the adminMaster section or remove it according to your preference.

addonNode

Set addonNode.autoResize.enabled to true or false according to your preference.

proxy

If the network that will have your admin cluster nodes is behind a proxy server, fill in the proxy section.

privateRegistry

Decide where you want to keep container images for the GKE on VMware components. The options are:

  • gcr.io. Do not fill in the privateRegistry section.

  • Your own private Docker registry.

To create your own private Docker registry:

  1. Fill in the privateRegistry section.

  2. Copy the registry certificate to the required location at /etc/docker/certs.d/REGISTRY_SERVER_ADDRESS/ca.crt if it does not exist. The registry certificate is set in privateRegistry.caCertPath in the admin seed configuration.

    mkdir -p /etc/docker/certs.d/REGISTRY_SERVER_ADDRESS
cp CERTIFICATE_PATH /etc/docker/certs.d/REGISTRY_SERVER_ADDRESS/ca.crt

    Replace the following:

    • REGISTRY_SERVER_ADDRESS: the IP address of your private registry
    • CERTIFICATE_PATH: the path to the registry certificate

componentAccessServiceAccountKeyPath

This field is already filled in for you.

gkeConnect

If you want to register your admin cluster to a Google Cloud fleet, fill in the gkeConnect section.

stackdriver

If you want to enable Cloud Logging and Cloud Monitoring for your cluster, fill in the stackdriver section.

cloudAuditLogging

If you want to enable Cloud Audit Logs for your cluster, fill in the cloudAuditLogging section.

clusterBackup

Set clusterBackup.datastore to true or false according to your preference.

autoRepair

Set autoRepair.enabled to true or false according to your preference.

secretsEncryption

If you want to enable always-on Secrets encryption, fill in the secretsEncryption section.

Validating your configuration file

After you've filled in your admin cluster configuration file, run gkectl check-config to verify that the file is valid:

gkectl check-config --config [CONFIG_PATH]

where [CONFIG_PATH] is the path of your admin cluster configuration file.

If the command returns any failure messages, fix the issues and validate the file again.

If you want to skip the more time-consuming validations, pass the --fast flag. To skip individual validations, use the --skip-validation-xxx flags. To learn more about the check-config command, see Running preflight checks.

Running gkectl prepare

Run gkectl prepare to initialize your vSphere environment:

gkectl prepare --config [CONFIG_PATH]

The gkectl prepare command performs the following preparatory tasks:

  • Imports the OS images to vSphere and marks them as VM templates.

  • If you are using a private Docker registry, this command pushes the Docker container images to your registry.

  • Optionally, this command validates the container images' build attestations, thereby verifying the images were built and signed by Google and are ready for deployment.

Creating a Seesaw load balancer for your admin cluster

If you have chosen to use the bundled Seesaw load balancer, do the step in this section. Otherwise, you can skip this section.

Create and configure the VMs for your Seesaw load balancer:

gkectl create loadbalancer --config [CONFIG_PATH]

Creating the admin cluster

Create the admin cluster:

gkectl create admin --config [CONFIG_PATH]

where [CONFIG_PATH] is the path of your admin cluster configuration file.

Locating the admin cluster kubeconfig file

The gkectl create admin command creates a kubeconfig file named kubeconfig in the current directory. You will need this kubeconfig file later to interact with your admin cluster.

If you like, you can change the name and location of your kubeconfig file.

Verifying that your admin cluster is running

Verify that your admin cluster is running:

kubectl get nodes --kubeconfig ADMIN_CLUSTER_KUBECONFIG

Replace ADMIN_CLUSTER_KUBECONFIG with the path of your admin cluster kubeconfig file.

The output shows the admin cluster nodes. For example:

gke-admin-master-hdn4z            Ready    control-plane,master ...
gke-admin-node-7f46cc8c47-g7w2c   Ready ...
gke-admin-node-7f46cc8c47-kwlrs   Ready ...

Managing the checkpoint.yaml file

When you ran the gkectl create admin command to create the admin cluster, it created a checkpoint.yaml file in the same datastore folder as the admin cluster data disk. By default, this file has the name DATA_DISK_NAME‑checkpoint.yaml. If the length of DATA_DISK_NAME is greater than or equal to 245 characters, then, due to the vSphere limit on filename length, the name is DATA_DISK_NAME.yaml. Replace DATA_DISK_NAME with the name of the data disk.

This file contains the admin cluster state and credentials and is used for future upgrades. As with the admin cluster datadisk, do not delete this file unless you are following the process for deleting the admin cluster.

If you have enabled VM encryption in your vCenter, then you must have Cryptographer.Access permission before you create or upgrade your admin cluster, if you want the checkpoint file to be uploaded. If you cannot obtain this permission, then you can disable uploading the checkpoint file with the hidden flag --disable-checkpoint when you run a relevant command.

The checkpoint.yaml file is automatically updated when you run the gkectl upgrade admin command, or when you run a gkectl update command that affect the admin cluster.

Troubleshooting

See Troubleshooting cluster creation and upgrade

What's next

Creating a user cluster