This page shows how to create a user cluster for Anthos clusters on VMware (GKE on-prem).
The instructions here are complete. For a shorter introduction to creating a user cluster, see Create a user cluster (quickstart).
Before you begin
Get an SSH connection to your admin workstation
Get an SSH connection to your admin workstation.
Recall that gkeadm activated your
component access service account on the admin workstation.
Do all the remaining steps in this topic on your admin workstation in the home directory.
Credentials configuration file
When you used gkeadm to create your admin workstation, you filled in a
credentials configuration file named credential.yaml. This file holds the
username and password for your vCenter server.
Admin cluster configuration file
When gkeadm created your admin workstation, it generated a configuration file
named user-cluster.yaml. This configuration file is for creating your user
cluster.
Filling in your configuration file
name
Set the
name
field to a name of your choice for the user cluster.
gkeOnPremVersion
This field is already filled in for you.
vCenter
The values you set in the vCenter section of your
admin cluster configuration file
are global. That is, they apply to your admin cluster and your user clusters.
For each user cluster that you create, you have the option of overriding some of
the global vCenter values.
If you want to override any of the global vCenter values, fill in the relevant
fields in the
vCenter
section of your user cluster configuration file.
network
Set network.ipMode.type to the same value that you set in your
admin cluster configuration file:
either "dhcp" or "static".
If you set ipMode.type to "static", create an
IP block file
that provides the static IP addresses for the nodes in your user cluster. Then
set
network.ipMode.ipBlockFilePath
to the path of your IP block file.
Provide values for the remaining fields in the
network
section.
Regardless of whether you rely on a DHCP server or specify a list of static IP addresses, you need enough IP addresses to satisfy the following:
The nodes in your user cluster
An additional node in the user cluster to be used temporarily during upgrades
As mentioned previously, if you want to use static IP addresses, then you need to provide an IP block file. Here is an example of an IP block file with six hosts. This is enough addresses for a cluster that has five nodes and an occasional sixth node for upgrades:
blocks:
- netmask: 255.255.252.0
gateway: 172.16.23.254
ips:
- ip: 172.16.20.21
hostname: user-host1
- ip: 172.16.20.22
hostname: user-host2
- ip: 172.16.20.23
hostname: user-host3
- ip: 172.16.20.24
hostname: user-host4
- ip: 172.16.20.25
hostname: user-host5
- ip: 172.16.20.26
hostname: user-host6
loadBalancer
Set aside a VIP for the Kubernetes API server of your user cluster. Set aside
another VIP for the ingress service of your user cluster. Provide your VIPs as
values for
loadBalancer.vips.controlPlaneVIP
and
loadBalancer.vips.ingressVIP.
Set loadBalancer.kind to the same value that you set in your
admin cluster configuration file:
"ManualLB", "F5BigIP", or "Seesaw". Then fill in the corresponding
section:
manualLB,
f5BigIP,
or
seesaw.
proxy
If the network that will have your user cluster nodes is behind a proxy server,
fill in the
proxy
section.
masterNode
Fill in the
masterNode
section.
nodePools
Fill in the
nodePools
section.
antiAffinityGroups
Set
antiAffinityGroups.enabled
to true or false.
authentication
If you want to use OpenID Connect (OIDC) to authenticate users, fill in the
authentication.oidc
section.
If you want to provide an additional serving certificate for your user cluster's
vCenter server, fill in the
authentication.sni
section.
stackdriver
Fill in the
stackdriver
section.
gkeConnect
Fill in the
gkeConnect
section.
cloudRun
Set
cloudRun.enabled
to true or false.
usageMetering
If you want to enable usage metering for your cluster, then fill in the
usageMetering
section.
cloudAuditLogging
If you want to integrate the audit logs from your cluster's Kubernetes API
server with Cloud Audit Logs, fill in the
cloudAuditLogging
section.
Validating your configuration file
After you've filled in your user cluster configuration file, run
gkectl check-config to verify that the file is valid:
gkectl check-config --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] --config [CONFIG_PATH]
where:
[ADMIN_CLUSTER_KUBECONFIG] is the path of the kubeconfig file for your admin cluster.
[CONFIG_PATH] is the path of your user cluster configuration file.
If the command returns any failure messages, fix the issues and validate the file again.
If you want to skip the more time-consuming validations, pass the --fast flag.
To skip individual validations, use the --skip-validation-xxx flags. To
learn more about the check-config command, see
Running preflight checks.
Creating a Seesaw load balancer for your user cluster
If you have chosen to use the bundled Seesaw load balancer, do the step in this section. Otherwise, skip this section.
Create and configure the VMs for your Seesaw load balancer:
gkectl create loadbalancer --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] --config [CONFIG_PATH]
where:
[ADMIN_CLUSTER_KUBECONFIG] is the path of the kubeconfig file for your admin cluster.
[CONFIG_PATH] is the path of your user cluster configuration file.
Creating the user cluster
Create the user cluster:
gkectl create cluster --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] \ --config [CONFIG_PATH] --skip-validation-all
where
[CONFIG_PATH] is the path of your user cluster configuration file.
[ADMIN_CLUSTER_KUBECONFIG] is the path of the kubeconfig file for your admin cluster.
The gkectl create cluster command creates a kubeconfig file named
[USER_CLUSTER_NAME]-kubeconfig in the current directory. You will need this
kubeconfig file later to interact with your user cluster.
Do not use the --skip-validation-all flag if you did not previously validate your user cluster configuration file.
Verifying that your user cluster is running
Verify that your user cluster is running:
kubectl get nodes --kubeconfig [USER_CLUSTER_KUBECONFIG]
where [USER_CLUSTER_KUBECONFIG] is the path of your kubeconfig file.
The output shows the user cluster nodes.
Troubleshooting
See Troubleshooting cluster creation and upgrade.