This page shows how to create a user cluster.
Generating a configuration file
To create a user cluster, you need a user cluster configuration file. If
you used gkeadm
to create your admin workstation, then gkeadm
generated a
template for your configuration file and filled in some of the fields.
If you did not use gkeadm
to create your admin workstation, you can generate
a template by running this command:
gkectl create-config cluster --config [OUTPUT_PATH]
where [OUTPUT_PATH] is a path of your choice for the generated
template. If you do not include the --config
flag, gkectl
names the file
user-cluster.yaml
and puts it in the current directory.
Filling in your configuration file
name
Set the
name
field to a name of your choice for the user cluster.
gkeOnPremVersion
Set the
gkeOnPremVersion
field.
vCenter
The values you set in the vCenter
section of your
admin cluster configuration file
are global. That is, they apply to your admin cluster and your user clusters.
For each user cluster that you create, you have the option of overriding some of
the global vCenter
values.
If you want to override any of the global vCenter
values, fill in the relevant
fields in the
vCenter
section of your user cluster configuration file.
network
Set network.ipMode.type
to the same value that you set in your
admin cluster configuration file:
either "dhcp"
or "static"
.
If you set ipMode.type
to "static"
, create a
host configuration file
that provides the static IP addresses for the nodes in your user cluster. Then
set
network.ipBlockFilePath
to the path of your host configuration file.
Provide values for the remaining fields in the
network
section.
loadBalancer
Set aside a VIP for the Kubernetes API server of your user cluster. Set aside
another VIP for the ingress service of your user cluster. Provide your VIPs as
values for
loadBalancer.controlPlaneVIP
and
loadBalancer.ingressVIP
.
Set loadBalancer.kind
to the same value that you set in your
admin cluster configuration file:
"ManualLB"
, "F5BigIP"
, or "Seesaw"
. Then fill in the corresponding
section:
manualLB
,
f5BigIP
,
or
seesaw
.
proxy
If the network that will have your user cluster nodes is behind a proxy server,
fill in the
proxy
section.
masterNode
Fill in the
masterNode
section.
nodePools
Fill in the
nodePools
section.
authentication
If you want to use OpenID Connect (OIDC) to authenticate users, fill in the
authentication.oidc
section.
If you want to provide an additional serving certificate for your user cluster's
vCenter server, fill in the
authentication.sni
section.
stackdriver
Fill in the
stackdriver
section.
gkeConnect
Fill in the
gkeConnect
section.
cloudRun
Set
cloudRun.enabled
to true
or false
.
Validating your configuration file
After you've filled in your user cluster configuration file, run
gkectl check-config
to verify that the file is valid:
gkectl check-config --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] --config [CONFIG_PATH]
where:
[ADMIN_CLUSTER_KUBECONFIG] is the path of the kubeconfig file for your admin cluster.
[CONFIG_PATH] is the path of your user cluster configuration file.
If the command returns any failure messages, fix the issues and validate the file again.
If you want to skip the more time-consuming validations, pass the --fast
flag.
To skip individual validations, use the --skip-validation-xxx
flags. To
learn more about the check-config
command, see
Running preflight checks.
Creating the user cluster
Create the user cluster:
gkectl create cluster --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] \ --config [CONFIG_PATH] --skip-validation-all
where
[CONFIG_PATH] is the path of your user cluster configuration file.
[ADMIN_CLUSTER_KUBECONFIG] is the path of the kubeconfig file for your admin cluster.
The gkectl create cluster
command creates a kubeconfig file named
[USER_CLUSTER_NAME]-kubeconfig
in the current directory. You will need this
kubeconfig file later to interact with your user cluster.
Do not use the --skip-validation-all
flag if you did not previously validate your configuration file.
Verifying that your user cluster is running
Verify that your user cluster is running:
kubectl get nodes --kubeconfig [USER_CLUSTER_KUBECONFIG]
where [USER_CLUSTER_KUBECONFIG] is the path of your kubeconfig file.
The output shows the user cluster nodes.
Troubleshooting
Diagnosing cluster issues using gkectl
Use gkectl diagnose
commands to identify cluster issues
and share cluster information with Google. See
Diagnosing cluster issues.
Default logging behavior
For gkectl
and gkeadm
it is sufficient to use the
default logging settings:
-
By default, log entries are saved as follows:
-
For
gkectl
, the default log file is/home/ubuntu/.config/gke-on-prem/logs/gkectl-$(date).log
, and the file is symlinked with thelogs/gkectl-$(date).log
file in the local directory where you rungkectl
. -
For
gkeadm
, the default log file islogs/gkeadm-$(date).log
in the local directory where you rungkeadm
.
-
For
- All log entries are saved in the log file, even if they are not printed in
the terminal (when
--alsologtostderr
isfalse
). - The
-v5
verbosity level (default) covers all the log entries needed by the support team. - The log file also contains the command executed and the failure message.
We recommend that you send the log file to the support team when you need help.
Specifying a non-default location for the log file
To specify a non-default location for the gkectl
log file, use
the --log_file
flag. The log file that you specify will not be
symlinked with the local directory.
To specify a non-default location for the gkeadm
log file, use
the --log_file
flag.
Locating Cluster API logs in the admin cluster
If a VM fails to start after the admin control plane has started, you can try debugging this by inspecting the Cluster API controllers' logs in the admin cluster:
Find the name of the Cluster API controllers Pod in the
kube-system
namespace, where [ADMIN_CLUSTER_KUBECONFIG] is the path to the admin cluster's kubeconfig file:kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] -n kube-system get pods | grep clusterapi-controllers
Open the Pod's logs, where [POD_NAME] is the name of the Pod. Optionally, use
grep
or a similar tool to search for errors:kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] -n kube-system logs [POD_NAME] vsphere-controller-manager
Debugging F5 BIG-IP issues using the admin cluster control plane node's kubeconfig
After an installation, GKE on-prem generates a kubeconfig file in
the home directory of your admin workstation named
internal-cluster-kubeconfig-debug
. This kubeconfig file is
identical to your admin cluster's kubeconfig, except that it points directly at
the admin cluster's control plane node, where the admin control plane runs. You can use
the internal-cluster-kubeconfig-debug
file to debug F5 BIG-IP
issues.
gkectl check-config
validation fails: can't find F5 BIG-IP partitions
- Symptoms
Validation fails because F5 BIG-IP partitions can't be found, even though they exist.
- Potential causes
An issue with the F5 BIG-IP API can cause validation to fail.
- Resolution
Try running
gkectl check-config
again.
gkectl prepare --validate-attestations
fails: could not validate build attestation
- Symptoms
Running
gkectl prepare
with the optional--validate-attestations
flag returns the following error:could not validate build attestation for gcr.io/gke-on-prem-release/.../...: VIOLATES_POLICY
- Potential causes
An attestation might not exist for the affected image(s).
- Resolution
Try downloading and deploying the admin workstation OVA again, as instructed in Creating an admin workstation. If the issue persists, reach out to Google for assistance.
Debugging using the bootstrap cluster's logs
During installation, GKE on-prem creates a temporary bootstrap cluster. After a successful installation, GKE on-prem deletes the bootstrap cluster, leaving you with your admin cluster and user cluster. Generally, you should have no reason to interact with this cluster.
If something goes wrong during an installation, and you did pass
--cleanup-external-cluster=false
to gkectl create cluster
,
you might find it useful to debug using the bootstrap cluster's logs. You can
find the Pod, and then get its logs:
kubectl --kubeconfig /home/ubuntu/.kube/kind-config-gkectl get pods -n kube-system
kubectl --kubeconfig /home/ubuntu/.kube/kind-config-gkectl -n kube-system get logs [POD_NAME]
For more information, refer to Troubleshooting.