Creating a user cluster

This page shows how to create a user cluster.

Generating a configuration file

To create a user cluster, you need a user cluster configuration file. If you used gkeadm to create your admin workstation, then gkeadm generated a template for your configuration file and filled in some of the fields.

If you did not use gkeadm to create your admin workstation, you can generate a template by running this command:

gkectl create-config cluster --config [OUTPUT_PATH]

where [OUTPUT_PATH] is a path of your choice for the generated template. If you do not include the --config flag, gkectl names the file user-cluster.yaml and puts it in the current directory.

Filling in your configuration file

name

Set the name field to a name of your choice for the user cluster.

gkeOnPremVersion

Set the gkeOnPremVersion field.

vCenter

The values you set in the vCenter section of your admin cluster configuration file are global. That is, they apply to your admin cluster and your user clusters.

For each user cluster that you create, you have the option of overriding some of the global vCenter values.

If you want to override any of the global vCenter values, fill in the relevant fields in the vCenter section of your user cluster configuration file.

network

Set network.ipMode.type to the same value that you set in your admin cluster configuration file: either "dhcp" or "static".

If you set ipMode.type to "static", create a host configuration file that provides the static IP addresses for the nodes in your user cluster. Then set network.ipBlockFilePath to the path of your host configuration file.

Provide values for the remaining fields in the network section.

loadBalancer

Set aside a VIP for the Kubernetes API server of your user cluster. Set aside another VIP for the ingress service of your user cluster. Provide your VIPs as values for loadBalancer.controlPlaneVIP and loadBalancer.ingressVIP.

Set loadBalancer.kind to the same value that you set in your admin cluster configuration file: "ManualLB", "F5BigIP", or "Seesaw". Then fill in the corresponding section: manualLB, f5BigIP, or seesaw.

proxy

If the network that will have your user cluster nodes is behind a proxy server, fill in the proxy section.

masterNode

Fill in the masterNode section.

nodePools

Fill in the nodePools section.

authentication

If you want to use OpenID Connect (OIDC) to authenticate users, fill in the authentication.oidc section.

If you want to provide an additional serving certificate for your user cluster's vCenter server, fill in the authentication.sni section.

stackdriver

Fill in the stackdriver section.

gkeConnect

Fill in the gkeConnect section.

cloudRun

Set cloudRun.enabled to true or false.

Validating your configuration file

After you've filled in your user cluster configuration file, run gkectl check-config to verify that the file is valid:

gkectl check-config --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] --config [CONFIG_PATH]

where:

  • [ADMIN_CLUSTER_KUBECONFIG] is the path of the kubeconfig file for your admin cluster.

  • [CONFIG_PATH] is the path of your user cluster configuration file.

If the command returns any failure messages, fix the issues and validate the file again.

If you want to skip the more time-consuming validations, pass the --fast flag. To skip individual validations, use the --skip-validation-xxx flags. To learn more about the check-config command, see Running preflight checks.

Creating the user cluster

Create the user cluster:

gkectl create cluster --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] \
  --config [CONFIG_PATH] --skip-validation-all

where

  • [CONFIG_PATH] is the path of your user cluster configuration file.

  • [ADMIN_CLUSTER_KUBECONFIG] is the path of the kubeconfig file for your admin cluster.

The gkectl create cluster command creates a kubeconfig file named [USER_CLUSTER_NAME]-kubeconfig in the current directory. You will need this kubeconfig file later to interact with your user cluster.

Do not use the --skip-validation-all flag if you did not previously validate your configuration file.

Verifying that your user cluster is running

Verify that your user cluster is running:

kubectl get nodes --kubeconfig [USER_CLUSTER_KUBECONFIG]

where [USER_CLUSTER_KUBECONFIG] is the path of your kubeconfig file.

The output shows the user cluster nodes.

Troubleshooting

Diagnosing cluster issues using gkectl

Use gkectl diagnosecommands to identify cluster issues and share cluster information with Google. See Diagnosing cluster issues.

Default logging behavior

For gkectl and gkeadm it is sufficient to use the default logging settings:

  • By default, log entries are saved as follows:

    • For gkectl, the default log file is /home/ubuntu/.config/gke-on-prem/logs/gkectl-$(date).log, and the file is symlinked with the logs/gkectl-$(date).log file in the local directory where you run gkectl.
    • For gkeadm, the default log file is logs/gkeadm-$(date).log in the local directory where you run gkeadm.
  • All log entries are saved in the log file, even if they are not printed in the terminal (when --alsologtostderr is false).
  • The -v5 verbosity level (default) covers all the log entries needed by the support team.
  • The log file also contains the command executed and the failure message.

We recommend that you send the log file to the support team when you need help.

Specifying a non-default location for the log file

To specify a non-default location for the gkectl log file, use the --log_file flag. The log file that you specify will not be symlinked with the local directory.

To specify a non-default location for the gkeadm log file, use the --log_file flag.

Locating Cluster API logs in the admin cluster

If a VM fails to start after the admin control plane has started, you can try debugging this by inspecting the Cluster API controllers' logs in the admin cluster:

  1. Find the name of the Cluster API controllers Pod in the kube-system namespace, where [ADMIN_CLUSTER_KUBECONFIG] is the path to the admin cluster's kubeconfig file:

    kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] -n kube-system get pods | grep clusterapi-controllers

  2. Open the Pod's logs, where [POD_NAME] is the name of the Pod. Optionally, use grep or a similar tool to search for errors:

    kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] -n kube-system logs [POD_NAME] vsphere-controller-manager

Debugging F5 BIG-IP issues using the admin cluster control plane node's kubeconfig

After an installation, GKE on-prem generates a kubeconfig file in the home directory of your admin workstation named internal-cluster-kubeconfig-debug. This kubeconfig file is identical to your admin cluster's kubeconfig, except that it points directly at the admin cluster's control plane node, where the admin control plane runs. You can use the internal-cluster-kubeconfig-debug file to debug F5 BIG-IP issues.

gkectl check-config validation fails: can't find F5 BIG-IP partitions

Symptoms

Validation fails because F5 BIG-IP partitions can't be found, even though they exist.

Potential causes

An issue with the F5 BIG-IP API can cause validation to fail.

Resolution

Try running gkectl check-config again.

gkectl prepare --validate-attestations fails: could not validate build attestation

Symptoms

Running gkectl prepare with the optional --validate-attestations flag returns the following error:

could not validate build attestation for gcr.io/gke-on-prem-release/.../...: VIOLATES_POLICY
Potential causes

An attestation might not exist for the affected image(s).

Resolution

Try downloading and deploying the admin workstation OVA again, as instructed in Creating an admin workstation. If the issue persists, reach out to Google for assistance.

Debugging using the bootstrap cluster's logs

During installation, GKE on-prem creates a temporary bootstrap cluster. After a successful installation, GKE on-prem deletes the bootstrap cluster, leaving you with your admin cluster and user cluster. Generally, you should have no reason to interact with this cluster.

If something goes wrong during an installation, and you did pass --cleanup-external-cluster=false to gkectl create cluster, you might find it useful to debug using the bootstrap cluster's logs. You can find the Pod, and then get its logs:

kubectl --kubeconfig /home/ubuntu/.kube/kind-config-gkectl get pods -n kube-system
kubectl --kubeconfig /home/ubuntu/.kube/kind-config-gkectl -n kube-system get logs [POD_NAME]

For more information, refer to Troubleshooting.