This document shows how to create an admin cluster for Google Distributed Cloud. The admin cluster runs the Kubernetes control plane for the admin cluster itself and for the associated user clusters. You must create an admin cluster before you create any user clusters to run your workloads.
For more details about the admin cluster, see the installation overview.
Procedure overview
These are the primary steps involved in creating an admin cluster:
- Connect to your admin workstation.
- This VM has the necessary tools to create new clusters.
- Fill in your configuration files.
- Specify the details for your new admin cluster by completing and validating an admin cluster configuration file, a credentials configuration file, and possibly an IP block file.
- Import OS images to vSphere, and push container images to the private registry if applicable.
- Run
gkectl prepare
.
- (Optional) Create a Seesaw load balancer.
- If you have decided to use the Seesaw load balancer, run
gkectl create loadbalancer
.
- Create an admin cluster.
- Use
gkectl
to create a new admin cluster as specified in your completed configuration files. Theonprem-admin-cluster-controller
in the temporary bootstrap cluster manages admin cluster creation.
- Verify that your admin cluster is running.
- Use
kubectl
to view your cluster nodes.
At the end of this procedure, you will have a running admin cluster that you can use to create and manage user clusters.
Before you begin
Ensure that you have created an admin workstation.
Review the IP addresses planning document. Ensure that you have enough IP addresses available, and revisit your decision about how you want your cluster nodes to get their IP addresses: DHCP or static. If you decide to use static IP addresses, you must fill in an IP block file that contains your chosen addresses.
Review the load balancing overview and revisit your decision about the kind of load balancer you want to use. For certain load balancers, you must set up the load balancer before you create your admin cluster.
Look ahead at the
privateRegistry
section, and decide whether you want to use a public or private registry for Google Distributed Cloud components.Look ahead at the osImageType field, and decide what type of operating system you want to run on your admin cluster nodes
1. Connect to your admin workstation
Follow the instructions to get an SSH connection to your admin workstation. The admin workstation has the tools you need to create your admin cluster. The admin workstation also has your component access service account activated.
Do all the remaining steps in this topic on your admin workstation in the home directory.
2. Fill in your configuration file
When gkeadm
created your admin workstation, it generated a configuration file
named admin-cluster.yaml
. This configuration file is for creating your admin
cluster.
Familiarize yourself with the configuration file by scanning the admin cluster configuration file document. You might want to keep this document open in a separate tab or window, because you will refer to it as you complete the following steps.
name
If you want to specify a name for your admin cluster, fill in the
name
field.
bundlePath
The bundle is a zipped file that contains cluster components. It is included with the admin workstation. This field is already filled in for you.
vCenter
Most of the fields in this section are already filled in with values that you
entered when you created your admin workstation. The exception is the
dataDisk
field, which you must fill in now.
network
Decide how you want your cluster nodes to get their IP addresses. The options are:
From a DHCP server that you set up ahead of time. Set
network.ipMode.type
to"dhcp"
.From a list of static IP addresses that you provide. Set
network.ipMode.type
to"static"
, and create an IP block file that provides the static IP addresses. For an example of an IP block file, see Example of filled-in configuration files.
Fill in the rest of the fields in the network section of the configuration file as needed:
If you have decided to use static IP addresses for your cluster nodes, the
network.ipMode.ipBlockFilePath
field and thenetwork.hostconfig
section are required. Thenetwork.hostconfig
section holds information about NTP servers, DNS servers, and DNS search domains used by your cluster nodes.If you are using the Seesaw load balancer, the
network.hostconfig
section is required even if you intend to use DHCP for your cluster nodes.The network.podCIDR and network.serviceCIDR fields have prepopulated values that you can leave unchanged unless they conflict with addresses already being used in your network. Kubernetes uses these ranges to assign IP addresses to Pods and Services in your cluster.
Regardless of whether you rely on a DHCP server or specify a list of static IP addresses, you need to have enough IP addresses available for your admin cluster nodes. This includes the nodes in the admin cluster that run the control planes for any associated user clusters. For an explanation of how many IP addresses you need, see Plan your IP addresses.
loadBalancer
Set aside a VIP for the Kubernetes API server of your admin cluster. Set aside
another VIP for the add-ons server. Provide your VIPs as values for
loadBalancer.vips.controlPlaneVIP
and
loadBalancer.vips.addonsVIP
.
For more information, see VIPs in the admin cluster subnet.
Decide what type of load balancing you want to use. The options are:
MetalLB bundled load balancing. Set
loadBalancer.kind
to"MetalLB"
.Seesaw bundled load balancing. Set
loadBalancer.kind
to"Seesaw"
, and fill in theloadBalancer.seesaw
section.Integrated load balancing with F5 BIG-IP. Set
loadBalancer.kind
to"F5BigIP"
, and fill in thef5BigIP
section.Manual load balancing. Set
loadBalancer.kind
to"ManualLB"
, and fill in themanualLB
section.
For more information about load balancing options, see Overview of load balancing.
antiAffinityGroups
Set antiAffinityGroups.enabled
to true
or false
according to your preference.
Use this field to specify whether you want Google Distributed Cloud to create VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your admin cluster nodes, causing them to be spread across at least three physical hosts in your data center.
adminMaster
If you want to specify CPU and memory for the control-plane node of the admin
cluster, fill in the
adminMaster
.
addonNode
Set
addonNode.autoResize.enabled
to true
or false
according to your preference.
proxy
If the network that will have your admin cluster nodes is behind a proxy server,
fill in the
proxy
section.
privateRegistry
Decide where you want to keep container images for the Google Distributed Cloud components. The options are:
Container Registry
Your own private Docker registry.
If you want to use your own private registry, fill in the
privateRegistry
section.
componentAccessServiceAccountKeyPath
Google Distributed Cloud uses your component access service account to download cluster components from Container Registry. This field holds the path of a JSON key file for your component access service account.
This field is already filled in for you.
gkeConnect
Register your admin cluster
to a Google Cloud fleet by filling in the
gkeConnect
section.
stackdriver
If you want to enable
Cloud Logging and Cloud Monitoring
for your cluster, fill in the
stackdriver
section.
This is section is required by default. That is, if you don't fill in this
section, you must you include the --skip-validation-stackdriver
flag when you
run gkectl create admin
.
cloudAuditLogging
If you want to integrate the audit logs from your cluster's Kubernetes API
server with Cloud Audit Logs, fill in the
cloudAuditLogging
section.
clusterBackup
If you want to enable
backing up of the admin cluster,
set
clusterBackup.datastore
to the
vSphere datastore
where you want to save cluster backups.
autoRepair
If you want to
enable automatic node repair
for your admin cluster, set
autoRepair.enabled
to true
.
secretsEncryption
If you want to enable
always-on Secrets encryption,
fill in the
secretsEncryption
section.
osImageType
Decide what type of OS image you want to use for the admin cluster nodes, and
fill in the
osImageType
section accordingly.
Example of filled-in configuration files
Here is an example of a filled-in IP block file and a filled-in admin cluster configuration file. The configuration enables some, but not all, of the available features.
vc-01-ipblock.yaml
blocks: - netmask: 255.255.252.0 gateway: 172.16.23.254 ips: - ip: 172.16.20.10 hostname: admin-host1 - ip: 172.16.20.11 hostname: admin-host2 - ip: 172.16.20.12 hostname: admin-host3 - ip: 172.16.20.13 hostname: admin-host4 - ip: 172.16.20.14 hostname: admin-host5 - ip: 172.16.20.15 hostname: admin-host6 - ip: 172.16.20.16 hostname: admin-host7 - ip: 172.16.20.17 hostname: admin-host8
vc-01-admin-cluster.yaml
apiVersion: v1 kind: AdminCluster name: "gke-admin-01" bundlePath: "/var/lib/gke/bundles/gke-onprem-vsphere-1.11.0-gke.543-full.tgz" vCenter: address: "vc01.example" datacenter: "vc-01" cluster: "vc01-workloads-1" resourcePool: "vc-01-pool-1" datastore: "vc01-datastore-1" caCertPath: "/usr/local/google/home/me/certs/vc01-cert.pem"" credentials: fileRef: path: "credential.yaml" entry: "vCenter" dataDisk: "vc01-admin-disk.vmdk" network: hostConfig: dnsServers: - "203.0.113.1" - "198.51.100.1" ntpServers: - "216.239.35.4" ipMode: type: "static" ipBlockFilePath: "vc-01-ipblock.yaml" serviceCIDR: "10.96.232.0/24" podCIDR: "192.168.0.0/16" vCenter: networkName: "vc01-net-1" loadBalancer: vips: controlPlaneVIP: "172.16.20.59" addonsVIP: "172.16.20.60" kind: "MetalLB" antiAffinityGroups: enabled: true componentAccessServiceAccountKeyPath: "sa-key.json" gkeConnect: projectID: "my-project-123" registerServiceAccountKeyPath: "connect-register-sa-2203040617.json" stackdriver: projectID: "my-project-123" clusterLocation: "us-central1" enableVPC: false serviceAccountKeyPath: "log-mon-sa-2203040617.json" disableVsphereResourceMetrics: false clusterBackup: datastore: "vc-01-datastore-bu" autoRepair: enabled: true osImageType: "ubuntu_containerd"
Validate your configuration file
After you've filled in your admin cluster configuration file, run
gkectl check-config
to verify that the file is valid:
gkectl check-config --config ADMIN_CLUSTER_CONFIG
Replace ADMIN_CLUSTER_CONFIG with the path of your admin cluster configuration file.
If the command returns any failure messages, fix the issues and validate the file again.
If you want to skip the more time-consuming validations, pass the --fast
flag.
To skip individual validations, use the --skip-validation-xxx
flags. To
learn more about the check-config
command, see
Running preflight checks.
3. Get OS images
Run gkectl prepare
to initialize your vSphere environment:
gkectl prepare --config ADMIN_CLUSTER_CONFIG
The gkectl prepare
command performs the following preparatory tasks:
Imports OS images to vSphere and marks them as VM templates.
If you are using a private Docker registry, pushes the container images to your registry.
Optionally, validates the container images' build attestations, thereby verifying the images were built and signed by Google and are ready for deployment.
4. (Optional) Create a Seesaw load balancer
Recall that you have several load balancing options for your admin cluster: Metal LB, Seesaw, F5 BIG-IP, or manual.
If you have chosen to use the Seesaw load balancer, do the step in this section. Otherwise, you can skip this section.
Create and configure the VMs for your Seesaw load balancer:
gkectl create loadbalancer --config ADMIN_CLUSTER_CONFIG
5. Create the admin cluster
Create the admin cluster:
gkectl create admin --config ADMIN_CLUSTER_CONFIG
Resume creation of the admin cluster after a failure
If the admin cluster creation fails or is canceled, you can run the create
command again:
gkectl create admin --config ADMIN_CLUSTER_CONFIG
Locate the admin cluster kubeconfig file
The gkectl create admin
command creates a kubeconfig file named
kubeconfig
in the current directory. You will need this kubeconfig file
later to interact with your admin cluster.
The kubeconfig file contains the name of your admin cluster. To view the cluster name, you can run:
kubectl config get-clusters --kubeconfig ADMIN_CLUSTER_KUBECONFIG
The output shows the name of the cluster. For example:
NAME gke-admin-tqk8x
If you like, you can change the name and location of your kubeconfig file.
Manage the checkpoint.yaml
file
When you ran the gkectl create admin
command to create the admin cluster, it
created a checkpoint file in the same datastore folder as the admin
cluster data disk. By default, this file has the name
DATA_DISK_NAME‑checkpoint.yaml
. If the length of
DATA_DISK_NAME is greater than or equal to 245 characters, then, due
to the vSphere limit on filename length, the name is
DATA_DISK_NAME.yaml
.
This file contains the admin cluster state and credentials, and is used for future upgrades. Do not delete this file unless you are following the process for deleting an admin cluster.
If you have enabled VM encryption in your instance of vCenter Server, then you
must have the
Cryptographic operations.Direct Access
privilege before you create or upgrade your admin
cluster. Otherwise the checkpoint will not be uploaded. If you cannot obtain
this privilege, then you can disable uploading the checkpoint file by using the
hidden flag --disable-checkpoint
when you run a relevant command.
The checkpoint.yaml
file is automatically updated when you run the
gkectl upgrade admin
command, or when you run a gkectl update
command that
affects the admin cluster.
6. Verify that your admin cluster is running
Verify that your admin cluster is running:
kubectl get nodes --kubeconfig ADMIN_CLUSTER_KUBECONFIG
Replace ADMIN_CLUSTER_KUBECONFIG with the path of your admin cluster kubeconfig file.
The output shows the admin cluster nodes. For example:
gke-admin-master-hdn4z Ready control-plane,master ... gke-admin-node-7f46cc8c47-g7w2c Ready ... gke-admin-node-7f46cc8c47-kwlrs Ready ...
7. Back up files
We recommend that you back up your admin cluster kubeconfig file. That is, copy the kubeconfig file from your admin workstation to another location. Then if you lose access to the admin workstation, or if the kubeconfig file on your admin workstation gets accidentally deleted, you still have access to the admin cluster.
We also recommend that you back up the private SSH key for your admin cluster. Then if you lose access to the admin cluster, you can still use SSH to connect to the admin cluster nodes. This will allow you to troubleshoot and investigate any issues with connectivity to the admin cluster.
Extract the SSH key from the admin cluster to a file named
admin-cluster-ssh-key
:
kubectl --kubeconfig ADMIN_CLUSTER_KUBECONFIG get secrets -n kube-system sshkeys \ -o jsonpath='{.data.vsphere_tmp}' | base64 -d > admin-cluster-ssh-key
Now you can back up admin-cluster-ssh-key
to another location of your choice.
Troubleshooting
See Troubleshooting cluster creation and upgrade.