Installing using DHCP

Caution: Be sure to plan your installation ahead of time. Your clusters' configurations, as declared in the GKE On-Prem configuration file, become immutable after you create the clusters. You can't change most aspects of your clusters after they are created, including networking and authentication. The only configurable exceptions are:

• Resizing user clusters
• Creating additional user clusters
• Upgrading clusters
• Creating and managing node pools

This page explains how to install GKE On-Prem to a VMware vSphere 6.5 environment using an existing Dynamic Host Configuration Protocol (DHCP) server to assign IP addresses to cluster nodes. You can also install using static IPs.

Overview

The instructions on this page show you how to create an admin cluster and one user cluster with three nodes. Each node runs on a virtual machine (VM) in a vSphere cluster, and each node has an IP address assigned to it by a DHCP server in your environment.

After you've created the clusters, you can create additional user clusters and add or remove nodes in a user cluster.

Before you begin

1. Set up your on-prem environment as described in System requirements.

2. Complete the procedures in Preparing to install.

3. Create an admin workstation in vSphere.

4. SSH into your admin workstation:

   ssh -i ~/.ssh/vsphere_workstation ubuntu@[IP_ADDRESS]
   

5. If you are using a proxy, you need to configure Google Cloud CLI for the proxy, so that you can run gcloud commands. For instructions, see Configuring gcloud CLI for use behind a proxy/firewall.

6. Log in to Google Cloud using your account credentials:

   gcloud auth login

7. Register gcloud as a Docker credential helper. (Read more about this command):

   gcloud auth configure-docker

8. Set a default project. Setting a default Google Cloud causes all gcloud CLI commands to run against the project, so that you don't need to specify your project for each command:

   gcloud config set project [PROJECT_ID]
   

   Replace [PROJECT_ID] with your project ID. (You can find your project ID in Google Cloud console, or by running gcloud config get-value project.)

Using DHCP reservations for cluster nodes

In Kubernetes, it's important that node IP addresses never change. If a node IP address changes or becomes unavailable, it can break the cluster. To prevent this, consider using DHCP reservations to assign permanent addresses nodes in your admin and user clusters. Using DHCP reservations ensures that each node is assigned the same IP addresses after restart or lease renewal.

Choosing a container image registry for installation

To install, GKE On-Prem needs to know where to pull its containerized cluster components. You have two options:

Container Registry

By default, GKE On-Prem uses an existing, Google-owned container image registry hosted by Container Registry. Apart from setting up your proxy to allow traffic from gcr.io, this doesn't require additional setup.

Private Docker registry

You can choose to use a private Docker registry for installation. GKE On-Prem pushes its cluster components to that Docker registry.

Before you install, you need to configure the registry. During installation, you need to populate the GKE On-Prem configuration file with information about the registry.

Configuring a private Docker registry for installation (optional)

This section explains how to configure an existing Docker registry for installing GKE On-Prem. To learn how to create a Docker registry, see Run an externally-accessible registry. After you've configured the registry, you popuate the privateregistryconfig field of the GKE On-Prem configuration file.

If you want to use your private Docker registry for installation, your admin workstation VM must trust the CA that signed your certificate. GKE On-Prem does not support unsecured Docker registries. When you start your Docker registry, you must provide a certificate and a key. The certificate can be signed by a public certificate authority (CA), or it can be self-signed.

To establish this trust, perform the following steps from your admin workstation VM:

1. Create a folder to hold the certificate:

   sudo mkdir -p /etc/docker/certs.d/[REGISTRY_SERVER]
   

   where [REGISTRY_SERVER] is the IP address or hostname of the VM that runs your Docker registry.

2. Copy your certificate file to /etc/docker/certs.d/[REGISTRY_SERVER]/ca.crt. You must name the file ca.crt, even if it had a different name originally.

3. Restart the Docker service:

   sudo service docker restart

4. Verify that you can log in to Docker:

   docker login -u [USERNAME] -p [PASSWORD] [REGISTRY_SERVER]

   where [USERNAME] and [PASSWORD] are the credentials for logging in to the Docker registry.

Now, when you run gkectl prepare during installation, the images needed for installation are pushed to your Docker registry.

Troubleshooting registry configuration

• GET https://[REGISTRY_SERVER]/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers): Make sure you have the correct IP address for the VM that runs your Docker registry.

• login attempt to https://[REGISTRY_SERVER]/v2/ failed with status: 401 Unauthorized: Make sure your username and password are correct.

• GET https://[REGISTRY_SERVER]/v1/users/: x509: certificate signed by unknown authority: Your admin workstation VM doesn't trust the certificate.

Create service accounts' private keys in your admin workstation

In Preparing to install, you created four service accounts. Now, you need to create a JSON private key file for each of those service accounts. You'll provide these keys during installation.

List service accounts' email addresses

First, list the service accounts in your Google Cloud project:

gcloud iam service-accounts list

For a Google Cloud project named my-gcp-project, this command's output looks like this:

gcloud iam service-accounts list
NAME                                    EMAIL
                                        [e-mail]
                                        [e-mail]
                                        [e-mail]
                                        [e-mail]

Take note of each accounts' email address. For each of the following sections, you provide the relevant account's email account.

Access service account

Note: If you purchased GKE Enterprise, you might have already created a allowlisted service account. If so, be sure to provide that account's email address to the following command.

gcloud iam service-accounts keys create access-key.json \
--iam-account [ACCESS_SERVICE_ACCOUNT_EMAIL]

where [ACCESS_SERVICE_ACCOUNT_EMAIL] is the access service account's email address.

Register service account

gcloud iam service-accounts keys create register-key.json \
--iam-account [REGISTER_SERVICE_ACCOUNT_EMAIL]

where [REGISTER_SERVICE_ACCOUNT_EMAIL] is the register service account's email address.

Connect service account

gcloud iam service-accounts keys create connect-key.json \
--iam-account [CONNECT_SERVICE_ACCOUNT_EMAIL]

where [CONNECT_SERVICE_ACCOUNT_EMAIL] is the connect service account's email address.

Cloud Monitoring service account

gcloud iam service-accounts keys create stackdriver-key.json \
--iam-account [STACKDRIVER_SERVICE_ACCOUNT_EMAIL]

where [STACKDRIVER_SERVICE_ACCOUNT_EMAIL] is the Cloud Monitoring service account's email address.

Activating your access service account for Google Cloud CLI

Note: If you purchased GKE Enterprise, you might have already created a allowlisted service account. If so, be sure to provide that account's private key to the following command.

Activating your access service account for gcloud CLI causes all gcloud commands to run as that service account. Since your access service account is allowlisted to access the GKE On-Prem binaries, activating the account for gcloud CLI gives you permission to download GKE On-Prem's binaries from Cloud Storage.

To activate your access service account, run the following command. Be sure to provide the path to the account's key file, if it isn't in the current working directory:

gcloud auth activate-service-account --key-file=access-key.json

Generating a configuration file

To start an installation, you run gkectl create-config to generate a configuration file. You modify the file with your environment's specifications and with the cluster specifications you want.

To generate the file, run the following command, where --config [PATH] is optional and accepts a path and name for the configuration file. Omitting --config creates config.yaml in the current working directory:

gkectl create-config [--config [PATH]]

Modifying the configuration file

Important: Be sure to plan your installation. While you are able to create additional user clusters, changing an existing cluster's configuration is not supported.

Now that you've generated the configuration file, you need to modify it to be suitable for your environment and to meet your expectations for your clusters. The following sections explain each field, the values it expects, and where you might find the information. Some fields are commented out by default. If any of those fields are relevant to your installation, uncomment them and provide values.

bundlepath

A GKE On-Prem bundle is a set of YAML files. Collectively, the YAML files describe all of the components in a particular release of GKE On-Prem.

When you create an admin workstation, it comes with a bundle at /var/lib/gke/bundles/gke-onprem-vsphere-[VERSION]-full.tgz. This bundle's version matches the version of the OVA you used to create the admin workstation.

Note: When you install GKE On-Prem for the first time, your gkectl and bundle versions must be the same. If you created an admin workstation using an OVA that is at a different version than the version you want to install, you might need to upgrade your admin workstation.

Set the value of bundlepath to the path of your admin workstation's bundle file. That is, set bundlepath to:

/var/lib/gke/bundles/gke-onprem-vsphere-[VERSION]-full.tgz

where [VERSION] is the version of GKE On-Prem that you are installing. The latest version is 1.1.2-gke.0.

Note that you are free to keep your bundle file in a different location or give it a different name. Just make sure that in your configuration file, the value of bundlepath is the path to your bundle file, whatever that might be.

vCenter specification

The vCenter Server specification, vcenter, holds information about your vCenter Server instance that GKE On-Prem needs to install to your environment.

vcenter.credentials.address

The vcenter.credentials.address field holds the IP address or the hostname of your vCenter server.

Before you fill in the vsphere.credentials.address field, download and inspect the serving certificate of your vCenter server. Enter the following command to download the certificate and save it to a file named vcenter.pem.

true | openssl s_client -connect [VCENTER_IP]:443 -showcerts 2>/dev/null | sed -ne '/-BEGIN/,/-END/p' > vcenter.pem

Open the certificate file to see the Subject Common Name and the Subject Alternative Name:

openssl x509 -in vcenter.pem -text -noout

The output shows the Subject Common Name (CN). This might be an IP address, or it might be a hostname. For example:

Subject: ... CN = 203.0.113.100

Subject: ... CN = my-host.my-domain.example

The output might also include one or more DNS names under Subject Alternative Name:

X509v3 Subject Alternative Name:
    DNS:vcenter.my-domain.example

Choose the Subject Common Name or one of the DNS names under Subject Alternative Name to use as the value of vcenter.credentials.address in your configuration file. For example:

vcenter:
  credentials:
    address: "203.0.113.1"
    ...

vcenter:
  credentials:
    address: "my-host.my-domain.example"
    ...

You must choose a value that appears in the certificate. For example, if the IP address does not appear in the certificate, you cannot use it for vcenter.credentials.address.

vcenter.credentials

GKE On-Prem needs to know your vCenter Server's username, and password. To provide this information, set the username and password values under vcenter.credentials. For example:

vcenter:
  credentials:
    ...
    username: "my-name"
    password: "my-password"

vcenter.datacenter, .datastore, .cluster, .network

GKE On-Prem needs some information about the structure of your vSphere environment. Set the values under vcenter to provide this information. For example:

vcenter:
  ...
  datacenter: "MY-DATACENTER"
  datastore: "MY-DATASTORE"
  cluster: "MY-VSPHERE-CLUSTER"
  network: "MY-VIRTUAL-NETWORK"

Note: You can specify different vCenter networks for admin and user clusters in the admincluster.vcenter.network and usercluster.vcenter.network fields, respectively.

vcenter.resourcepool

A vSphere resource pool is a logical grouping of vSphere VMs in your vSphere cluster. If you are using a resource pool other than the default, provide its name to vcenter.resourcepool. For example:

vcenter:
  ...
  resourcepool: "my-pool"

If you want GKE On-Prem to deploy its nodes to the vSphere cluster's default resource pool, provide an empty string to vcenter.resourcepool. For example:

vcenter:
  ...
  resourcepool: ""

vcenter.datadisk

GKE On-Prem creates a virtual machine disk (VMDK) to hold the Kubernetes object data for the admin cluster. The installer creates the VMDK for you, but you must provide a name for the VMDK in the vcenter.datadisk field. For example:

vcenter:
  ...
  datadisk: "my-disk.vmdk"

vSAN datastore: Creating a folder for the VMDK

If you are using a vSAN datastore, you need to put the VMDK in a folder. You must manually create the folder ahead of time. To do so, you could use govc to create a folder:

govc datastore.mkdir my-gke-on-prem-folder

Currently, a known issue requires that you provide the folder's universally unique identifier (UUID) path, rather than its file path, to vcenter.datadisk. To find the folder's UUID, open the vCenter Client, select your data store, then the select folder. Copy the folder's UUID. You could also run the following command, where [ADMIN_CONTROL_PLANE_VM] is the vSphere VM that runs the admin control plane:

govc vm.info -json ./vm/[ADMIN_CONTROL_PLANE_VM] | jq '.VirtualMachines[0].Config.Hardware.Device[] | select(.Backing | has("FileName")) | .Backing.FileName'

Then, provide the folder's UUID in the vcenter.datadisk field. For example:

vcenter:
...
datadisk: "14159b5d-4265-a2ba-386b-246e9690c588/my-disk.vmdk"

vcenter.cacertpath

When a client, like GKE On-Prem, sends a request to vCenter Server, the server must prove its identity to the client by presenting a certificate. The certificate is signed by a certificate authority (CA). The client verifies the server's certificate by using the CA's certificate.

Set vcenter.cacertpath to the path of the CA's certificate. For example:

vcenter:
  ...
  cacertpath: "/my-cert-folder/altostrat.crt"

Regardless of whether your vCenter server is using a self-signed certificate or a certificate signed by a public CA, you can get the CA's certificate by running this command on your admin workstation:

true | openssl s_client -connect [VCENTER_IP]:443 -showcerts 2>/dev/null | sed -ne '/-BEGIN/,/-END/p' > vcenter.pem

where [VCENTER_IP] is the IP address of your vCenter server.

Proxy specification

The proxy field holds information of user-provided HTTPS proxy and the addresses that should be excluded from proxying. For example:

proxy:
  url: "https://password:username@domain"
  noproxy: "100.151.222.0/24,corp.domain,100.151.2.1"

• proxy.url is the URL of the HTTPS proxy.
• proxy.noproxy includes the CIDR, domains and IP addresses that should not use the proxy. Typically this includes the vSphere subnet and the private registry address if you're using a private Docker registry.

Admin cluster specification

The admin cluster pecification, admincluster, holds information that GKE On-Prem needs to create the admin cluster.

admincluster.vcenter.network

In admincluster.vcenter.network, you can specify a vCenter network for your admin cluster nodes. Note that this overrides the global setting you provided in vcenter. For example:

admincluster:
  vcenter:
    network: MY-ADMIN-CLUSTER-NETWORK

admincluster.ipblockfilepath

This field is used if you are using static IPs. Since you are using a DHCP server to allocate IP addresses, leave the admincluster.ipblockfilepath field commented out.

admincluster.bigip.credentials (integrated load balancing mode)

If you are using integrated load balancing mode, GKE On-Prem needs to know the IP address or hostname, username, and password of your F5 BIG-IP load balancer. Set the values under admincluster.bigip to provide this information. For example:

admincluster:
  ...
  bigip:
    credentials:
      address: "203.0.113.2"
      username: "my-admin-f5-name"
      password: "rJDlm^%7aOzw"

admincluster.bigip.credentials (integrated load balancing mode)

If you are using integrated load balancing mode, you must create a BIG-IP partition for your admin cluster. Set admincluster.bigip.partition to the name of your partition. For example:

admincluster:
  ...
  bigip:
    partition: "my-admin-f5-partition"

admincluster.vips

Set the value of admincluster.vips.controlplanevip to the IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the admin cluster. Set the value of ingressvip to the IP address you have chosen to configure on the load balancer for the admin cluster's ingress controller. For example:

admincluster:
  ...
  vips:
    controlplanevip: 203.0.113.3
    ingressvip: 203.0.113.4

admincluster.serviceiprange and admincluster.podiprange

The admin cluster must have a range of IP addresses to use for Services and a range of IP addresses to use for Pods. These ranges are specified by the admincluster.serviceiprange and admincluster.podiprange fields. These fields are populated when you run gkectl create-config. If you like, you can change the populated values to values of your choice.

The Service and Pod ranges must not overlap. Also, the Service and Pod ranges must not overlap with IP addresses that are used for nodes in any cluster.

Example:

admincluster:
  ...
  serviceiprange: 10.96.232.0/24
  podiprange: 192.168.0.0/16

User cluster specification

The user cluster specification, usercluster, holds information that GKE On-Prem needs to create the initial user cluster.

Disabling VMware DRS anti-affinity rules (optional)

As of version 1.1.0-gke.6, GKE On-Prem automatically creates VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your user cluster's nodes, causing them to be spread across at least three physical hosts in your datacenter. As of version 1.1.0-gke.6, this feature is automatically enabled for new clusters and existing clusters.

This feature requires that your vSphere environment meets the following conditions:

• VMware DRS is enabled. VMware DRS requires vSphere Enterprise Plus license edition. To learn how to enable DRS, see Creating a DRS Cluster.
• The vSphere user account provided in the vcenter field has the Host.Inventory.EditCluster permission.
• There are at least three physical hosts available.

If you do not have DRS enabled, or if you do not have at least three hosts to which vSphere VMs can be scheduled, add usercluster.antiaffinitygroups.enabled: false to your configuration file. For example:

usercluster:
  ...
  antiaffinitygroups:
    enabled: false

For clusters running more than three nodes

If vSphere vMotion moves a node to a different host, the node's workloads will need to be restarted before they are distributed across hosts again.

Note: You can distribute workloads across multiple nodes using Pod affinity rules. In your workload's Pod template specification, add a topologyKey field with the value onprem.gke.io/failure-domain-name. This adds the onprem.gke.io/failure-domain-name label to nodes periodically and matches the name of the host on which the vSphere VM is running.

usercluster.vcenter.network

In usercluster.vcenter.network, you can specify a vCenter network for your user cluster nodes. Note that this overrides the global setting you provided in vcenter. For example:

usercluster:
  vcenter:
    network: MY-USER-CLUSTER-NETWORK

usercluster.ipblockfilepath

This field is used if you are using static IPs. Since you are using a DHCP server to allocate IP addresses, leave the usercluster.ipblockfilepath field commented out.

usercluster.bigip.credentials (integrated load balancing mode)

If you are using integrated load balancing mode, GKE On-Prem needs to know the IP address or hostname, username, and password of the F5 BIG-IP load balancer that you intend to use for the user cluster. Set the values under usercluster.bigip to provide this information. For example:

usercluster:
  ...
  bigip:
    credentials:
      address: "203.0.113.5"
      username: "my-user-f5-name"
      password: "8%jfQATKO$#z"
  ...

usercluster.bigip.partition (integrated load balancing mode)

You must create a BIG-IP partition for your user cluster. Set usercluster.bigip.partition to the name of your partition. For example:

usercluster:
  ...
  bigip:
    partition: "my-user-f5-partition"
  ...

usercluster.vips

Set the value of usercluster.vips.controlplanevip to the IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the user cluster. Set the value of ingressvip to the IP address you have chosen to configure on the load balancer for the user cluster's ingress controller. For example:

usercluster:
  ...
  vips:
    controlplanevip: 203.0.113.6
    ingressvip: 203.0.113.7

usercluster.serviceiprange and usercluster.podiprange

The user cluster must have a range of IP addresses to use for Services and a range of IP addresses to use for Pods. These ranges are specified by the usercluster.serviceiprange and usercluster.podiprange fields. These fields are populated when you run gkectl create-config. If you like, you can change the populated values to values of your choice.

The Service and Pod ranges must not overlap. Also, the Service and Pod ranges must not overlap with IP addresses that are used for nodes in any cluster.

Example:

usercluster:
  ...
  serviceiprange: 10.96.233.0/24
  podiprange: 172.16.0.0/12

usercluster.clustername

Set the value of usercluster.clustername to a name of your choice. Choose a name that is no longer than 40 characters. For example:

usercluster:
  ...
  clustername: "my-user-cluster-1"

usercluster.masternode.replicas

The usercluster.masternode.replicas field specifies how many control plane nodes you want the user cluster to have. A user cluster's control plane node runs the user control plane, the Kubernetes control plane components. This value must be 1 or 3:

• Set this field to 1 to run one user control plane.
• Set this field to 3 if you want to have a high availability (HA) user control plane composed of three control plane nodes that each run a user control plane.

usercluster.masternode.cpus and usercluster.masternode.memorymb

The usercluster.masternode.cpus and usercluster.masternode.memorymb fields specify how many CPUs and how much memory, in megabytes, is allocated to each control plane node of the user cluster. For example:

usercluster:
  ...
  masternode:
    cpus: 4
    memorymb: 8192

usercluster.workernode.replicas

The usercluster.workernode.replicas field specifies how many worker nodes you want the user cluster to have. The worker nodes run the cluster workloads.

usercluster.workernode.cpus and usercluster.workernode.memorymb

The usercluster.masternode.cpus and usercluster.masternode.memorymb fields specify how many CPUs and how much memory, in megabytes, is allocated to each worker node of the user cluster. For example:

usercluster:
  ...
  workernode:
    cpus: 4
    memorymb: 8192
    replicas: 3

usercluster.oidc

If you intend for clients of the user cluster to use OIDC authentication, set values for the fields under usercluster.oidc. Configuring OIDC is optional.

To learn how to configure OIDC, see Authenticating with OIDC.

About installing version 1.0.2-gke.3

Version 1.0.2-gke.3 introduces the following OIDC fields (usercluster.oidc). These fields enable logging in to a cluster from Google Cloud console:

• usercluster.oidc.kubectlredirecturl
• usercluster.oidc.clientsecret
• usercluster.oidc.usehttpproxy

In version 1.0.2-gke.3, if you want to use OIDC, the clientsecret field is required even if you don't want to log in to a cluster from Google Cloud console. In that case, you can provide a placeholder value for clientsecret:

oidc:
clientsecret: "secret"

usercluster.sni

Note: Some legacy browsers and devices might not support SNI.

Server Name Indication (SNI), an extension to Transport Layer Security (TLS), allows servers to present multiple certificates on a single IP address and TCP port, depending on the client-indicated hostname.

If your CA is already distributed as a trusted CA to clients outside your user cluster and you want to rely on this chain to identify trusted clusters, you can configure the Kubernetes API server with an additional certificate that is presented to external clients of the load balancer IP address.

To use SNI with your user clusters, you need to have your own CA and Public Key Infrastructure (PKI). You provision a separate serving certificate for each user cluster, and GKE On-Prem adds each additional serving certificate to its respective user cluster.

To configure SNI for the Kubernetes API server of the user cluster, provide values for usercluster.sni.certpath (path to the external certificate) and usercluster.sni.keypath (path to the external certificate's private key file). For example:

usercluster:
  ...
  sni:
    certpath: "/my-cert-folder/example.com.crt"
    keypath: "/my-cert-folder/example.com.key"

lbmode

You can use integrated load balancing with DHCP. Integrated load balancing mode applies to your admin cluster and your initial user cluster. It also applies to any additional user clusters that you create in the future. Integrates load balancing mode supports using F5 BIG-IP as your load balancer.

Set the value of lbmode to Integrated. For example:

lbmode: Integrated

gkeconnect

The gkeconnect specification holds information that GKE On-Prem needs to set up management of your on-prem clusters from Google Cloud console.

Set gkeconnect.projectid to the project ID of the Google Cloud project where you want to manage your on-prem clusters.

Set the value of gkeconnect.registerserviceaccountkeypath to the path of the JSON key file for your register service account. Set the value of gkeconnect.agentserviceaccountkeypath to the path of the JSON key file for your connect service account.

If you want the Connect agent to use a proxy to communicate with Google Cloud, set the value of gkeconnect.proxy to the URL of the proxy. Use the format http(s)://[PROXY_ADDRESS].

Example:

gkeconnect:
  projectid: "my-project"
  registerserviceaccountkeypath: "/my-key-folder/register-key.json"
  agentserviceaccountkeypath: "/my-key-folder/connect-key.json"
  proxy: https://203.0.113.20

stackdriver

Note: The stackdriver.proxyconfigsecretname field is removed as of version 1.1.0-gke.6. Don't include it in your configuration file.

The stackdriver specification holds information that GKE On-Prem needs to store log entries generated by your on-prem clusters.

Set stackdriver.projectid to the project ID of the Google Cloud project where you want to view Stackdriver logs that pertain to your on-prem clusters.

Set stackdriver.clusterlocation to a Google Cloud region where you want to store Stackdriver logs. It is a good idea to choose a region that is near your on-prem data center.

Set stackdriver.enablevpc to true if you have your cluster's network controlled by a VPC. This ensures that all telemetry flows through Google's restricted IP addresses.

Set stackdriver.serviceaccountkeypath to the path of the JSON key file for your Stackdriver Logging service account. For example:

stackdriver:
  projectid: "my-project"
  clusterlocation: "us-west1"
  enablevpc: false
  serviceaccountkeypath: "/my-key-folder/stackdriver-key.json"

privateregistryconfig

Note: If you are creating a cluster at version 1.1.1-gke.2, see the known issue affecting use of a Docker registry.

If you have a private Docker registry, the privateregistryconfig field holds information that GKE On-Prem uses to push images to your private registry. If you don't specify a private registry, gkectl pulls GKE On-Prem's container images from its Container Registry repository, gcr.io/gke-on-prem-release, during installation.

Under privatedockerregistry.credentials, set address to the IP address of the machine that runs your private Docker registry. Set username and password to the username and password of your private Docker registry.

When Docker pulls an image from your private registry, the registry must prove its identity by presenting a certificate. The registry's certificate is signed by a certificate authority (CA). Docker uses the CA's certificate to validate the registry's certificate.

Set privateregistryconfig.cacertpath to the path of the CA's certificate. For example:

privateregistryconfig
  ...
  cacertpath: /my-cert-folder/registry-ca.crt

gcrkeypath

Set the value of gcrkeypath to the path of the JSON key file for your access service account. For example:

gcrkeypath: "/my-key-folder/access-key.json"

cloudauditlogging

Note: Currently, this feature is only available to allowlisted users.

If you want to send your Kubernetes audit logs to your Google Cloud project, populate the cloudauditlogging specification. For example:

cloudauditlogging:
  projectid: "my-project"
  # A GCP region where you would like to store audit logs for this cluster.
  clusterlocation: "us-west1"
  # The absolute or relative path to the key file for a GCP service account used to
  # send audit logs from the cluster
  serviceaccountkeypath: "/my-key-folder/audit-logging-key.json"

Learn more about using audit logging.

Validating the configuration file

After you've modified the configuration file, run gkectl check-config to verify that the file is valid and can be used for installation:

gkectl check-config --config [PATH_TO_CONFIG]

If the command returns any FAILURE messages, fix the issues and validate the file again.

Skipping validations

The following gkectl commands automatically run validations against your config file:

• gkectl prepare
• gkectl create cluster
• gkectl upgrade

To skip a command's validations, pass in --skip-validation-all. For example, to skip all validations for gkectl prepare:

gkectl prepare --config [PATH_TO_CONFIG] --skip-validation-all

To see all available flags for skipping specific validations:

gkectl check-config --help

Running gkectl prepare

Note: You need to run gkectl prepare each time you when you upgrade your clusters to a new GKE On-Prem version.

Before you install, you need to run gkectl prepare on your admin workstation to initialize your vSphere environment. The gkectl prepare performs the following tasks:

• Import the node OS image to vSphere and mark it as a template.

• Optionally, validate the container images' build attestations, thereby verifying the images were built and signed by Google and are ready for deployment.

Note: If you're using internal DNS names, or your workstation cannot access Google public DNS addresses (8.8.8.8 or 8.8.4.4), you must specify your internal nameserver in /etc/resolv.conf.

Run gkectl prepare with the GKE On-Prem configuration file, where --validate-attestations is optional:

gkectl prepare --config [CONFIG_FILE] --validate-attestations

Positive output from --validate-attestations is Image [IMAGE_NAME] validated.

Installing GKE On-Prem

You've created a configuration file that specifies how your environment looks and how you'd like your clusters to look, and you've validated the file. You ran gkectl prepare to initialize your environment with the GKE On-Prem software. Now you're ready to initiate a fresh installation of GKE On-Prem.

To install GKE On-Prem, run the following command:

gkectl create cluster --config [CONFIG_FILE]

where [CONFIG_FILE] is the configuration file you generated and modified.

You can reuse the configuration file to create additional user clusters.

Resuming an installation

In the event that your installation is interrupted after the admin cluster was created, you can resume installation by:

• Removing the admincluster specification from the config file.
• Running gkectl create cluster again, passing in the admin cluster's kubeconfig file.

gkectl create cluster --config [CONFIG_FILE] \
--kubeconfig [ADMIN_CLUSTER_KUBECONFIG]

where [ADMIN_CLUSTER_NAME] is the admin cluster's kubeconfig, which was created in the working directory when you started the installation.

Known issues

Currently, you are not able to resume an installation if you are creating a HA user cluster. This issue will be resolved in a future release.

Connecting clusters to Google

• When you populate the gkeconnect specification, your user cluster is automatically registered with Google Cloud console. You can view a registered GKE On-Prem cluster in Google Cloud console's Kubernetes clusters menu. From there, you can sign into the cluster to view its workloads.

• If you don't see your cluster in Google Cloud console within one hour of creating it, refer to Connect troubleshooting.

Enabling ingress

After your user cluster is running, you must enable ingress by creating a Gateway object. The first part of the Gateway manifest is always this:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-autogenerated-k8s-ingress
  namespace: gke-system
spec:
  selector:
    istio: ingress-gke-system

You can tailor the rest of the manifest according to your needs. For example, this manifest says that clients can send requests on port 80 using the HTTP/2 protocol and any hostname:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-autogenerated-k8s-ingress
  namespace: gke-system
spec:
  selector:
    istio: ingress-gke-system
  servers:
  - port:
      number: 80
      protocol: HTTP2
      name: http
    hosts:
    - "*"

If you want to accept HTTPS requests, then you must provide one or more certificates that your ingress controller can present to clients.

To provide a certificate:

1. Create a Secret that holds your certificate and key.
2. Create a Gateway object, or modify an existing Gateway object, that refers to your Secret. The name of the Gateway object must be istio-autogenerated-k8s-ingress.

For example, suppose you have already created a certificate file, ingress-wildcard.crt, and a key file ingress-wildcard.key.

Create a Secret named ingressgateway-wildcard-certs:

Note: As of Istio 1.1.3, the ingress Secret's name cannot start with istio.

kubectl create secret tls \
    --namespace gke-system \
    ingressgateway-wildcard-certs \
    --cert ./ingress-wildcard.crt \
    --key ./ingress-wildcard.key

Here's a manifest for a Gateway that refers to your Secret. Clients can call on port 443 using the HTTPS protocol and any hostname that matches *.example.com. Note that the hostname in the certificate must match the hostname in the manifest, *.example.com in this example:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-autogenerated-k8s-ingress
  namespace: gke-system
spec:
  selector:
    istio: ingress-gke-system
  servers:
  - port:
      number: 80
      protocol: HTTP2
      name: http
    hosts:
    - "*"
  - hosts:
    - "*.example.com"
    port:
      name: https-demo-wildcard
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: ingressgateway-wildcard-certs

You can create multiple TLS certs for different hosts by modifying your Gateway manifest.

Save your manifest to a file named my-gateway.yaml, and create the Gateway:

kubectl apply -f my-gateway.yaml

Now you can use Kubernetes Ingress objects in the standard way.

Limitations

Note: For current known issues, refer to the [Release notes](/anthos/clusters/docs/on-prem/1.1/release-notes).

Limitation	Description
Maximum and minimum limits for clusters and nodes	See Quotas and limits. Your environment's performance might impact these limits.
Uniqueness for user cluster names	All user clusters registered to the same Google Cloud project must have unique names.
Cannot deploy to more than one vCenter and/or vSphere datacenter	Currently, you can only deploy an admin cluster and a set of associated user clusters to a single vCenter and/or vSphere datacenter. You cannot deploy the same admin and user clusters to more than one vCenter and/or vSphere datacenter.
Cannot declaratively change cluster configurations after creation	While you can create additional clusters and resize existing clusters, you cannot change an existing cluster through its configuration file.

Troubleshooting

For more information, refer to Troubleshooting.

Diagnosing cluster issues using gkectl

Use gkectl diagnosecommands to identify cluster issues and share cluster information with Google. See Diagnosing cluster issues.

Default logging behavior

For gkectl and gkeadm it is sufficient to use the default logging settings:

• By default, log entries are saved as follows:

  • For gkectl, the default log file is /home/ubuntu/.config/gke-on-prem/logs/gkectl-$(date).log, and the file is symlinked with the logs/gkectl-$(date).log file in the local directory where you run gkectl.
  • For gkeadm, the default log file is logs/gkeadm-$(date).log in the local directory where you run gkeadm.
• All log entries are saved in the log file, even if they are not printed in the terminal (when --alsologtostderr is false).
• The -v5 verbosity level (default) covers all the log entries needed by the support team.
• The log file also contains the command executed and the failure message.

We recommend that you send the log file to the support team when you need help.

Specifying a non-default location for the log file

To specify a non-default location for the gkectl log file, use the --log_file flag. The log file that you specify will not be symlinked with the local directory.

To specify a non-default location for the gkeadm log file, use the --log_file flag.

Locating Cluster API logs in the admin cluster

If a VM fails to start after the admin control plane has started, you can try debugging this by inspecting the Cluster API controllers' logs in the admin cluster:

1. Find the name of the Cluster API controllers Pod in the kube-system namespace, where [ADMIN_CLUSTER_KUBECONFIG] is the path to the admin cluster's kubeconfig file:

   kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] -n kube-system get pods | grep clusterapi-controllers

2. Open the Pod's logs, where [POD_NAME] is the name of the Pod. Optionally, use grep or a similar tool to search for errors:

   kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] -n kube-system logs [POD_NAME] vsphere-controller-manager

Debugging F5 BIG-IP issues using the admin cluster control plane node's kubeconfig

After an installation, GKE On-Prem generates a kubeconfig file in the home directory of your admin workstation named internal-cluster-kubeconfig-debug. This kubeconfig file is identical to your admin cluster's kubeconfig, except that it points directly at the admin cluster's control plane node, where the admin control plane runs. You can use the internal-cluster-kubeconfig-debug file to debug F5 BIG-IP issues.

gkectl check-config validation fails: can't find F5 BIG-IP partitions

Symptoms

Validation fails because F5 BIG-IP partitions can't be found, even though they exist.

Potential causes

An issue with the F5 BIG-IP API can cause validation to fail.

Resolution

Try running gkectl check-config again.

gkectl prepare --validate-attestations fails: could not validate build attestation

Symptoms

Running gkectl prepare with the optional --validate-attestations flag returns the following error:

could not validate build attestation for gcr.io/gke-on-prem-release/.../...: VIOLATES_POLICY

Potential causes

An attestation might not exist for the affected image(s).

Resolution

Try downloading and deploying the admin workstation OVA again, as instructed in Creating an admin workstation. If the issue persists, reach out to Google for assistance.

Debugging using the bootstrap cluster's logs

During installation, GKE On-Prem creates a temporary bootstrap cluster. After a successful installation, GKE On-Prem deletes the bootstrap cluster, leaving you with your admin cluster and user cluster. Generally, you should have no reason to interact with this cluster.

If something goes wrong during an installation, and you did pass --cleanup-external-cluster=false to gkectl create cluster, you might find it useful to debug using the bootstrap cluster's logs. You can find the Pod, and then get its logs:

kubectl --kubeconfig /home/ubuntu/.kube/kind-config-gkectl get pods -n kube-system

kubectl --kubeconfig /home/ubuntu/.kube/kind-config-gkectl -n kube-system get logs [POD_NAME]

What's next

• Learn how to create additional user clusters.
• View your clusters in Google Cloud console.
• Log in to your clusters.