Installing using static IPs

This page explains how to install GKE On-Prem to a VMware vSphere 6.5 environment using static IPs. You can also install using a DHCP server.

Overview

The instructions on this page show you how to create an admin cluster and one user cluster with three nodes.

After you've created the clusters, you can create additional user clusters and add or remove nodes in a user cluster.

Before you begin

  1. Set up your on-prem environment as described in System requirements.

  2. Complete the procedures in Preparing to install.

  3. Create an admin workstation in vSphere.

  4. Learn how to enable manual load balancing, if you want to use it.

  5. SSH into your admin workstation:

    ssh -i ~/.ssh/vsphere_workstation ubuntu@[IP_ADDRESS]

  6. If you are using a proxy, you need to configure Google Cloud CLI for the proxy, so that you can run gcloud and gsutil commands. For instructions, see Configuring gcloud CLI for use behind a proxy/firewall.

  7. Log in to Google Cloud using your account credentials:

    gcloud auth login

  8. Register gcloud as a Docker credential helper. (Read more about this command):

    gcloud auth configure-docker

  9. Set a default project. Setting a default Google Cloud causes all gcloud CLI commands to run against the project, so that you don't need to specify your project for each command:

    gcloud config set project [PROJECT_ID]

    Replace [PROJECT_ID] with your project ID. (You can find your project ID in Google Cloud console, or by running gcloud config get-value project.)

Choosing a container image registry for installation

To install, GKE On-Prem needs to know where to pull its containerized cluster components. You have two options:

Container Registry

By default, GKE On-Prem uses an existing, Google-owned container image registry hosted by Container Registry. Apart from setting up your proxy to allow traffic from gcr.io, this doesn't require additional setup.

Private Docker registry

You can choose to use a private Docker registry for installation. GKE On-Prem pushes its cluster components to that Docker registry.

Before you install, you need to configure the registry. During installation, you need to populate the GKE On-Prem configuration file with information about the registry.

Configuring a private Docker registry for installation (optional)

This section explains how to configure an existing Docker registry for installing GKE On-Prem. To learn how to create a Docker registry, see Run an externally-accessible registry. After you've configured the registry, you popuate the privateregistryconfig field of the GKE On-Prem configuration file.

If you want to use your private Docker registry for installation, your admin workstation VM must trust the CA that signed your certificate. GKE On-Prem does not support unsecured Docker registries. When you start your Docker registry, you must provide a certificate and a key. The certificate can be signed by a public certificate authority (CA), or it can be self-signed.

To establish this trust, perform the following steps from your admin workstation VM:

  1. Create a folder to hold the certificate:

    sudo mkdir -p /etc/docker/certs.d/[REGISTRY_SERVER]

    where [REGISTRY_SERVER] is the IP address or hostname of the VM that runs your Docker registry.

  2. Copy your certificate file to /etc/docker/certs.d/[REGISTRY_SERVER]/ca.crt. You must name the file ca.crt, even if it had a different name originally.

  3. Restart the Docker service:

    sudo service docker restart

  4. Verify that you can log in to Docker:

    docker login -u [USERNAME] -p [PASSWORD] [REGISTRY_SERVER]

    where [USERNAME] and [PASSWORD] are the credentials for logging in to the Docker registry.

Now, when you run gkectl prepare during installation, the images needed for installation are pushed to your Docker registry.

Troubleshooting registry configuration

  • GET https://[REGISTRY_SERVER]/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers): Make sure you have the correct IP address for the VM that runs your Docker registry.

  • login attempt to https://[REGISTRY_SERVER]/v2/ failed with status: 401 Unauthorized: Make sure your username and password are correct.

  • GET https://[REGISTRY_SERVER]/v1/users/: x509: certificate signed by unknown authority: Your admin workstation VM doesn't trust the certificate.

Configuring static IPs

During installation, you generate a GKE On-Prem configuration file. The configuration file you generate includes two ipblockfilepath fields:

  • admincluster.ipblockfilepath
  • usercluster.ipblockfilepath.

The ipblockfilepath accepts the path to a YAML file containing a host configuration (or hostconfig) file, as described below.

If you want to use static IPs, you need to create two YAML files in your admin workstation: one containing a hostconfig to be used by your admin cluster, and another to be used by your user cluster.

You need a minimum of N + 4 IP/hostname pairs in the admin cluster IP configuration, where N is the number of user clusters you plan to create.

You can choose to create a high availability user cluster. A HA user cluster uses three user control planes. Each VM that runs a user control plane requires its own static IP. Since the user control planes are managed by the admin cluster, these values should be provided in the admin cluster's hostconfig file.

Example

The following is an example of a hostconfig file with three hosts. Your file might look different depending on your environment. For example, you might expand the ips array with more ip/hostname pairs:

hostconfig:
  dns: 172.16.255.1
  tod: 192.138.210.214
  otherdns:
  - 8.8.8.8
  - 8.8.4.4
  othertod:
  - ntp.ubuntu.com
blocks:
  - netmask: 255.255.252.0
    gateway: 110.116.232.1
    ips:
    - ip: 10.116.232.23
      hostname: host1.enterprise.net  # will be trimmed to host1
    - ip: 10.116.232.65
      hostname: host2.enterprise.net  # will be trimmed to host2
    - ip: 10.116.232.66
      hostname: host3.enterprise.net  # will be trimmed to host3

The YAML file contains two sections, hostconfig and blocks.

hostconfig

The hostconfig contains networking parameters that apply statically to all of a cluster's nodes. hostconfig configures the following values:

  • dns: IP address of the DNS server to use for nodes.
  • tod: IP address of the time server to use for nodes.
  • otherdns: Alternate DNS servers to use for nodes.
  • othertod: Alternate time servers to use for nodes.

blocks

blocks contains an array of static IP addresses blocks. Currently, GKE On-Prem only considers the first block for IP allocation. Each block represents a network and its IP addresses within it.

netmask and gateway

netmask and gateway represent the network mask and default gateway to use for nodes.

blocks:
  - netmask: 255.255.252.0
    gateway: 110.116.232.1

ips

An ips array lists the IPs you've allocated. Each object in the array contains an IPv4 address and its hostname:

blocks:
...
  ips:
  - ip: [IPV4_ADDRESS]
    hostname: [HOSTNAME]
  - ip: [IPV4_ADDRESS]
    hostname: [HOSTNAME]
  - ip: [IPV4_ADDRESS]
    hostname: [HOSTNAME]
...

GKE On-Prem keeps tracks of free and assigned IP addresses within this block, and allocates one available IP address to each node in a cluster. Ensure that the number of IP addresses in the array is strictly greater than the number of nodes in the cluster, and that each IP address is unique to your environment's network.

hostname, interpreted as the local hostname without its domain. If you specify a fully-qualified domain name (FQDN), the domain name is trimmed. For example, host1.enterprise.net becomes host1. hostname values must be lowercase.

Creating hostconfig file

The following is an example of a hostconfig file for a user cluster that has three nodes:

  1. Copy the following template to a YAML file:

    hostconfig:
  dns:
  tod:
blocks:
  - netmask:
    gateway:
    ips:
    - ip:
      hostname:
    - ip:
      hostname:
    - ip:
      hostname:

  2. Save the files by different names, like admin-cluster-hostconfig.yaml and user-cluster-hostconfig.yaml.

  3. During installation, modify the configuration file's admincluster.ipblockfilepath and usercluster.ipblockfilepath fields with the appropriate files.

Create service accounts' private keys in your admin workstation

In Preparing to install, you created four service accounts. Now, you need to create a JSON private key file for each of those service accounts. You'll provide these keys during installation.

List service accounts' email addresses

First, list the service accounts in your Google Cloud project:

gcloud iam service-accounts list

For a Google Cloud project named my-gcp-project, this command's output looks like this:

gcloud iam service-accounts list
NAME                                    EMAIL
                                        access-service-account@my-gcp-project.iam.gserviceaccount.com
                                        register-service-account@my-gcp-project.iam.gserviceaccount.com
                                        connect-service-account@my-gcp-project.iam.gserviceaccount.com
                                        stackdriver-service-account@my-gcp-project.iam.gserviceaccount.com

Take note of each account's email address. For each of the following sections, you provide the relevant account's email account.

Access service account

gcloud iam service-accounts keys create access-key.json \
--iam-account [ACCESS_SERVICE_ACCOUNT_EMAIL]

where [ACCESS_SERVICE_ACCOUNT_EMAIL] is the access service account's email address.

Register service account

gcloud iam service-accounts keys create register-key.json \
--iam-account [REGISTER_SERVICE_ACCOUNT_EMAIL]

where [REGISTER_SERVICE_ACCOUNT_EMAIL] is the register service account's email address.

Connect service account

gcloud iam service-accounts keys create connect-key.json \
--iam-account [CONNECT_SERVICE_ACCOUNT_EMAIL]

where [CONNECT_SERVICE_ACCOUNT_EMAIL] is the connect service account's email address.

Cloud Monitoring service account

gcloud iam service-accounts keys create stackdriver-key.json \
--iam-account [STACKDRIVER_SERVICE_ACCOUNT_EMAIL]

where [STACKDRIVER_SERVICE_ACCOUNT_EMAIL] is the Cloud Monitoring service account's email address.

Activating your access service account for Google Cloud CLI

Activating your access service account for gcloud CLI causes all gcloud and gsutil commands to run as that service account. Since your access service account is allowlisted to access the GKE On-Prem binaries, activating the account for gcloud CLI gives you permission to download GKE On-Prem's binaries from Cloud Storage.

To activate your access service account, run the following command. Be sure to provide the path to the account's key file, if it isn't in the current working directory:

gcloud auth activate-service-account --key-file=access-key.json

Generating a configuration file

To start an installation, you run gkectl create-config to generate a configuration file. You modify the file with your environment's specifications and with the cluster specifications you want.

To generate the file, run the following command, where --config [PATH] is optional and accepts a path and name for the configuration file. Omitting --config creates config.yaml in the current working directory:

gkectl create-config [--config [PATH]]

Modifying the configuration file

Now that you've generated the configuration file, you need to modify it to be suitable for your environment and to meet your expectations for your clusters. The following sections explain each field, the values it expects, and where you might find the information. Some fields are commented out by default. If any of those fields are relevant to your installation, uncomment them and provide values.

bundlepath

The GKE On-Prem bundle file contains all of the components in a particular release of GKE On-Prem. When you create an admin workstation, it comes with a full bundle at /var/lib/gke/bundles/gke-onprem-vsphere-[VERSION]-full.tgz. This bundle's version matches the version of the OVA you imported to create the admin workstation.

Set the value of bundlepath to the path of your admin workstation's bundle file. That is, set bundlepath to:

/var/lib/gke/bundles/gke-onprem-vsphere-[VERSION]-full.tgz

where [VERSION] is the version of GKE On-Prem that you are installing. The latest version is 1.1.2-gke.0.

Note that you are free to keep your bundle file in a different location or give it a different name. Just make sure that in your configuration file, the value of bundlepath is the path to your bundle file, whatever that might be.

vCenter specification

The vcenter specification holds information about your vCenter Server instance. GKE On-Prem needs this information to communicate with your vCenter Server.

vcenter.credentials.address

The vcenter.credentials.address field holds the IP address or the hostname of your vCenter server.

Before you fill in the vsphere.credentials.address field, download and inspect the serving certificate of your vCenter server. Enter the following command to download the certificate and save it to a file named vcenter.pem.

true | openssl s_client -connect [VCENTER_IP]:443 -showcerts 2>/dev/null | sed -ne '/-BEGIN/,/-END/p' > vcenter.pem

Open the certificate file to see the Subject Common Name and the Subject Alternative Name:

openssl x509 -in vcenter.pem -text -noout

The output shows the Subject Common Name (CN). This might be an IP address, or it might be a hostname. For example:

Subject: ... CN = 203.0.113.100
Subject: ... CN = my-host.my-domain.example

The output might also include one or more DNS names under Subject Alternative Name:

X509v3 Subject Alternative Name:
    DNS:vcenter.my-domain.example

Choose the Subject Common Name or one of the DNS names under Subject Alternative Name to use as the value of vcenter.credentials.address in your configuration file. For example:

vcenter:
  credentials:
    address: "203.0.113.1"
    ...

vcenter:
  credentials:
    address: "my-host.my-domain.example"
    ...

You must choose a value that appears in the certificate. For example, if the IP address does not appear in the certificate, you cannot use it for vcenter.credentials.address.

vcenter.credentials

GKE On-Prem needs to know your vCenter Server's username, and password. To provide this information, set the username and password values under vcenter.credentials. For example:

vcenter:
  credentials:
    ...
    username: "my-name"
    password: "my-password"

vcenter.datacenter, .datastore, .cluster, .network

GKE On-Prem needs some information about the structure of your vSphere environment. Set the values under vcenter to provide this information. For example:

vcenter:
  ...
  datacenter: "MY-DATACENTER"
  datastore: "MY-DATASTORE"
  cluster: "MY-VSPHERE-CLUSTER"
  network: "MY-VIRTUAL-NETWORK"

vcenter.resourcepool

A vSphere resource pool is a logical grouping of vSphere VMs in your vSphere cluster. If you are using a resource pool other than the default, provide its name to vcenter.resourcepool. For example:

vcenter:
  ...
  resourcepool: "my-pool"

If you want GKE On-Prem to deploy its nodes to the vSphere cluster's default resource pool, provide an empty string to vcenter.resourcepool. For example:

vcenter:
  ...
  resourcepool: ""

vcenter.datadisk

GKE On-Prem creates a virtual machine disk (VMDK) to hold the Kubernetes object data for the admin cluster. The installer creates the VMDK for you, but you must provide a name for the VMDK in the vcenter.datadisk field. For example:

vcenter:
  ...
  datadisk: "my-disk.vmdk"
vSAN datastore: Creating a folder for the VMDK

If you are using a vSAN datastore, you need to put the VMDK in a folder. You must manually create the folder ahead of time. To do so, you could use govc to create a folder:

govc datastore.mkdir my-gke-on-prem-folder

Currently, a known issue requires that you provide the folder's universally unique identifier (UUID) path, rather than its file path, to vcenter.datadisk. To find the folder's UUID, open the vCenter Client, select your data store, then the select folder. Copy the folder's UUID. You could also run the following command, where [ADMIN_CONTROL_PLANE_VM] is the vSphere VM that runs the admin control plane:

govc vm.info -json ./vm/[ADMIN_CONTROL_PLANE_VM] | jq '.VirtualMachines[0].Config.Hardware.Device[] | select(.Backing | has("FileName")) | .Backing.FileName'

Then, provide the folder's UUID in the vcenter.datadisk field. For example:

vcenter:
...
datadisk: "14159b5d-4265-a2ba-386b-246e9690c588/my-disk.vmdk"

vcenter.cacertpath

When a client, like GKE On-Prem, sends a request to vCenter Server, the server must prove its identity to the client by presenting a certificate. The certificate is signed by a certificate authority (CA). The client verifies the server's certificate by using the CA's certificate.

Set vcenter.cacertpath to the path of the CA's certificate. For example:

vcenter:
  ...
  cacertpath: "/my-cert-folder/altostrat.crt"

Regardless of whether your vCenter server is using a self-signed certificate or a certificate signed by a public CA, you can get the CA's certificate by running this command on your admin workstation:

true | openssl s_client -connect [VCENTER_IP]:443 -showcerts 2>/dev/null | sed -ne '/-BEGIN/,/-END/p' > vcenter.pem

where [VCENTER_IP] is the IP address of your vCenter server.

Proxy specification

If your network is behind a proxy server, populate the proxy field with HTTPS proxy and the addresses that should be excluded from proxying. For example:

proxy:
  url: "https://password:username@domain"
  noproxy: "100.151.222.0/24,corp.domain,100.151.2.1"
  • proxy.url is the URL of the HTTPS proxy.
  • proxy.noproxy includes the CIDR, domains and IP addresses that should not use the proxy. Typically this includes the vSphere subnet and the private registry address if you're using a private Docker registry.

Admin cluster specification

The admincluster specification holds information that GKE On-Prem needs to create the admin cluster.

admincluster.vcenter.network

In admincluster.vcenter.network, you can specify a vCenter network for your admin cluster nodes. Note that this overrides the global setting you provided in vcenter. For example:

admincluster:
  vcenter:
    network: MY-ADMIN-CLUSTER-NETWORK

admincluster.ipblockfilepath

Since you are using static IP addresses, you must have a host configuration file as described in Configuring static IPs. Provide the path to your host configuration file in the admincluster.ipblockfilepath field. For example:

admincluster:
  ipblockfilepath: "/my-config-folder/my-admin-hostconfig.yaml"

admincluster.manuallbspec (manual load balancing mode)

The ingress controller in the admin cluster is implemented as a Service of type NodePort. The Service has one ServicePort for HTTP and another ServicePort for HTTPS. If you are using manual load balancing mode, you must choose nodePort values for these ServicePorts. Specify the nodePort values in ingresshttpnodeport and ingresshttpsnodeport. For example:

admincluster:
  ...
  manuallbspec:
    ingresshttpnodeport: 32527
    ingresshttpsnodeport: 30139

The Kubernetes API server in the admin cluster is implemented as a Service of type NodePort. If you are using manual load balancing, you must choose a nodePort value for the Service. Specify the nodePort value in controlplanenodeport For example:

admincluster:
  ...
  manuallbspec:
    ...
    controlplanenodeport: 30968

The addons server in the admin cluster is implemented as a Service of type NodePort. If you are using manual load balancing, you must choose a nodePort value for the Service. Specify the nodePort value in controlplanenodeport For example:

admincluster:
  manuallbspec:
    ...
    addonsnodeport: 30562

admincluster.bigip.credentials (integrated load balancing mode)

If you are not using integrated load balancing mode, leave admincluster.bigip commented out.

If you are using integrated load balancing mode, GKE On-Prem needs to know the IP address or hostname, username, and password of your F5 BIG-IP load balancer. Set the values under admincluster.bigip to provide this information. For example:

admincluster:
  ...
  bigip:
    credentials:
      address: "203.0.113.2"
      username: "my-admin-f5-name"
      password: "rJDlm^%7aOzw"

admincluster.bigip.partition (integrated load balancing mode)

If you are using integrated load balancing mode, you must create a BIG-IP partition for your admin cluster. Set admincluster.bigip.partition to the name of your partition. For example:

admincluster:
  ...
  bigip:
    partition: "my-admin-f5-partition"

admincluster.vips

Regardless of whether you are using integrated or manual load balancing for the admin cluster, you need to fill in the admincluster.vips field.

Set the value of admincluster.vips.controlplanevip to the IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the admin cluster. Set the value of ingressvip to the IP address you have chosen to configure on the load balancer for the admin cluster's ingress controller. For example:

admincluster:
  ...
  vips:
    controlplanevip: 203.0.113.3
    ingressvip: 203.0.113.4

admincluster.serviceiprange and admincluster.podiprange

The admin cluster must have a range of IP addresses to use for Services and a range of IP addresses to use for Pods. These ranges are specified by the admincluster.serviceiprange and admincluster.podiprange fields. These fields are populated when you run gkectl create-config. If you like, you can change the populated values to values of your choice.

The Service and Pod ranges must not overlap. Also, the Service and Pod ranges must not overlap with IP addresses that are used for nodes in any cluster.

Example:

admincluster:
  ...
  serviceiprange: 10.96.232.0/24
  podiprange: 192.168.0.0/16

User cluster specification

The user cluster specification, usercluster, holds information that GKE On-Prem needs to create the initial user cluster.

Disabling VMware DRS anti-affinity rules (optional)

As of version 1.1.0-gke.6, GKE On-Prem automatically creates VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your user cluster's nodes, causing them to be spread across at least three physical hosts in your datacenter. As of version 1.1.0-gke.6, this feature is automatically enabled for new clusters and existing clusters.

This feature requires that your vSphere environment meets the following conditions:

If you do not have DRS enabled, or if you do not have at least three hosts to which vSphere VMs can be scheduled, add usercluster.antiaffinitygroups.enabled: false to your configuration file. For example:

usercluster:
  ...
  antiaffinitygroups:
    enabled: false
For clusters running more than three nodes
If vSphere vMotion moves a node to a different host, the node's workloads will need to be restarted before they are distributed across hosts again.

usercluster.vcenter.network

In usercluster.vcenter.network, you can specify a vCenter network for your user cluster nodes. Note that this overrides the global setting you provided in vcenter. For example:

usercluster:
  vcenter:
    network: MY-USER-CLUSTER-NETWORK

usercluster.ipblockfilepath

Since you are using static IP addresses, you must have a host configuration file as described in Configuring static IPs. Provide the path to your host configuration file in the usercluster.ipblockfilepath field. For example:

usercluster:
  ipblockfilepath: "/my-config-folder/my-user-hostconfig.yaml"

usercluster.manuallbspec (manual load balancing mode)

The ingress controller in the user cluster is implemented as a Service of type NodePort. The Service has one ServicePort for HTTP and another ServicePort for HTTPS. If you are using manual load balancing mode, you must choose nodePort values for these ServicePorts. Specify the nodePort values in ingresshttpnodeport and ingresshttpsnodeport. For example:

usercluster:
  manuallbspec:
    ingresshttpnodeport: 30243
    ingresshttpsnodeport: 30879

The Kubernetes API server in the user cluster is implemented as a Service of type NodePort. If you are using manual load balancing, you must choose a nodePort value for the Service. Specify the nodePort value in controlplanenodeport. For example:

usercluster:
  ...
  manuallbspec:
    ...
    controlplanenodeport: 30562

usercluster.bigip.credentials (integrated load balancing mode)

If you are using integrated load balancing mode, GKE On-Prem needs to know the IP address or hostname, username, and password of the F5 BIG-IP load balancer that you intend to use for the user cluster. Set the values under usercluster.bigip to provide this information. For example:

usercluster:
  ...
  bigip:
    credentials:
      address: "203.0.113.5"
      username: "my-user-f5-name"
      password: "8%jfQATKO$#z"
  ...

usercluster.bigip.partition (integrated load balancing mode)

If you are using integrated load balancing mode, you must create a BIG-IP partition for your user cluster. Set usercluster.bigip.partition to the name of your partition. For example:

usercluster:
  ...
  bigip:
    partition: "my-user-f5-partition"
  ...

usercluster.vips

Regardless of whether you are using integrated or manual load balancing for the user cluster, you need to fill in the usercluster.vips field.

Set the value of usercluster.vips.controlplanevip to the IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the user cluster. Set the value of ingressvip to the IP address you have chosen to configure on the load balancer for the user cluster's ingress controller. For example:

usercluster:
  ...
  vips:
    controlplanevip: 203.0.113.6
    ingressvip: 203.0.113.7

usercluster.serviceiprange and usercluster.podiprange

The user cluster must have a range of IP addresses to use for Services and a range of IP addresses to use for Pods. These ranges are specified by the usercluster.serviceiprange and usercluster.podiprange fields. These fields are populated when you run gkectl create-config. If you like, you can change the populated values to values of your choice.

The Service and Pod ranges must not overlap. Also, the Service and Pod ranges must not overlap with IP addresses that are used for nodes in any cluster.

Example:

usercluster:
  ...
  serviceiprange: 10.96.233.0/24
  podiprange: 172.16.0.0/12

usercluster.clustername

Set the value of usercluster.clustername to a name of your choice. Choose a name that is no longer than 40 characters. For example:

usercluster:
  ...
  clustername: "my-user-cluster-1"

usercluster.masternode.replicas

The usercluster.masternode.replicas field specifies how many control plane nodes you want the user cluster to have. A user cluster's control plane node runs the user control plane, the Kubernetes control plane components. This value must be 1 or 3:

  • Set this field to 1 to run one user control plane.
  • Set this field to 3 if you want to have a high availability (HA) user control plane composed of three control plane nodes that each run a user control plane.

usercluster.masternode.cpus and usercluster.masternode.memorymb

The usercluster.masternode.cpus and usercluster.masternode.memorymb fields specify how many CPUs and how much memory, in megabytes, is allocated to each control plane node of the user cluster. For example:

usercluster:
  ...
  masternode:
    cpus: 4
    memorymb: 8192

usercluster.workernode.replicas

The usercluster.workernode.replicas field specifies how many worker nodes you want the user cluster to have. The worker nodes run the cluster workloads.

usercluster.workernode.cpus and usercluster.workernode.memorymb

The usercluster.masternode.cpus and usercluster.masternode.memorymb fields specify how many CPUs and how much memory, in megabytes, is allocated to each worker node of the user cluster. For example:

usercluster:
  ...
  workernode:
    cpus: 4
    memorymb: 8192
    replicas: 3

usercluster.oidc

If you intend for clients of the user cluster to use OIDC authentication, set values for the fields under usercluster.oidc. Configuring OIDC is optional.

To learn how to configure OIDC, see Authenticating with OIDC.

About installing version 1.0.2-gke.3

Version 1.0.2-gke.3 introduces the following OIDC fields (usercluster.oidc). These fields enable logging in to a cluster from Google Cloud console:

  • usercluster.oidc.kubectlredirecturl
  • usercluster.oidc.clientsecret
  • usercluster.oidc.usehttpproxy

In version 1.0.2-gke.3, if you want to use OIDC, the clientsecret field is required even if you don't want to log in to a cluster from Google Cloud console. In that case, you can provide a placeholder value for clientsecret:

oidc:
clientsecret: "secret"

usercluster.sni

Server Name Indication (SNI), an extension to Transport Layer Security (TLS), allows servers to present multiple certificates on a single IP address and TCP port, depending on the client-indicated hostname.

If your CA is already distributed as a trusted CA to clients outside your user cluster and you want to rely on this chain to identify trusted clusters, you can configure the Kubernetes API server with an additional certificate that is presented to external clients of the load balancer IP address.

To use SNI with your user clusters, you need to have your own CA and Public Key Infrastructure (PKI). You provision a separate serving certificate for each user cluster, and GKE On-Prem adds each additional serving certificate to its respective user cluster.

To configure SNI for the Kubernetes API server of the user cluster, provide values for usercluster.sni.certpath (path to the external certificate) and usercluster.sni.keypath (path to the external certificate's private key file). For example:

usercluster:
  ...
  sni:
    certpath: "/my-cert-folder/example.com.crt"
    keypath: "/my-cert-folder/example.com.key"

lbmode

You can use integrated load balancing or manual load balancing. Your choice of load balancing mode applies to your admin cluster and your initial user cluster. It also applies to any additional user clusters that you create in the future.

Specify your load balancing choice by setting the value of lbmode to Integrated or Manual. For example:

lbmode: Integrated

gkeconnect

The gkeconnect specification holds information that GKE On-Prem needs to set up management of your on-prem clusters from Google Cloud console.

Set gkeconnect.projectid to the project ID of the Google Cloud project where you want to manage your on-prem clusters.

Set the value of gkeconnect.registerserviceaccountkeypath to the path of the JSON key file for your register service account. Set the value of gkeconnect.agentserviceaccountkeypath to the path of the JSON key file for your connect service account.

If you want the Connect agent to use a proxy to communicate with Google Cloud, set the value of gkeconnect.proxy to the URL of the proxy. Use the format http(s)://[PROXY_ADDRESS].

Example:

gkeconnect:
  projectid: "my-project"
  registerserviceaccountkeypath: "/my-key-folder/register-key.json"
  agentserviceaccountkeypath: "/my-key-folder/connect-key.json"
  proxy: https://203.0.113.20

stackdriver

The stackdriver specification holds information that GKE On-Prem needs to store log entries generated by your on-prem clusters.

Set stackdriver.projectid to the project ID of the Google Cloud project where you want to view Stackdriver logs that pertain to your on-prem clusters.

Set stackdriver.clusterlocation to a Google Cloud region where you want to store Stackdriver logs. It is a good idea to choose a region that is near your on-prem data center.

Set stackdriver.enablevpc to true if you have your cluster's network controlled by a VPC. This ensures that all telemetry flows through Google's restricted IP addresses.

Set stackdriver.serviceaccountkeypath to the path of the JSON key file for your Stackdriver Logging service account.

Example:

stackdriver:
  projectid: "my-project"
  clusterlocation: "us-west1"
  enablevpc: false
  serviceaccountkeypath: "/my-key-folder/stackdriver-key.json"

privateregistryconfig

If you have a private Docker registry, the privateregistryconfig field holds information that GKE On-Prem uses to push images to your private registry. If you don't specify a private registry, gkectl pulls GKE On-Prem's container images from its Container Registry repository, gcr.io/gke-on-prem-release, during installation.

Under privatedockerregistry.credentials, set address to the IP address of the machine that runs your private Docker registry. Set username and password to the username and password of your private Docker registry.

When Docker pulls an image from your private registry, the registry must prove its identity by presenting a certificate. The registry's certificate is signed by a certificate authority (CA). Docker uses the CA's certificate to validate the registry's certificate.

Set privateregistryconfig.cacertpath to the path of the CA's certificate. For example:

privateregistryconfig
  ...
  cacertpath: /my-cert-folder/registry-ca.crt

gcrkeypath

Set the value of gcrkeypath to the path of the JSON key file for your access service account. For example:

gcrkeypath: "/my-key-folder/access-key.json"

cloudauditlogging

If you want to send your Kubernetes audit logs to your Google Cloud project, populate the cloudauditlogging specification. For example:

cloudauditlogging:
  projectid: "my-project"
  # A GCP region where you would like to store audit logs for this cluster.
  clusterlocation: "us-west1"
  # The absolute or relative path to the key file for a GCP service account used to
  # send audit logs from the cluster
  serviceaccountkeypath: "/my-key-folder/audit-logging-key.json"

Learn more about using audit logging.

Validating the configuration file

Complete this step from your admin workstation.

After you've modified the configuration file, run gkectl check-config to verify that the file is valid and can be used for installation:

gkectl check-config --config [PATH_TO_CONFIG]

If the command returns any FAILURE messages, fix the issues and validate the file again.

Skipping validations

The following gkectl commands automatically run validations against your config file:

  • gkectl prepare
  • gkectl create cluster
  • gkectl upgrade

To skip a command's validations, pass in --skip-validation-all. For example, to skip all validations for gkectl prepare:

gkectl prepare --config [PATH_TO_CONFIG] --skip-validation-all

To see all available flags for skipping specific validations:

gkectl check-config --help

Running gkectl prepare

Before you install, you need to run gkectl prepare on your admin workstation to initialize your vSphere environment. The gkectl prepare performs the following tasks:

  • Import the node OS image to vSphere and mark it as a template.

  • If you are using a private Docker registry, push GKE On-Prem images to your registry.

  • Optionally, validate the container images' build attestations, thereby verifying the images were built and signed by Google and are ready for deployment.

Run gkectl prepare with the GKE On-Prem configuration file, where --validate-attestations is optional:

gkectl prepare --config [CONFIG_FILE] --validate-attestations

Positive output from --validate-attestations is Image [IMAGE_NAME] validated.

Installing GKE On-Prem

You've created a configuration file that specifies how your environment looks and how you'd like your clusters to look, and you've validated the file. You ran gkectl prepare to initialize your environment with the GKE On-Prem software. Now you're ready to initiate a fresh installation of GKE On-Prem.

To install GKE On-Prem, run the following command:

gkectl create cluster --config [CONFIG_FILE]

where [CONFIG_FILE] is the configuration file you generated and modified.

You can reuse the configuration file to create additional user clusters.

Resuming an installation

In the event that your installation is interrupted after the admin cluster was created, you can resume installation by:

  • Removing the admincluster specification from the config file.
  • Running gkectl create cluster again, passing in the admin cluster's kubeconfig file.
gkectl create cluster --config [CONFIG_FILE] \
--kubeconfig [ADMIN_CLUSTER_KUBECONFIG]

where [ADMIN_CLUSTER_NAME] is the admin cluster's kubeconfig, which was created in the working directory when you started the installation.

Known issues

Currently, you are not able to resume an installation if you are creating a HA user cluster. This issue will be resolved in a future release.

Connecting clusters to Google

Enabling ingress

After your user cluster is running, you must enable ingress by creating a Gateway object. The first part of the Gateway manifest is always this:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-autogenerated-k8s-ingress
  namespace: gke-system
spec:
  selector:
    istio: ingress-gke-system

You can tailor the rest of the manifest according to your needs. For example, this manifest says that clients can send requests on port 80 using the HTTP/2 protocol and any hostname:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-autogenerated-k8s-ingress
  namespace: gke-system
spec:
  selector:
    istio: ingress-gke-system
  servers:
  - port:
      number: 80
      protocol: HTTP2
      name: http
    hosts:
    - "*"

If you want to accept HTTPS requests, then you must provide one or more certificates that your ingress controller can present to clients.

To provide a certificate:

  1. Create a Secret that holds your certificate and key.
  2. Create a Gateway object, or modify an existing Gateway object, that refers to your Secret. The name of the Gateway object must be istio-autogenerated-k8s-ingress.

For example, suppose you have already created a certificate file, ingress-wildcard.crt, and a key file ingress-wildcard.key.

Create a Secret named ingressgateway-wildcard-certs:

kubectl create secret tls \
    --namespace gke-system \
    ingressgateway-wildcard-certs \
    --cert ./ingress-wildcard.crt \
    --key ./ingress-wildcard.key

Here's a manifest for a Gateway that refers to your Secret. Clients can call on port 443 using the HTTPS protocol and any hostname that matches *.example.com. Note that the hostname in the certificate must match the hostname in the manifest, *.example.com in this example:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-autogenerated-k8s-ingress
  namespace: gke-system
spec:
  selector:
    istio: ingress-gke-system
  servers:
  - port:
      number: 80
      protocol: HTTP2
      name: http
    hosts:
    - "*"
  - hosts:
    - "*.example.com"
    port:
      name: https-demo-wildcard
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: ingressgateway-wildcard-certs

You can create multiple TLS certs for different hosts by modifying your Gateway manifest.

Save your manifest to a file named my-gateway.yaml, and create the Gateway:

kubectl apply -f my-gateway.yaml

Now you can use Kubernetes Ingress objects in the standard way.

GKE On-Prem limitations

Limitation Description
Maximum and minimum limits for clusters and nodes

See Quotas and limits. Your environment's performance might impact these limits.
Uniqueness for user cluster names

All user clusters registered to the same Google Cloud project must have unique names.
Cannot deploy to more than one vCenter and/or vSphere datacenter

Currently, you can only deploy an admin cluster and a set of associated user clusters to a single vCenter and/or vSphere datacenter. You cannot deploy the same admin and user clusters to more than one vCenter and/or vSphere datacenter.
Cannot declaratively change cluster configurations after creation While you can create additional clusters and resize existing clusters, you cannot change an existing cluster through its configuration file.

Troubleshooting

For more information, refer to Troubleshooting.

Diagnosing cluster issues using gkectl

Use gkectl diagnosecommands to identify cluster issues and share cluster information with Google. See Diagnosing cluster issues.

Default logging behavior

For gkectl and gkeadm it is sufficient to use the default logging settings:

  • By default, log entries are saved as follows:

    • For gkectl, the default log file is /home/ubuntu/.config/gke-on-prem/logs/gkectl-$(date).log, and the file is symlinked with the logs/gkectl-$(date).log file in the local directory where you run gkectl.
    • For gkeadm, the default log file is logs/gkeadm-$(date).log in the local directory where you run gkeadm.
  • All log entries are saved in the log file, even if they are not printed in the terminal (when --alsologtostderr is false).
  • The -v5 verbosity level (default) covers all the log entries needed by the support team.
  • The log file also contains the command executed and the failure message.

We recommend that you send the log file to the support team when you need help.

Specifying a non-default location for the log file

To specify a non-default location for the gkectl log file, use the --log_file flag. The log file that you specify will not be symlinked with the local directory.

To specify a non-default location for the gkeadm log file, use the --log_file flag.

Locating Cluster API logs in the admin cluster

If a VM fails to start after the admin control plane has started, you can try debugging this by inspecting the Cluster API controllers' logs in the admin cluster:

  1. Find the name of the Cluster API controllers Pod in the kube-system namespace, where [ADMIN_CLUSTER_KUBECONFIG] is the path to the admin cluster's kubeconfig file:

    kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] -n kube-system get pods | grep clusterapi-controllers

  2. Open the Pod's logs, where [POD_NAME] is the name of the Pod. Optionally, use grep or a similar tool to search for errors:

    kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] -n kube-system logs [POD_NAME] vsphere-controller-manager

Debugging F5 BIG-IP issues using the admin cluster control plane node's kubeconfig

After an installation, GKE On-Prem generates a kubeconfig file in the home directory of your admin workstation named internal-cluster-kubeconfig-debug. This kubeconfig file is identical to your admin cluster's kubeconfig, except that it points directly at the admin cluster's control plane node, where the admin control plane runs. You can use the internal-cluster-kubeconfig-debug file to debug F5 BIG-IP issues.

gkectl check-config validation fails: can't find F5 BIG-IP partitions

Symptoms

Validation fails because F5 BIG-IP partitions can't be found, even though they exist.

Potential causes

An issue with the F5 BIG-IP API can cause validation to fail.

Resolution

Try running gkectl check-config again.

gkectl prepare --validate-attestations fails: could not validate build attestation

Symptoms

Running gkectl prepare with the optional --validate-attestations flag returns the following error:

could not validate build attestation for gcr.io/gke-on-prem-release/.../...: VIOLATES_POLICY
Potential causes

An attestation might not exist for the affected image(s).

Resolution

Try downloading and deploying the admin workstation OVA again, as instructed in Creating an admin workstation. If the issue persists, reach out to Google for assistance.

Debugging using the bootstrap cluster's logs

During installation, GKE On-Prem creates a temporary bootstrap cluster. After a successful installation, GKE On-Prem deletes the bootstrap cluster, leaving you with your admin cluster and user cluster. Generally, you should have no reason to interact with this cluster.

If something goes wrong during an installation, and you did pass --cleanup-external-cluster=false to gkectl create cluster, you might find it useful to debug using the bootstrap cluster's logs. You can find the Pod, and then get its logs:

kubectl --kubeconfig /home/ubuntu/.kube/kind-config-gkectl get pods -n kube-system
kubectl --kubeconfig /home/ubuntu/.kube/kind-config-gkectl -n kube-system get logs [POD_NAME]

What's next