GKE On-Prem overview

GKE On-Prem is hybrid cloud software that brings Google Kubernetes Engine (GKE) to on-premises data centers. With GKE On-Prem, you can create, manage, and upgrade Kubernetes clusters in your on-prem environment and connect those clusters to Google Cloud console.

This topic is a technical overview of GKE On-Prem. After you've finished this topic, be sure to read System requirements.

Learning about Kubernetes

GKE On-Prem and GKE are built on top of Kubernetes, an open-source, extensible platform for managing containerized applications. Kubernetes orchestrates clusters, which are sets of nodes (also called "machines") that run containerized applications.

• What is a Container?
• What is Kubernetes

Getting GKE On-Prem

GKE On-Prem is a core component of GKE Enterprise. To use GKE On-Prem, you first reach out to Google's sales team and purchase an GKE Enterprise subscription.

• GKE Enterprise overview
• Contact sales

Preparing for GKE On-Prem

Installing GKE On-Prem entails completing some prerequisite setup tasks. The following sections outline these tasks.

VMware vSphere virtualization environment

You install GKE On-Prem to a VMware vSphere 6.5 cluster running in your on-prem environment. vSphere is VMware's server virtualization software.

You also procure VMware vCenter and VMware ESXi. ESXi is a type-1 hypervisor that you install to a set of the physical hosts/servers in your data center. vCenter is vSphere's web-based user interface for managing data center services; it provides a central view across ESXi hosts.

Specifically, you need the following licensed VMware products:

• VMware ESXI 6.5, with an Enterprise Plus license edition, installed on a set of the hosts in your data center.
• VMware vCenter 6.5, with a Standard license edition, installed on one host in your data center.

For more information, see the following VMware resources:

• vSphere requirements
• Compare VMware vSphere editions
• VMware vSphere feature comparison

Load balancing

For layer 4 load balancing, you can choose from two load balancing modes: integrated or manual. Integrated mode supports F5 BIG-IP (GKE On-Prem's integrated load balancer). With manual mode, you can choose any load balancer you want; GKE On-Prem currently offers documented instructions for the following load balancers:

• Citrix
• Seesaw

You configure a load balancer in your data center before you install GKE On-Prem. During installation, you choose whether to enable integrated (F5 BIG-IP) or manual load balancing mode (a load balancer of your choice). Manual load balancing mode requires that you allocate static IPs for your nodes; you can't use a Dynamic Host Configuration Protocol (DHCP) server with manual load balancing mode.

• Setting up your load balancer
• Load balancer requirements
• Enabling manual load balancing
• F5 requirements

Authentication

If you have clients that need to authenticate against your clusters and access your containerized workloads, you can use OpenID Connect (OIDC) with a provider of your choice or with Active Directory Federated Services (ADFS).

• Authenticating with OIDC
• Authenticating with OIDC and ADFS

Preparing your Google Cloud project

Preparing your Google Cloud project includes:

• Creating a Google Cloud project and enabling the necessary APIs
• Binding Identity and Access Management roles to the service accounts.
• Downloading the necessary command line interface (CLI) tools, including HashiCorp Terraform version 11

See the following topics for more information:

• GKE On-Prem system requirements
• Preparing to install

Creating the admin workstation

Creating the admin workstation includes:

• Downloading the admin workstation OVA file, which includes a GKE On-Prem bundle file.
• Copying a Terraform configuration (TF) and Terraform variables (TFVARS) file, and modifying them to reflect your vCenter and vSphere configuration.
• Using Terraform to create an admin workstation virtual machine (VM) in vSphere.

See the following topics for more information:

• Creating an admin workstation
• Downloads

How installing GKE On-Prem works

Here is a high-level summary of steps taken during an installation: (See also Overview of installation.)

1. You SSH into your admin workstation.
2. You run gkectl create-config to generate a GKE On-Prem configuration file. The configuration file declares a specification for installing GKE On-Prem.

3. You modify the configuration file with values appropriate for your needs and environment, such as your clusters' specifications, OIDC authentication, load balancing, Stackdriver logging, the Google Cloud project with which to register, and more.

   Important: After you have successfully deployed clusters, you are not able to change this configuration and use it to update your clusters' configurations. However, you are able to add more nodes to an existing cluster, and you are able to create additional user clusters.

4. You run gkectl check-config to validate that the modified configuration file can be used for an installation.

5. You run gkectl prepare to move GKE On-Prem's OS image to vSphere and mark it as a template for VMs. If you configure a private Docker registry, this command also pushes GKE On-Prem's container images to the registry.

6. You run gkectl create cluster --config with the configuration file to create the GKE On-Prem clusters in your data center.

At the end of a successful installation, you should have the following in vSphere:

• an admin workstation.
• an admin cluster.
• a user cluster.

Architecture

Note: See also the high availability architecture.

Figure: GKE On-Prem architecture with one user control plane. (Click to enlarge)

In GKE On-Prem, there is an admin cluster and, by default, three user clusters. There are also two add-ons VMs and one admin workstation VM. All of these VMs run in a single vSphere cluster.

Admin workstation

The admin workstation is the VM in vSphere from which cluster administrators install and interact with GKE On-Prem. It is the first VM that you create before you create anything else.

If you're a cluster admin, you use Terraform to create the admin workstation in vSphere. The admin workstation includes:

• A full bundle file.
• CLI tools for interacting with GKE On-Prem, vSphere, and Google Cloud.

To create the admin workstation, you download three files:

• The admin workstation Open Virtual Appliance (OVA) file. This is a versioned VM image of the admin workstation. You import this file to vSphere and mark it as a VM template. Then, Terraform uses the template to deploy the admin workstation to your vSphere cluster.
• Terraform configuration file (TF). These are instructions that Terraform uses to create the admin workstation in your cluster. You copy this file to your local workstation or laptop.
• Terraform configuration variables file (TFVARS). You populate the variables in this file with values from your environment. The TF configuration file references the TFVARS file's variables. You copy this file to your local workstation or laptop.

Admin cluster

The admin cluster is the base layer of GKE On-Prem. It runs the following GKE On-Prem components:

• Admin control plane: The admin control plane handles all gkectl and Kubernetes API calls to and from GKE On-Prem. The admin control plane makes some calls to and from vCenter APIs.
• User control planes: A user cluster's control plane. Routes API requests to the cluster's nodes. Each cluster has its own control plane that runs in the admin cluster. User control planes also make some calls to and from vCenter APIs.
• Add-on VMs: VMs that run the admin cluster's add-ons, like Grafana, Prometheus, Istio components, and Stackdriver.

Note that user control planes are managed by the admin cluster. They run on nodes in admin clusters, not in the user clusters themselves. To manage user control planes, admin clusters need to:

• Manage the machines that run the user cluster control planes.
• Create, update, and delete the control plane components.
• Expose the Kubernetes API server to the user cluster.
• Manage cluster certificates.

User cluster

User clusters are where you deploy and run your containerized workloads and services.

Add-ons VMs

GKE On-Prem creates two VMs to run the admin cluster's add-ons, like Grafana, Prometheus, Istio components, and Stackdriver.

About the bundle

GKE On-Prem's bundle is a versioned TGZ archive that contains all of the components needed to create and upgrade GKE On-Prem clusters.

There are two types of bundles:

Full bundle

The full bundle, gke-onprem-vsphere-[VERSION]-full.tgz, is included with the admin workstation. You can find it at /var/lib/gke/bundles. The full bundle is used for installing GKE On-Prem for the first time. It's a large file that includes:

• a TAR file with container images of all cluster components.
• YAML files of those cluster components.
• GKE On-Prem's node image.

Upgrade bundle

The upgrade bundle, gke-onprem-vsphere-[VERSION].tgz, is provided for upgrading clusters. Each time you upgrade your clusters, you download the version's bundle from Downloads. The upgrade bundle only has YAML configuration files, which are used to upgrade your cluster's components.

CLI tools

When you install GKE On-Prem, you download the following CLI tools to your local workstation or laptop:

• govc
• terraform
• gkectl
• kubectl (included in the Google Cloud CLI)
• gcloud (included in the gcloud CLI)

govc

govc is the CLI to vSphere. You use govc when you create the admin workstation, and you can use it to administer your vSphere cluster.

terraform

terraform is the CLI to HashiCorp Terraform. You use terraform to create and upgrade the admin workstation.

gkectl

gkectl is the CLI to GKE On-Prem. See its reference guide.

You use gkectl for many cluster administration tasks, including:

• Cluster creation and management.
• Diagnosing and troubleshooting issues.
• Capturing and exporting cluster logs.

kubectl

kubectl is the CLI to Kubernetes. You use kubectl to interact with Kubernetes and Kubernetes clusters, and for tasks including:

• Deploying, managing, and deleting containerized workloads running in clusters.
• Managing, editing, and deleting Kubernetes resources.

gcloud

Google Cloud CLI is the CLI to Google Cloud. You use the gcloud CLI for several purposes, including:

• Authenticating against your Google Cloud project.
• Creating service accounts and their private keys.
• Binding Identity and Access Management roles to accounts.

Registering clusters with Google Cloud console

When you create GKE On-Prem user clusters, you can choose enable Connect to automatically register them with Google Cloud console. Connect enables you to view and sign in to your on-premises and on-cloud Kubernetes clusters from the same Google Cloud user interface.

Enabling Connect creates a Connect Agent in each user cluster. The Connect Agent is a Deployment that establishes a long-lived, encrypted connection to Google Cloud from the user cluster on which it runs.

The Connect Agent's container image is pulled from a Container Registry repository that lives at gcr.io. If your user cluster doesn't or can't have a connection to gcr.io, you need use a private Docker registry to connect it to Google Cloud console.

Versioning

To learn all about how versioning works, see Versions.

Troubleshooting and diagnosing issues

See the following topics for troubleshooting:

• To get troubleshooting help for specific topics, see Troubleshooting.
• To diagnose cluster issues using gkectl, see Diagnosing cluster issues.
• To diagnose cluster node issues, see Debugging node issues using debug-toolbox.
• To get help from google, see Getting support.

Placeholder values in GKE On-Prem documentation

As you use GKE On-Prem's documentation, you can change placeholder values in code blocks by clicking them. In the following block, click [YOUR_NAME] and observe that it is editable:

Enter your name: [YOUR_NAME]

You might find this useful if you want to fill in placeholder values before copying commands and running them in your environment.

What's next

Next, read System requirements to learn more about how to prepare your on-prem environment.