GKE On-Prem is hybrid cloud software that brings Google Kubernetes Engine (GKE) to on-premises data centers. With GKE On-Prem, you can create, manage, and upgrade Kubernetes clusters in your on-prem environment and connect those clusters to Google Cloud Console.
This topic is a technical overview of GKE On-Prem. After you've finished this topic, be sure to read System requirements.
Learning about Kubernetes
GKE On-Prem and GKE are built on top of Kubernetes, an open-source, extensible platform for managing containerized applications. Kubernetes orchestrates clusters, which are sets of nodes (also called "machines") that run containerized applications.
Getting GKE On-Prem
GKE On-Prem is a core component of Anthos. To use GKE On-Prem, you first reach out to Google's sales team and purchase an Anthos subscription.
Preparing for GKE On-Prem
Installing GKE On-Prem entails completing some prerequisite setup tasks. The following sections outline these tasks.
VMware vSphere virtualization environment
You install GKE On-Prem to a VMware vSphere 6.5 cluster running in your on-prem environment. vSphere is VMware's server virtualization software.
You also procure VMware vCenter and VMware ESXi. ESXi is a type-1 hypervisor that you install to a set of the physical hosts/servers in your data center. vCenter is vSphere's web-based user interface for managing data center services; it provides a central view across ESXi hosts.
Specifically, you need the following licensed VMware products:
- VMware ESXI 6.5, with an Enterprise Plus license edition, installed on a set of the hosts in your data center.
- VMware vCenter 6.5, with a Standard license edition, installed on one host in your data center.
For more information, see the following VMware resources:
For layer 4 load balancing, you can choose from two load balancing modes: integrated or manual. Integrated mode supports F5 BIG-IP (GKE On-Prem's integrated load balancer). With manual mode, you can choose any load balancer you want; GKE On-Prem currently offers documented instructions for the following load balancers:
You configure a load balancer in your data center before you install GKE On-Prem. During installation, you choose whether to enable integrated (F5 BIG-IP) or manual load balancing mode (a load balancer of your choice). Manual load balancing mode requires that you allocate static IPs for your nodes; you can't use a Dynamic Host Configuration Protocol (DHCP) server with manual load balancing mode.
- Setting up your load balancer
- Load balancer requirements
- Enabling manual load balancing
- F5 requirements
If you have clients that need to authenticate against your clusters and access your containerized workloads, you can use OpenID Connect (OIDC) with a provider of your choice or with Active Directory Federated Services (ADFS).
Preparing your Google Cloud project
Preparing your Google Cloud project includes:
- Creating a Google Cloud project and enabling the necessary APIs
- Binding Identity and Access Management roles to the service accounts.
- Downloading the necessary command line interface (CLI) tools, including HashiCorp Terraform version 11
See the following topics for more information:
Creating the admin workstation
Creating the admin workstation includes:
- Downloading the admin workstation OVA file, which includes a GKE On-Prem bundle file.
- Copying a Terraform configuration (TF) and Terraform variables (TFVARS) file, and modifying them to reflect your vCenter and vSphere configuration.
- Using Terraform to create an admin workstation virtual machine (VM) in vSphere.
See the following topics for more information:
How installing GKE On-Prem works
- You SSH into your admin workstation.
- You run
gkectl create-configto generate a GKE On-Prem configuration file. The configuration file declares a specification for installing GKE On-Prem.
You modify the configuration file with values appropriate for your needs and environment, such as your clusters' specifications, OIDC authentication, load balancing, Stackdriver logging, the Google Cloud project with which to register, and more.
gkectl check-configto validate that the modified configuration file can be used for an installation.
gkectl prepareto move GKE On-Prem's OS image to vSphere and mark it as a template for VMs. If you configure a private Docker registry, this command also pushes GKE On-Prem's container images to the registry.
gkectl create cluster --configwith the configuration file to create the GKE On-Prem clusters in your data center.
At the end of a successful installation, you should have the following in vSphere:
In GKE On-Prem, there is an admin cluster and, by default, three user clusters. There are also two add-ons VMs and one admin workstation VM. All of these VMs run in a single vSphere cluster.
The admin workstation is the VM in vSphere from which cluster administrators install and interact with GKE On-Prem. It is the first VM that you create before you create anything else.
If you're a cluster admin, you use Terraform to create the admin workstation in vSphere. The admin workstation includes:
To create the admin workstation, you download three files:
- The admin workstation Open Virtual Appliance (OVA) file. This is a versioned VM image of the admin workstation. You import this file to vSphere and mark it as a VM template. Then, Terraform uses the template to deploy the admin workstation to your vSphere cluster.
- Terraform configuration file (TF). These are instructions that Terraform uses to create the admin workstation in your cluster. You copy this file to your local workstation or laptop.
- Terraform configuration variables file (TFVARS). You populate the variables in this file with values from your environment. The TF configuration file references the TFVARS file's variables. You copy this file to your local workstation or laptop.
The admin cluster is the base layer of GKE On-Prem. It runs the following GKE On-Prem components:
- Admin control plane: The admin control plane handles all
gkectland Kubernetes API calls to and from GKE On-Prem. The admin control plane makes some calls to and from vCenter APIs.
- User control planes: A user cluster's control plane. Routes API requests to the cluster's nodes. Each cluster has its own control plane that runs in the admin cluster. User control planes also make some calls to and from vCenter APIs.
- Add-on VMs: VMs that run the admin cluster's add-ons, like Grafana, Prometheus, Istio components, and Stackdriver.
Note that user control planes are managed by the admin cluster. They run on nodes in admin clusters, not in the user clusters themselves. To manage user control planes, admin clusters need to:
- Manage the machines that run the user cluster control planes.
- Create, update, and delete the control plane components.
- Expose the Kubernetes API server to the user cluster.
- Manage cluster certificates.
User clusters are where you deploy and run your containerized workloads and services.
GKE On-Prem creates two VMs to run the admin cluster's add-ons, like Grafana, Prometheus, Istio components, and Stackdriver.
About the bundle
GKE On-Prem's bundle is a versioned TGZ archive that contains all of the components needed to create and upgrade GKE On-Prem clusters.
There are two types of bundles:
The full bundle,
gke-onprem-vsphere-[VERSION]-full.tgz, is included with
the admin workstation. You can find it at
/var/lib/gke/bundles. The full
bundle is used for installing GKE On-Prem for the first time.
It's a large file that includes:
- a TAR file with container images of all cluster components.
- YAML files of those cluster components.
- GKE On-Prem's node image.
The upgrade bundle,
gke-onprem-vsphere-[VERSION].tgz, is provided for
Each time you upgrade your clusters, you download the version's bundle from
Downloads. The upgrade bundle only has YAML configuration files, which
are used to upgrade your cluster's components.
When you install GKE On-Prem, you download the following CLI tools to your local workstation or laptop:
kubectl(included in the Cloud SDK)
gcloud(included in the Cloud SDK)
is the CLI to vSphere. You use
govc when you create the admin workstation,
and you can use it to administer your vSphere cluster.
terraform is the
CLI to HashiCorp Terraform. You use
terraform to create and upgrade the admin
gkectl is the CLI to GKE On-Prem. See its
gkectl for many cluster administration tasks, including:
- Cluster creation and management.
- Diagnosing and troubleshooting issues.
- Capturing and exporting cluster logs.
is the CLI to Kubernetes. You use
kubectl to interact with Kubernetes and
Kubernetes clusters, and for tasks including:
- Deploying, managing, and deleting containerized workloads running in clusters.
- Managing, editing, and deleting Kubernetes resources.
gcloud is the CLI to Google Cloud. You use
several purposes, including:
- Authenticating against your Google Cloud project.
- Creating service accounts and their private keys.
- Binding Identity and Access Management roles to accounts.
Registering clusters with Cloud Console
When you create GKE On-Prem user clusters, you can choose enable Connect to automatically register them with Cloud Console. Connect enables you to view and sign in to your on-premises and on-cloud Kubernetes clusters from the same Google Cloud user interface.
Enabling Connect creates a Connect Agent in each user cluster. The Connect Agent is a Deployment that establishes a long-lived, encrypted connection to Google Cloud from the user cluster on which it runs.
The Connect Agent's container image is pulled from a
Container Registry repository that lives at
If your user cluster doesn't or can't have a connection to gcr.io, you need
use a private Docker registry to connect it to Cloud Console.
To learn all about how versioning works, see Versioning and upgrades.
Troubleshooting and diagnosing issues
See the following topics for troubleshooting:
- To get troubleshooting help for specific topics, see Troubleshooting.
- To diagnose cluster issues using
gkectl, see Diagnosing cluster issues.
- To diagnose cluster node issues, see Debugging node issues using debug-toolbox.
- To get help from google, see Getting support.
Placeholder values in GKE On-Prem documentation
As you use GKE On-Prem's documentation, you can change placeholder values in code blocks by clicking them. In the following block, click [YOUR_NAME] and observe that it is editable:
Enter your name: [YOUR_NAME]
You might find this useful if you want to fill in placeholder values before copying commands and running them in your environment.
Next, read System requirements to learn more about how to prepare your on-prem environment.