Preparing to install

This page takes you through some steps you should take before you install GKE On-Prem in your environment.

Before you begin

Review the following topics:

Limitations

Limitation Description
Maximum and minimum limits for clusters and nodes

See Quotas and limits. Your environment's performance might impact these limits.
Uniqueness for user cluster names

All user clusters registered to the same Google Cloud project must have unique names.
Cannot deploy to more than one vCenter and/or vSphere datacenter

Currently, you can only deploy an admin cluster and a set of associated user clusters to a single vCenter and/or vSphere datacenter. You cannot deploy the same admin and user clusters to more than one vCenter and/or vSphere datacenter.
Cannot declaratively change cluster configurations after creation While you can create additional clusters and resize existing clusters, you cannot change an existing cluster through its configuration file.

Creating a Google Cloud project

Create a Google Cloud project, if you don't already have one. You need a project to run GKE On-Prem.

Installing the required command-line interface tools

  • Install the Google Cloud CLI, which includes gcloud, the command-line interface (CLI) to Google Cloud.

  • Install govc, the CLI to VMware vSphere.

  • Install Terraform 0.11, which includes the terraform CLI. Follow Terraform's installation instructions to verify the installation and set up your PATH variable.

Configuring Google Cloud CLI to use your proxy/firewall from your laptop/workstation

If you are using a proxy to connect to the internet from your laptop or workstation, you might need to configure Google Cloud CLI for the proxy, so that you can run gcloud and gsutil commands. For instructions, see Configuring gcloud CLI for use behind a proxy/firewall.

Authorizing gcloud to access Google Cloud

After you install gcloud CLI, log in to Google Cloud using your account credentials:

gcloud auth login

Setting a default Google Cloud project

Setting a default Google Cloud causes all gcloud CLI commands to run against the project, so that you don't need to specify your project for each command. To set a default project, run the following command:

gcloud config set project [PROJECT_ID]

Replace [PROJECT_ID] with your project ID. (You can find your project ID in Google Cloud console, or by running gcloud config get-value project.)

Creating Google Cloud service accounts

Before you install GKE On-Prem for the first time, you use gcloud to create four Google Cloud service accounts. GKE On-Prem uses these service accounts to complete tasks on your behalf; the following sections describe each account's purpose.

Access service account

You use this service account to download GKE On-Prem's binaries from Cloud Storage. It is the only service account that Google allowlists.

Run the following command to create access-service-account:

gcloud iam service-accounts create access-service-account

Register service account

Connect uses this service account to register your GKE On-Prem clusters with Google Cloud console.

Run the following command to create register-service-account:

gcloud iam service-accounts create register-service-account

Connect service account

Connect uses this service account to maintain a connection between GKE On-Prem clusters and Google Cloud.

Run the following command to create connect-service-account:

gcloud iam service-accounts create connect-service-account

Google Cloud Observability service account

This service account allows GKE On-Prem to write logging and monitoring data to Google Cloud Observability:

Run the following command to create stackdriver-service-account:

gcloud iam service-accounts create stackdriver-service-account

Allowlisting your project and accounts

After you purchase GKE Enterprise, Google allowlists the following to grant you access to GKE On-Prem and Connect:

  • Your Google Cloud project.
  • Your Google account, and individual Google accounts of team members.
  • Your access service account.

If you want to use a different project or service account, or if you'd like to enable additional users, Google Cloud Support or your Technical Account Manager can help. Open a support case via Google Cloud console or the Google Cloud Support Center.

Enabling the required APIs in your project

You need to enable the following APIs in your Google Cloud project:

  • cloudresourcemanager.googleapis.com
  • container.googleapis.com
  • gkeconnect.googleapis.com
  • gkehub.googleapis.com
  • serviceusage.googleapis.com
  • stackdriver.googleapis.com
  • monitoring.googleapis.com
  • logging.googleapis.com

To enable these APIs, run the following command:

gcloud services enable \
cloudresourcemanager.googleapis.com \
container.googleapis.com \
gkeconnect.googleapis.com \
gkehub.googleapis.com \
serviceusage.googleapis.com \
stackdriver.googleapis.com \
monitoring.googleapis.com \
logging.googleapis.com

Assigning Identity and Access Management roles to your service accounts

IAM grants accounts permissions to call Google Cloud APIs. Assign dedicated IAM roles to these service accounts for privilege isolation.

List service accounts' email addresses

First, list the service accounts in your Google Cloud project:

gcloud iam service-accounts list

For a Google Cloud project named my-gcp-project, this command's output looks like this:

gcloud iam service-accounts list
NAME                                    EMAIL
                                        access-service-account@my-gcp-project.iam.gserviceaccount.com
                                        register-service-account@my-gcp-project.iam.gserviceaccount.com
                                        connect-service-account@my-gcp-project.iam.gserviceaccount.com
                                        stackdriver-service-account@my-gcp-project.iam.gserviceaccount.com

Take note of each accounts' email address. For each of the following sections, you provide the relevant account's email account.

Register service account

Grant the gkehub.admin and serviceuseage.serviceUsageViewer roles to your register service account:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[REGISTER_SERVICE_ACCOUNT_EMAIL]" \
--role "roles/gkehub.admin"
gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[REGISTER_SERVICE_ACCOUNT_EMAIL]" \
--role "roles/serviceusage.serviceUsageViewer"

Connect service account

Grant the gkehub.connect role to your connect service account:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[CONNECT_SERVICE_ACCOUNT_EMAIL]" \
--role "roles/gkehub.connect"

Google Cloud Observability service account

Grant the stackdriver.resourceMetadata.writer, logging.logWriter, and monitoring.metricWriter roles to your Google Cloud Observability service account:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[STACKDRIVER_SERVICE_ACCOUNT_EMAIL]" \
--role "roles/stackdriver.resourceMetadata.writer"
gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[STACKDRIVER_SERVICE_ACCOUNT_EMAIL]" \
--role "roles/logging.logWriter"
gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[STACKDRIVER_SERVICE_ACCOUNT_EMAIL]" \
--role "roles/monitoring.metricWriter"

Configuring Logging and Monitoring

Stackdriver Logging and Stackdriver Monitoring is enabled by default for GKE On-Prem.

Allowlisting addresses for your proxy

If your organization requires Internet access to pass through an HTTP proxy, you need to allowlist the following addresses for your proxy.

The following sections explain the addresses in detail.

Google addresses

GKE On-Prem uses several Google APIs to create and manage clusters. Allowlist the following Google addresses in the proxy:

Address Purpose
accounts.google.com Allows access to Google accounts for the purpose of accessing your Google Cloud project.
cloudresourcemanager.googleapis.com Creates, reads, and updates metadata for Google Cloud resource containers.
console.cloud.google.com Allows access to Google Cloud console.
container.googleapis.com Allows access to the Google Kubernetes Engine API.
gcr.io Allows access to Container Registry repositories, including the GKE On-Prem repository.
gkeconnect.googleapis.com Allows access to Connect for establishing a long-lived, encrypted connection with Google Cloud.
gkehub.googleapis.com Allows access to Google Cloud console for cluster registration with your Google Cloud project.
logging.googleapis.com Allows access to Cloud Logging's API for cluster metrics logging features.
monitoring.googleapis.com Allows access to Cloud Monitoring's API for cluster monitoring features.
oauth2.googleapis.com Allow access to Google's OAuth2 API for authentication.
serviceusage.googleapis.com
storage.googleapis.com Allows access to Cloud Storage buckets.
googleapis.com Allows access to Google Cloud product-specific endpoints.

HashiCorp addresses

You use HashiCorp Terraform version 0.11 to create an admin workstation VM in vSphere. To run Terraform in an environment with a proxy or firewall, you need to allowlist the following HashiCorp addresses:

Address Purpose
checkpoint-api.hashicorp.com Allows access to HashiCorp's version and alert information for various open source and proprietary products.
releases.hashicorp.com Allows access to HashiCorp's binaries.

VMware, load balancer, and other addresses

Lastly, be sure to allowlist the following addresses for your proxy. These addresses can vary:

Address Purpose
vCenter Server's IP address Allow internet traffic for the vCenter Server.
All ESXi hosts' IP addresses Allow internet traffic for your ESXi hosts running GKE On-Prem clusters.
Other IP addresses that you intend to configure on your load balancer Allow internet traffic for other IP addresses, like clients and workloads.

Setting aside Pod and Service ranges

For the admin cluster, and for each user cluster you intend to create, you need to set aside two distinct CIDR IPv4 blocks: one for Pod IPs, and one for Service IPs.

The sizes of these ranges depend on how many Pods and Services you intend to create. For example, if you intend to create fewer than 256 Services in a cluster, you could set aside a /24 Service range, like 10.96.233.0/24. If you intend to create fewer than 4096 Pods in your cluster, you could set aside a /20 Pod range, like 172.16.0.0/20.

For a given cluster, the Service and Pod ranges must not overlap. Also, the Service and Pod ranges must not overlap with IP addresses that are used for nodes in any cluster.

Preparing your load balancer

GKE On-Prem clusters can run with one of two load balancing modes, "Integrated" and "Manual." With Integrated mode, GKE On-Prem clusters run with the F5 BIG-IP load balancer. With Manual mode, you manually configure a different load balancer.

Preparing F5 BIG-IP partitions

If you choose to use the Integrated mode, you need to create an F5 BIG-IP partition to handle load balancing for each GKE On-Prem cluster you intend to create.

Initially, you need to create at least two partitions: one for the admin cluster, and one for a user cluster. You must create a partition before you create the corresponding cluster.

Do not use your cluster partitions for anything else. Each of your clusters must have a partition that is for the sole use of that cluster.

To learn how to create partitions, read Creating an administrative partition in the F5 BIG-IP documentation.

Using Manual load balancing mode

The Manual load balancing mode requires more configuration than the Integrated mode. For details, see Enabling manual load balancing.