Version 1.0. This version is no longer supported as outlined in the Anthos version support policy. For the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware (GKE on-prem), upgrade to a supported version. You can find the most recent version here.

Getting started

This page takes you through some steps you should take before you install GKE On-Prem in your environment.

Before you begin

Review the following topics:

Placeholder values in GKE On-Prem documentation

As you use GKE On-Prem's documentation, you can change placeholder values in code blocks by clicking them:

Enter your name: [YOUR_NAME]

You might find this useful if you want to fill in placeholder values before copying and running commands.

Limitations

Limitation Description
Maximum and minimum limits for clusters and nodes

See Quotas and limits. Your environment's performance might impact these limits.

One admin cluster per project

You can only register one admin cluster per Google Cloud project. If you need to run multiple admin clusters, you need to use a separate Google Cloud project.

Creating a Google Cloud project

Create a Google Cloud project, if you don't already have one. You need a project to run GKE On-Prem.

Installing the required command-line interface tools

  • Install Cloud SDK, which includes gcloud, the command-line interface (CLI) to Google Cloud.
  • Install govc, the CLI to VMware vSphere.
  • Install Terraform 0.11, which includes the terraform CLI. Follow Terraform's installation instructions to verify the installation and set up your PATH variable.

Authorizing gcloud to access Google Cloud

After you install Cloud SDK, authorize gcloud to access Google Cloud:

gcloud auth login

Setting a default Google Cloud project

Setting a default Google Cloud causes all Cloud SDK commands to run against the project, so that you don't need to specify your project for each command. To set a default project, run the following command:

gcloud config set project [PROJECT_ID]

Replace [PROJECT_ID] with your project ID. (You can find your project ID in Cloud Console, or by running gcloud config get-value project.)

Creating Google Cloud service accounts

Before you install GKE On-Prem for the first time, you use gcloud to create four Google Cloud service accounts. GKE On-Prem uses these service accounts to complete tasks on your behalf; the following sections describe each account's purpose.

Access service account

You use this service account to download GKE On-Prem's binaries from Cloud Storage. It is the only service account that Google allowlists.

Run the following command to create access-service-account:

gcloud iam service-accounts create access-service-account

Register service account

Connect uses this service account to register your GKE On-Prem clusters with Google Cloud Console.

Run the following command to create register-service-account:

gcloud iam service-accounts create register-service-account

Connect service account

Connect uses this service account to maintain a connection between GKE On-Prem clusters and Google Cloud.

Run the following command to create connect-service-account:

gcloud iam service-accounts create connect-service-account

Google Cloud's operations suite service account

This service account allows GKE On-Prem to write logging and monitoring data to Google Cloud's operations suite:

Run the following command to create stackdriver-service-account:

gcloud iam service-accounts create stackdriver-service-account

Allowlisting your project and accounts

After you purchase Anthos, Google allowlists the following to grant you access to GKE On-Prem and Connect:

  • Your Google Cloud project.
  • Your Google account, and individual Google accounts of team members.
  • Your access service account.

If you want to use a different project or service account, or if you'd like to enable additional users, Google Cloud Support or your Technical Account Manager can help. Open a support case via Cloud Console or the Google Cloud Support Center.

Enabling the required APIs in your project

You need to enable the following APIs in your Google Cloud project:

  • cloudresourcemanager.googleapis.com
  • container.googleapis.com
  • gkeconnect.googleapis.com
  • gkehub.googleapis.com
  • serviceusage.googleapis.com
  • stackdriver.googleapis.com
  • monitoring.googleapis.com
  • logging.googleapis.com

To enable these APIs, run the following command:

gcloud services enable \
cloudresourcemanager.googleapis.com \
container.googleapis.com \
gkeconnect.googleapis.com \
gkehub.googleapis.com \
serviceusage.googleapis.com \
stackdriver.googleapis.com \
monitoring.googleapis.com \
logging.googleapis.com

Assigning Identity and Access Management roles to your service accounts

IAM grants accounts permissions to call Google Cloud APIs. Assign dedicated IAM roles to these service accounts for privilege isolation.

List service accounts' email addresses

First, list the service accounts in your Google Cloud project:

gcloud iam service-accounts list

For a Google Cloud project named my-gcp-project, this command's output looks like this:

gcloud iam service-accounts list
NAME                                    EMAIL
                                        access-service-account@my-gcp-project.iam.gserviceaccount.com
                                        register-service-account@my-gcp-project.iam.gserviceaccount.com
                                        connect-service-account@my-gcp-project.iam.gserviceaccount.com
                                        stackdriver-service-account@my-gcp-project.iam.gserviceaccount.com

Take note of each accounts' email address. For each of the following sections, you provide the relevant account's email account.

Register service account

Grant the gkehub.admin and serviceuserage.serviceUsageViewer roles to your register service account:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member="serviceAccount:[REGISTER_SERVICE_ACCOUNT_EMAIL] \
--role="roles/gkehub.admin"
gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member="serviceAccount:[REGISTER_SERVICE_ACCOUNT_EMAIL] \
--role="roles/serviceusage.serviceUsageViewer"

Connect service account

Grant the gkehub.connect role to your connect service account:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member="serviceAccount:[CONNECT_SERVICE_ACCOUNT_EMAIL] \
--role="roles/gkehub.connect"

Google Cloud's operations suite service account

Grant the stackdriver.resourceMetadata.writer, logging.logWriter, and monitoring.metricWriter roles to your Google Cloud's operations suite service account:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[STACKDRIVER_SERVICE_ACCOUNT_EMAIL] \
--role "roles/stackdriver.resourceMetadata.writer"
gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[STACKDRIVER_SERVICE_ACCOUNT_EMAIL] \
--role "roles/logging.logWriter"
gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[STACKDRIVER_SERVICE_ACCOUNT_EMAIL] \
--role "roles/monitoring.metricWriter"

Creating a Stackdriver Workspace

Stackdriver Logging and Stackdriver Monitoring is enabled by default for GKE On-Prem. For Stackdriver Monitoring to function properly, you need to create a Stackdriver Workspace.

To learn how to create a Stackdriver Workspace, refer to Get a Workspace quickly in the Stackdriver Monitoring documentation.

Allowlisting Google and HashiCorp addresses for your proxy

If your organization requires Internet access to pass through an HTTP proxy, allowlist the following Google addresses in the proxy:

  • binaryauthorization.googleapis.com
  • googlecode.l.googleusercontent.com
  • storage.l.googleusercontent.com
  • storage.googleapis.com

You use HashiCorp Terraform version 0.11 to create an admin workstation VM in vSphere. To run Terraform in an environment with a proxy or firewall, you need to allowlist the following HashiCorp addresses:

  • checkpoint-api.hashicorp.com
  • releases.hashicorp.com

Preparing your load balancer

GKE On-Prem clusters can run with one of two load balancing modes, "Integrated" and "Manual." With Integrated mode, GKE On-Prem clusters run with the F5 BIG-IP load balancer. With Manual mode, you manually configure a different load balancer.

Preparing F5 BIG-IP partitions

If you choose to use the Integrated mode, you need to create an F5 BIG-IP partition to handle load balancing for each GKE On-Prem cluster you intend to create.

Initially, you need to create at least two partitions: one for the admin cluster, and one for a user cluster. You must create a partition before you create the corresponding cluster.

Do not use your cluster partitions for anything else. Each of your clusters must have a partition that is for the sole use of that cluster.

To learn how to create partitions, read Creating an administrative partition in the F5 BIG-IP documentation.

Using Manual load balancing mode

The Manual load balancing mode requires more configuration than the Integrated mode. For details, see Enabling manual load balancing.