This document shows how to create an admin workstation that you can use to create Anthos clusters on VMware (GKE on-prem) clusters.
In this document, you create an admin workstation that has a static IP address. This is consistent with the basic installation journey, which creates clusters that use static IP addresses.
If you want to create an admin workstation that uses Dynamic Host Configuration Protocol (DHCP) to get its IP address, see Creating an admin workstation using DHCP.
The steps in this document use the
gkeadm command-line tool, which is
available for 64-bit Linux, Windows 10, Windows Server 2019, and macOS 10.15
gkeadm command-line tool and make it executable:
gsutil cp gs://gke-on-prem-release-public/gkeadm/1.6.0-gke.7/linux/gkeadm ./ chmod +x gkeadm
gkeadm command-line tool:
gsutil cp gs://gke-on-prem-release-public/gkeadm/1.6.0-gke.7/windows/gkeadm.exe ./
gkeadm command-line tool and make it executable:
gsutil cp gs://gke-on-prem-release-public/gkeadm/1.6.0-gke.7/darwin/gkeadm ./ chmod +x gkeadm
Generating a template for your configuration file
To create an admin workstation, the
gkeadm tool requires a configuration file.
To generate a template for your configuration file:
./gkeadm create config
The output is a file named
The following fields in your configuration file are populated with default or generated values. For example:
adminWorkstation: name: gke-admin-ws-200617-113711 cpus: 4 memoryMB: 8192 diskGB: 50 dataDiskName: gke-on-prem-admin-workstation-data-disk/gke-admin-ws-200617-113711-data-disk.vmdk dataDiskMB: 512 ntpServer: ntp.ubuntu.com
Keep all of the default and generated values.
Notice the generated value of
dataDiskName. This virtual machine disk (VMDK)
is created for you. Later, if you manually delete your admin workstation, the
VMDK is automatically deleted.
Filling in the rest of your configuration file
In your configuration file, enter field values as described in this section.
String. The path of the JSON key file for your component access service account. This is the account that Anthos clusters on VMware uses to download components from Container Registry. For information about creating a JSON key file, see Service accounts and keys.
gkeadm creates your admin workstation, it activates your component access
service account on the admin workstation.
gcp: whitelistedServiceAccountKeyPath: "my-key-folder/component-access-key.json"
String. The IP address or the hostname of your vCenter server.
Before you fill in the
address field, download and inspect the serving
certificate of your vCenter server. Enter the following command to
download the certificate and save it to a file named
true | openssl s_client -connect VCENTER_IPVCENTER_IP:443 -showcerts 2>/dev/null | sed -ne '/-BEGIN/,/-END/p' > vcenter.pem
VCENTER_IP with the IP address of your
Open the certificate file to see the Subject Common Name and the Subject Alternative Name:
openssl x509 -in vcenter.pem -text -noout
The output shows the
Subject Common Name (CN). This might be an IP address, or
it might be a hostname. For example:
Subject: ... CN = 203.0.113.100
Subject: ... CN = my-host.my-domain.example
The output might also include one or more DNS names under
Subject Alternative Name:
X509v3 Subject Alternative Name: DNS:vcenter.my-domain.example
Subject Common Name or one of the DNS names under
Subject Alternative Name to use as the value of
in your configuration file. For example:
vCenter: credentials: address: "203.0.113.1"
vCenter: credentials: address: "my-host.my-domain.example"
String. The path of a credentials configuration file that holds the username and password of your vCenter user account. The user account should have the Administrator role or equivalent privileges. See vSphere requirements.
When you run
gkeadm create config, it creates a template for a credentials
configuration file, named
credential.yaml, in the same directory as
vCenter: credentials: fileRef: path: "credential.yaml"
String. The name of the credentials block, in your credentials configuration file, that holds the username and password of your vCenter user account. For example:
vCenter: credentials: fileRef: entry: "vCenter"
String. The name of your vSphere datacenter. For example:
vCenter: datacenter: "MY-DATACENTER"
String. The name of your vSphere datastore. For example:
vCenter: datastore: "MY-DATASTORE"
String. The name of your vSphere cluster. For example:
vCenter: cluster: "MY-CLUSTER"
String. The name of your vSphere VM folder where the admin workstation and admin workstation will be created.
Note that this should be a VM Folder, as opposed to other types of inventory folders in vCenter.
vCenter: folder: "MY-FOLDER"
String. The name of the vSphere network where you want to create your admin workstation.
If the name contains a special character, you must use an escape sequence for it.
|Special characters||Escape sequence|
|Percent sign (
If the network name is not unique, it is possible to specify a path to the
network, such as
vCenter: network: "MY-VM-NETWORK"
String. If you are using a non-default resource pool, provide the name of your vSphere resource pool. For example:
vCenter: resourcePool: "MY-POOL"
If you are using the default resource pool, provide the following value:
vCenter: resourcePool: "MY_CLUSTER/Resources"
MY_CLUSTER with the name of your vSphere cluster.
String. When a client, like Anthos clusters on VMware, sends a request to your vCenter server, the server must prove its identity to the client by presenting a certificate or a certificate bundle. To verify the certificate or bundle, Anthos clusters on VMware must have the root certificate in the chain of trust.
vCenter.caCertPath to the path of the root certificate. For example:
vCenter: caCertPath: "/usr/local/google/home/me/certs/vcenter-ca-cert.pem"
Your VMware installation has a certificate authority (CA) that issues a certificate to your vCenter server. The root certificate in the chain of trust is a self-signed certificate created by VMware.
If you do not want to use the VMWare CA, which is the default, you can configure VMware to use a different certificate authority.
If your vCenter server uses a certificate issued by the default VMware CA, download the certificate as follows:
curl "https://SERVER_ADDRESS/certs/download.zip" > download.zip
SERVER_ADDRESS with the address of your vCenter server.
unzip command and unzip the certificate file:
sudo apt-get install unzip unzip downloads.zip
If the unzip command doesn't work the first time, enter the command again.
Find the certificate file in
String: If the machine you are using to run
gkeadm uses a proxy server for access to
the internet, set this field to the URL of the proxy server. For example:
String. Set this to
"static". For example:
adminWorkstation: network: ipAllocationMode: "static"
String. An IP address of your choice for your admin workstation. For example:
adminWorkstation: network: hostconfig: ip: "172.16.5.1"
String. The IP address of the default gateway for the network that contains your admin workstation. For example:
adminWorkstation: network: hostconfig: gateway: "172.16.6.254"
String. The netmask for the network that contains your admin workstation. For example:
adminWorkstation: network: hostconfig: netmask: "255.255.248.0"
String array. An array of IP addresses for DNS servers that your admin workstation can use. For example:
adminWorkstation: network: hostconfig: dns: - "172.16.255.1" - "172.16.255.2"
String. If your network is behind a proxy server, and you want both your admin
workstation and your Anthos clusters on VMware clusters to use the same proxy
server, then set
adminworkstation.proxyURL to the URL of the proxy server.
adminworkstation: proxyUrl: "http://aw-proxy.example"
String. The hostname or IP address of the Network Time Protocol server that your admin workstation is to use. For example:
adminWorkstation: ntpServer: "126.96.36.199"
If you leave this empty, Anthos clusters on VMware uses
Here is an example of a completed
gcp: whitelistedServiceAccountKeyPath: "my-key-folder/component-access-key.json" vCenter: credentials: address: "203.0.113.1" username: "administrator.vsphere.local" password: "#STyZ2T#Ko2o" datacenter: "MY-DATACENTER" datastore: "MY-DATASTORE" cluster: "MY-CLUSTER" network: "MY-VM-NETWORK" resourcePool: "MY-POOL" caCertPath: "/usr/local/google/home/me/certs/the-root.cert" proxyUrl: "" adminWorkstation: name: "gke-admin-ws-200617-113711" cpus: 4 memoryMB: 8192 diskGB: 50 dataDiskName: "gke-on-prem-admin-workstation-data-disk/gke-admin-ws-200617-113711-data-disk.vmdk" dataDiskMB: 512 network: ipAllocationMode: "static" hostConfig: ip: "172.16.5.1" gateway: "172.16.6.254" netmask: "255.255.248.0" dns: - "172.16.255.1" - "172.16.255.2" proxyUrl: "" ntpServer: "ntp.ubuntu.com"
Creating your admin workstation
gkeadm to create an admin workstation and a set of
service accounts, you must grant the
serviceusage.serviceUsageAdmin IAM roles to the default Cloud Billing account.
To view the default account:
gcloud config get-value account
To grant the required roles:
gcloud projects add-iam-policy-binding PROJECT_ID \ --member="user:DEFAULT_ACCOUNT" \ --role="roles/resourcemanager.projectIamAdmin" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="user:DEFAULT_ACCOUNT" \ --role="roles/serviceusage.serviceUsageAdmin"
Replace the following:
PROJECT_ID: the ID of your connect project.
DEFAULT_ACCOUNT: the default Cloud Billing account.
Create your admin workstation and a set of service accounts. If you prefer to
create and configure your service accounts manually, you can omit the
./gkeadm create admin-workstation --auto-create-service-accounts
The output gives detailed information about the creation of your admin
workstation. For example, you can see that
gkeadm performed these tasks:
Ran preflight checks.
Downloaded an OVA image and uploaded it as a vSphere template.
Created an SSH key.
Created an admin workstation VM.
gkectland Docker versions installed on the admin workstation.
Printed information about the NTP server on the admin workstation.
Created service accounts and their JSON key files. These service accounts are members of the project that is the parent of your component access service account.
Assigned the appropriate IAM roles to the service accounts. These roles are granted on the project that is the parent of your component access service account.
Copied JSON key files for the service accounts to the admin workstation.
Copied the CA certificate file for your vCenter server to the admin workstation.
user-cluster.yamlon the admin workstation. Later, you use these configuration files to create clusters.
... - Validation Category: vCenter - [SUCCESS] Credentials - [SUCCESS] Version - [SUCCESS] Datacenter - [SUCCESS] Datastore - [SUCCESS] Resource Pool - [SUCCESS] Folder - [SUCCESS] Network - [SUCCESS] Datadisk All validation results were SUCCESS. ****************************************** Admin workstation VM successfully created: - Name: my-admin-workstation - IP: 172.16.5.1 - SSH Key: /usr/local/google/home/me/.ssh/gke-admin-workstation ****************************************** Printing gkectl and docker versions on admin workstation... gkectl version gkectl 1.4.0-gke.13 (git-b39331512) docker version Client: Version: 19.03.2 API version: 1.40 Go version: go1.12.9 Git commit: 6a30dfca03 Built: Mon Sep 30 22:47:02 2019 OS/Arch: linux/amd64 Experimental: false Getting ... service account...< Creating other service accounts and JSON key files... - connect-agent-sa-2007081316 - connect-register-sa-2007081316 - log-mon-sa-2007081316 Enabling APIs... - project my-project - serviceusage.googleapis.com - iam.googleapis.com - cloudresourcemanager.googleapis.com Configuring IAM roles for service accounts... - my-component-access-sa for project my-project - roles/serviceusage.serviceUsageViewer - roles/iam.serviceAccountCreator - roles/iam.roleViewer - connect-agent-sa-2007081316 for project my-project - roles/gkehub.connect - connect-register-sa-2007081316 for project my-project - roles/gkehub.admin - roles/serviceusage.serviceUsageViewer - log-mon-sa-2007081316 for project my-project - roles/stackdriver.resourceMetadata.writer - roles/logging.logWriter - roles/monitoring.metricWriter Copying files to admin workstation... - /usr/local/google/home/me/certs/vcenter-ca-cert.pem - /usr/local/google/home/me/service-account-keys/component-access-key.json - connect-agent-sa-2007081316.json - connect-register-sa-2007081316.json - log-mon-sa-2007081316.json Preparing "admin-cluster.yaml" for gkectl... Preparing "user-cluster.yaml" for gkectl... ******************************************************************** Admin workstation is ready to use. Admin workstation information saved to /usr/local/google/home/me/my-admin-workstation This file is required for future upgrades SSH into the admin workstation with the following command: ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation email@example.com ********************************************************************
Getting an SSH connection to your admin workstation
Near the end of the preceding output there is a command you can use to get an SSH connection to your admin workstation. Enter that command now. For example:
ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation firstname.lastname@example.org
List the files on your admin workstation:
In the output, you can see two cluster configuration files, your CA certificate file, and the JSON key files for your service accounts:
admin-cluster.yaml connect-agent-sa-2007081316.json connect-register-sa-2007081316.json log-mon-sa-2007081316.json user-cluster.yaml vcenter-ca-cert.pem component-access-key.json
Inspecting the cluster configuration files
On your admin workstation,
gkeadm created two cluster configuration
user-cluster.yaml You use these files later
to create your admin and user clusters.
Use a text editor to open
admin-cluster.yaml. Notice that many of the
fields are already populated with values that you provided in your
admin-ws-config.yaml file. For example:
apiVersion: v1 kind: AdminCluster bundlePath: /var/lib/gke/bundles/gke-onprem-vsphere-1.4.0-gke.13-full.tgz vCenter: address: 203.0.113.1 datacenter: MY-DATACENTER cluster: MY-CLUSTER resourcePool: MY-POOL datastore: MY-DATASTORE caCertPath: vcenter-ca-cert.pem credentials: username: email@example.com password: STyZ2T#Ko2o ... gcrKeyPath: component-access-key.json stackdriver: projectID: my-project-id serviceAccountKeyPath: log-mon-sa-2007081316.json
user-cluster.yaml. Notice that many of the fields are already
populated with values that you provided in your
apiVersion: v1 kind: UserCluster gkeOnPremVersion: 1.4.0-gke.13 ... stackdriver: projectID: my-project-id serviceAccountKeyPath: log-mon-sa-2007081316.json gkeConnect: projectID: my-project-id registerServiceAccountKeyPath: connect-register-sa-2007081316.json agentServiceAccountKeyPath: connect-agent-sa-2007081316.json
Using a single Cloud project
The Anthos clusters on VMware configuration file has several fields where you can
specify a Google Cloud project ID. For example, you can specify one
project ID for
gkeconnect and another project ID for
gkeconnect: projectid: "" ... stackdriver: projectid: ""
gkeadm prepared your Anthos clusters on VMware configuration file, it set
all of the
projectid fields to the project that is the parent of your
component access service account. Also,
gkeadm granted roles to your
service accounts on the project that is the parent of your service
account. In short,
gkeadm used a single Cloud project.
If you want the flexibility of using multiple Google Cloud projects, you
must manually fill in the
projectid fields and manually grant roles to your
service accounts. For more information, see
Using Multiple Google Cloud projects.