This document shows how to create an admin workstation that you can use to create GKE on-prem clusters.
In this document, you create an admin workstation that has a static IP address. This is consistent with the basic installation journey, which creates clusters that use static IP addresses.
If you want to create an admin workstation that uses Dynamic Host Configuration Protocol (DHCP) to get its IP address, see Creating an admin workstation using DHCP.
The steps in this document use the
gkeadm command-line tool, which is available
only for Linux. In a future version of GKE on-prem,
gkeadm will be
available for Windows and MAC OS. If you want to use a computer running
Windows or MAC OS to create an admin workstation, see
Creating an admin workstation with a static IP address
Creating an admin workstation using DHCP.
gkeadm command-line tool, and make it executable:
gsutil cp gs://gke-on-prem-release-public/gkeadm/1.3.2-gke.1/linux/gkeadm ./ chmod +x gkeadm
Generating a template for your configuration file
To create an admin workstation, the
gkeadm tool requires a configuration file.
To generate a template for your configuration file:
./gkeadm create config
The output is a file named
gcp: # Path of the allowlisted service account's JSON key file whitelistedServiceAccountKeyPath: "" # Specify which vCenter resources to use vCenter: # The credentials and address GKE On-Prem should use to connect to vCenter credentials: address: "" username: "" ... adminWorkstation: ... network: # The IP allocation mode: 'dhcp' or 'static' ipAllocationMode: "" # # The host config in static IP mode. Do not include if using DHCP # hostConfig: # # The IPv4 static IP address for the admin workstation ...
Filling in your configuration file
In your configuration file, enter field values as described in this section.
String. The path of the JSON key file for your allowlisted service account. For information about creating a JSON key file, see allowlisted service account. For example:
String. The IP address or the hostname of your vCenter server.
Before you fill in the
address field, download and inspect
the serving certificate of your vCenter server. Enter the following command to
download the certificate and save it to a file named
true | openssl s_client -connect VCENTER_IPVCENTER_IP:443 -showcerts 2>/dev/null | sed -ne '/-BEGIN/,/-END/p' > vcenter.pem
VCENTER_IP with the IP address of your vCenter Server.
Open the certificate file to see the Subject Common Name and the Subject Alternative Name:
openssl x509 -in vcenter.pem -text -noout
The output shows the
Subject Common Name (CN). This might be an IP address, or
it might be a hostname. For example:
Subject: ... CN = 203.0.113.100
Subject: ... CN = my-host.my-domain.example
The output might also include one or more DNS names under
Subject Alternative Name:
X509v3 Subject Alternative Name: DNS:vcenter.my-domain.example
Subject Common Name or one of the DNS names under
Subject Alternative Name to use as the value of
in your configuration file. For example:
vcenter: credentials: address: "203.0.113.1" ...
vcenter: credentials: address: "my-host.my-domain.example" ...
String. A vCenter Server user account. The user account should have the Administrator role or equivalent privileges. See vSphere requirements.
String. The password for the vCenter Server user account. For example:
String. The name of your vSphere datacenter. For example:
String. The name of your vSphere datastore. For example:
String. The name of your vSphere cluster. For example:
String. The name of the vSphere network where you want to create your admin workstation. For example:
String. If you are using a non-default resource pool, provide the name of your vSphere resource pool. For example:
If you are using the default resource pool, provide the following value:
MY_CLUSTER with the name of your vSphere cluster.
String. When a client, like GKE on-prem, sends a request to your vCenter server, the server must prove its identity to the client by presenting a certificate or a certificate bundle. To verify the certificate or bundle, GKE on-prem must have the root certificate in the chain of trust.
vCenter.caCertPath to the path of the root certificate. For example:
Your VMware installation has a certificate authority (CA) that issues a certificate to your vCenter server. The root certificate in the chain of trust is a self-signed certificate created by VMware.
If you do not want to use the VMWare CA, which is the default, you can configure VMware to use a different certificate authority.
If your vCenter server uses a certificate issued by the default VMware CA, download the certificate as follows:
curl -k "https://SERVER_ADDRESS/certs/download.zip" > download.zip
SERVER_ADDRESS with the address of your vCenter server.
unzip command and unzip the certificate file:
sudo apt-get install unzip unzip downloads.zip
If the unzip command doesn't work the first time, enter the command again.
Find the certificate file in
String: If the machine you are using to run
gkeadm uses a proxy server for access to
the internet, set this field to the URL of the proxy server. For example:
String. A name of your choice for your admin workstation. For example:
Integer. The number of virtual CPUs for your admin workstation. For example:
Integer. The number of megabytes of memory for your admin workstation. For example:
Integer. The number of gigabytes of virtual disk space for your admin workstation. Must be at least 50. For example:
String. Set this to
"static". For example:
String. An IP address of your choice for your admin workstation. For example:
String. The IP address of the default gateway for the network that contains your admin workstation. For example:
String. The netmask for the network that contains your admin workstation. For example:
String array. An array of IP addresses for DNS servers that your admin workstation can use. For example:
dns: - "172.16.255.1" - "172.16.255.2"
String. If your network is behind a proxy server, and you want both your admin
workstation and your GKE on-prem clusters to use the same proxy
server, then set
adminworkstation.proxyURL to the URL of the proxy server.
adminworkstation: proxyUrl: "http://aw-proxy.example"
String. The hostname or IP address of the Network Time Protocol server that your admin workstation is to use. For example:
If you leave this empty, GKE on-prem uses
Here is an example of a completed
gcp: whitelistedServiceAccountKeyPath: "my-key-folder/whitelisted-key.json" vCenter: credentials: address: "203.0.113.1" username: "administrator.vsphere.local" password: "#STyZ2T#Ko2o" datacenter: "MY-DATACENTER" datastore: "MY-DATASTORE" cluster: "MY-CLUSTER" network: "MY-VM-NETWORK" resourcePool: "MY-POOL" caCertPath: "/usr/local/google/home/me/certs/the-root.cert" proxyUrl: "" adminWorkstation: name: "my-admin-workstation" cpus: 4 memoryMB: 8192 diskGB: 50 network: ipAllocationMode: "static" hostConfig: ip: "172.16.5.1" gateway: "172.16.6.254" netmask: "255.255.248.0" dns: - "172.16.255.1" - "172.16.255.2 proxyUrl: "" ntpServer: "220.127.116.11"
Creating your admin workstation
gkeadm to create an admin workstation and a set of
service accounts, you must grant the
serviceusage.serviceUsageAdmin IAM roles to the default Cloud Billing account.
To view the default account:
gcloud config get-value account
To grant the required roles:
gcloud projects add-iam-policy-binding PROJECT_ID \ --member="user:DEFAULT_ACCOUNT" \ --role="roles/resourcemanager.projectIamAdmin" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="user:DEFAULT_ACCOUNT" \ --role="roles/serviceusage.serviceUsageAdmin"
Replace the following:
PROJECT_ID: the ID of your connect project.
DEFAULT_ACCOUNT: the default Cloud Billing account.
Create your admin workstation and a set of service accounts. If you prefer to
create and configure your service accounts manually, you can omit the
./gkeadm create admin-workstation --auto-create-service-accounts
The output gives detailed information about the creation of your admin
workstation. For example, you can see that
gkeadm performed these tasks:
Ran preflight checks.
Downloaded an OVA image and uploaded it as a vSphere template.
Created an SSH key.
Created an admin workstation VM.
gkectland Docker versions installed on the admin workstation.
Printed information about the NTP server on the admin workstation.
Created service accounts and their JSON key files. These service accounts are members of the project that is the parent of your allowlisted service account.
Assigned the appropriate IAM roles to the service accounts. These roles are granted on the project that is the parent of your allowlisted service account.
Copied JSON key files for the service accounts to the admin workstation.
Copied the CA certificate file for your vCenter server to the admin workstation.
config.yamlon the admin workstation. Later, you will use
config.yamlto create clusters.
... - Validation Category: vCenter - [SUCCESS] Credentials - [SUCCESS] Version - [SUCCESS] Datacenter - [SUCCESS] Datastore - [SUCCESS] Resource Pool - [SUCCESS] Network All validation results were SUCCESS. ****************************************** Admin workstation VM successfully created: - Name: my-admin-workstation - IP: 172.16.5.1 - SSH Key: /usr/local/google/home/me/.ssh/gke-admin-workstation ****************************************** Printing gkectl and docker versions on admin workstation... gkectl version gkectl 1.3.1-gke.0 (git-3aaf91927) docker version Client: Version: 19.03.2 API version: 1.40 Go version: go1.12.9 Git commit: 6a30dfca03 Built: Mon Sep 30 22:47:02 2019 OS/Arch: linux/amd64 Experimental: false ... Getting allowlisted service account... Creating other service accounts and JSON key files... - log-mon-sa-2004211642 - connect-agent-sa-2004211642 - connect-register-sa-2004211642 Enabling APIs... - project my-project - serviceusage.googleapis.com - iam.googleapis.com - cloudresourcemanager.googleapis.com Configuring IAM roles for service accounts... - my-allowlisted-sa for project my-project - roles/serviceusage.serviceUsageViewer - roles/iam.serviceAccountCreator - roles/iam.roleViewer - log-mon-sa-2004211642 for project my-project - roles/stackdriver.resourceMetadata.writer - roles/logging.logWriter - roles/monitoring.metricWriter - connect-agent-sa-2004211642 for project my-project - roles/gkehub.connect - connect-register-sa-2004211642 for project my-project - roles/gkehub.admin - roles/serviceusage.serviceUsageViewer Copying files to admin workstation... - /usr/local/google/home/me/certs/vcenter-ca-cert.pem - /usr/local/google/home/me/service-account-keys/whitelisted-key.json - log-mon-sa-2004211642.json - connect-agent-sa-2004211642.json - connect-register-sa-2004211642.json Preparing config.yaml for gkectl... ******************************************************************** Admin workstation is ready to use. SSH into the admin workstation with the following command: ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation email@example.com ******************************************************************** Admin workstation information saved to /usr/local/google/home/me/my-admin-workstation
Getting an SSH connection to your admin workstation
Near the end of the preceding output there is a command you can use to get an SSH connection to your admin workstation. Enter that command now. For example:
ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation firstname.lastname@example.org
List the files on your admin workstation:
In the output, you can see a GKE on-prem configuration file, your CA certificate file, and the JSON key files for your service accounts:
config.yaml connect-agent-sa-200303-143027.json connect-register-sa-200303-143027.json log-mon-sa-200303-143027.json the-root.crt whitelisted-key.json
Inspecting the GKE on-prem configuration file
On your admin workstation, your
configuration file was created and
config.yaml. You use this file later to create your admin and user
clusters. Use a text editor to open
config.yaml. Notice that many of the
fields are already populated with values that you provided in your
admin-ws-config.yaml file. For example:
vcenter: credentials: address: 203.0.113.1 username: email@example.com password: STyZ2T#Ko2o datacenter: MY-DATACENTER datastore: MY-DATASTORE cluster: MY-CLUSTER network: MY-VM-NETWORK … gkeconnect: projectid: my-project-id registerserviceaccountkeypath: /home/ubuntu/connect-register-sa-xxx.json agentserviceaccountkeypath: /home/ubuntu/connect-agent-sa-xxx.json stackdriver: projectid: my-project-id clusterlocation: "" enablevpc: false serviceaccountkeypath: /home/ubuntu/log-mon-sa-xxx.json
Using a single Cloud project
The GKE on-prem configuration file has several fields where you can
specify a Google Cloud project ID. For example, you can specify one
project ID for
gkeconnect and another project ID for
gkeconnect: projectid: "" ... stackdriver: projectid: ""
gkeadm prepared your GKE on-prem configuration file, it set
all of the
projectid fields to the project that is the parent of your
allowlisted service account. Also,
gkeadm granted roles to your service
accounts on the project that is the parent of your allowlisted service account.
gkeadm used a single Cloud project.
If you want the flexibility of using multiple Google Cloud projects, you
must manually fill in the
projectid fields and manually grant roles to your
service accounts. For more information, see
Using Multiple Google Cloud projects.