Creating an admin workstation

This document shows how to create an admin workstation that you can use to create GKE on-prem clusters.

In this document, you create an admin workstation that has a static IP address. This is consistent with the basic installation journey, which creates clusters that use static IP addresses.

If you want to create an admin workstation that uses Dynamic Host Configuration Protocol (DHCP) to get its IP address, see Creating an admin workstation using DHCP.

The steps in this document use the gkeadm command-line tool, which is available for Linux, Windows 10, and Windows Server 2019, but not for MAC OS. In a future version of GKE on-prem, gkeadm will be available for MAC OS. If you want to use a computer running MAC OS to create an admin workstation, see Creating an admin workstation with a static IP address or Creating an admin workstation using DHCP.

Downloading gkeadm

Download the gkeadm command-line tool, and make it executable:

gcloud storage cp gs://gke-on-prem-release-public/gkeadm/1.4.3-gke.3/linux/gkeadm ./
chmod +x gkeadm

Generating a template for your configuration file

To create an admin workstation, the gkeadm tool requires a configuration file. To generate a template for your configuration file:

./gkeadm create config

The output is a file named admin-ws-config.yaml.

Populated values

The following fields in your configuration file are populated with default or generated values. For example:

adminWorkstation:
  name: gke-admin-ws-200617-113711
  cpus: 4
  memoryMB: 8192
  diskGB: 50
  dataDiskName: gke-on-prem-admin-workstation-data-disk/gke-admin-ws-200617-113711-data-disk.vmdk
  dataDiskMB: 512
  ntpServer: ntp.ubuntu.com

Keep all of the default and generated values.

Notice the generated value of dataDiskName. This virtual machine disk (VMDK) is created for you. Later, if you manually delete your admin workstation, the VMDK is automatically deleted.

Filling in the rest of your configuration file

In your configuration file, enter field values as described in this section.

gcp.whitelistedServiceAccountKeyPath

String. The path of the JSON key file for your allowlisted service account. For information about creating a JSON key file, see allowlisted service account. For example:

gcp:
  whitelistedServiceAccountKeyPath: "my-key-folder/whitelisted-key.json"

vCenter.credentials.address

String. The IP address or the hostname of your vCenter server.

Before you fill in the address field, download and inspect the serving certificate of your vCenter server. Enter the following command to download the certificate and save it to a file named vcenter.pem.

true | openssl s_client -connect VCENTER_IP:443 -showcerts 2>/dev/null | sed -ne '/-BEGIN/,/-END/p' > vcenter.pem

Replace VCENTER_IP with the IP address of your vCenter Server.

Open the certificate file to see the Subject Common Name and the Subject Alternative Name:

openssl x509 -in vcenter.pem -text -noout

The output shows the Subject Common Name (CN). This might be an IP address, or it might be a hostname. For example:

Subject: ... CN = 203.0.113.100
Subject: ... CN = my-host.my-domain.example

The output might also include one or more DNS names under Subject Alternative Name:

X509v3 Subject Alternative Name:
    DNS:vcenter.my-domain.example

Choose the Subject Common Name or one of the DNS names under Subject Alternative Name to use as the value of vcenter.credentials.address in your configuration file. For example:

vCenter:
  credentials:
    address: "203.0.113.1"

vCenter:
  credentials:
    address: "my-host.my-domain.example"

vCenter.credentials.username

String. A vCenter Server user account. The user account should have the Administrator role or equivalent privileges. See vSphere requirements.

For example:

vCenter:
  credentials:
    username: "administrator@vsphere.local"

vCenter.credentials.password

String. The password for the vCenter Server user account. For example:

vCenter:
  credentials:
    password: "#STyZ2T#Ko2o"

vCenter.datacenter

String. The name of your vSphere datacenter. For example:

vCenter:
  datacenter: "MY-DATACENTER"

vCenter.datastore

String. The name of your vSphere datastore. For example:

vCenter:
  datastore: "MY-DATASTORE"

vCenter.cluster

String. The name of your vSphere cluster. For example:

vCenter:
  cluster: "MY-CLUSTER"

vCenter.network

String. The name of the vSphere network where you want to create your admin workstation. For example:

vCenter:
  network: "MY-VM-NETWORK"

vCenter.resourcePool

String. If you are using a non-default resource pool, provide the name of your vSphere resource pool. For example:

vCenter:
  resourcePool: "MY-POOL"

If you are using the default resource pool, provide the following value:

vCenter:
  resourcePool: "MY_CLUSTER/Resources"

Replace MY_CLUSTER with the name of your vSphere cluster.

See Specifying the root resource pool for a standalone host.

vCenter.caCertPath

String. When a client, like GKE on-prem, sends a request to your vCenter server, the server must prove its identity to the client by presenting a certificate or a certificate bundle. To verify the certificate or bundle, GKE on-prem must have the root certificate in the chain of trust.

Set vCenter.caCertPath to the path of the root certificate. For example:

vCenter:
  caCertPath: "/usr/local/google/home/me/certs/vcenter-ca-cert.pem"

Your VMware installation has a certificate authority (CA) that issues a certificate to your vCenter server. The root certificate in the chain of trust is a self-signed certificate created by VMware.

If you do not want to use the VMWare CA, which is the default, you can configure VMware to use a different certificate authority.

If your vCenter server uses a certificate issued by the default VMware CA, download the certificate as follows:

curl -k "https://SERVER_ADDRESS/certs/download.zip" > download.zip

Replace SERVER_ADDRESS with address of your vCenter server.

Install the unzip command and unzip the certificate file:

sudo apt-get install unzip
unzip downloads.zip

If the unzip command doesn't work the first time, enter the command again.

Find the certificate file in certs/lin.

proxyUrl

String: If the machine you are using to run gkeadm uses a proxy server for access to the internet, set this field to the URL of the proxy server. For example:

proxyUrl: "https://my-proxy.example.local"

adminWorkstation.network.ipAllocationMode

String. Set this to "static". For example:

adminWorkstation:
  network:
    ipAllocationMode: "static"

adminWorkstation.network.hostConfig.ip

String. An IP address of your choice for your admin workstation. For example:

adminWorkstation:
  network:
    hostconfig:
      ip: "172.16.5.1"

adminWorkstation.network.hostConfig.gateway

String. The IP address of the default gateway for the network that contains your admin workstation. For example:

adminWorkstation:
  network:
    hostconfig:
      gateway: "172.16.6.254"

adminWorkstation.network.hostConfig.netmask

String. The netmask for the network that contains your admin workstation. For example:

adminWorkstation:
  network:
    hostconfig:
      netmask: "255.255.248.0"

adminWorkstation.network.hostConfig.dns

String array. An array of IP addresses for DNS servers that your admin workstation can use. For example:

adminWorkstation:
  network:
    hostconfig:
      dns:
      - "172.16.255.1"
      - "172.16.255.2"

adminWorkstation.proxyUrl

String. If your network is behind a proxy server, and you want both your admin workstation and your GKE on-prem clusters to use the same proxy server, then set adminworkstation.proxyURL to the URL of the proxy server. For example:

adminworkstation:
  proxyUrl: "http://aw-proxy.example"

adminWorkstation.ntpServer

String. The hostname or IP address of the Network Time Protocol server that your admin workstation is to use. For example:

adminWorkstation:
  ntpServer: "216.239.35.0"

If you leave this empty, GKE on-prem uses "ntp.ubuntu.com".

Here is an example of a completed admin-ws-config.yaml file:

gcp:
  whitelistedServiceAccountKeyPath: "my-key-folder/whitelisted-key.json"
vCenter:
  credentials:
    address: "203.0.113.1"
    username: "administrator.vsphere.local"
    password: "#STyZ2T#Ko2o"
  datacenter: "MY-DATACENTER"
  datastore: "MY-DATASTORE"
  cluster: "MY-CLUSTER"
  network: "MY-VM-NETWORK"
  resourcePool: "MY-POOL"
  caCertPath: "/usr/local/google/home/me/certs/the-root.cert"
proxyUrl: ""
adminWorkstation:
  name: "gke-admin-ws-200617-113711"
  cpus: 4
  memoryMB: 8192
  diskGB: 50
  dataDiskName: "gke-on-prem-admin-workstation-data-disk/gke-admin-ws-200617-113711-data-disk.vmdk"
  dataDiskMB: 512
  network:
    ipAllocationMode: "static"
    hostConfig:
      ip: "172.16.5.1"
      gateway: "172.16.6.254"
      netmask: "255.255.248.0"
      dns:
      - "172.16.255.1"
      - "172.16.255.2"
  proxyUrl: ""
  ntpServer: "ntp.ubuntu.com"

Creating your admin workstation

The Google Account that is set as your SDK account property must have these IAM roles so that gkeadm can create and manage service accounts for you:

  • resourcemanager.projectIamAdmin
  • serviceusage.serviceUsageAdmin
  • iam.serviceAccountCreator
  • iam.serviceAccountKeyAdmin

To view the value of your SDK account property:

gcloud config get-value account

To grant roles, you must have certain permissions on your Google Cloud project. For details, see Granting, changing, and revoking access to resources.

If you have the required permissions, you can grant the roles yourself. Otherwise, someone else in your organization must grant the roles for you.

To grant the roles:

Linux and macOS

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="user:ACCOUNT" \
    --role="roles/resourcemanager.projectIamAdmin"

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="user:ACCOUNT" \
    --role="roles/serviceusage.serviceUsageAdmin"

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="user:ACCOUNT" \
    --role="roles/iam.serviceAccountCreator"

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="user:ACCOUNT" \
    --role="roles/iam.serviceAccountKeyAdmin"

Windows

gcloud projects add-iam-policy-binding PROJECT_ID ^
    --member="user:ACCOUNT" ^
    --role="roles/resourcemanager.projectIamAdmin"

gcloud projects add-iam-policy-binding PROJECT_ID ^
    --member="user:ACCOUNT" ^
    --role="roles/serviceusage.serviceUsageAdmin"

gcloud projects add-iam-policy-binding PROJECT_ID ^
    --member="user:ACCOUNT" ^
    --role="roles/iam.serviceAccountCreator"

gcloud projects add-iam-policy-binding PROJECT_ID ^
    --member="user:ACCOUNT" ^
    --role="roles/iam.serviceAccountKeyAdmin"

Replace the following:

  • PROJECT_ID: the value of your SDK project property
  • ACCOUNT: the value of your SDK account property.

Create your admin workstation and a set of service accounts. If you prefer to create and configure your service accounts manually, you can omit the --auto-create-service-accounts flag.

./gkeadm create admin-workstation --auto-create-service-accounts

The output gives detailed information about the creation of your admin workstation. For example, you can see that gkeadm performed these tasks:

  • Ran preflight checks.

  • Downloaded an OVA image and uploaded it as a vSphere template.

  • Created an SSH key.

  • Created an admin workstation VM.

  • Printed the gkectl and Docker versions installed on the admin workstation.

  • Printed information about the NTP server on the admin workstation.

  • Created service accounts and their JSON key files. These service accounts are members of the project that is the parent of your allowlisted service account.

  • Assigned the appropriate IAM roles to the service accounts. These roles are granted on the project that is the parent of your allowlisted service account.

  • Copied JSON key files for the service accounts to the admin workstation.

  • Copied the CA certificate file for your vCenter server to the admin workstation.

  • Prepared admin-cluster.yaml and user-cluster.yaml on the admin workstation.n Later, you will use these configuration files to create clusters.

...
- Validation Category: vCenter
    - [SUCCESS] Credentials
    - [SUCCESS] Version
    - [SUCCESS] Datacenter
    - [SUCCESS] Datastore
    - [SUCCESS] Resource Pool
    - [SUCCESS] Folder
    - [SUCCESS] Network
    - [SUCCESS] Datadisk

All validation results were SUCCESS.

******************************************
Admin workstation VM successfully created:
- Name:    my-admin-workstation
- IP:      172.16.5.1
- SSH Key: /usr/local/google/home/me/.ssh/gke-admin-workstation
******************************************

Printing gkectl and docker versions on admin workstation...
gkectl version
gkectl 1.4.0-gke.13 (git-b39331512)

docker version
Client:
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.9
 Git commit:        6a30dfca03
 Built:             Mon Sep 30 22:47:02 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Getting allowlisted service account...

Creating other service accounts and JSON key files...
    - connect-agent-sa-2007081316
    - connect-register-sa-2007081316
    - log-mon-sa-2007081316

Enabling APIs...
    - project my-project
        - serviceusage.googleapis.com
        - iam.googleapis.com
        - cloudresourcemanager.googleapis.com

Configuring IAM roles for service accounts...
    - my-allowlisted-sa for project my-project
        - roles/serviceusage.serviceUsageViewer
        - roles/iam.serviceAccountCreator
        - roles/iam.roleViewer
    - connect-agent-sa-2007081316 for project my-project
        - roles/gkehub.connect
    - connect-register-sa-2007081316 for project my-project
        - roles/gkehub.admin
        - roles/serviceusage.serviceUsageViewer
    - log-mon-sa-2007081316 for project my-project
        - roles/stackdriver.resourceMetadata.writer
        - roles/logging.logWriter
        - roles/monitoring.metricWriter

Copying files to admin workstation...
    - /usr/local/google/home/me/certs/vcenter-ca-cert.pem
    - /usr/local/google/home/me/service-account-keys/whitelisted-key.json
    - connect-agent-sa-2007081316.json
    - connect-register-sa-2007081316.json
    - log-mon-sa-2007081316.json

Preparing "admin-cluster.yaml" for gkectl...
Preparing "user-cluster.yaml" for gkectl...

********************************************************************
Admin workstation is ready to use.

Admin workstation information saved to /usr/local/google/home/me/my-admin-workstation
This file is required for future upgrades
SSH into the admin workstation with the following command:
ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation ubuntu@172.16.5.1
********************************************************************

Getting an SSH connection to your admin workstation

Near the end of the preceding output there is a command you can use to get an SSH connection to your admin workstation. Enter that command now. For example:

ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation ubuntu@172.16.5.1

List the files on your admin workstation:

ls -1

In the output, you can see two cluster configuration files, your CA certificate file, and the JSON key files for your service accounts:

admin-cluster.yaml
connect-agent-sa-2007081316.json
connect-register-sa-2007081316.json
log-mon-sa-2007081316.json
user-cluster.yaml
vcenter-ca-cert.pem
whitelisted-key.json

Inspecting the cluster configuration files

On your admin workstation, gkeadm created two cluster configuration files: admin-cluster.yaml and user-cluster.yaml You use these files later to create your admin and user clusters.

Use a text editor to open admin-cluster.yaml. Notice that many of the fields are already populated with values that you provided in your admin-ws-config.yaml file. For example:

apiVersion: v1
kind: AdminCluster
bundlePath: /var/lib/gke/bundles/gke-onprem-vsphere-1.4.0-gke.13-full.tgz
vCenter:
  address: 203.0.113.1
  datacenter: MY-DATACENTER
  cluster: MY-CLUSTER
  resourcePool: MY-POOL
  datastore: MY-DATASTORE
  caCertPath: vcenter-ca-cert.pem
  credentials:
    username: administrator@vsphere.local
    password: STyZ2T#Ko2o
...
gcrKeyPath: whitelisted-key.json
stackdriver:
  projectID: my-project-id
  serviceAccountKeyPath: log-mon-sa-2007081316.json

Open user-cluster.yaml. Notice that many of the fields are already populated with values that you provided in your admin-ws-config.yaml file. For example:

apiVersion: v1
kind: UserCluster
gkeOnPremVersion: 1.4.0-gke.13
...
stackdriver:
  projectID: my-project-id
  serviceAccountKeyPath: log-mon-sa-2007081316.json
gkeConnect:
  projectID: my-project-id
  registerServiceAccountKeyPath: connect-register-sa-2007081316.json
  agentServiceAccountKeyPath: connect-agent-sa-2007081316.json

Using a single Google Cloud project

The GKE on-prem configuration file has several fields where you can specify a Google Cloud project ID. For example, you can specify one project ID for gkeconnect and another project ID for stackdriver:

gkeconnect:
  projectid: ""
...
stackdriver:
  projectid: ""

When gkeadm prepared your GKE on-prem configuration file, it set all of the projectid fields to the project that is the parent of your allowlisted service account. Also, gkeadm granted roles to your service accounts on the project that is the parent of your allowlisted service account. In short, gkeadm used a single Google Cloud project.

If you want the flexibility of using multiple Google Cloud projects, you must manually fill in the projectid fields and manually grant roles to your service accounts. For more information, see Using Multiple Google Cloud projects.