Creating an admin workstation with a static IP address

This page explains how to create an admin workstation that you can use to install GKE on-prem.

In this topic, you create an admin workstation that has a static IP address.

If you want to create an admin workstation that uses Dynamic Host Configuration Protocol (DHCP) to get its IP address, see Creating an admin workstation using DHCP.

If you run your commands on Linux, there is an alternative to following the steps in this topic. The alternative is to use the gkeadm tool which automates much of the admin workstation creation.

Overview

The admin workstation is a vSphere VM that contains all the tools you need to create and manage GKE on-prem clusters. To create the admin workstation, you perform the following steps described in this topic:

• Download the admin workstation Open Virtual Appliance (OVA) file, a compressed image of the admin workstation VM.
• Use govc, the command line interface to vSphere, to import the OVA to vSphere as a VM template.
• Copy and populate HashiCorp Terraform configuration files.
• Use Terraform version 0.11 to create the admin workstation VM.

Creating a key for your allowlisted service account

If you don't already have a JSON key file for your allowlisted service account, create one now:

1. gcloud auth login

2. gcloud iam service-accounts keys create whitelisted-key.json \
   --iam-account [ALLOWLISTED_SERVICE_ACCOUNT_EMAIL]
   

where [ALLOWLISTED_SERVICE_ACCOUNT_EMAIL] is the email address of your allowlisted service account.

Downloading the admin workstation OVA

The admin workstation OVA includes all of the cluster components, command line tools, and other entities needed to install and manage GKE on-prem clusters.

Activate your allowlisted service account:

gcloud auth activate-service-account --key-file [KEY_ALLOWLISTED_ACCOUNT]

where [KEY_ALLOWLISTED_ACCOUNT] is the path of the JSON key file for your allowlisted service account.

Download the latest version of the admin workstation OVA and its signature file:

gcloud storage cp gs://gke-on-prem-release/admin-appliance/1.3.2-gke.1/gke-on-prem-admin-appliance-vsphere-1.3.2-gke.1.{ova,ova.1.sig} ./

Verifying the OVA using openssl

Verify the OVA file you downloaded against the public key using openssl:

openssl dgst -verify - -signature gke-on-prem-admin-appliance-vsphere-1.3.2-gke.1.ova.1.sig gke-on-prem-admin-appliance-vsphere-1.3.2-gke.1.ova <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

Expected output of this command is Verified OK.

Using govc to import the OVA to vSphere and mark it as a VM template

In the following sections, you:

1. Create some variables declaring elements of your vCenter Server and vSphere environment.
2. Import the admin workstation OVA to vSphere and mark it as a VM template.

Creating variables for govc

Before you import the admin workstation OVA to vSphere, you need to provide govc some variables declaring elements of your vCenter Server and vSphere environment:

Note: You can find many of these values by logging in to the vCenter Server.

export GOVC_URL=https://[VCENTER_SERVER_ADDRESS]/sdk
export GOVC_USERNAME=[VCENTER_SERVER_USERNAME]
export GOVC_PASSWORD=[VCENTER_SERVER_PASSWORD]
export GOVC_DATASTORE=[VSPHERE_DATASTORE]
export GOVC_DATACENTER=[VSPHERE_DATACENTER]
export GOVC_INSECURE=true

You can choose to use vSphere's default resource pool or create your own:

# If you want to use a resource pool you've configured yourself, export this variable:
export GOVC_RESOURCE_POOL=[VSPHERE_CLUSTER]/Resources/[VSPHERE_RESOURCE_POOL]

# If you want to use vSphere's default resource pool, export this variable instead:
export GOVC_RESOURCE_POOL=[VSPHERE_CLUSTER]/Resources

where:

• [VCENTER_SERVER_ADDRESS] is your vCenter Server's IP address or hostname.
• [VCENTER_SERVER_USERNAME] is the username of an account that holds the Administrator role or equivalent privileges in vCenter Server.
• [VCENTER_SERVER_PASSWORD] is the vCenter Server account's password.
• [VSPHERE_DATASTORE] is the name of the datastore you've configured in your vSphere environment.
• [VSPHERE_DATACENTER] is the name of the datacenter you've configured in your vSphere environment.
• [VSPHERE_CLUSTER] is the name of the cluster you've configured in your vSphere environment.
For using a non-default resource pool,
• [VSPHERE_RESOURCE_POOL] is the name of the resource pool you've configured to your vSphere environment.

Creating variables for your proxy

If you are using a proxy, export variables for its the HTTP and HTTPS address, where [PROXY_ADDRESS] is the proxy's IP address or hostname:

export HTTP_PROXY=http://[PROXY_ADDRESS]
export HTTPS_PROXY=https://[PROXY_ADDRESS]

Importing the OVA to vSphere: Standard switch

If you are using a vSphere Standard Switch, import the OVA to vSphere using this command:

govc import.ova -options - gke-on-prem-admin-appliance-vsphere-1.3.2-gke.1.ova <<EOF
{
  "DiskProvisioning": "thin",
  "MarkAsTemplate": true
}
EOF

Importing the OVA to vSphere: Distributed switch

If you are using a vSphere Distributed Switch, import the OVA to vSphere using this command, where [YOUR_DISTRIBUTED_PORT_GROUP_NAME] is the name of your distributed port group:

govc import.ova -options - gke-on-prem-admin-appliance-vsphere-1.3.2-gke.1.ova <<EOF
{
  "DiskProvisioning": "thin",
  "MarkAsTemplate": true,
  "NetworkMapping": [
      {
          "Name": "VM Network",
          "Network": "[YOUR_DISTRIBUTED_PORT_GROUP_NAME]"
      }
  ]
}
EOF

Copying the Terraform configuration files

Create a directory for your Terraform files:

mkdir [TERRAFORM_DIR]

where [TERRAFORM_DIR] is the path of a directory where you want to keep your Terraform files.

Copy the following TF and TFVARS files and save them to [TERRAFORM_DIR]/terraform.tf and [TERRAFORM_DIR]/terraform.tfvars, respectively.

The TF file is the Terraform HCL config that performs the VM creation.

The Terraform files given here apply to an admin workstation that has a static IP address. If you want to create an admin workstation that uses DHCP to get its IP address, use the Terraform files given in Creating an admin workstation using DHCP.

<-- Click the disclosure arrow to reveal the Terraform files.

terraform.tfvars

# vCenter Server username
vcenter_user = ""

# vCenter Server password
vcenter_password = ""

# vCenter Server IP or hostname
vcenter_server = ""

# Path in which the admin workstation's VM's public key should be saved
ssh_public_key_path = "~/.ssh/vsphere_workstation.pub"

# Hostname for the VM
vm_name = ""

# vSphere datastore to use for storage
datastore = ""

# vSphere datacenter in which to create the VM
datacenter = ""

# vSphere cluster in which to create the VM
cluster = ""

# vSphere resource pool in which to create VM, if you are using a non-default resource pool
# If you are using the default resource pool, provide a value like "[CLUSTER-NAME]/Resources"
resource_pool = ""

# vSphere network to use for the VM
network = ""

# Number of CPUs for this VM. Recommended minimum 4.
num_cpus = 4

# Memory in MB for this VM. Recommended minimum 8192.
memory = 8192

# The VM template (OVA) to clone. Change the version if you imported a different version of the OVA.
vm_template = "gke-on-prem-admin-appliance-vsphere-1.3.2-gke.1"

################################################
# Static IP settings                           #
################################################

# An IPv4 static IP for the admin workstataion
ipv4_address = ""

# The netmask prefix length to use
ipv4_netmask_prefix_length = ""

# The IPv4 gateway the admin workstation should use
ipv4_gateway = ""

# Comma-delimited DNS servers
dns_nameservers = ""

# The NTP server the admin workstation should use
ntp_server = ""

terraform.tf

#########################
####### VARIABLES #######
#########################

# The following variables are declared in the accompanying TFVARS file

# vCenter Server username
variable "vcenter_user" { }

# vCenter Server password
variable "vcenter_password" { }

# vCenter Server address
variable "vcenter_server" { }

# Path in which the VM's public key should be saved
variable "ssh_public_key_path" { default = "~/.ssh/vsphere_workstation.pub" }

# vSphere network to use for the VM
variable "network" { default = "VM Network"}

# Hostname for the VM
variable "vm_name" { default = "vsphere-workstation" }

# vSphere datacenter in which to create the admin workstation VM
variable "datacenter" { }

# vSphere datastore to use for storage
variable "datastore" { }

# vSphere cluster in which to create the VM
variable "cluster" { }

# vSphere resource pool in which to create the VM
variable "resource_pool" { }

# Number of CPUs for this VM. Recommended minimum 4.
variable "num_cpus" { default = 4 }

# Memory in MB for this VM. Recommended minimum 8192.
variable "memory" { default = 8192 }

# The VM template (OVA) to clone
variable "vm_template" { }

# The IP address to assign to the VM
variable "ipv4_address" { }

# Netmask prefix length
variable "ipv4_netmask_prefix_length" { }

# Default gateway to use
variable "ipv4_gateway" { }

# DNS resolvers to use
variable "dns_nameservers" { }

# NTP server to use
variable "ntp_server" { default = "ntp.ubuntu.com" }

##########################
##########################

provider "vsphere" {
  version        = "1.15"
  user           = "${var.vcenter_user}"
  password       = "${var.vcenter_password}"
  vcenter_server = "${var.vcenter_server}"

  # if you have a self-signed cert
  allow_unverified_ssl = true
}

### vSphere Data ###

data "vsphere_datastore" "datastore" {
  name          = "${var.datastore}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_datacenter" "dc" {
  name = "${var.datacenter}"
}

data "vsphere_compute_cluster" "cluster" {
  name          = "${var.cluster}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_resource_pool" "pool" {
  name          = "${var.resource_pool}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_network" "network" {
  name          = "${var.network}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_virtual_machine" "template_from_ovf" {
  name          = "${var.vm_template}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

##########################
### IF USING STATIC IP ###
##########################
data "template_file" "static_ip_config" {
  template = <<EOF
network:
  version: 2
  ethernets:
    ens192:
      dhcp4: no
      dhcp6: no
      addresses: ["${var.ipv4_address}/${var.ipv4_netmask_prefix_length}"]
      gateway4: ${var.ipv4_gateway}
      nameservers:
        addresses: [${var.dns_nameservers}]
EOF
}

data "template_file" "user_data" {
  template = <<EOF
#cloud-config
ntp:
  enabled: true
  servers: [${var.ntp_server}]
  ntp_client: systemd-timesyncd
apt:
  primary:
    - arches: [default]
      uri: http://us-west1.gce.archive.ubuntu.com/ubuntu/
write_files:
  - path: /tmp/static-ip.yaml
    permissions: '0644'
    encoding: base64
    content: |
      $${static_ip_config}
runcmd:
  - /var/lib/gke/guest-startup.sh
EOF
  vars = {
    static_ip_config = "${base64encode(data.template_file.static_ip_config.rendered)}"

  }
}
##########################
### IF USING STATIC IP ###
##########################

### vSphere Resources ###

resource "vsphere_virtual_machine" "vm" {
  name             = "${var.vm_name}"
  resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
  datastore_id     = "${data.vsphere_datastore.datastore.id}"
  num_cpus         = "${var.num_cpus}"
  memory           = "${var.memory}"
  guest_id         = "${data.vsphere_virtual_machine.template_from_ovf.guest_id}"
  enable_disk_uuid = "true"
  scsi_type = "${data.vsphere_virtual_machine.template_from_ovf.scsi_type}"
  network_interface {
    network_id   = "${data.vsphere_network.network.id}"
    adapter_type = "${data.vsphere_virtual_machine.template_from_ovf.network_interface_types[0]}"
  }

  wait_for_guest_net_timeout = 15

  nested_hv_enabled = false
  cpu_performance_counters_enabled = false

  disk {
    label            = "disk0"
    size             = "${max(50, data.vsphere_virtual_machine.template_from_ovf.disks.0.size)}"
    eagerly_scrub    = "${data.vsphere_virtual_machine.template_from_ovf.disks.0.eagerly_scrub}"
    thin_provisioned = "${data.vsphere_virtual_machine.template_from_ovf.disks.0.thin_provisioned}"
  }

  cdrom {
    client_device = true
  }

  vapp {
    properties = {
      hostname    = "${var.vm_name}"
      public-keys = "${file(var.ssh_public_key_path)}"
      user-data   = "${base64encode(data.template_file.user_data.rendered)}"
    }
  }

  clone {
    template_uuid = "${data.vsphere_virtual_machine.template_from_ovf.id}"
  }
}

output "ip_address" {
  value = "${vsphere_virtual_machine.vm.default_ip_address}"
}

Creating an SSH key

Create an SSH key, so that you can SSH into the admin workstation from your local laptop or workstation. On Linux-based operating systems, you can use ssh-keygen:

ssh-keygen -t rsa -f ~/.ssh/vsphere_workstation -N ""

Modifying the TFVARS file

Open terraform.tfvars in a text editor and provide values for the following variables. You can find many of these values by logging in to the vCenter Client:

vcenter_user

Provide a vCenter Server user account as a string. The user account should have the Administrator role or equivalent privileges (see vSphere requirements).

For example:

vcenter_user = "[e-mail]"

vcenter_password

Provide the vCenter Server user account's password as a string. For example:

vcenter_password = "#STyZ2T#Ko2o"

vcenter_server

Provide your vCenter Server's address (IP or hostname) as a string. For example:

vcenter_server = "198.51.100.2"

ssh_public_key_path

Provide the path to your SSH public key. You created this in a previous step:

ssh_public_key_path = "~/.ssh/vsphere_workstation.pub"

vm_name

Provide a name of your choice for the admin workstation. For example:

vm_name = "my-admin-workstation"

datastore

Provide the name of your vSphere datastore as a string. For example:

datastore = "MY-DATASTORE"

datacenter

Provide the name of your vSphere datacenter as a string. For example:

datacenter = "MY-DATACENTER"

cluster

Provide the name of your vSphere cluster as a string. For example.

cluster = "MY-CLUSTER"

resource_pool

If you are using a non-default resource pool, provide the name of your vSphere resource pool as a string. For example:

resource_pool = "MY-POOL"

If you are using the default resource pool, provide the following value:

resource_pool = "[MY_CLUSTER]/Resources"

where [MY_CLUSTER] is the name of your vSphere cluster.

See Specifying the root resource pool for a standalone host.

network

Provide the vSphere network where you want to create your admin workstation, as a string. For example:

network = "MY-VM-NETWORK"

vm_template

Provide the VM template name as a string. You created imported the OVA and marked it as a template in a previous step. Notice that the template name does not have the .ova extension.

vm_template = "gke-on-prem-admin-appliance-vsphere-1.3.2-gke.1"

ipv4_address

Provide an IPv4 static IP address for the admin workstation. For example:

ipv4_address = "203.0.113.1"

ipv4_netmask_prefix_length

Provide the number of bits in the subnet mask of the network where you want to create your admin workstation. For example:

ipv4_netmask_prefix_length = "22"

ipv4_gateway

Provide the IP address of the default gateway of the subnet in which the admin workstation is to be created. For example:

ipv4_gateway = "198.51.100.1

dns_nameservers

Provide DNS nameservers to be used by the admin workstation, separated by commas. For example:

dns_nameservers = "8.8.8.8,8.8.4.4"

Creating the admin workstation

Now you are ready to create the admin workstation VM. Use Terraform version 0.11 for the steps in this section.

1. Go to the directory that contains your Terraform configuration files (TF and TFVARS):

2. Initialize Terraform in the directory and apply the configuration. This might take a few minutes:

   terraform init && terraform apply -auto-approve -input=false

   Note: The name of the TFVARS file must be terraform.tfvars. Otherwise, you have to pass in the --var-file parameter and specify the TFVARS file.

SSH in to your admin workstation

1. Go to the directory that contains your Terraform configuration files.

2. Retrieve the IP address of the admin workstation:

   terraform output ip_address

   Make note of the admin workstation's IP address.

3. SSH in to the admin workstation by using your SSH key and the IP address:

   ssh -i ~/.ssh/vsphere_workstation ubuntu@[ADMIN_WORKSTATION_IP_ADDRESS]
   

Verifying that the admin workstation is set up correctly

Verify that gkectl and docker are installed on your admin workstation:

gkectl version
docker version

Configuring the NTP server on your admin workstation

By default, the admin workstation uses ntp.ubuntu.com as its Network Time Protocol (NTP) server. If your organization uses a different time server, configure your admin workstation to use the same NTP server as the rest of your organization.

Enter the following commands to configure the NTP server on your admin workstation:

sudo mkdir -p /etc/systemd/timesyncd.conf.d/

sudo bash -c "cat >> /etc/systemd/timesyncd.conf.d/cloud-init.conf" << EOF
[Time]
NTP=[NTP_SERVER]
EOF

sudo systemctl restart systemd-timesyncd
timedatectl status

where [NTP_SERVER] is the hostname or IP address of your NTP server.

To verify that your NTP server is working, enter the following command:

timedatectl

The output is similar to this:

                      Local time: Tue 2019-12-17 00:21:50 UTC
                  Universal time: Tue 2019-12-17 00:21:50 UTC
                        RTC time: Tue 2019-12-17 00:21:50
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: yes
                 RTC in local TZ: no

Troubleshooting

AccessDeniedException while downloading OVA

Symptoms

Attempting to download the admin workstation OVA and signature returns the following error:

AccessDeniedException: 403 [e-mail] does not have storage.objects.list access to gke-on-prem-release

Potential causes

Your allowlisted service account is not activated.

Resolution

Make sure you have activated your allowlisted service account. If the issue persists, reach out to Google for assistance.

openssl can't validate admin workstation OVA

Symptoms

Running openssl dgst against the admin workstation OVA file doesn't return Verified OK

Potential causes

An issue is present in the OVA file that prevents successful validation.

Resolution

Try downloading and deploying the admin workstation OVA again, as instructed in Download the admin workstation OVA . If the issue persists, reach out to Google for assistance.

For more information, refer to Troubleshooting.