Admin cluster configuration file

This page describes the fields in the admin cluster configuration file for Google Distributed Cloud.

Generating a template for your configuration file

If you used gkeadm to create your admin workstation, then gkeadm generated a template for your admin cluster configuration file. Also, gkeadm filled in some of the fields for you.

If you did not use gkeadm to create your admin workstation, you can use gkectl to generate a template for your admin cluster configuration file.

To generate a template for your admin cluster configuration file:

gkectl create-config admin --config=OUTPUT_FILENAME --gke-on-prem-version=VERSION

Replace OUTPUT_FILENAME with a path of your choice for the generated template. If you omit this flag, gkectl names the file admin-cluster.yaml and puts it in the current directory.

Replace VERSION with the desired version number, which must be equal to or less than your gkectl version. For example: gkectl create-config admin --gke-on-prem-version=1.6.2-gke.0. If you omit this flag, the generated config template is populated with values based on the latest cluster version.

Template

Filling in your configuration file

In your configuration file, enter field values as described in the following sections.

name

Optional
Immutable string
Default: A random name with prefix "gke-admin-"

A name of your choice for the cluster.

Example:

name: "my-admin-cluster"

bundlePath

Required
Mutable string

The path of your Google Distributed Cloud bundle file.

The Google Distributed Cloud full bundle file contains all of the components in a particular release of Google Distributed Cloud. When you create an admin workstation, it comes with a full bundle at:

/var/lib/gke/bundles/gke-onprem-vsphere-VERSION-full.tgz

Example:

bundlePath: "/var/lib/gke/bundles/gke-onprem-vsphere-1.10.0-gke.8.full.tgz"

vCenter

Required

This sections holds information about your vSphere environment and your connection to vCenter Server.

vCenter.address

Required
Immutable string

The IP address or the hostname of your vCenter server.

For more information, see Finding your vCenter server address.

Examples:

vCenter:
  address: "203.0.113.100"

vCenter:
  address: "my-vcenter-server.my-domain.example"

vCenter.datacenter

Required
Immutable string

The name of a vCenter data center that has the physical ESXi hosts where your admin cluster VMs will run.

Example:

vCenter:
  datacenter: "my-datacenter""

vCenter.cluster

Required
Immutable string

The name of a vSphere cluster that has the ESXi hosts where your admin cluster VMs will run. This vSphere cluster is a set of physical ESXi hosts that form a subset of the physical ESXi hosts in your vCenter data center.

Example:

vCenter:
  cluster: "my-vsphere-cluster"

vCenter.resourcePool

Required
Immutable string

The name of a vCenter resource pool for your admin cluster VMs.

Example:

vCenter:
  resourcePool: "my-resource-pool-2"

To use the default resource pool, set this to VSPHERE_CLUSTER/Resources.

Example:

vCenter:
  resourcePool: "my-vsphere-cluster-2/Resources"

For more information, see Specifying the root resource pool for a standalone host.

vCenter.datastore

Required
Immutable string

The name of a vCenter datastore for your admin cluster VMs.

Example:

vCenter:
  datastore: "my-datastore"

vCenter.caCertPath

Required
Mutable string

The path of the CA certificate for your vCenter server.

For more information, see Getting your vCenter CA certificate.

For information about updating this field for an existing cluster, see Update vCenter certificate references.

Example:

vCenter:
  caCertPath: "/usr/local/google/home/me/certs/vcenter-ca-cert.pem"

vCenter.credentials.fileRef.path

Required
Mutable string

The path of a credentials configuration file that holds the username and password of your vCenter user account. The user account should have the Administrator role or equivalent privileges. See vSphere requirements.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your vCenter credentials, see Updating cluster credentials.

Example:

vCenter:
  credentials:
    fileRef:
      path: "my-config-folder/admin-creds.yaml"

vCenter.credentials.fileRef.entry

Required
Mutable string

The name of the credentials block, in your credentials configuration file, that holds the username and password of your vCenter user account.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your vCenter credentials, see Updating cluster credentials.

Example:

vCenter:
  credentials:
    fileRef:
      entry: "vcenter-creds"

vCenter.folder

Optional
Immutable string
Default: The datacenter-wide folder

The name of a vCenter folder where your admin cluster VMs will be located.

Example:

vCenter:
  folder: "my-folder"

vCenter.dataDisk

Required
Immutable string

Google Distributed Cloud creates a virtual machine disk (VMDK) to hold Kubernetes object data. The installer creates the VMDK for you, but you must provide a name for the VMDK in the vCenter.dataDisk field.

Example:

vCenter:
  dataDisk: "my-disk.vmdk"

If you are using a vSAN datastore, you must put the VMDK in a folder, and you must manually create the folder ahead of time. You could use govc to create a folder:

govc datastore.mkdir -namespace=true my-folder

Then set vCenter.dataDisk to the path of the VMDK, including the folder.

Example:

vDenter:
  dataDisk: "my-folder/my-disk.vmdk"

network

Required

This section holds information about your admin cluster network.

network.hostConfig

Optional

This section holds information about NTP servers, DNS servers, and DNS search domains used by the VMs that are your cluster nodes. If you are using the Seesaw load balancer, this information also applies to your Seesaw VMs.

network.hostConfig.dnsServers

Optional
Immutable array of strings

The addresses of DNS servers for the VMs.

Example:

network:
  hostConfig:
    dnsServers:
    - "172.16.255.1"
    - "172.16.255.2"

network.hostConfig.ntpServers

Optional
Immutable array of strings

The addresses of time servers for the VMs to use.

Example:

network:
  hostConfig:
    ntpServers:
    - "216.239.35.0"

network.hostConfig.searchDomainsForDNS

Optional
Immutable array of strings

DNS search domains for the VMs to use. These domains are used as part of a domain search list.

Example:

network:
  hostConfig:
    searchDomainsForDNS:
    - "my.local.com"

network.ipMode.type

Required
Immutable string
Prepopulated: "dhcp"
Default: "dhcp"

If you want your cluster nodes to get their IP address from a DHCP server, set this to "dhcp". If you want your cluster nodes to have static IP addresses chosen from a list that you provide, set this to "static".

Example:

network:
  ipMode:
    type: "static"

network.serviceCIDR

Required
Immutable string
Smallest possible range: /24
Largest possible range: /12
Prepopulated: "10.96.232.0/24"
Default: "10.96.232.0/24"

A range of IP addresses, in CIDR format, to be used for Services in your cluster.

Example:

network:
  serviceCIDR: "10.96.232.0/24"

network.podCIDR

Required
Immutable string
Smallest possible range: /18
Largest possible range: /8
Prepopulated: "192.168.0.0/16"
Default: "192.168.0.0/16"

A range of IP addresses, in CIDR format, to be used for Pods in your cluster.

Example:

network:
  podCIDR: "192.168.0.0/16"

The Service range must not overlap with the Pod range.

The Service and Pod ranges must not overlap with any address outside the cluster that you want to reach from inside the cluster.

For example, suppose your Service range is 10.96.232.0/24, and your Pod range is 192.168.0.0/16. Any traffic sent from a Pod to an address in either of those ranges will be treated as in-cluster and will not reach any destination outside the cluster.

In particular, the Service and Pod ranges must not overlap with:

  • IP addresses of nodes in any cluster

  • IP addresses used by load balancer machines

  • VIPs used by control-plane nodes and load balancers

  • IP address of vCenter servers, DNS servers, and NTP servers

We recommend that your Service and Pod ranges be in the RFC 1918 address space.

Here is one reason for the recommendation to use RFC 1918 addresses. Suppose your Pod or Service range contains external IP addresses. Any traffic sent from a Pod to one of those external addresses will be treated as in-cluster traffic and will not reach the external destination.

network.vCenter.networkName

Required
Immutable string

The name of the vSphere network for your cluster nodes.

If the name contains a special character, you must use an escape sequence for it.

Special characters Escape sequence
Slash (/) %2f
Backslash (\) %5c
Percent sign (%) %25

If the network name is not unique, it is possible to specify a path to the network, such as /DATACENTER/network/NETWORK_NAME.

Example:

network:
  vCenter:
    networkName: "my-network"

loadBalancer

This section holds information about the load balancer for your admin cluster.

loadBalancer.vips.controlPlaneVIP

Required
Immutable string

The IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the admin cluster.

Example:

loadBalancer:
  vips:
    controlplaneVIP: "203.0.113.3"

loadBalancer.vips.addonsVIP

Optional
Immutable string

The IP address you have chosen to configure on the load balancer for add-ons.

Example:

loadBalancer:
  vips:
    addonsVIP: "203.0.113.4"

loadBalancer.kind

Required
Immutable string

String. Set this to "ManualLB", "F5BigIP", "Seesaw", or "MetalLB"

Example:

loadBalancer:
  kind: "MetalLB"

loadBalancer.manualLB

If you set loadbalancer.kind to "ManualLB", fill in this section. Otherwise, remove this section.

loadBalancer.manualLB.ingressHTTPNodePort

Remove this field from your configuration file. It is not used in an admin cluster.

loadBalancer.manualLB.ingressHTTPSNodePort

Remove this field from your configuration file. It is not used in an admin cluster.

loadBalancer.manualLB.konnectivityServerNodePort

Remove this field from your configuration file. It is not used in an admin cluster.

loadBalancer.manualLB.controlPlaneNodePort

Required if loadBalancer.kind = "ManualLB"
Immutable integer

The Kubernetes API server in the admin cluster is exposed by a Kubernetes Service. You must choose a nodePort value for the Service.

Set this field to the nodePort value.

For example:

loadBalancer:
  manualLB:
    contolPlaneNodePort: 30968

loadBalancer.manualLB.addonsNodePort

Required if loadBalancer.kind = "ManualLB"
Immutable integer

The add-ons server in the admin cluster is exposed by a Kubernetes Service. You must choose a nodePort value for the Service.

Set this field to the nodePort value.

Example:

loadBalancer:
  manualLB:
    addonsNodePort: 31405

loadBalancer.f5BigIP

If you set loadbalancer.kind to "f5BigIP", fill in this section. Otherwise, remove this section.

loadBalancer.f5BigIP.address

Required if loadBalancer.kind = "f5BigIp"
Immutable string

The address of your F5 BIG-IP load balancer. For example:

loadBalancer:
  f5BigIP:
    address: "203.0.113.2"

loadBalancer.f5BigIP.credentials.fileRef.path

Required if loadBalancer.kind = "f5BigIp"
Mutable string

The path of a credentials configuration file that holds the username and password of an account that Google Distributed Cloud can use to connect to your F5 BIG-IP load balancer.

The user account must have a user role that has sufficient permissions to set up and manage the load balancer. Either the Administrator role or the Resource Administrator role is sufficient.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your F5 BIG-IP credentials, see Updating cluster credentials.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        path: ""my-config-folder/admin-creds.yaml"

loadBalancer.f5BigIP.credentialsfileRef.entry

Required if loadBalancer.kind = "f5BigIp"
Immutable string

The name of the credentials block, in your credentials configuration file, that holds the username and password of your F5 BIG-IP account.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your F5 BIG-IP credentials, see Updating cluster credentials.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        entry: "f5-creds"

loadBalancer.f5BigIP.partition

Required if loadBalancer.kind = "f5BigIp"
Immutable string

The name of a BIG-IP partition that you created for your admin cluster.

Example:

loadBalancer:
  f5BigIP:
    partition: "my-f5-admin-partition"

loadBalancer.f5BigIP.snatPoolName

Optional
Relevant if loadBalancer.kind = "f5BigIp"
Immutable string

If you are using SNAT, the name of your SNAT pool. If you are not using SNAT, remove this field.

Example:

loadBalancer:
  f5BigIP:
    snatPoolName: "my-snat-pool"

loadBalancer.seesaw

If you set loadbalancer.kind to "Seesaw", fill in this section. Otherwise, remove this section.

For information on setting up the Seesaw load balancer, see Seesaw load balancer quickstart and Bundled load balancing with Seesaw.

loadBalancer.seesaw.ipBlockFilePath

Required if loadBalancer.kind = "Seesaw"
Immutable string

The path of the IP block file for your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    ipBlockFilePath: "config-folder/admin-seesaw-ipblock.yaml"

loadBalancer.seesaw.vrid

Required if loadBalancer.kind = "Seesaw"
Immutable integer
Possible values: 1 - 255

The virtual router identifier of your Seesaw VM. This identifier, which is an integer of your choice, must be unique in a VLAN.

Example:

loadBalancer:
  seesaw:
    vrid: 125

loadBalancer.seesaw.masterIP

Required if loadBalancer.kind = "Seesaw"
Immutable integer

String. The virtual IP address configured on your Master Seesaw VM.

Example:

loadBalancer:
  seesaw:
    masterIP: 172.16.20.21

loadBalancer.seesaw.cpus

Required if loadBalancer.kind = "Seesaw"
Immutable integer
Prepopulated: 2
Default:

The number of CPUs for each of your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    cpus: 8

loadBalancer.seesaw.memoryMB

Required if loadBalancer.kind = "Seesaw"
Immutable integer
Prepopulated: 3072 Default: 1024

The number of mebibytes of memory for each of your Seesaw VM.

Example:

loadBalancer:
  seesaw:
    memoryMB: 8192

loadBalancer.seesaw.vCenter.networkName

Optional
Relevant if loadBalancer.kind = "Seesaw"
Immutable string

The name of the vCenter network that contains your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    vCenter:
      networkName: "my-seesaw-network"

loadBalancer.seesaw.enableHA

Optional
Relevant if loadBalancer.kind = "Seesaw"
Immutable boolean
Prepopulated: false
Default: false

If you want to create a highly-available (HA) Seesaw load balancer, set this to true. Otherwise set this to false. An HA Seesaw load balancer uses a (Master, Backup) pair of VMs.

Example:

loadBalancer:
  seesaw:
    enableHA: true

loadBalancer.seesaw.disableVRRPMAC

Optional
Relevant if loadBalancer.kind = "Seesaw"
Immutable boolean
Prepopulatd: false
Default: false

If you set this to true, the Seesaw load balancer does not use MAC learning for failover. Instead, it uses gratuitous ARP. If you set this to false, the Seesaw load balancer uses MAC learning. We recommend that you set this to true. If you are using vSphere 7 or later, and you have a high-availability Seesaw load balancer, then you must set this to true.

Example:

loadBalancer:
  seesaw:
    disableVRRPMAC: true

antiAffinityGroups.enabled

Required
Mutable boolean
Prepopulated: true

Set this to true to enable DRS rule creation. Otherwise, set this to false.

Example:

antiAffinityGroups:
  enabled: true

Google Distributed Cloud automatically creates VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your admin cluster's nodes, causing them to be spread across at least three physical hosts in your datacenter.

This feature requires that your vSphere environment meets the following conditions:

  • VMware DRS is enabled. VMware DRS requires vSphere Enterprise Plus license edition.

  • Your vSphere user account has the Host.Inventory.Modify cluster privilege.

  • There are at least three physical hosts available.

Recall that if you have a vSphere Standard license, you cannot enable VMware DRS.

If you do not have DRS enabled, or if you do not have at least three hosts where vSphere VMs can be scheduled, set antiAffinityGroups.enabled to false.

adminMaster

Preview

If you want to specify CPU and memory for the control-plane node of the admin cluster, fill in this section. Otherwise, remove this section or leave it commented out.

adminMaster.cpus

Preview
Optional
Immutable integer
Prepopulated 4
Default: 4

The number of vCPUs for the control-plane node of the admin cluster.

Example:

adminMaster:
  cpus: 4

adminMaster.memoryMB

Preview
Optional
Immutable integer
Prepopulated 16384
Default: 16384

The number of mebibytes of memory for the control-plane node of the admin cluster.

Example:

adminMaster:
  memoryMB: 16384

addonNode.autoResize.enabled

Preview
Optional
Mutable boolean
Prepopulated: false
Default: false

Set this to true to enable automatic resizing of the add-on nodes in the admin cluster. Othersise set it to false.

To update the value of this field, use gkectl update admin.

Example:

addonNode:
  autoResize:
    enabled: true

connectivity

Optional
Immutable string
Possible value: "connected"
Prepopulated: "connected"
Default: "connected"

Specify the connectivity to Google Cloud. The only possible value for connectivity is "connected".

Example:

connectivity: "connected"

proxy

If your network is behind a proxy server, fill in this section. Otherwise, remove this section or leave it commented out.

proxy.url

Optional
Immutable string

The HTTP address of your proxy server. Include the port number even if it's the same as the scheme's default port.

Example:

proxy:
  url: "http://my-proxy.example.local:80"

The proxy server you specify here is used by your Google Distributed Cloud clusters. Also, your admin workstation is automatically configured to use this same proxy server unless you set the HTTPS_PROXY environment variable on your admin workstation.

If you specify proxy.url, you must also specify proxy.noProxy.

After the proxy configuration for the admin cluster has been set, it cannot be modified or deleted, unless the cluster is rebuilt.

proxy.noProxy

Optional
Immutable string

A comma-separated list of IP addresses, IP address ranges, host names, and domain names that should not go through the proxy server. When Google Distributed Cloud sends a request to one of these addresses, hosts, or domains, the request is sent directly.

Example:

proxy:
  noProxy: "10.151.222.0/24,my-host.example.local,10.151.2.1"

privateRegistry

If you have a private Docker registry, fill in this section. Otherwise, remove this section or leave it commented out.

The settings you select in the privateregistry section are reflected not only to the admin cluster, but also to the user clusters.

privateRegistry.address

Required for private registry
Immutable string

The IP address or FQDN (Fully Qualified Domain Name) of the machine that runs your private Docker registry.

Examples:

privateRegistry:
  address: "203.0.113.10"

privateRegistry:
  address: "fqdn.example.com"

privateRegistry.credentials.fileRef.path

Required for private registry
Immutable string

The path of a credentials configuration file that holds the username and password of an account that Google Distributed Cloud can use to access your private Docker registry.

Example:

privateRegistry:
  credentials:
    fileRef:
      path: "my-config-folder/admin-creds.yaml"

privateRegistry.credentials.fileRef.entry

Required for private registry
Immutable string

The name of the credentials block, in your credentials configuration file, that holds the username and password of your private Docker registry account.

privateRegistry:
  credentials:
    fileRef:
      entry: "private-registry-creds"

privateRegistry.caCertPath

Required for private registry
Immutable string

When Docker pulls an image from your private registry, the registry must prove its identity by presenting a certificate. The registry's certificate is signed by a certificate authority (CA). Docker uses the CA's certificate to validate the registry's certificate.

Set this field to the path of the CA's certificate.

Example:

privateRegistry:
  caCertPath: "my-cert-folder/registry-ca.crt"

componentAccessServiceAccountKeyPath

Required
Immutable string

The path of the JSON key file for your component access service account.

Example:

componentAccessServiceAccountKeyPath: "my-key-folder/access-key.json"

gkeConnect

Optional.

The gkeConnect section, which allows you to register an admin cluster, is a generally available feature for cluster creation, but a preview feature for cluster update.

This section holds information about the Google Cloud project and service account you want to use to register your cluster to a Google Cloud fleet.

gkeConnect.projectID

Required for registration
Immutable string

The ID of your fleet host project.

Example:

gkeConnect:
  projectID: "my-fleet-host-project-123"

gkeConnect.registerServiceAccountKeyPath

Required for registration
Immutable string

The path of the JSON key file for your connect-register service account.

Example:

gkeConnect:
  registerServiceAccountKeyPath: "my-key-folder/connect-register-key.json"

stackdriver

Optional

If you want to enable Cloud Logging and Cloud Monitoring for your cluster, fill in this section. Otherwise remove this section.

stackdriver.projectID

Required for Logging and Monitoring
Immutable string

The ID of your logging-monitoring project. This is the Google Cloud project where you will view logs and metrics.

Example:

stackdriver:
  projectID: "my-logs-project"

stackdriver.clusterLocation

Required for Logging and Monitoring
Immutable string
Prepopulated: "us-central1"

The Google Cloud region where you want to store logs. It is a good idea to choose a region that is near your on-premises data center.

Example:

stackdriver:
  clusterLocation: "us-central1"

stackdriver.enableVPC

Optional
Immutable boolean
Prepopulated: false

If your cluster's network is controlled by a VPC, set this totrue. This ensures that all telemetry flows through Google's restricted IP addresses. Otherwise, set this to false.

Example:

stackdriver:
  enableVPC: false

stackdriver.serviceAccountKeyPath

Required for Logging and Monitoring
Mutablee string

The path of the JSON key file for your logging-monitoring service account.

For information on how to update this field in an existing cluster, see Rotating service account keys.

Example:

stackdriver:
  serviceAccountKeyPath: "my-key-folder/log-mon-key.json"

stackdriver.disableVsphereResourceMetrics

Optional
Relevant for Logging and Monitoring
Immutable boolean
Prepopulated: false
Default: false

Set this to true to disable the collection of metrics from vSphere. Otherwise, set it to false.

Example:

stackdriver:
  disableVsphereResourceMetrics: true

cloudAuditLogging

Preview

If you want to integrate the audit logs from your cluster's Kubernetes API server with Cloud Audit Logs, fill in this section. Otherwise, remove this section or leave it commented out.

cloudAuditLogging.projectID

Preview
Required for Cloud Audit Logs
Immutable string

The project ID of the Google Cloud project where you want to store audit logs.

Example:

cloudAuditLogging:
  projectID: "my-audit-project"

cloudAuditLogging.clusterLocation

Preview
Required for Cloud Audit Logs
Immutable string

The Google Cloud region where you want to store audit logs. It is a good idea to choose a region that is near your on-premises data center

Example:

cloudAuditLogging:
  clusterLocation: "us-central1"

cloudAuditLogging.serviceAccountKeyPath

Preview
Required for Cloud Audit Logs
Mutable string

The path of the JSON key file for your audit-logging service account.

For information on how to update this field in an existing cluster, see Rotating service account keys.

Example:

cloudAuditLogging:
  serviceAccountKeyPath: "my-key-folder/audit-log-key.json"

clusterBackup.datastore

Preview
Optional
Immutable string

If you want to enable backing up of the admin cluster, set this to the vSphere datastore where you want to save cluster backups.

Example:

clusterBackup:
  datastore: "my-datastore"

autoRepair.enabled

Required
Mutable boolean
Prepopulated: true

Set this to true to enable node auto repair. Otherwise, set this to false.

To update the value of this field, use gkectl update admin.

Example:

autoRepair:
  enabled: true

secretsEncryption

If you want to encrypt Secrets without the need for an external KMS (Key Management Service), or any other dependencies, fill in this section. Otherwise, remove this section or leave it commented out.

secretsEncryption.mode

Required for Secrets encryption
Immutable string
Possible value: "GeneratedKey"
Prepopulated: "GeneratedKey"

The Secret encryption mode.

secretsEncryption:
  mode: "GeneratedKey"

secretsEncryption.generatedKey.keyVersion

Required for Secrets encryption
Immutable integer
Prepopulated: 1

An integer of your choice to use for the key version number. We recommended that you start with 1.

Example:

secretsEncryption:
  generatedKey:
    keyVersion: 1

secretsEncryption.generatedKey.disabled

Required for Secrets encryption
Mutable boolean
Prepopulated: false

Set this to true to disable Secrets encryption. Otherwise set it to false.

Example:

secretsEncryption:
  generatedKey:
    disabled: false

osImageType

Optional
Immutable string
Possible values: "ubuntu", "ubuntu_containerd", or "cos"
Prepopulated: "ubuntu_containerd"
Default: "ubuntu_containerd"

The type of OS image to run on your admin cluster nodes.

Example:

osImageType: "cos"