User cluster configuration file

This page describes the fields in the user cluster configuration file.

Generating a template for your configuration file

If you used gkeadm to create your admin workstation, then gkeadm generated a template for your user cluster configuration file. And gkeadm filled in some of the fields for you.

If you did not use gkeadm to create your admin workstation, you can use gkectl to generate a template for your user cluster configuration file.

To generate a template for your user cluster configuration file:

gkectl create-config cluster --config=OUTPUT_FILENAME

Replace OUTPUT_FILENAME with a path of your choice for the generated template. If you omit this flag, gkectl names the file user-cluster.yaml and puts it in the current directory.

Template

Filling in your configuration file

In your configuration file, enter field values as described in the following sections.

name

String. A name of your choice for your user cluster. For example:

name: "my-user-cluster"

gkeOnPremVersion

String. The GKE on-prem version for your user cluster. For example:

gkeOnPremVersion: 1.5.2-gke.3

vCenter

If you want all aspects of your vCenter environment to be the same as what you specified for your admin cluster, remove this section or leave it commented out.

If you want some aspects of your vCenter environment to be different from what you specified for your admin cluster, fill in the relevant fields in this section. Any fields that you set here in the vCenter section override the corresponding fields in your admin cluster configuration file.

vCenter.resourcePool

String. The name of the vCenter resource pool for your user cluster. If you are using a non-default resource pool, provide the name of your vCenter resource pool. For example:

vCenter:
  resourcePool: "MY-USER-POOL"

If you are using the default resource pool, provide the following value:

vCenter:
  resourcePool: "VCENTER_CLUSTER/Resources"

Replace VCENTER_CLUSTER with the name of your vCenter cluster.

See Specifying the root resource pool for a standalone host.

vCenter.datastore

String. The name of the vCenter datastore for your user cluster. For example:

vCenter:
  datastore: "MY-USER-DATASTORE"

vCenter.caCertPath

String. When a client, like GKE on-prem, sends a request to your vCenter server, the server must prove its identity to the client by presenting a certificate or a certificate bundle. To verify the certificate or bundle, GKE on-prem must have the root certificate in the chain of trust.

Set vCenter.caCertPath to the path of the root certificate. For example:

vCenter:
  caCertPath: "/usr/local/google/home/me/certs/user-vcenter-ca-cert.pem"

Your VMware installation has a certificate authority (CA) that issues a certificate to your vCenter server. The root certificate in the chain of trust is a self-signed certificate created by VMware.

If you do not want to use the VMWare CA, which is the default, you can configure VMware to use a different certificate authority.

If your vCenter server uses a certificate issued by the default VMware CA, download the certificate as follows:

curl -k "https://SERVER_ADDRESS/certs/download.zip" > download.zip

Replace SERVER_ADDRESS with the address of your vCenter server.

Install the unzip command and unzip the certificate file:

sudo apt-get install unzip
unzip downloads.zip

If the unzip command doesn't work the first time, enter the command again.

Find the certificate file in certs/lin.

vCenter.credentials.fileRef.path

String. The path of a credentials configuration file that holds the username and password of your vCenter user account. The user account should have the Administrator role or equivalent privileges. See vSphere requirements. For example:

vCenter:
  credentials:
    fileRef:
      path: "my-config-directory/user-creds.yaml"

vCenter.credentials.fileRef.entry

String. The name of the credentials block, in your credentials configuration file, that holds the username and password of your vCenter user account. For example:

vCenter:
  credentials:
    fileRef:
      entry: "vcenter-creds"

enableDataplaneV2

Boolean. If you want to enable Dataplane V2 (Beta), set this to true. Otherwise set this to false. For example:

enableDataplaneV2: true

network

This section holds information about your user cluster network.

network.hostConfig

This section holds information about NTP servers, DNS servers, and DNS search domains used by your cluster.

If you provided a value for one or both of the following fields, then fill in this section. Otherwise, remove this section.

  • loadBalancer.seesaw.ipBlockFilePath
  • network.ipMode.ipBlockFilePath

network.hostConfig.dnsServers

Array of strings. The addresses of DNS servers for the hosts to use. For example:

network:
  hostConfig:
    dnsServers:
    - "172.16.255.1"
    - "172.16.255.2"

network.hostConfig.ntpServers

Array of strings. The addresses of time servers for the hosts to use. For example:

network:
  hostConfig:
    ntpServers:
    - "216.239.35.0"

network.hostConfig.searchDomainsForDNS

Array of strings. DNS search domains for the hosts to use. These domains are used as part of a domain search list. For example:

network:
  hostConfig:
    searchDomainsForDNS:
    - "my.local.com"

network.ipMode.type

String. If you want your cluster nodes to get their IP address from a DHCP server, set this to "dhcp". If you want your cluster nodes to have static IP addresses chosen from a list that you provide, set this to "static". For example:

network:
  ipMode:
    type: "static"

network.ipMode.ipBlockFilePath

If you set ipMode.type to "static", fill in this field. If you set ipMode.type to "dhcp", remove this field.

String. The path of the IP block file for your cluster. For example:

network:
  ipMode:
    ipBlockFilePath: "/my-config-folder/user-cluster-ipblock.yaml"

network.serviceCIDR and network.podCiDR

Strings. Your user cluster must have a range of IP addresses to use for Services and a range of IP addresses to use for Pods. These ranges are specified by the network.serviceCIDR and network.podCIDR fields. These fields are populated with default values. If you like, you can change the populated values to values of your choice.

The Service and Pod ranges must not overlap. Also, the Service and Pod ranges must not overlap with IP addresses that are used for nodes in any cluster.

Example:

network:
  serviceCIDR: "10.96.232.0/24"
  podCIDR: "192.168.0.0/16"

network.vCenter.networkName

String. The name of the vSphere network for your user cluster nodes.

If the name contains a special character, you must use an escape sequence for it.

Special characters Escape sequence
Slash (/) %2f
Backslash (\) %5c
Percent sign (%) %25

If the network name is not unique, it is possible to specify a path to the network, such as /DATACENTER/network/NETWORK_NAME.

For example:

network:
  vCenter:
    networkName: "MY-USER-CLUSTER-NETWORK"

loadBalancer

This section holds information about the load balancer for your user cluster.

loadBalancer.vips.controlPlaneVIP

The IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the admin cluster. For example:

loadBalancer:
  vips:
    controlplaneVIP: "203.0.113.3"

loadBalancer.vips.ingressVIP

The IP address that you have chosen to configure on the load balancer for ingress traffic. For example:

loadBalancer:
  vips:
    ingressVIP: "203.0.113.4"

loadBalancer.kind

String. Set this to "Seesaw", "F5BigIP", or "ManualLB" For example:

loadBalancer:
  kind: "Seesaw"

loadBalancer.manualLB

If you set loadbalancer.kind to "manualLB", fill in this section. Otherwise, remove this section or leave it commented out.

loadBalancer.manualLB.ingressHTTPNodePort

Integer. The ingress service in a user cluster is implemented as a Kubernetes Services of type LoadBalancer. The Service has a ServicePort for HTTP. You must choose a nodePort value for the HTTP ServicePorts.

Set this field to the nodePort value. For example:

loadBalancer:
  manualLB:
    ingressHTTPNodePort: 32527

loadBalancer.manualLB.ingressHTTPSNodePort

Integer. The ingress service in a user cluster is implemented as a Service of type LoadBalancer. The Service has a ServicePort for HTTPS. You must choose a nodePort value for the HTTPS ServicePort.

Set this field to the nodePort value. For example:

loadBalancer:
  manualLB:
    ingressHTTPSNodePort: 30139

loadBalancer.manualLB.controlPlaneNodePort

Integer. The Kubernetes API server in the admin cluster is implemented as a Service of type NodePort. You must choose a nodePort value for the Service.

Set this field to the nodePort value. For example:

loadBalancer:
  manualLB:
    controlPlaneNodePort: 30968

loadBalancer.manualLB.addonsNodePort

Remove this field. It is not used in a user cluster.

loadBalancer.f5BigIP

If you set loadbalancer.kind to "f5BigIP", fill in this section. Otherwise, remove this section or leave it commented out.

loadBalancer.f5BigIP.address

String. The address of your F5 BIG-IP load balancer. For example:

loadBalancer:
  f5BigIP:
      address: "203.0.113.2"

loadBalancer.f5BigIP.fileRef.path

String. The path of a credentials configuration file that holds the username and password of an account that GKE on-prem can use to connect to your F5 BIG-IP load balancer. For example:

loadBalancer:
  f5BigIP:
    fileRef:
      path: ""my-config-folder/user-creds.yaml"

loadBalancer.f5BigIP.fileRef.entry

String. The name of the credentials block, in your credentials configuration file, that holds the username and password of your F5 BIG-IP account. For example:

loadBalancer:
  f5BigIP:
    fileRef:
      entry: "f5-creds"

loadBalancer.f5BigIP.partition

String. The name of a BIG-IP partition that you created for your admin cluster. For example:

loadBalancer:
  f5BigIP:
    partition: "my-f5-admin-partition"

loadBalancer.f5BigIP.snatPoolName

String. If you are using SNAT, the name of your SNAT pool. If you are not using SNAT, remove this field or leave it commented out. For example:

loadBalancer:
  f5BigIP:
    snatPoolName: "my-snat-pool"

loadBalancer.seesaw

If you set loadbalancer.kind to "Seesaw", fill in this section. Otherwise, remove this section or leave it commented out.

loadBalancer.seesaw.ipBlockFilePath

String. Set this to the path of the IP block file for your Seesaw VM. For example:

loadbalancer:
  seesaw:
    ipBlockFilePath: "user-seesaw-ipblock.yaml"

loadBalancer.seesaw.vird

Integer. The virtual router identifier of your Seesaw VM or VM pair. This identifier must be unique in a VLAN. Valid range is 1-255. For example:

loadBalancer:
  seesaw:
    vrid: 125

loadBalancer.seesaw.masterIP

String. A VIP of your choice that your control plane Seesaw VM will advertise. For example:

loadBalancer:
  seesaw:
    masterIP: 172.16.20.21

loadBalancer.seesaw.cpus

Integer. The number of CPUs for each of your Seesaw VMs. For example:

loadBalancer:.
  seesaw:
    cpus: 8

loadBalancer.seesaw.memoryMB

Integer. The number of megabytes of memory for each of your Seesaw VMs. For example:

loadBalancer:.
  seesaw:
    memoryMB: 8192

loadBalancer.seesaw.vCenter.networkName

String. The name of the network that contains your Seesaw VMs. For example:

loadBalancer:
  seesaw:
    vCenter:
      networkName: "my-seesaw-network"

loadBalancer.seesaw.enableHA

Boolean. If you want to create a highly-available Seesaw load balancer, set this to true. Otherwise set this to false. For example:

loadBalancer:.
  seesaw:
    enableHA: true

masterNode

This section holds information about the nodes, in the admin cluster, that serve as control plane nodes for your user cluster.

masterNode.cpus

Integer. The number of CPUs for each admin cluster node that serve as control planes for this user cluster. For example:

masterNode:
  cpus: 8

masterNode.memoryMB

Integer. The megabytes of memory for each admin cluster node that serves as a control plane for this user cluster. For example:

masterNode:
  memoryMB: 8192

masterNode.replicas

Integer. The number of control plane nodes for this user cluster. Set this field to 1 or 3. For example:

masterNode:
  replicas: 3

nodePools

Array of objects, each of which describes a node pool.

nodePools[i].name

String. A name of your choice for the node pool. For example:

nodePools:
- name: "my-node-pool"

nodePools[i].cpus

Integer. The number of CPUs for each node in the pool. For example:

nodePools"
- name: "my-node-pool"
  cpus: 8

nodePools[i].memoryMB

Integer. The megabytes of memory for each node in the pool. For example:

nodePools"
- name: "my-node-pool"
  memoryMB: 8192

nodePools[i].replicas

Integer. The number of nodes in the pool. For example:

nodePools:
- name: "my-node-pool"
  replicas: 5

noodePools[i].labels

Mapping. Labels to apply to each node in the pool. For example:

nodePools:
- name: "my-node-pool"
  labels:
    environment: "production"
    tier: "cache"

nodePools[i].taints

Array of objects, each of which describes a taint. For example:

nodePools:
- name: "my-node-pool"
  taints:
  - key: "staging"
    value: "true"
    effect: "NoSchedule"

nodePools[i].bootDiskSizeGB

Integer. The size of boot disk in gigabytes for each node in the pool. This configuration is available starting from GKE on-prem version 1.5.0. For example:

nodePools"
- name: "my-node-pool"
  bootDiskSizeGB: 40

nodePools[i].vsphere.datastore

String. The name of the vCenter datastore on which each node in the pool will be created. For example:

nodePools:
- name: "my-node-pool"
  vsphere:
    datastore: "my-datastore"

antiAffinityGroups.enabled

Boolean. Set this to true to enable DRS rule creation. Otherwise, set this to false. For example:

antiAffinityGroups:
  enabled: true

GKE on-prem automatically creates VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your user cluster's nodes, causing them to be spread across at least three physical hosts in your datacenter.

This feature requires that your vSphere environment meets the following conditions:

  • VMware DRS is enabled. VMware DRS requires vSphere Enterprise Plus license edition.

  • Your vSphere user account has the Host.Inventory.Modify cluster privilege.

  • There are at least three physical hosts available.

Recall that if you have a vSphere Standard license, you cannot enable VMware DRS.

If you do not have DRS enabled, or if you do not have at least three hosts where vSphere VMs can be scheduled, set antiAffinityGroups.enabled to false.

authentication

This section holds information about how cluster users are authenticated and authorized.

authentication.oidc

If you want to use OpenID Connect (OIDC) to manage access to this cluster, fill in this section. Otherwise, remove this section or leave it commented out.

authentication.oidc.issuerURL

String. The URL of your OpenID provider. Client applications, like the gcloud CLI and Google Cloud console, send authorization requests to this URL. The Kubernetes API server uses this URL to discover public keys for verifying tokens. Must use HTTPS. For example:

authentication:
  oidc:
    issuerURL: "https://example.com/adfs"

authentication.oidc.kubectlRedirectURL

String. The redirect URL for the gcloud CLI. For example:

authentication:
  oidc:
    kubectlRedirectURL: "https://localhost:1025/callback"

authentication.oidc.clientID

String. ID for the client application that makes authentication requests to the OpenID provider. Both the gcloud CLI and Google Cloud console use this ID. For example:

authentication:
  oidc:
    clientID: "my-big-hex-string"

authentication.oidc.clientSecret

String. Secret for the client application. Both the gcloud CLI and Google Cloud console use this secret. For example:

authentication:
  oidc:
    clientSecret: "N3i&JlLZoD!W"

authentication.oidc.username

String. The JWT claim to use as the user's name. The default is sub, which is expected to be a unique identifier of the end user. You can choose other claims, such as email or name, depending on the OpenID provider. However, claims other than email are prefixed with the issuer URL to prevent naming clashes with other plugins. For example:

authentication:
  oidc:
    username: "sub"

authentication.oidc.usernamePrefix

String. Prefix prepended to username claims to prevent clashes with existing names. If you do not provide this field, and username is a value other than email, the prefix defaults to issuerurl#. You can use the value - to disable all prefixing. For example:

authentication:
  oidc:
    usernamePrefix: "my-prefix"

authentication.oidc.group

String. The JWT claim that the provider will use to return your security groups. For example:

authentication:
  oidc:
    group: "sec-groups"

authentication.oidc.groupPrefix

String. Prefix prepended to group claims to prevent clashes with existing names. For example, given a group foobar and a prefix gid-, gid-foobar. By default, this value is empty, and there is no prefix. For example:

authentication:
  oidc:
    groupPrefix: "gid-"

authentication.oidc.scopes

String. A comma-delimited list of additional scopes to send to the OpenID provider. For example:

authentication:
  oidc:
    scopes: "offline-access"

For authentication with Microsoft Azure or Okta, set this to offline_access.

authentication.oidc.extraParams

String. A comma-delimited list of additional key-value parameters to send to the OpenID provider.

For example:

authentication:
  oidc:
    extraparams: "prompt=consent"

authentication.oidc.deployCloudConsoleProxy

String. Specifies whether to deploy a reverse proxy in the cluster to allow Google Cloud console access to the on-premises OIDC provider for authenticating users. Mmust be a string: "true" or "false". If your identity provider is not reachable over the public internet, and you wish to authenticate using Google Cloud console, then you must set this field to "true". If left blank, this field defaults to "false".

authentication.oidc.caPath

String. Path to the certificate for the certificate authority (CA) that issued your identity provider's web certificate. This value might not be necessary. For example, if your identity provider's certificate was issued by a well-known public CA, then you would not need to provide a value here. However, if deployCloudConsoleProxy is "true", then you must provide this value, even for a well-known public CA.

For example:

authentication:
  oidc:
    caPath: "my-cert-folder/provider-root-cert.pem"

authentication.sni

If you want to provide an additional serving certificate for the cluster's Kubernetes API server, fill in this section. Otherwise, remove this section or leave it commented out.

authentication.sni.certPath

String. The path to a serving certificate for the Kubernetes API server. For example:

authentication:
  sni:
    certPath: "my-cert-folder/example.com.crt"

authentication.sni.keyPath

String. Path to the certificate's private key file. For example:

authentication:
  sni:
    keyPath: "my-cert-folder/example.com.key"

authentication.ldap

If you want to provide information about your LDAP provider, fill in this section. Otherwise, remove this section.

authentication.ldap.name

String. The name of your LDAP provider. For example:

authentication:
  ldap:
    name: "my-ldap-provider"

authentication.ldap.host

String. The hostname or IP address of your LDAP provider. For example:

authentication:
  ldap:
    host" "203.0.113.20"

authentication.ldap.connectionType

String. Set this to "insecure" to use an unsecured connection for LDAP. To use a secure connection, remove this field.

The default is to use LDAP over TLS. Set connectionType to"insecure" to change this and use an unsecured connection. If you set this to "insecure", there is no need to supply a value for ldap.caPath.

Example:

authentication:
  ldap:
    connectionType: "insecure"

authentication.ldap.caPath

String. The path of the root certificate for your LDAP provider. For example:

authentication:
  ldap:
    caPath: "my-cert-folder/ldap-cert.pem"

authentication.ldap.user.baseDN

String. The location for the user entries in the LDAP directory. For example:

authentication:
  ldap:
    user:
      baseDN: "DC=example-domain,DC=com"

authentication.ldap.user.userAttribute

String: The name of the attribute that precedes the user name in a DN. The default is "CN". For example:

authentication:
  ldap:
    user:
      userAttribute: "CN"

authentication.ldap.user.memberAttribute

String. The name of the attribute that records a user's group membership. The default is "memberOf". For example:

authentication:
  ldap:
    user:
      memberAttribute: "memberOf"

stackdriver

This section holds information about the Google Cloud project and service account you want to use for storing logs and metrics.

stackdriver.projectID

String. The ID of the Google Cloud project where you want to view logs. For example:

stackdriver:
  projectID: "my-logs-project"

stackdriver.clusterLocation

String. The Google Cloud region where you want to store logs. It is a good idea to choose a region that is near your on-premises data center. For example:

stackdriver:
  clusterLocation: "us-central1"

stackdriver.enableVPC

Boolean. If your cluster's network is controlled by a VPC, set this to this field totrue. This ensures that all telemetry flows through Google's restricted IP addresses. Otherwise, set this field to false. For example:

stackdriver:
  enableVPC: false

stackdriver.serviceAccountKeyPath

String. The path of the JSON key file for your logging-monitoring service account. For example:

stackdriver:
  serviceAccountKeyPath: "my-key-folder/log-mon-key.json"

stackdriver.disableVsphereResourceMetrics

Boolean. Set this to true to disable the collection of metrics from vSphere. Otherwise, set it to false. For example:

stackdriver:
  disableVsphereResourceMetrics: true

gkeConnect

This section holds information about the Google Cloud project and service accounts you want to use to connect your cluster to Google Cloud.

gkeConnect.projectID

String. The ID of the Google Cloud project that you want to use for connecting your cluster to Google Cloud. For example:

gkeConnect:
  projectID: "my-connect-project-123"

gkeConnect.registerServiceAccountKeyPath

String. The path of the JSON key file for your connect-register service account. For example:

gkeConnect:
  registerServiceAccountKeyPath: "my-key-folder/connect-register-key.json"

gkeConnect.agentServiceAccountKeyPath

String. The path of the JSON key file for your connect-agent service account. For example:

gkeConnect:
  agentServiceAccountKeyPath: "my-key-folder/connect-agent-key.json"

cloudRun.enabled

Boolean. Set this to true if you want to enable Cloud Run. Otherwise set this to false. For example:

cloudRun:
  enabled: true

usageMetering

If you want to enable usage metering for your cluster, then fill in this section. Otherwise, remove this section or leave it commented out.

usageMetering.bigQueryProjectID

String. The ID of the Google Cloud project where you want to store usage metering data. For example:

usageMetering:
  bigQueryProjectID: "my-bq-project"

usageMetering.bigQueryDatasetID

String. The ID of the BigQuery dataset where you want to store usage metering data. For example:

usageMetering:
  bigQueryDatasetID: "my-bq-dataset"

usageMetering.bigQueryServiceAccountKeyPath

String. The path of the JSON key file for your BigQuery service account. For example:

usageMetering:
  bigQueryServiceAccountKeyPath: "my-key-folder/bq-key.json"

usageMetering.enableConsumptionMetering

Boolean. Set this to true if you want to enable consumption-based metering. Otherwise set this to false. For example:

usageMetering:
  enableConsumptionMetering: true

cloudAuditLogging

If you want to integrate the audit logs from your cluster's Kubernetes API server with Cloud Audit Logs, fill in this section. Otherwise, remove this section or leave it commented out.

cloudAuditLogging.projectid`

String. The project ID of the Google Cloud project where you want to store audit logs. For example:

cloudAuditLogging:
  projectid: "my-audit-project"

cloudAuditLogging.clusterlocation

String. The Google Cloud region where you want to store audit logs. It is a good idea to choose a region that is near your on-premises data center. For example:

cloudAuditLogging:
  clusterlocation: "us-central1"

cloudAuditLogging.serviceaccountkeypath

String. The path of the JSON key file for your audit-logging service account. For example:

cloudAuditLogging:
  serviceaccountkeypath: "my-key-folder/audit-log-key.json"

autoRepair.enabled

Boolean. Set this to true to enable node auto repair. Otherwise, set it to false. For example:

autoRepair:
  enabled: true