Admin cluster configuration file

This page describes the fields in the admin cluster configuration file for Google Distributed Cloud.

Generating a template for your configuration file

If you used gkeadm to create your admin workstation, then gkeadm generated a template for your admin cluster configuration file. Also, gkeadm filled in some of the fields for you.

If you did not use gkeadm to create your admin workstation, you can use gkectl to generate a template for your admin cluster configuration file.

To generate a template for your admin cluster configuration file:

gkectl create-config admin --config=OUTPUT_FILENAME --gke-on-prem-version=VERSION

Replace OUTPUT_FILENAME with a path of your choice for the generated template. If you omit this flag, gkectl names the file admin-cluster.yaml and puts it in the current directory.

Replace VERSION with the desired version number, which must be equal to or less than your gkectl version. For example: gkectl create-config admin --gke-on-prem-version=1.6.2-gke.0. If you omit this flag, the generated config template is populated with values based on the latest cluster version.

Template

Filling in your configuration file

In your configuration file, enter field values as described in the following sections.

name

Optional
String
Default: A random name with prefix "gke-admin-"

A name of your choice for the cluster.

Example:

name: "my-admin-cluster"

bundlePath

Required
Mutable
String

The path of your Google Distributed Cloud bundle file.

The Google Distributed Cloud full bundle file contains all of the components in a particular release of Google Distributed Cloud. When you create an admin workstation, it comes with a full bundle at:

/var/lib/gke/bundles/gke-onprem-vsphere-VERSION-full.tgz

Example:

bundlePath: "/var/lib/gke/bundles/gke-onprem-vsphere-1.10.0-gke.8.full.tgz"

preparedSecrets.enabled

Preview
Optional
Immutable
Boolean
Prepopulated: false
Default: false

Set this to true if you want to use prepared credentials in the admin cluster. Otherwise, set it to false.

Example:

preparedSecrets:
  enabled: true

vCenter

Required
Immutable

This sections holds information about your vSphere environment and your connection to vCenter Server.

vCenter.address

Required
Immutable
String

The IP address or the hostname of your vCenter server.

For more information, see Finding your vCenter server address.

Examples:

vCenter:
  address: "203.0.113.100"

vCenter:
  address: "my-vcenter-server.my-domain.example"

vCenter.datacenter

Required
Immutable
String

The relative path of a vSphere data center.

The value you specify is relative to the root folder named /.

If your data center is in the root folder, the value is the name of the data center.

Example:

vCenter:
  datacenter: "my-data-center"

Otherwise, the value is a relative path that includes one or more folders along with the name of the data center.

Example:

vCenter:
  datacenter: "data-centers/data-center-1"

vCenter.cluster

Required
Immutable
String

The relative path of a vSphere cluster that represents the ESXi hosts where your admin cluster VMs will run. This vSphere cluster represents a subset of the physical ESXi hosts in your vCenter data center.

The value you specify is relative to /.../DATA_CENTER/vm/.

If your vSphere cluster is in the /.../DATA_CENTER/vm/ folder, the value is the name of the vSphere cluster.

Example:

vCenter:
  cluster: "my-vsphere-cluster"

Otherwise, the value is a relative path that includes one or more folders along with the name of the vSphere cluster.

Example:

vCenter:
  cluster: "clusters/vsphere-cluster-1"

vCenter.resourcePool

Required
Immutable
String

A vCenter resource pool for your admin cluster VMs.

If you want to use the default resource pool, set this to VSPHERE_CLUSTER/Resources.

Example:

vCenter:
  resourcePool: "my-vsphere-cluster/Resources"

If you want to use a resource pool that you have already created, set this to the relative path of your resource pool.

The value you specify is relative to /.../DATA_CENTER/host/.../VSPHERE_CLUSTER/Resources/

If your resource pool is a direct child of /.../DATA_CENTER/host/.../VSPHERE_CLUSTER/Resources/ the value is the name of the resource pool.

Example:

vCenter:
  resourcePool: "my-resource-pool"

If your resource pool is nested under another resource pool, set this to the full path to your resource pool.

Example:

vCenter:
  resourcePool: "my-vsphere-cluster/Resources/parent-resource-pool/child-resource-pool"

vCenter.datastore

Required
Immutable
String

The name of a vSphere datastore for your admin cluster.

The value you specify must be a name, not a path. Do not include any folders in the value.

Example:

vCenter:
  datastore: "my-datastore"

vCenter.caCertPath

Required
Mutable
String

The path of the CA certificate for your vCenter server.

For more information, see Getting your vCenter CA certificate.

For information about updating this field for an existing cluster, see Update vCenter certificate references.

Example:

vCenter:
  caCertPath: "/usr/local/google/home/me/certs/vcenter-ca-cert.pem"

vCenter.credentials.fileRef.path

Required
String

The path of a credentials configuration file that holds the username and password of your vCenter user account. The user account should have the Administrator role or equivalent privileges. See vSphere requirements.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your vCenter credentials, see Updating cluster credentials.

Example:

vCenter:
  credentials:
    fileRef:
      path: "my-config-folder/admin-creds.yaml"

vCenter.credentials.fileRef.entry

Required
String

The name of the credentials block, in your credentials configuration file, that holds the username and password of your vCenter user account.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your vCenter credentials, see Updating cluster credentials.

Example:

vCenter:
  credentials:
    fileRef:
      entry: "vcenter-creds"

vCenter.folder

Optional
Immutable
String
Default: The datacenter-wide folder

The relative path of a vSphere folder that you have already created. This folder will hold your admin cluster VMs.

If you do not specify a value, your admin cluster VMs will be put in /.../DATA_CENTER/vm/.

If you specify a value, it is relative to /.../DATA_CENTER/vm/.

The value can be the name of a folder.

Example:

vCenter:
  folder: "my-folder"

If your vm folder is nested under another vm folder, set this to the full path to your vm folder.

Example:

vCenter:
  folder: "/my-datacenter/vm/vm-folder/nested-vm-folder"

Note that you must include the first "/" in your vm folder full path.

vCenter.dataDisk

Required
Immutable
String

Google Distributed Cloud creates a virtual machine disk (VMDK) to hold Kubernetes object data. The installer creates the VMDK for you, but you must provide a relative path for the VMDK in the vCenter.dataDisk field.

The value you specify is relative to /.../DATA_CENTER/datastore/.

If you want the VMDK to be in the /.../DATA_CENTER/datastore/ folder, the value is the name of the VMDK.

If you set adminMaster.replicas to 3, then you must NOT set this field. For HA admin cluster, the data disk paths for the 3 admin control plane machines will be auto generated under /anthos/ADMIN_CLUSTER/default/`.

Example:

vCenter:
  dataDisk: "my-disk.vmdk"

Otherwise, the value is a relative path that includes one or more folders along with the VMDK name.

Example:

vCenter:
  dataDisk: "data-disks/data-disk-1.vmdk"

If you are using a vSAN datastore, the VMDK must be in a folder that you have already created.

You can use govc to create a folder:

govc datastore.mkdir -namespace=true data-disks

network

Required
Immutable

This section holds information about your admin cluster network.

network.hostConfig

Immutable

This section is required if one or more of the following are true:

  • network.ipMode.type is set to "static"
  • loadBalancer.kind is set to "Seesaw"
  • adminMaster.replicas is set to 3

This section holds information about NTP servers, DNS servers, and DNS search domains used by the VMs that are your cluster nodes. If you are using the Seesaw load balancer, this information also applies to your Seesaw VMs.

network.hostConfig.dnsServers

Required if network.hostConfig section is filled in.
Immutable
Array of strings.
The maximum number of elements in the array is three.

The addresses of DNS servers for the VMs.

Example:

network:
  hostConfig:
    dnsServers:
    - "172.16.255.1"
    - "172.16.255.2"

network.hostConfig.ntpServers

Required if network.hostConfig section is filled in.
Immutable
Array of strings

The addresses of time servers for the VMs to use.

Example:

network:
  hostConfig:
    ntpServers:
    - "216.239.35.0"

network.hostConfig.searchDomainsForDNS

Optional
Immutable
Array of strings

DNS search domains for the VMs to use. These domains are used as part of a domain search list.

Example:

network:
  hostConfig:
    searchDomainsForDNS:
    - "my.local.com"

network.ipMode.type

Required
Immutable
String
Prepopulated: "dhcp"
Default: "dhcp"

If you want your cluster nodes to get their IP address from a DHCP server, set this to "dhcp". If you want your cluster nodes to have static IP addresses chosen from a list that you provide, set this to "static".

Example:

network:
  ipMode:
    type: "static"

network.ipMode.ipBlockFilePath

Required if network.ipMode.type = static
Immutable
String

The path of the IP block file for your cluster.

Example:

network:
  ipMode:
    ipBlockFilePath: "/my-config-folder/admin-cluster-ipblock.yaml"

network.serviceCIDR

Required
Immutable
String
Smallest possible range: /24
Largest possible range: /12
Prepopulated: "10.96.232.0/24"
Default: "10.96.232.0/24"

A range of IP addresses, in CIDR format, to be used for Services in your cluster.

Example:

network:
  serviceCIDR: "10.96.232.0/24"

network.podCIDR

Required
Immutable
String
Smallest possible range: /18
Largest possible range: /8
Prepopulated: "192.168.0.0/16"
Default: "192.168.0.0/16"

A range of IP addresses, in CIDR format, to be used for Pods in your cluster.

Example:

network:
  podCIDR: "192.168.0.0/16"

The Service range must not overlap with the Pod range.

The Service and Pod ranges must not overlap with any address outside the cluster that you want to reach from inside the cluster.

For example, suppose your Service range is 10.96.232.0/24, and your Pod range is 192.168.0.0/16. Any traffic sent from a Pod to an address in either of those ranges will be treated as in-cluster and will not reach any destination outside the cluster.

In particular, the Service and Pod ranges must not overlap with:

  • IP addresses of nodes in any cluster

  • IP addresses used by load balancer machines

  • VIPs used by control-plane nodes and load balancers

  • IP address of vCenter servers, DNS servers, and NTP servers

We recommend that your Service and Pod ranges be in the RFC 1918 address space.

Here is one reason for the recommendation to use RFC 1918 addresses. Suppose your Pod or Service range contains external IP addresses. Any traffic sent from a Pod to one of those external addresses will be treated as in-cluster traffic and will not reach the external destination.

network.vCenter.networkName

Required
Immutable
String

The name of the vSphere network for your cluster nodes.

Example:

network:
  vCenter:
    networkName: "my-network"

If the name contains a special character, you must use an escape sequence for it.

Special characters Escape sequence
Slash (/) %2f
Backslash (\) %5c
Percent sign (%) %25

If the network name is not unique in your data center, you can specify a full path.

Example:

network:
  vCenter:
    networkName: "/data-centers/data-center-1/network/my-network"

network.controlPlaneIPBlock

If you want to create a high-availability admin cluster, fill in this section. Otherwise, remove this section or leave it commented out.

A high-availability admin cluster has three nodes that run control-plane components.

If you fill in this section, you must also set adminMaster.replicas to 3.

network.controlPlaneIPBlock.netmask

Required for a high-availability control plane
Immutable
String

The netmask for the network that has your control-plane nodes.

Example:

network:
  controlPlaneIPBlock:
    netmask: "255.255.255.0"

network.controlPlaneIPBlock.gateway

Required for a high-availability control plane
Immutable
String

The IP address of the default gateway for your control-plane nodes.

Example:

network:
  controlPlaneIPBlock:
    gateway: "172.16.22.1"

network.controlPlaneIPBlock.ips

Required for a high-availability control plane
Immutable
Array of three objects, each of which has an IP address and an optional hostname.

These are the IP addresses that will be assigned to your control-plane nodes.

Example:

network:
  controlPlaneIPBlock:
    ips:
    - ip: "172.16.22.6"
      hostname: "admin-cp-vm-1"
    - ip: "172.16.22.7"
      hostname: "admin-cp-vm-2"
    - ip: "172.16.22.8"
      hostname: "admin-cp-vm-3"

loadBalancer

This section holds information about the load balancer for your admin cluster.

loadBalancer.vips.controlPlaneVIP

Required
Immutable
String

The IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the admin cluster.

Example:

loadBalancer:
  vips:
    controlplaneVIP: "203.0.113.3"

loadBalancer.kind

Required
Immutable
String
Prepopulated: "MetalLB"

String. Set this to "ManualLB", "F5BigIP", "Seesaw", or "MetalLB"

If you set adminMaster.replicas to 3, then you cannot use the Seesaw load balancer.

Example:

loadBalancer:
  kind: "MetalLB"

loadBalancer.manualLB

If you set loadbalancer.kind to "ManualLB", fill in this section. Otherwise, remove this section.
Immutable

loadBalancer.manualLB.ingressHTTPNodePort

Remove this field from your configuration file. It is not used in an admin cluster.

loadBalancer.manualLB.ingressHTTPSNodePort

Remove this field from your configuration file. It is not used in an admin cluster.

loadBalancer.manualLB.konnectivityServerNodePort

Remove this field from your configuration file. It is not used in an admin cluster.

loadBalancer.manualLB.controlPlaneNodePort

Required if loadBalancer.kind = "ManualLB" and adminMaster.replicas = 1
Immutable
Integer
Prepopulated: 30968

The Kubernetes API server in the admin cluster is exposed by a Kubernetes Service. You must choose a nodePort value for the Service.

Set this field to the nodePort value.

For example:

loadBalancer:
  manualLB:
    controlPlaneNodePort: 30968

loadBalancer.f5BigIP

If you set loadbalancer.kind to "f5BigIP", fill in this section. Otherwise, remove this section.

loadBalancer.f5BigIP.address

Required if loadBalancer.kind = "f5BigIp"
Immutable
String

The address of your F5 BIG-IP load balancer. For example:

loadBalancer:
  f5BigIP:
    address: "203.0.113.2"

loadBalancer.f5BigIP.credentials.fileRef.path

Required if loadBalancer.kind = "f5BigIp"
Mutable
String

The path of a credentials configuration file that holds the username and password of an account that Google Distributed Cloud can use to connect to your F5 BIG-IP load balancer.

The user account must have a user role that has sufficient permissions to set up and manage the load balancer. Either the Administrator role or the Resource Administrator role is sufficient.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your F5 BIG-IP credentials, see Updating cluster credentials.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        path: ""my-config-folder/admin-creds.yaml"

loadBalancer.f5BigIP.credentialsfileRef.entry

Required if loadBalancer.kind = "f5BigIp"
Mutable
String

The name of the credentials block, in your credentials configuration file, that holds the username and password of your F5 BIG-IP account.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your F5 BIG-IP credentials, see Updating cluster credentials.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        entry: "f5-creds"

loadBalancer.f5BigIP.partition

Required if loadBalancer.kind = "f5BigIp"
Immutable
String

The name of a BIG-IP partition that you created for your admin cluster.

Example:

loadBalancer:
  f5BigIP:
    partition: "my-f5-admin-partition"

loadBalancer.f5BigIP.snatPoolName

Optional
Relevant if loadBalancer.kind = "f5BigIp"
Immutable
String

If you are using SNAT, the name of your SNAT pool. If you are not using SNAT, remove this field.

Example:

loadBalancer:
  f5BigIP:
    snatPoolName: "my-snat-pool"

loadBalancer.seesaw

If you set loadbalancer.kind to "Seesaw", fill in this section. Otherwise, remove this section.
Immutable

For information on setting up the Seesaw load balancer, see Bundled load balancing with Seesaw.

loadBalancer.seesaw.ipBlockFilePath

Required if loadBalancer.kind = "Seesaw"
Immutable
String

The path of the IP block file for your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    ipBlockFilePath: "config-folder/admin-seesaw-ipblock.yaml"

loadBalancer.seesaw.vrid

Required if loadBalancer.kind = "Seesaw"
Immutable
Integer
Possible values: 1 - 255

The virtual router identifier of your Seesaw VM. This identifier, which is an integer of your choice, must be unique in a VLAN.

Example:

loadBalancer:
  seesaw:
    vrid: 125

loadBalancer.seesaw.masterIP

Required if loadBalancer.kind = "Seesaw"
Immutable
Integer

String. The virtual IP address configured on your Master Seesaw VM.

Example:

loadBalancer:
  seesaw:
    masterIP: 172.16.20.21

loadBalancer.seesaw.cpus

Required if loadBalancer.kind = "Seesaw"
Mutable
Integer
Prepopulated: 2
Default:

The number of CPUs for each of your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    cpus: 8

loadBalancer.seesaw.memoryMB

Required if loadBalancer.kind = "Seesaw"
Mutable
Integer
Prepopulated: 3072 Default: 1024

The number of mebibytes of memory for each of your Seesaw VM.

Example:

loadBalancer:
  seesaw:
    memoryMB: 8192

loadBalancer.seesaw.vCenter.networkName

Optional
Relevant if loadBalancer.kind = "Seesaw"
Immutable
String

The name of the vCenter network that contains your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    vCenter:
      networkName: "my-seesaw-network"

loadBalancer.seesaw.enableHA

Optional
Relevant if loadBalancer.kind = "Seesaw"
Immutable
Boolean
Prepopulated: false
Default: false

If you want to create a highly-available (HA) Seesaw load balancer, set this to true. Otherwise set this to false. An HA Seesaw load balancer uses a (Master, Backup) pair of VMs.

Example:

loadBalancer:
  seesaw:
    enableHA: true

loadBalancer.seesaw.disableVRRPMAC

Optional
Relevant if loadBalancer.kind = "Seesaw"
Immutable
Boolean
Prepopulatd: false
Default: false

If you set this to true, the Seesaw load balancer does not use MAC learning for failover. Instead, it uses gratuitous ARP. If you set this to false, the Seesaw load balancer uses MAC learning. We recommend that you set this to true. If you are using vSphere 7 or later, and you have a high-availability Seesaw load balancer, then you must set this to true.

Example:

loadBalancer:
  seesaw:
    disableVRRPMAC: true

antiAffinityGroups.enabled

Optional
Mutable
Boolean
Prepopulated: true

Set this to true to enable DRS rule creation. Otherwise, set this to false.

Example:

antiAffinityGroups:
  enabled: true

If this field is true, Google Distributed Cloud creates VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your admin cluster nodes, causing them to be spread across at least three physical ESXi hosts in your datacenter.

This feature requires that your vSphere environment meets the following conditions:

  • VMware DRS is enabled. VMware DRS requires vSphere Enterprise Plus license edition.

  • Your vSphere user account has the Host.Inventory.Modify cluster privilege.

  • There are at least four ESXi hosts available.

Even though the rule requires that the cluster nodes are spread across three ESXi hosts, we strongly recommend that you have at least four ESXi hosts available. This protects you from losing your admin cluster control plane. For example, suppose you have only three ESXi hosts, and your admin cluster control-plane node is on an ESXi host that fails. The DRS rule will prevent the control-plane node from being placed on one of the remaining two ESXi hosts.

Recall that if you have a vSphere Standard license, you cannot enable VMware DRS.

If you do not have DRS enabled, or if you do not have at least four hosts where vSphere VMs can be scheduled, set antiAffinityGroups.enabled to false.

adminMaster

Preview
Optional
Immutable

Edit this section if you want to do any of the following:

  • Specify CPUs for the control-plane nodes of the admin cluster
  • Specify memory for the control-plane nodes of the admin cluster
  • Create high-availability admin cluster

Otherwise, leave the prepopulated values in place.

adminMaster.cpus

Preview
Optional
Immutable
Integer
Prepopulated: 4
Default: 4

The number of vCPUs for each control-plane node in the admin cluster.

Example:

adminMaster:
  cpus: 4

adminMaster.memoryMB

Preview
Optional
Immutable
Integer
Prepopulated: 16384
Default: 16384

The number of mebibytes of memory for each control-plane node in the admin cluster.

Example:

adminMaster:
  memoryMB: 16384

adminMaster.replicas

Optional
Immutable
Integer
Possible values: 1 or 3
Prepopulated: 1
Default: 1

The number of control-plane nodes in the admin cluster. Set this to 3 if you want to create a high-availability admin cluster. Otherwise, set it to 1.

If you set this to 3, then you must also fill in the network.controlPlaneIPBlock section and the network.hostConfig section.

If you set this to 3, then you cannot use the Seesaw load balancer.

Example:

adminMaster:
  replicas: 3

addonNode.autoResize.enabled

Optional
Mutable
Boolean
Prepopulated: false
Default: false

Set this to true to enable automatic resizing of the add-on nodes in the admin cluster. Othersise set it to false.

To update the value of this field, use gkectl update admin.

Example:

addonNode:
  autoResize:
    enabled: true

proxy

If your network is behind a proxy server, fill in this section. Otherwise, remove this section or leave it commented out. The proxy server you specify here is used by the user clusters that this admin cluster manages.
Immutable

proxy.url

Required if proxy section is filled in.
Immutable
String

The HTTP address of your proxy server. Include the port number even if it's the same as the scheme's default port.

Example:

proxy:
  url: "http://my-proxy.example.local:80"

The proxy server you specify here is used by your Google Distributed Cloud clusters. Also, your admin workstation is automatically configured to use this same proxy server unless you set the HTTPS_PROXY environment variable on your admin workstation.

If you specify proxy.url, you must also specify proxy.noProxy.

After the proxy configuration for the admin cluster has been set, it cannot be modified or deleted, unless the cluster is rebuilt.

proxy.noProxy

Optional
Immutable
String

A comma-separated list of IP addresses, IP address ranges, host names, and domain names that should not go through the proxy server. When Google Distributed Cloud sends a request to one of these addresses, hosts, or domains, the request is sent directly.

Example:

proxy:
  noProxy: "10.151.222.0/24,my-host.example.local,10.151.2.1"

privateRegistry

If you have a private container registry, fill in this section. Otherwise, remove this section or leave it commented out.

privateRegistry.address

Required for private registry
Immutable
String

The IP address or FQDN (Fully Qualified Domain Name) of the machine that runs your private registry.

Examples:

privateRegistry:
  address: "203.0.113.10"

privateRegistry:
  address: "fqdn.example.com"

privateRegistry.credentials.fileRef.path

Required for private registry
Mutable
String

The path of a credentials configuration file that holds the username and password of an account that Google Distributed Cloud can use to access your private registry.

Example:

privateRegistry:
  credentials:
    fileRef:
      path: "my-config-folder/admin-creds.yaml"

privateRegistry.credentials.fileRef.entry

Required for private registry
Mutable
String

The name of the credentials block, in your credentials configuration file, that holds the username and password of your private registry account.

privateRegistry:
  credentials:
    fileRef:
      entry: "private-registry-creds"

privateRegistry.caCertPath

Required for private registry
Mutable
String

When the container runtime pulls an image from your private registry, the registry must prove its identity by presenting a certificate. The registry's certificate is signed by a certificate authority (CA). The container runtime uses the CA's certificate to validate the registry's certificate.

Set this field to the path of the CA's certificate.

Example:

privateRegistry:
  caCertPath: "my-cert-folder/registry-ca.crt"

componentAccessServiceAccountKeyPath

Required
Mutable
String

The path of the JSON key file for your component access service account.

Example:

componentAccessServiceAccountKeyPath: "my-key-folder/access-key.json"

gkeConnect

Required
Mutable

When you fill in the gkeConnect section, the admin cluster is automatically registered to a fleet after it is created. This section holds information about the Google Cloud project and service account needed to register the cluster.

During cluster creation or update, several RBAC policies are configured on the admin cluster. These RBAC policies are needed so that you can create user clusters in the Google Cloud console.

gkeConnect.projectID

Required
Immutable
String

The ID of your fleet host project. For new clusters, this project ID must be the same as the ID set in stackdriver.projectID and cloudAuditLogging.projectID. If the project IDs aren't the same, cluster creation fails. This requirement isn't applied to existing clusters.

Example:

gkeConnect:
  projectID: "my-fleet-host-project"

gkeConnect.registerServiceAccountKeyPath

Required
Mutable
String

The path of the JSON key file for your connect-register service account.

Example:

gkeConnect:
  registerServiceAccountKeyPath: "my-key-folder/connect-register-key.json"

gkeOnPremAPI

Optional

In 1.16 and later, if the GKE On-Prem API is enabled in your Google Cloud project, all clusters in the project are enrolled in the GKE On-Prem API automatically in the region configured in stackdriver.clusterLocation.

  • If you want to enroll all clusters in the project in the GKE On-Prem API, be sure to do the steps in Before you begin to activate and use the GKE On-Prem API in the project.

  • If you don't want to enroll the cluster in the GKE On-Prem API, include this section and set gkeOnPremAPI.enabled to false. If you don't want to enroll any clusters in the project, disable gkeonprem.googleapis.com (the service name for the GKE On-Prem API) in the project. For instructions, see Disabling services.

Enrolling your admin cluster in the GKE On-Prem API lets you use standard tools—the Google Cloud console, the Google Cloud CLI, or Terraform—to upgrade user clusters that the admin cluster manages. Enrolling your cluster also lets you run gcloud commands to get information about your clusters.

After you add this section and create or update the admin cluster, if subsequently you remove the section and update the cluster, the update will fail.

gkeOnPremAPI.enabled

Required if the gkeOnPremAPI section is included.
Mutable
Boolean
Default: true

By default, the cluster is enrolled in the GKE On-Prem API if the GKE On-Prem API is enabled in your project. Set to false if you don't want to enroll the cluster.

After the cluster is enrolled in the GKE On-Prem API, if you need to unenroll the cluster, make the following change and then update the cluster:

gkeOnPremAPI:
  enabled: false

gkeOnPremAPI.location

Immutable
String
Default:stackdriver.clusterLocation

The Google Cloud region where the GKE On-Prem API runs and stores cluster metadata. Choose one of the supported regions. You must use the same region that is configured in stackdriver.clusterLocation. If gkeOnPremAPI.enabled is false, don't include this field.

stackdriver

Required by default
Mutable

If you want to enable Cloud Logging and Cloud Monitoring for your cluster, fill in this section. Otherwise remove this section.

This section is required by default. That is, if you don't include this section, you must include the --skip-validation-stackdriver flag when you run gkectl create admin.

This section is required in the admin cluster if you want to manage the lifecycle of user clusters using GKE On-Prem API clients.

stackdriver.projectID

Required for Logging and Monitoring
Immutable
String

The ID of your fleet host project. For new clusters, this project ID must be the same as the ID set in gkeConnect.projectID and cloudAuditLogging.projectID. If the project IDs aren't the same, cluster creation fails. This requirement isn't applied to existing clusters.

If needed, you can configure a log router in this project to route logs into log buckets in another project. For information on how to configure the log router, see Supported destinations.

Example:

stackdriver:
  projectID: "my-fleet-host-project"

stackdriver.clusterLocation

Required for Logging and Monitoring
Immutable
String
Prepopulated: "us-central1"

The Google Cloud region where you want to store logs. It is a good idea to choose a region that is near your on-premises data center.

For new clusters, if you include the gkeOnPremAPI and cloudAuditLogging sections in the configuration file, the region that you set here must be the same region that you set in gkeOnPremAPI.location and cloudAuditLogging.clusterLocation. If the regions aren't the same, cluster creation fails.

Example:

stackdriver:
  clusterLocation: "us-central1"

stackdriver.enableVPC

Optional
Immutable
Boolean
Prepopulated: false

If your cluster's network is controlled by a VPC, set this totrue. This ensures that all telemetry flows through Google's restricted IP addresses. Otherwise, set this to false.

Example:

stackdriver:
  enableVPC: false

stackdriver.serviceAccountKeyPath

Required for Logging and Monitoring
Mutable
String

The path of the JSON key file for your logging-monitoring service account.

For information on how to update this field in an existing cluster, see Rotating service account keys.

Example:

stackdriver:
  serviceAccountKeyPath: "my-key-folder/log-mon-key.json"

stackdriver.disableVsphereResourceMetrics

Optional
Mutable
Relevant for Logging and Monitoring
Boolean
Prepopulated: false
Default: false

Set this to true to disable the collection of metrics from vSphere. Otherwise, set it to false.

This section is required in the admin cluster if you want to manage the lifecycle of user clusters in the Google Cloud console.

Example:

stackdriver:
  disableVsphereResourceMetrics: true

cloudAuditLogging

If you want to integrate the audit logs from your cluster's Kubernetes API server with Cloud Audit Logs, fill in this section. Otherwise, remove this section or leave it commented out.
Mutable

This section is required in the admin cluster if you want to manage the lifecycle of user clusters in using GKE On-Prem API clients.

cloudAuditLogging.projectID

Required for Cloud Audit Logs
Immutable
String

The ID of your fleet host project. For new clusters, this project ID must be the same as the ID set in gkeConnect.projecID and stackdriver.projectID. If the project IDs aren't the same, cluster creation fails. This requirement isn't applied to existing clusters.

If needed, you can configure a log router in this project to route logs into log buckets in another project. For information on how to configure the log router, see Supported destinations.

Example:

cloudAuditLogging:
  projectID: "my-fleet-host-project"

cloudAuditLogging.clusterLocation

Required for Cloud Audit Logs
Immutable
String

The Google Cloud region where you want to store audit logs. It is a good idea to choose a region that is near your on-premises data center

For new clusters, if you include the gkeOnPremAPI and stackdriver sections in the configuration file, the region that you set here must be the same region that you set in gkeOnPremAPI.location and stackdriver.clusterLocation. If the regions aren't the same, cluster creation fails.

Example:

cloudAuditLogging:
  clusterLocation: "us-central1"

cloudAuditLogging.serviceAccountKeyPath

Required for Cloud Audit Logs
Mutable
String

The path of the JSON key file for your audit-logging service account.

For information on how to update this field in an existing cluster, see Rotating service account keys.

Example:

cloudAuditLogging:
  serviceAccountKeyPath: "my-key-folder/audit-log-key.json"

clusterBackup.datastore

Preview
Optional
Mutable
String

If you want to enable backing up of the admin cluster, set this to the vSphere datastore where you want to save cluster backups.

Example:

clusterBackup:
  datastore: "my-datastore"

autoRepair.enabled

Optional
Mutable
Boolean
Prepopulated: true

Set this to true to enable node auto repair. Otherwise, set this to false.

To update the value of this field, use gkectl update admin.

Example:

autoRepair:
  enabled: true

secretsEncryption

If you want to encrypt Secrets without the need for an external KMS (Key Management Service), or any other dependencies, fill in this section. Otherwise, remove this section or leave it commented out.
Immutable

secretsEncryption.mode

Required for Secrets encryption
Immutable
String
Possible value: "GeneratedKey"
Prepopulated: "GeneratedKey"

The Secret encryption mode.

secretsEncryption:
  mode: "GeneratedKey"

secretsEncryption.generatedKey.keyVersion

Required for Secrets encryption
Mutable
Integer
Prepopulated: 1

An integer of your choice to use for the key version number. We recommended that you start with 1.

Example:

secretsEncryption:
  generatedKey:
    keyVersion: 1

secretsEncryption.generatedKey.disabled

Optional for Secrets encryption
Mutable
Boolean
Prepopulated: false

Set this to true to disable Secrets encryption. Otherwise set it to false.

Example:

secretsEncryption:
  generatedKey:
    disabled: false

osImageType

Optional
String
Possible values: "ubuntu_containerd" or "cos"
Prepopulated: "ubuntu_containerd"
Default: "ubuntu_containerd"

The type of OS image to run on your admin cluster nodes.

Example:

osImageType: "cos"