This page describes the fields in the Google Distributed Cloud user cluster configuration file.
Generating a template for your configuration file
If you used gkeadm
to create your admin workstation, then gkeadm
generated
a template for your user cluster configuration file. And gkeadm
filled in
some of the fields for you.
If you did not use gkeadm
to create your admin workstation, you can use
gkectl
to generate a template for your user cluster configuration file.
To generate a template for your user cluster configuration file:
gkectl create-config cluster --config=OUTPUT_FILENAME --gke-on-prem-version=VERSION
Replace the following:
OUTPUT_FILENAME
: a path of your choice for the
generated template. If you omit this flag, gkectl
names the file
user-cluster.yaml
and puts it in the current directory.
VERSION
: the desired version number. For example:
gkectl create-config cluster --gke-on-prem-version=1.10.0-gke.8
.
Template
Required fields and default values
If a field is marked as Required, then the completed configuration file must have a value filled in for the field.
Some fields are conditionally required. For example,
loadBalancer.metalLB.addressPools
is required if loadBalancer.kind
is equal to
MetalLB
.
If a Default value is given for a field, then the cluster will use that value if you don't enter anything for the field. You can override a default value by entering a value.
If a field is not marked as Required and no Default value is given, then the field is optional. You can fill it in if it is relevant for you, but you don't have to fill it in.
Filling in your configuration file
In your configuration file, enter field values as described in the following sections.
name
Required
String
A name of your choice for your user cluster. The name must:
- contain at most 40 characters
- contain only lowercase alphanumeric characters or a hyphen (
-
) - start with an alphabetic character
- end with an alphanumeric character
Example:
name: "my-user-cluster"
gkeOnPremVersion
Required
Mutable
String
The Google Distributed Cloud version for your user cluster.
Example:
gkeOnPremVersion: "1.29.0-gke.1456"
preparedSecrets.namespace
If you want to use prepared credentials, fill in this field. Otherwise remove this field or leave it commented out.
Immutable
String
Possible values: A string that begins with "gke-onprem-secrets-"
The name of a Kubernetes namespace in the admin cluster where prepared Secrets are kept for this user cluster.
Example:
preparedSecrets: namespace: "gke-onprem-secrets-alice"
enableControlplaneV2
1.30 and higher
Immutable
Boolean
Prepopulated: true
Default: true
In version 1.30 and higher, Controlplane V2 is required for new user
clusters. If you include this field in your configuration file, it must
be set to true
. With Controlplane V2, the control plane for a user cluster
runs on one or more nodes in the user cluster itself. The benefits of
Controlplane V2 include:
Architectural consistency between admin and user clusters.
Failure isolation. An admin cluster failure doesn't affect user clusters.
Operational separation. An admin cluster upgrade doesn't cause downtime for user clusters.
Deployment separation. You can put the admin and user clusters in different failure domains or geographical sites. For example, a user cluster in an edge location could be in a different geographical site from the admin cluster.
1.29 and lower
Immutable
Boolean
Prepopulated: true
Default: true
To enable Controlplane V2, set enableControlplaneV2
to true
or remove the
setting from your user cluster configuration file. Otherwise, set it to false
.
With Controlplane V2, the control plane for a user cluster runs on one or more
nodes in the user cluster itself. When Controlplane V2 isn't enabled, the user
cluster control plane runs in the admin cluster.
We recommend that you enable Controlplane V2.
Example:
enableControlplaneV2: true
vCenter
This section holds information about your vSphere environment and your connection to vCenter Server.
If you want a field in this section to be the same as what you specified for your admin cluster, remove the field or leave it commented out.
If you want a field to be different from what you specified for your admin
cluster, fill it in here. Any fields that you fill in here in the vCenter
section override the corresponding fields in your admin cluster configuration
file.
vCenter.address
Immutable
String
Default: Inherit from the admin cluster
The IP address or the hostname of the instance of vCenter Server that you want to use for this user cluster.
For more information, see Finding your vCenter server address.
If you specify a value that is different from the instance of vCenter Server
used by the admin cluster, then
Controlplane V2
must be enabled, and you must provide values for
network.vCenter.networkName
and all the
required fields in the section vCenter
.
Examples:
vCenter: address: "203.0.113.101"
vCenter: address: "my-vcenter-server-2.my-domain.example"
vCenter.datacenter
Immutable
String
Default: Inherit from the admin cluster
The relative path of a vSphere data center.
If you specify a value for this field, then you must also specify:
vCenter.networkName
- Either
vCenter.datastore
orvCenter.storagePolicyName
- Either
vCenter.cluster
orvCenter.resourcePool
The value you specify is relative to the root folder named /
.
If your data center is in the root folder, the value is the name of the data center.
Example:
vCenter: datacenter: "my-uc-data-center"
Otherwise, the value is a relative path that includes one or more folders along with the name of the data center.
Example:
vCenter: datacenter: "data-centers/data-center-2"
vCenter.cluster
Immutable
String
Default: Inherit from the admin cluster
The relative path of a vSphere cluster that represents the ESXi hosts where your user cluster VMs will run. This vSphere cluster represents a subset of the physical ESXi hosts in your vCenter data center.
If you specify a value for this field, then you must also specify:
vCenter.networkName
vCenter.datacenter
- Either
vCenter.datastore
orvCenter.storagePolicyName
The value you specify is relative to /.../DATA_CENTER/vm/
.
If your vSphere cluster is in the /.../DATA_CENTER/vm/
folder, the value is the name of the vSphere cluster
Example:
vCenter: cluster: "my-uc-vsphere-cluster"
Otherwise, the value is a relative path that includes one or more folders along with the name of the vSphere cluster.
Example:
vCenter: cluster: "clusters/vsphere-cluster-2"
vCenter.resourcePool
Immutable
String
Default: Inherit from the admin cluster
A vCenter resource pool for your user cluster VMs.
If you want to use the default resource pool, set this to
VSPHERE_CLUSTER/Resources
.
Example:
vCenter: resourcePool: "my-uc-vsphere-cluster/Resources"
The value you specify is relative to
/.../DATA_CENTER/host/.../VSPHERE_CLUSTER/Resources/
If your resource pool is a direct child of
/.../DATA_CENTER/host/.../VSPHERE_CLUSTER/Resources/
the value is the name of the resource pool.
Example:
vCenter: resourcePool: "my-uc-resource-pool"
Otherwise, the value is a relative path that has two or more resource pools.
Example:
vCenter: resourcePool: "uc-resource-pool-1/uc-resource-pool-2"
vCenter.datastore
Optional
Immutable
String
Default: Inherit from the admin cluster
The name of a vSphere datastore for your user cluster.
The value you specify must be a name, not a path. Don't include any folders in the value.
Example:
vCenter: datastore: "my-datastore"
If you specify a value for this field, don't specify a value for
vCenter.storagePolicyName
. The vCenter.datastore
field is immutable except
when you set the field to an empty string when you
migrate a datastore to Storage Policy Based Management (SPBM).
vCenter.storagePolicyName
Optional
Immutable
String
The name of a VM storage policy for the cluster nodes.
You can also specify a VM storage policy for the nodes in a particular node pool. But the policy specified here applies to any node pool that doesn't have its own policy.
For more information, see Configure a storage policy.
If you specify a value for this field, don't specify a value for
vCenter.datastore
.
vCenter.caCertPath
Mutable
String
Default: Inherit from the admin cluster
The path of the CA certificate for your vCenter server. For more information, see Getting your vCenter CA certificate.
Example:
vCenter: caCertPath: "/usr/local/google/home/me/certs/vcenter-ca-cert-2.pem"
For information on how to update this field, see Update vCenter certificate references.
vCenter.credentials.fileRef.path
Mutable
String
Default: Inherit from the admin cluster
If you provide a value for preparedSecrets.namespace, do not provide a value for this field. Instead, provide a value for vCenter.credentials.secretRef.version.
The path of a credentials configuration file that holds the username and password of your vCenter user account. The user account must have the Administrator role or equivalent privileges. See vSphere requirements.
You can use gkectl update credentials
to update this field in an existing
cluster. For more information, see
Rotating service account keys.
For information about updating your vCenter credentials, see Updating cluster credentials.
Example:
vCenter: credentials: fileRef: path: "my-config-directory/my-credentials-2.yaml"
vCenter.credentials.fileRef.entry
Mutable
String
Default: Inherit from the admin cluster
The name of the credentials block, in your credentials configuration file, that holds the username and password of your vCenter user account.
Example:
vCenter: credentials: fileRef: entry: "vcenter-credentials-2"
vCenter.credentials.secretRef.version
String
Possible values: An integer string or "latest"
Default value: "latest"
If you provide a value for preparedSecrets.namespace, fill in this field. Otherwise, remove this field or leave it commented out.
The version of a prepared Secret in the admin cluster that holds the username and password for the instance of vCenter Server that you intend to use for this user cluster.
Example:
vCenter: credentials: secretRef: version: "1"
vCenter.folder
Immutable
String
Default: Inherit from the admin cluster
The relative path of a vSphere folder that you have already created. This folder will hold your user cluster VMs.
If you do not specify a value, your user cluster VMs will be put in
/.../DATA_CENTER/vm/
.
If you specify a value, it is relative to
/.../DATA_CENTER/vm/
.
The value can be the name of a folder.
Example:
vCenter: folder: "my-uc-folder"
Or the value can be a relative path that includes more than one folder.
Example:
vCenter: folder: "folders/folder-2"
componentAccessServiceAccountKeyPath
Mutable
String
Default: Inherit from the admin cluster
The path of the JSON key file for your component access service account.
Example:
componentAccessServiceAccountKeyPath: "my-key-folder/uc-access-key.json"
componentAccessServiceAccountKey.secretRef.version
Mutable
String
Possible values: An integer string or "latest"
Default value: "latest"
The version of a prepared Secret in the admin cluster that holds a JSON key for your component access service account.
Example:
componentAccessServiceAccountKey: secretRef: version: "1"
privateRegistry
Fill in this section only if your admin cluster is using a private registry and you want your user cluster to use a different private registry. Otherwise, remove this section or leave it commented out.
If you want to have some different settings from admin cluster settings:
Different username, password, certificate
(Only available for Control Plane V2) Different private registry address
Fill the privateRegistry
fields here. The new settings are picked up
during cluster creation and update.
privateRegistry.address
Immutable
String
Default: Inherit from the admin cluster
The IP address or FQDN (Fully Qualified Domain Name) of the machine that runs your private Docker registry.
Examples:
privateRegistry: address: "203.0.113.10"
privateRegistry: address: "fqdn.example.com"
privateRegistry.credentials.fileRef.path
Mutable
String
Default: Inherit from the admin cluster
If you are planning to use prepared secret for the private registry, do not provide a value for this field. Instead, provide a value for privateRegistry.credentials.secretRef.version.
The path of a credentials configuration file that holds the username and password of an account that Google Distributed Cloud can use to access your private Docker registry.
Example:
privateRegistry: credentials: fileRef: path: "my-config-folder/admin-creds.yaml"
privateRegistry.credentials.fileRef.entry
Mutable
String
Default: Inherit from the admin cluster
The name of the credentials block, in your credentials configuration file, that holds the username and password of your private Docker registry account.
privateRegistry: credentials: fileRef: entry: "private-registry-creds"
privateRegistry.credentials.secretRef.version
String
Possible values: An integer string or "latest"
Default value: "latest"
If you provide a value for preparedSecrets.namespace and want to use prepared secret for a private registry, fill in this field. Otherwise, remove this field or leave it commented out.
The version of a prepared Secret in the admin cluster that holds the username and password for the instance of vCenter Server that you intend to use for this user cluster.
Example:
privateRegistry: credentials: secretRef: version: "1"
privateRegistry.caCertPath
Mutable
String
Default: Inherit from the admin cluster
When Docker pulls an image from your private registry, the registry must prove its identity by presenting a certificate. The registry's certificate is signed by a certificate authority (CA). Docker uses the CA's certificate to validate the registry's certificate.
Set this field to the path of the CA's certificate.
Example:
privateRegistry: caCertPath: "my-cert-folder/registry-ca.crt"
network
This section holds information about your user cluster network.
network.hostConfig
1.30 and higher
This section holds information about NTP servers, DNS servers, and DNS search domains used by the VMs that are your cluster nodes.
This section is required if one or more of the following are true:
network.ipMode.type
is set tostatic
.enableControlplaneV2
is set totrue
or allowed to default totrue
. Note that Conrolplane V2 is required for new user clusters.
1.29 and lower
This section holds information about NTP servers, DNS servers, and DNS search domains used by the VMs that are your cluster nodes. If you are using the Seesaw load balancer, this information also applies to your Seesaw VMs.
This section is required if one or more of the following are true:
network.ipMode.type
is set tostatic
.enableControlplaneV2
is set totrue
or allowed to default totrue
. The Seesaw load balancer isn't supported on clusters with Controlplane V2 enabled.loadBalancer.kind
is set to"Seesaw"
.
network.hostConfig.dnsServers
Required
Immutable
Array of strings.
The maximum number of elements in the array is three.
The addresses of DNS servers for the VMs.
Example:
network: hostConfig: dnsServers: - "172.16.255.1" - "172.16.255.2"
network.hostConfig.ntpServers
Required
Immutable
Array of strings
The addresses of time servers for the VMs to use.
Example:
network: hostConfig: ntpServers: - "216.239.35.0"
network.hostConfig.searchDomainsForDNS
Immutable
Array of strings
DNS search domains for the hosts to use. These domains are used as part of a domain search list.
Example:
network: hostConfig: searchDomainsForDNS: - "my.local.com"
network.ipMode.type
Immutable
String
Prepopulated: "static"
Default: "dhcp"
If you want your cluster nodes to get their IP address from a DHCP server,
set this to "dhcp"
. If you want your cluster nodes to have static IP
addresses chosen from a list that you provide, set this to "static"
.
Example:
network: ipMode: type: "static"
network.ipMode.ipBlockFilePath
Required if network.ipMode.type
= static
Immutable
String
The path of the IP block file for your cluster.
Example:
network: ipMode: ipBlockFilePath: "/my-config-folder/user-cluster-ipblock.yaml"
network.serviceCIDR
Immutable
String
Prepopulated: "10.96.0.0/20"
>
A range of IP addresses, in CIDR format, to be used for Services in your cluster. Must be at least a /24 range.
Example:
network: serviceCIDR: "10.96.0.0/20"
network.podCIDR
Immutable
String
Prepopulated: "192.168.0.0/16"
Default: "192.168.0.0/16"
A range of IP addresses, in CIDR format, to be used for Pods in your cluster. Must be at least a /18 range.
Example:
network: podCIDR: "192.168.0.0/16"
The Service range must not overlap with the Pod range.
The Service and Pod ranges must not overlap with any address outside the cluster that you want to reach from inside the cluster.
For example, suppose your Service range is 10.96.232.0/24, and your Pod range is 192.168.0.0/16. Any traffic sent from a Pod to an address in either of those ranges will be treated as in-cluster and will not reach any destination outside the cluster.
In particular, the Service and Pod ranges must not overlap with:
IP addresses of nodes in any cluster
IP addresses used by load balancer machines
VIPs used by control-plane nodes and load balancers
IP address of vCenter servers, DNS servers, and NTP servers
We recommend that your Service and Pod ranges be in the RFC 1918 address space.
Here is one reason for the recommendation to use RFC 1918 addresses. Suppose your Pod or Service range contains external IP addresses. Any traffic sent from a Pod to one of those external addresses will be treated as in-cluster traffic and will not reach the external destination.
network.vCenter.networkName
Required if
vCenter.address
is different from the vCenter address you are using for the admin cluster.
Immutable
String
The name of the vSphere network for your user cluster nodes.
Example:
network: vCenter: networkName: "my-network"
If the name contains a special character, you must use an escape sequence for it.
Special characters | Escape sequence |
---|---|
Slash (/ ) |
%2f |
Backslash (\ ) |
%5c |
Percent sign (% ) |
%25 |
If the network name is not unique in your data center, you can specify a full path.
Example:
network: vCenter: networkName: "/my-uc-datacenter/network/my-network"
network.additionalNodeInterfaces
If you set multipleNetworkInterfaces
to true
, fill in this section.
Otherwise, remove this section or leave it commented out.
Immutable
An array of objects, each of which describes a network interface that can be used on all or some of your cluster nodes.
network.additionalNodeInterfaces[i].networkName
Required if multipleNetworkInterfaces
= true
Immutable
String
The name of the vSphere network that this additional node interface will connect to.
Example:
network: additionalNodeInterfaces: - networkName: "my-node-interface-1"
network.additionalNodeInterfaces[i].type
Required if multipleNetworkInterfaces
= true
Immutable
String
Example:
network: additionalNodeInterfaces: - name: "my-node-interface-1" type: "static"
network.additionalNodeInterfaces[i].ipBlockFilePath
Required if network.additionalNodeInterfaces[i].type
= static
Immutable
String
The path of an IP block file that holds IP addresses to be used for this network interface on the nodes that have this network interface.
Example:
network: additionalNodeInterfaces: - name: "my-node-interface-1" type: "static" ipBlockFilePath: "my-ipblock-file-1"
network.controlPlaneIPBlock
If you set enableControlplaneV2
to true
, fill in this section. Otherwise, remove this section or leave it commented out.
network.controlPlaneIPBlock.netmask
Required if enableControlplaneV2
= true
Immutable
String
Example:
network: controlPlaneIPBlock: netmask: "255.255.255.0"
network.controlPlaneIPBlock.gateway
Required if enableControlplaneV2
= true
Immutable
String
Example:
network: controlPlaneIPBlock: gateway: "172.16.21.1"
network.controlPlaneIPBlock.ips
Required if enableControlplaneV2
= true
Immutable
Array of objects, each of which has an IP address and an optional hostname.
For a high-availability (HA) user cluster, the array has three elements. For a non-HA user cluster, the array has one element.
Example:
network: controlPlaneIPBlock: ips: - ip: "172.16.21.6" hostname: "cp-vm-1" - ip: "172.16.21.7" hostname: "cp-vm-2" - ip: "172.16.21.8" hostname: "cp-vm-3"
loadBalancer
This section holds information about the load balancer for your user cluster.
loadBalancer.vips.controlPlaneVIP
Required
Immutable
String
The IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the user cluster.
Example:
loadBalancer: vips: controlplaneVIP: "203.0.113.3"
loadBalancer.vips.ingressVIP
Required
Immutable
String
The IP address that you have chosen to configure on the load balancer for the ingress proxy.
Example:
loadBalancer: vips: ingressVIP: "203.0.113.4"
loadBalancer.kind
1.30 and higher
Required
Immutable
String
Prepopulated: "MetalLB"
String. Set this to "ManualLB"
or "MetalLB"
Use "ManualLB"
if you have a third-party load balancer (such as F5 BIG-IP
or Citrix) or "MetalLB"
for our bundled solution.
Although you can upgrade a cluster that has kind
set to "F5BigIP"
or
"Seesaw"
, you can't use Dataplane V2 or Controlplane V2 on existing
clusters with either of these load balancer configurations. For information
on migrating load balancing configurations, see the following:
Example:
loadBalancer:
kind: "MetalLB"
1.29 and earlier
Required
Immutable
String
Prepopulated: "MetalLB"
Set this to "ManualLB"
, "F5BigIP"
, "Seesaw"
, or "MetalLB"
.
To enable Dataplane V2 and Controlplane V2, we recommend that you use
"ManualLB"
if you have a third-party load balancer (such as F5 BIG-IP or
Citrix) or "MetalLB"
for our bundled solution.
Example:
loadBalancer:
kind: "MetalLB"
loadBalancer.manualLB
If you set loadbalancer.kind
to "manualLB"
, fill in this section. Otherwise,
remove this section or leave it commented out.
loadBalancer.manualLB.ingressHTTPNodePort
Required if loadBalancer.kind
= ManualLB
Immutable
Integer
Prepopulated: 30243
The ingress proxy in a user cluster is exposed by a
Kubernetes Services of type LoadBalancer
.
The Service has a
ServicePort
for HTTP. You must choose a nodePort
value for the HTTP ServicePort.
Set this field to the nodePort
value.
Example:
loadBalancer: manualLB: ingressHTTPNodePort: 32527
loadBalancer.manualLB.ingressHTTPSNodePort
Required if loadBalancer.kind
= ManualLB
Immutable
Integer
Prepopulated: 30879
The ingress proxy in a user cluster is exposed by a Service
of type LoadBalancer. The Service has a ServicePort for HTTPS. You must choose a
nodePort
value for the HTTPS ServicePort.
Set this field to the nodePort
value.
Example:
loadBalancer: manualLB: ingressHTTPSNodePort: 30139
loadBalancer.manualLB.controlPlaneNodePort
Required if loadBalancer.kind
= ManualLB
and
enableControlplaneV2
= false
Immutable
Integer
Prepopulated: 30562
The Kubernetes API server of a user cluster runs in the admin cluster,
and is exposed by a Service of type LoadBalancer
. You must choose a nodePort
value for the Service.
Set this field to the nodePort
value.
Example:
loadBalancer: manualLB: controlPlaneNodePort: 30968
loadBalancer.manualLB.konnectivityServerNodePort
Required if loadBalancer.kind
= ManualLB
and
enableControlplaneV2
= false
Immutable
Integer
Prepopulated: 30563
The Kubernetes API server of a user cluster that uses kubeception runs in the
admin cluster, and is exposed by a Service of type LoadBalancer
. The
Konnectivity server reuses this service with a different nodePort
value. You
must choose a nodePort
value for the Konnectivity server.
Set this field to the nodePort
value for the Konnectivity server.
Example:
loadBalancer: manualLB: konnectivityServerNodePort: 30969
loadBalancer.f5BigIP
1.30 and higher
In 1.30 and higher, the value "F5BigIP"
isn't allowed for
loadbalancer.kind
for new user clusters. If the loadBalancer.f5BigIP
section is in your configuration file, remove it or comment it out.
You can still use your F5 BIG-IP load balancer with new user clusters, but the configuration is different. For configuration details, see Enabling manual load balancing mode.
1.29 and lower
If you set loadbalancer.kind
to "f5BigIP"
, fill in this section. Otherwise,
remove this section or leave it commented out.
To enable new and advanced features, we recommend that you configure manual
load balancing for your F5 BIG-IP load balancer. To enable manual load
balancing, set loadbalancer.kind
to"ManualLB"
and fill in the
loadBalancer.manualLB
section. For more
information, see
Enabling manual load balancing mode.
If you have an existing F5-BIG-IP load balancer and the cluster configuration uses this section, after you have upgraded to 1.29 or higher, we recommend that you migrate to manual load balancing.
loadBalancer.f5BigIP.address
1.30 and higher
Not allowed for new clusters
Required if loadBalancer.kind
= "f5BigIP"
String
The address of your F5 BIG-IP load balancer.
Example:
loadBalancer:
f5BigIP:
address: "203.0.113.2"
1.29 and lower
Required if loadBalancer.kind
= "f5BigIP"
String
The address of your F5 BIG-IP load balancer.
Example:
loadBalancer:
f5BigIP:
address: "203.0.113.2"
loadBalancer.f5BigIP.credentials.fileRef.path
1.30 and higher
Not allowed for new clusters
Required if loadBalancer.kind
= "f5BigIP"
String
The path of a credentials file that holds the username and password of an account that Google Distributed Cloud can use to connect to your F5 BIG-IP load balancer.
The user account must have a user role that has sufficient permissions to set up and manage the load balancer. Either the Administrator role or the Resource Administrator role is sufficient.
For information about updating your F5 BIG-IP credentials, see Updating cluster credentials.
Example:
loadBalancer:
f5BigIP:
credentials:
fileRef:
path: "my-config-folder/user-creds.yaml"
1.29 and lower
Required if loadBalancer.kind
= "f5BigIP"
String
The path of a credentials file that holds the username and password of an account that Google Distributed Cloud can use to connect to your F5 BIG-IP load balancer.
The user account must have a user role that has sufficient permissions to set up and manage the load balancer. Either the Administrator role or the Resource Administrator role is sufficient.
For information about updating your F5 BIG-IP credentials, see Updating cluster credentials.
Example:
loadBalancer:
f5BigIP:
credentials:
fileRef:
path: "my-config-folder/user-creds.yaml"
loadBalancer.f5BigIP.credentials.fileRef.entry
1.30 and higher
Not allowed for new clusters
Required if loadBalancer.kind
= "f5BigIP"
String
The name of the credentials block, in your credentials configuration file, that holds the username and password of your F5 BIG-IP account.
Example:
loadBalancer:
f5BigIP:
credentials:
fileRef:
entry: "f5-creds"
1.29 and lower
Required if loadBalancer.kind
= "f5BigIP"
String
The name of the credentials block, in your credentials configuration file, that holds the username and password of your F5 BIG-IP account.
Example:
loadBalancer:
f5BigIP:
credentials:
fileRef:
entry: "f5-creds"
loadBalancer.f5BigIP.partition
1.30 and higher
Not allowed for new clusters
Required if loadBalancer.kind
= "f5BigIP"
String
The name of a BIG-IP partition that you created for your user cluster.
Example:
loadBalancer:
f5BigIP:
partition: "my-f5-user-partition"
1.29 and lower
Required if loadBalancer.kind
= "f5BigIP"
String
The name of a BIG-IP partition that you created for your user cluster.
Example:
loadBalancer:
f5BigIP:
partition: "my-f5-user-partition"
loadBalancer.f5BigIP.snatPoolName
1.30 and higher
Not allowed for new clusters
Required if loadBalancer.kind
= "f5BigIP"
and you are using SNAT
String
The name of your SNAT pool.
Example:
loadBalancer:
f5BigIP:
snatPoolName: "my-snat-pool"
1.29 and lower
Required if loadBalancer.kind
= "f5BigIP"
and you are using SNAT
String
The name of your SNAT pool.
Example:
loadBalancer:
f5BigIP:
snatPoolName: "my-snat-pool"
loadBalancer.seesaw
1.30 and higher
In 1.30 and higher, the value "Seesaw"
isn't allowed for
loadbalancer.kind
for new user clusters because this load balancer isn't
supported with new and advanced features. If the loadBalancer.seesaw
section is in your configuration file, remove it or comment it out.
Instead, you can configure the bundled MetalLB load balancer. To enable the
MetalLB load balancer, set loadbalancer.kind
to"MetalLB"
and fill in the
loadBalancer.metalLB
section. For more
information, see
Bundled load balancing with MetalLB.
1.29 and lower
If you set loadbalancer.kind
to "Seesaw"
, fill in this section. Otherwise,
remove this section or leave it commented out.
Note the following limitations with the SeeSaw load balancer:
- If a high-availability admin cluster manages this user cluster, then this user cluster can't use the Seesaw load balancer.
- User clusters with Controlplane V2 enabled can't use the Seesaw load balancer.
To use these features, we recommend that you configure the MetalLB
load balancer. To enable the MetalLB load balancer, set loadbalancer.kind
to"MetalLB"
and fill in the
loadBalancer.metalLB
section. For more
information, see
Bundled load balancing with MetalLB.
loadBalancer.seesaw.ipBlockFilePath
1.30 and higher
Not allowed for new clusters
Required if loadBalancer.kind
= Seesaw
Immutable
String
The path of the IP block file for your Seesaw VMs.
Example:
loadBalancer:
seesaw:
ipBlockFilePath: "config-folder/user-seesaw-ipblock.yaml"
1.29 and lower
Required if loadBalancer.kind
= Seesaw
Immutable
String
The path of the IP block file for your Seesaw VMs.
Example:
loadBalancer:
seesaw:
ipBlockFilePath: "config-folder/user-seesaw-ipblock.yaml"
loadBalancer.seesaw.vrid
1.30 and higher
Not allowed for new clusters
Required if loadBalancer.kind
= Seesaw
Immutable
Integer
Possible values: 1 - 255
Prepopulated: 0
The virtual router identifier of your Seesaw VM. This identifier, which is an integer of your choice, must be unique in a VLAN.
Example:
loadBalancer:
seesaw:
vrid: 125
1.29 and lower
Required if loadBalancer.kind
= Seesaw
Immutable
Integer
Possible values: 1 - 255
Prepopulated: 0
The virtual router identifier of your Seesaw VM. This identifier, which is an integer of your choice, must be unique in a VLAN.
Example:
loadBalancer:
seesaw:
vrid: 125
loadBalancer.seesaw.masterIP
1.30 and higher
Not allowed for new clusters
Required if loadBalancer.kind
= Seesaw
Immutable
String
The virtual IP address configured on your Master Seesaw VM.
Example:
loadBalancer:
seesaw:
masterIP: 172.16.20.21
1.29 and lower
Required if loadBalancer.kind
= Seesaw
Immutable
String
The virtual IP address configured on your Master Seesaw VM.
Example:
loadBalancer:
seesaw:
masterIP: 172.16.20.21
loadBalancer.seesaw.cpus
1.30 and higher
Not allowed for new clusters
Required if loadBalancer.kind
= Seesaw
Mutable
Integer
Prepopulated: 4
The number of CPUs for each of your Seesaw VMs.
Example:
loadBalancer:
seesaw:
cpus: 8
1.29 and lower
Required if loadBalancer.kind
= Seesaw
Mutable
Integer
Prepopulated: 4
The number of CPUs for each of your Seesaw VMs.
Example:
loadBalancer:
seesaw:
cpus: 8
loadBalancer.seesaw.memoryMB
1.30 and higher
Not allowed for new clusters
Required if loadBalancer.kind
= Seesaw
Mutable
Integer
Prepopulated: 3072
The number of mebibytes of memory for each of your Seesaw VMs.
Example:
loadBalancer:
seesaw:
memoryMB: 8192
Note: This field specifies the number of mebibytes of memory, not the number of megabytes. One mebibyte is 220 = 1,048,576 bytes. One megabyte is 106 = 1,000,000 bytes.
1.29 and lower
Required if loadBalancer.kind
= Seesaw
Mutable
Integer
Prepopulated: 3072
The number of mebibytes of memory for each of your Seesaw VMs.
Example:
loadBalancer:
seesaw:
memoryMB: 8192
Note: This field specifies the number of mebibytes of memory, not the number of megabytes. One mebibyte is 220 = 1,048,576 bytes. One megabyte is 106 = 1,000,000 bytes.
loadBalancer.seesaw.vCenter.networkName
1.30 and higher
Not allowed for new clusters
Immutable
String
Default: Same as the cluster nodes
The name of the vCenter network that contains your Seesaw VMs.
Example:
loadBalancer:
seesaw:
vCenter:
networkName: "my-seesaw-network"
1.29 and lower
Immutable
String
Default: Same as the cluster nodes
The name of the vCenter network that contains your Seesaw VMs.
Example:
loadBalancer:
seesaw:
vCenter:
networkName: "my-seesaw-network"
loadBalancer.seesaw.enableHA
1.30 and higher
Not allowed for new clusters
Immutable
Relevant if loadBalancer.kind
= Seesaw
Boolean
Prepopulated: false
Default: false
If you want to create a high-availability (HA) Seesaw load balancer, set this
to true
. Otherwise set this to false
. An HA Seesaw load balancer uses
a (Master, Backup) pair
of VMs.
Example:
loadBalancer:
seesaw:
enableHA: true
1.29 and lower
Immutable
Relevant if loadBalancer.kind
= Seesaw
Boolean
Prepopulated: false
Default: false
If you want to create a high-availability (HA) Seesaw load balancer, set this
to true
. Otherwise set this to false
. An HA Seesaw load balancer uses
a (Master, Backup) pair
of VMs.
Example:
loadBalancer:
seesaw:
enableHA: true
loadBalancer.seesaw.disableVRRPMAC
1.30 and higher
Not allowed for new clusters
Immutable
Relevant if loadBalancer.kind
= Seesaw
Boolean
Prepopulated: true
Default: true
If you set this to true
, the Seesaw load balancer does not use
MAC learning
for failover. Instead, it uses
gratuitous ARP.
If you set this to false
, the Seesaw load balancer uses MAC learning. We
recommend that you set this to true
. If you are using vSphere 7 or later,
and you have a high-availability Seesaw load balancer, then you must set this to
true
.
Example:
loadBalancer:
seesaw:
disableVRRPMAC: true
1.29 and lower
Immutable
Relevant if loadBalancer.kind
= Seesaw
Boolean
Prepopulated: true
Default: true
If you set this to true
, the Seesaw load balancer does not use
MAC learning
for failover. Instead, it uses
gratuitous ARP.
If you set this to false
, the Seesaw load balancer uses MAC learning. We
recommend that you set this to true
. If you are using vSphere 7 or later,
and you have a high-availability Seesaw load balancer, then you must set this to
true
.
Example:
loadBalancer:
seesaw:
disableVRRPMAC: true
loadBalancer.metalLB
If you set loadbalancer.kind
to "MetalLB"
, fill in this section. Otherwise,
remove this section or leave it commented out.
For more information, see Bundled load balancing with MetalLB.
loadBalancer.metalLB.addressPools
Required if loadBalancer.kind
= 'MetalLB`
Mutable
Array of objects, each of which holds information about an address pool to be used by the MetalLB load balancer.
loadBalancer.metalLB.addressPools[i].name
Required if loadBalancer.kind
= MetalLB
Mutable
String
A name of your choice for the address pool.
Example:
loadBalancer: metalLB: addressPools: - name: "my-address-pool-1"
loadBalancer.metalLB.addressPools[i].addresses
Required if loadBalancer.kind
= MetalLB
Mutable
Array of strings, each of which is a range of addresses. Each range must be in CIDR format or hyphenated range format.
Example:
loadBalancer: metalLB: addressPools: - name: "my-address-pool-1" addresses: - "192.0.2.0/26" - "192.0.2.64-192.0.2.72"
loadBalancer.metalLB.addressPools[i].avoidBuggyIPs
Relevant if loadBalancer.kind
= MetalLB
Mutable
Boolean
Prepopulated: false
Default: false
If you set this to true
, the MetalLB controller will not assign IP
addresses ending in .0 or .255 to Services. This avoids the problem of buggy
consumer devices mistakenly dropping traffic sent to those special IP addresses.
Example:
loadBalancer: metalLB: addressPools: - name: "my-address-pool-1" - "192.0.2.0/26" - "192.0.2.64-192.0.2.72" avoidBuggyIPs: true
loadBalancer.metalLB.addressPools[i].manualAssign
Relevant if loadBalancer.kind
= MetalLB
Mutable
Boolean
Prepopulated: false
Default: false
If you do not want the MetalLB controller to automatically assign IP addresses
from this pool to Services, set this to true
. Then a developer can create a
Service of type LoadBalancer
and manually specify one of the addresses from
the pool.
Otherwise, set this to false
.
Example:
loadBalancer: metalLB: addressPools: - name: "my-address-pool-1" manualAssign: true
enableDataplaneV2
1.30 and higher
Immutable
Boolean
Prepopulated: true
Default: true
In version 1.30 and higher,
Dataplane V2 is required
for new user clusters and must be set to true
. Dataplane V2 is required to
use Controlplane V2.
The benefits of Dataplane V2 include the following:
Dataplane V2 provides you with a more advanced and capable Container Network Interface (CNI) when compared with the previous CNI option, Calico. Calico is in maintenance mode, which means it receives only critical bug fixes and security updates, but no new feature development. In contrast, Dataplane V2 is actively developed and enhanced, ensuring that you have access to the latest networking innovations and capabilities.
Dataplane V2 is the preferred CNI for GKE and other Google Distributed Cloud products.
1.29 and lower
Immutable
Boolean
Prepopulated: true
Default: false
If you want to enable Dataplane V2,
or if you want to use Controlplane V2,
set this to true
. Otherwise set this to false
.
We recommend that you enable Dataplane V2.
Example:
enableDataplaneV2: true
See Dataplane V2 Troubleshooting for troubleshooting steps.
dataplaneV2.forwardMode
Immutable
String
Possible values: "snat", "dsr"
Prepopulated: "snat"
Default: "snat"
The forwarding mode for a cluster that has Dataplane V2 enabled.
With source network address translation (SNAT) mode, a packet is SNAT-translated when it is forwarded from a load balancer node to a backend Pod. The Pod cannot see the original source IP address, and the return packet must pass through the load balancer node.
With direct server return (DSR) mode, a packet retains its original source IP address when it is forwarded from a load balancer node to a backend Pod. The Pod can see the original source IP address, and the return packet goes directly to the client without passing through the load balancer node.
Example:
dataplaneV2: forwardMode: "dsr"
multipleNetworkInterfaces
Immutable
Boolean
Prepopulated: false
Default: false
If you want to enable multiple network interfaces for Pods, set this
to true
. Otherwise set this to false
.
For more information, see Configure multiple network interfaces for Pods.
Example:
multipleNetworkInterfaces: true
advancedNetworking
Immutable
boolean
Prepopulated: false
Default: false
If you plan to create an
egress NAT gateway, set this to
true
. Otherwise set it to false
.
If you set this field to true
, you must also set enableDataplaneV2
to
true
.
Example:
advancedNetworking: true
disableBundledIngress
Set this to true
if you want to
disable bundled ingress
for the cluster. Otherwise, set it to false
.
Boolean
Mutable
Prepopulated: false
Default: false
Example:
disableBundledIngress: true
storage.vSphereCSIDisabled
If you want to disable the deployment of vSphere CSI components, set this to
true
. Otherwise, set it to false
.
Mutable
Boolean
Prepopulated: false
Default: false
Example:
storage: vSphereCSIDisabled: false
masterNode
This section holds information about the nodes that serve as control-plane nodes for this user cluster.
masterNode.cpus
Mutable
Integer
Prepopulated: 4
Default: 4
The number of CPUs for each node that serves as a control plane for this user cluster.
Example:
masterNode: cpus: 8
masterNode.memoryMB
Mutable
Integer
Prepopulated: 8192
Default: 8192
The mebibytes of memory for each node that serves as a control plane for this user cluster. Must be a multiple of 4.
Example:
masterNode: memoryMB: 8192
Note: This field specifies the number of mebibytes of memory, not the number of megabytes. One mebibyte is 2^20 = 1,048,576 bytes. One megabyte is 10^6 = 1,000,000 bytes.
masterNode.replicas
Immutable
Integer
Possible values: 1 or 3
Prepopulated: 1
Default: 1
The number of control-plane nodes for this user cluster. This number cannot be changed once you have created the cluster. If you want to update the number of replicas later, you must recreate the user cluster.
Example:
masterNode: replicas: 3
masterNode.autoResize.enabled
Mutable
Boolean
Prepopulated: false
Default: false
Set this to true
to enable automatic resizing of the control-plane
nodes for the user cluster. Otherwise, set this to false
.
For more information, see Enable node resizing for the control-plane nodes of a user cluster.
Example:
masterNode: autoResize: enabled: true
masterNode.vsphere.datastore
Optional
Immutable
String
Default: The value of vCenter.datastore
The datastore where the control-plane nodes for this user cluster will be created.
The value you specify must be a name, not a path. Don't include any folders in the value.
Example:
masterNode: vSphere: datastore: "control-plane-datastore"
If you specify a value for this field, don't specify a value for
masterNode.vsphere.storagePolicyName
. The masterNode.vsphere.datastore
field
is immutable except when you set the field to an empty string when you
migrate a datastore to Storage Policy Based Management (SPBM).
masterNode.vsphere.storagePolicyName
Optional
Immutable
String
Default: The value of vCenter.storagePolicyName
The name of a VM storage policy for the control-plane nodes.
For more information, see Configure a storage policy.
Example:
masterNode: vSphere: storagePolicyName: "control-plane-storage-policy"
If you specify a value for this field, don't specify a value for
masterNode.vsphere.datastore
.
nodePools
Required
Mutable
Array of objects, each of which describes a node pool.
For more information, see Creating and managing node pools.
nodePools[i].name
Required
Immutable
String
A name of your choice for the node pool. The name must:
- contain at most 40 characters
- contain only lowercase alphanumeric characters or a hyphen (
-
) - start with an alphabetic character
- end with an alphanumeric character
Example:
nodePools: - name: "my-node-pool-1"
nodePools[i].gkeOnPremVersion
When you upgrade a user cluster, you can specify that selected node pools remain at the previous version.
If you want this node pool to remain at the previous version, set this to the previous version. Otherwise, remove this field or set it to the empty string. For more information, see Upgrade node pools.
Mutable
String
Default: The cluster gkeOnPremVersion
Example:
nodePools: - name: "my-node-pool" gkeOnPremVersion: "1.13.0-gke.16"
nodePools[i].cpus
Required
Mutable
Integer
Prepopulated: 4
The number of vCPUs for each node in the pool.
Example:
nodePools" - name: "my-node-pool" cpus: 8
nodePools[i].memoryMB
Required
Mutable
Integer
Prepopulated: 8192
The mebibytes of memory for each node in the pool. Must be a multiple of 4.
Example:
nodePools" - name: "my-node-pool" memoryMB: 8192
Note: This field specifies the number of mebibytes of memory, not the number of megabytes. One mebibyte is 2^20 = 1,048,576 bytes. One megabyte is 10^6 = 1,000,000 bytes.
nodePools[i].replicas
Required
Mutable
Integer
Prepopulated: 3
Possible values: The total number of untainted nodes across all node pools in
the cluster must be at least 3.
The number of nodes in the pool.
Example:
nodePools: - name: "my-node-pool" replicas: 5
nodePools[i].bootDiskSizeGB
Mutable
Integer
Prepopulated: 40
Default: 40
The size of the boot disk in gibibytes for each node in the pool.
Example:
nodePools - name: "my-node-pool" bootDiskSizeGB: 40
nodePools[i].osImageType
Mutable
String
Possible values: "ubuntu_containerd", "cos", "ubuntu_cgv2", "cos_cgv2", "windows"
Prepopulated: "ubuntu_cgv2"
Default: "ubuntu_containerd"
The type of OS image to run on the VMs in the node pool.
Example:
nodePools - name: "my-node-pool" osImageType: "ubuntu_cgv2"
nodePools[i].osImage
Required if nodePools[i].osImageType
= windows
Mutable
String
The name of a Windows VM template. The template must be based on a Windows Server 2019 ISO that has the en-US language/region tag.
For more information, see User guide for Windows Server OS node pools.
nodePools[i].labels
Mutable
Mapping
Labels to apply to each Kubernetes Node object in the pool.
Example:
nodePools: - name: "my-node-pool" labels: environment: "production" tier: "cache"
The kubelet can't apply labels to itself in certain namespaces for security reasons.
The reserved node label namespaces are : kubernetes.io
, k8s.io
, googleapis.com
.
nodePools[i].taints
Mutable
Array of objects, each of which describes a
Kubernetes taint that is applied to each
node in the pool. Taints are key-value pairs associated with an effect
. Taints
are used with tolerations for Pod scheduling. Specify one of the
following for the effect
: NoSchedule
, PreferNoSchedule
, NoExecute
.
Example:
nodePools: - name: "my-node-pool" taints: - key: "staging" value: "true" effect: "NoSchedule"
nodePools[i].vsphere.datastore
Mutable
String
Default: The value of vCenter.datastore
The name of the vCenter datastore where the nodes will be created.
Example:
nodePools: - name: "my-node-pool" vsphere: datastore: "my-datastore"
If you specify a value for this field, don't specify a value for
nodePools[i].vsphere.storagePolicyName
.
nodePools[i].vsphere.storagePolicyName
Mutable
String
Default: The value of vCenter.storagePolicyName
The name of a VM storage policy for the nodes in the pool.
For more information, see Configure a storage policy.
Example:
nodePools: - name: "my-node-pool" vsphere: storagePolicyName: "my-storage-policy"
If you specify a value for this field, don't specify a value for
nodePools[i].vsphere.datastore
.
nodePools[i].vsphere.hostgroups
Fill in this section if you want to configure your cluster to use VM-Host affinity.
Mutable
Array of strings, each of which is the name of a
host DRS group.
The array can have only one element.
Example:
nodePools: - name: "my-node-pool" vsphere: hostgroups: - "my-hostgroup"
nodePools[i].vsphere.tags
Mutable
Array of objects, each of which describes a
vSphere tag
to be placed on the VMs in the node pool.
Each tag has a category and a name.
Example:
nodePools: - name: "my-node-pool" vsphere: tags: - category: "purpose" name: "testing"
If you want to attach tags to all VMs in a node pool, your vCenter user account must have these vSphere tagging privileges:
- vSphere Tagging.Assign or Unassign vSphere Tag
- vSphere Tagging.Assign or Unassign vSphere Tag on Object (vSphere 7)
nodePools[i].autoscaling
Mutable
If you want to enable automatic scaling for the node pool, fill in this section. Otherwise, remove this section or leave it commented out.
nodePools[i].autoscaling.minReplicas
Mutable
Integer
Possible values: Greater than or equal to 1
Prepopulated: 0
The minimum number of nodes that the autoscaler can set for the pool.
Example:
nodePools: - name: "my-node-pool" autoscaling: minReplicas: 5
nodePools[i].autoscaling.maxReplicas
Mutable
Integer
Prepopulated: 0
The maximum number of nodes that the autoscaler can set for the pool.
Example:
nodePools: - name: "my-node-pool" autoscaling: maxReplicas: 10
nodePools[i].enableLoadBalancer
Relevant if loadBalancer.kind
= "MetalLB"
Mutable
Boolean
Prepopulated: true
Default: false
Set this to true
if you want to allow the MetalLB speaker to run on the nodes
in the pool. Otherwise, set it to false
.
For more information, see Bundled load balancing with MetalLB.
Example:
nodePools: - name: "my-node-pool" enableLoadBalancer: true
nodePools[i].updateStrategy.rollingUpdate.maxSurge
Preview
Mutable
Integer
Default: 1
The maximum number of machines in the node pool that can be updated simultaneously during an update or upgrade.
nodePools: - name: "my-node-pool" updateStrategy: rollingUpdate: maxSurge: 3
antiAffinityGroups.enabled
Mutable
Boolean
Prepopulated: true
Default: REVIEWERS: DOES THIS HAVE A DEFAULT VALUE?
Set this to true
to enable DRS rule creation. Otherwise, set this to
false
.
Example:
antiAffinityGroups: enabled: true
If this field is true
Google Distributed Cloud creates VMware
Distributed Resource Scheduler
(DRS) anti-affinity rules for your user cluster's nodes, causing them to be
spread across at least three physical ESXi hosts in your datacenter.
This feature requires that your vSphere environment meets the following conditions:
VMware DRS is enabled. VMware DRS requires vSphere Enterprise Plus license edition.
Your vSphere user account has the
Host.Inventory.Modify cluster
privilege.There are at least four ESXi hosts available.
Even though the rule requires that the cluster nodes are spread across three ESXi hosts, we strongly recommend that you have at least four ESXi hosts available.
Recall that if you have a vSphere Standard license, you cannot enable VMware DRS.
If you do not have DRS enabled, or if you do not have at least four hosts where
vSphere VMs can be scheduled, set antiAffinityGroups.enabled
to false
.
enableVMTracking
Preview
Immutable
Prepopulated: false
Set this to true
to enable VM tracking with vSphere tags. Otherwise, set it
to false
.
For more information, see Enable VM tracking.
Example:
enableVMTracking: true
nodePoolUpdatePolicy.maximumConcurrentNodePoolUpdate
Mutable
Integer
Possible values: 0 or 1
Prepopulated: 0
Default: TODO: REVIEWERS: DOES THIS HAVE A DEFAULT VALUE?
The number of node pools to update at a time. A value of 1
specifies that one
node pool can be updated at a time. A value of 0
specifies that an unlimited
number of node pools can be updated at a time.
Example:
nodePoolUpdatePolicy: maximumConcurrentNodePoolUpdate: 1
nodePoolUpdatePolicy.updateStrategy.rollingUpdate.maxSurge
Preview
Mutable
Integer
Default: 1
The maximum number of machines in a node pool that can be updated simultaneously during an update or upgrade. Applies to any node pool that doesn't specify its own update strategy.
nodePools: - name: "my-node-pool" updateStrategy: rollingUpdate: maxSurge: 3
authentication
This section holds information about how cluster users are authenticated and authorized.
authentication.oidc
Do not use this section. Instead, after cluster creation, edit the ClientConfig custom resource as described in Configuring clusters for Anthos Identity Service with OIDC .
authentication.sni
If you want to provide an additional serving certificate for the cluster's Kubernetes API server, fill in this section. Otherwise, remove this section or leave it commented out.
authentication.sni.certPath
String
The path of a serving certificate for the Kubernetes API server.
Example:
authentication: sni: certPath: "my-cert-folder/example.com.crt"
authentication.sni.keyPath
Optional
String
The path of the certificate's private key file.
Example:
authentication: sni: keyPath: "my-cert-folder/example.com.key"
gkeConnect
Required
Immutable
This section holds information about the Google Cloud project and service account you want to use to register your cluster to a Google Cloud fleet.
gkeConnect.projectID
Required
Immutable
String
The ID of your
fleet host project.
For new clusters, this project ID must be the same as the ID set in
stackdriver.projectID
and cloudAuditLogging.projectID
. If the project IDs
aren't the same, cluster creation fails. This requirement isn't applied to
existing clusters.
Example:
gkeConnect: projectID: "my-fleet-host-project"
gkeConnect.location
Immutable
String
Default: global
Each cluster's fleet membership is managed by the Fleet service
(gkehub.googleapis.com
) and the Connect service
(gkeconnect.googleapis.com
). The location of the services can be global
or regional. In 1.28 and later, you can optionally specify the Google Cloud
region in which the Fleet and the Connect services run. If not specified,
the global instances of the services are used. Note the following:
User clusters created prior to 1.28 are managed by the global Fleet and Connect services.
New clusters created using the GKE On-Prem API clients— the Google Cloud console, the Google Cloud CLI, or Terraform—use the same region that you specify for the GKE On-Prem API.
For new clusters, if you include this field, the region that you specify must be the same as the region configured in
cloudAuditLogging.clusterLocation
,stackdriver.clusterLocation
, andgkeOnPremAPI.location
. If the regions aren't the same, cluster creation fails.
Example:
gkeConnect: location: "us-central1"
gkeConnect.registerServiceAccountKeyPath
Required
Mutable
String
The path of the JSON key file for your connect-register service account.
To update the value of this field, use gkectl update cluster
.
For information about changing your connect-register service account key, see Rotating service account keys.
Example:
gkeConnect: registerServiceAccountKeyPath: "my-key-folder/connect-register-key.json"
gkeConnect.registerServiceAccountKey.secretRef.version
Mutable
String
Possible values: An integer string or "latest"
Default value: "latest"
The version of a prepared Secret in the admin cluster that holds a JSON key for your connect-register service account.
Example:
gkeConnect: registerServiceAccountKey: secretRef: version: "1"
gkeOnPremAPI
In 1.16 and later, if the GKE On-Prem API is enabled in your
Google Cloud project, all clusters in the project are
enrolled in the GKE On-Prem API
automatically in the region configured in stackdriver.clusterLocation
.
If you want to enroll all clusters in the project in the GKE On-Prem API, be sure to do the steps in Before you begin to activate and use the GKE On-Prem API in the project.
If you don't want to enroll the cluster in the GKE On-Prem API, include this section and set
gkeOnPremAPI.enabled
tofalse
. If you don't want to enroll any clusters in the project, disablegkeonprem.googleapis.com
(the service name for the GKE On-Prem API) in the project. For instructions, see Disabling services.
Enrolling your user cluster in the GKE On-Prem API lets you use
the standard tools—the Google Cloud console, the Google Cloud CLI, and
Terraform— to manage the
lifecycle of the cluster. Additionally, enrolling the cluster lets you
use the console or the gcloud CLI to view
cluster details. For example, you run can gcloud
commands to
get information about your user cluster.
After you add this section and create or update the cluster, if subsequently you remove the section and update the cluster, the update will fail.
gkeOnPremAPI.enabled
Mutable
Boolean
Default: true
By default, the cluster is enrolled in the GKE On-Prem API if the
GKE On-Prem API is enabled in your project. Set to false
if you
don't want to enroll the cluster.
After the cluster is enrolled in the GKE On-Prem API, if you need to unenroll the cluster, make the following change and then update the cluster:
gkeOnPremAPI: enabled: false
gkeOnPremAPI.location
Immutable
String
Default: stackdriver.clusterLocation
The Google Cloud region where the GKE On-Prem API runs and stores
cluster metadata. Choose one of the
supported regions.
You must use the same region that is configured in gkeConnect.location
,
stackdriver.clusterLocation
and cloudAuditLogging.clusterLocation
. If
gkeOnPremAPI.enabled
is false
, don't include this field.
stackdriver
Required by default
Mutable
If you want to enable Cloud Logging and Cloud Monitoring for your cluster, fill in this section. Otherwise remove this section or leave it commented out.
This section is required by default. That is, if you don't include this section,
you must include the --skip-validation-stackdriver
flag when you run
gkectl create cluster
.
This section is required in the user cluster if you want to manage the lifecycle of the user clusters in the Google Cloud console.
stackdriver.projectID
Required for Logging and Monitoring
Immutable
String
The ID of your
fleet host project.
For new clusters, this project ID must be the same as the ID set in
gkeConnect.projectID
and cloudAuditLogging.projectID
. If the project IDs
aren't the same, cluster creation fails. This requirement isn't applied to
existing clusters.
If needed, you can configure a log router in this project to route logs into log buckets in another project. For information on how to configure the log router, see Supported destinations.
Example:
stackdriver: projectID: "my-fleet-host-project"
stackdriver.clusterLocation
Required for Logging and Monitoring
Immutable
String
Prepopulated: "us-central1"
The Google Cloud region where you want to store logs. It is a good idea to choose a region that is near your on-premises data center.
For new clusters, if you include the gkeOnPremAPI
and cloudAuditLogging
sections in the configuration file, the region that you set here must be
the same region that you set in gkeConnect.location
, gkeOnPremAPI.location
,
and cloudAuditLogging.clusterLocation
. If the regions aren't the same,
cluster creation fails.
Example:
stackdriver: clusterLocation: "us-central1"
stackdriver.enableVPC
Immutable
Boolean
Prepopulated: false
If your cluster's network is controlled by a VPC, set this to true
.
This ensures that all telemetry flows through Google's restricted IP addresses.
Otherwise, set this to false
.
Example:
stackdriver: enableVPC: false
stackdriver.serviceAccountKeyPath
Required for Logging and Monitoring
Mutable
String
The path of the JSON key file for your logging-monitoring service account.
To update the value of this field, use gkectl update cluster
.
For information about changing your logging-monitoring service account key, see Rotating service account keys.
Example:
stackdriver: serviceAccountKeyPath: "my-key-folder/log-mon-key.json"
stackdriver.serviceAccountKey.secretRef.version
Mutable
String
Possible values: An integer string or "latest"
Default value: "latest"
The version of a prepared Secret in the admin cluster that holds a JSON key for your logging monitoring service account.
Example:
stackdriver: serviceAccountKey: secretRef: version: "1"
stackdriver.disableVsphereResourceMetrics
Mutable
Boolean
Prepopulated: false
Default: false
Set this to true
to disable the collection of metrics from vSphere.
Otherwise, set it to false
.
Example:
stackdriver: disableVsphereResourceMetrics: true
usageMetering
Preview
Immutable
If you want to enable usage metering for your cluster, then fill in this section. Otherwise, remove this section or leave it commented out.
usageMetering.bigQueryProjectID
Preview
Required for usage metering
Immutable
String
The ID of the Google Cloud project where you want to store usage metering data.
Example:
usageMetering: bigQueryProjectID: "my-bq-project"
usageMetering.bigQueryDatasetID
Preview
Required for usage metering
Immutable
String
The ID of the BigQuery dataset where you want to store usage metering data. Example:
usageMetering: bigQueryDatasetID: "my-bq-dataset"
usageMetering.bigQueryServiceAccountKeyPath
Preview
Required for usage metering
Immutable
String.
The path of the JSON key file for your BigQuery service account.
To update the value of this field, use gkectl update cluster
.
For information about changing your BigQuery service account key, see Rotating service account keys.
Example:
usageMetering: bigQueryServiceAccountKeyPath: "my-key-folder/bq-key.json"
usageMetering.bigQueryServiceAccountKey.secretRef.version
Mutable
String
Possible values: An integer string or "latest"
Default value: "latest"
The version of a prepared Secret in the admin cluster that holds a JSON key for your BigQuery service account.
Example:
gkeConnect: bigQueryServiceAccountKey: secretRef: version: "1"
usageMetering.enableConsumptionMetering
Preview
Required for usage metering
Immutable
Boolean
Prepopulated: false
Set this to true
if you want to enable consumption-based metering.
Otherwise set this to false.
Example:
usageMetering: enableConsumptionMetering: true
cloudAuditLogging
If you want to integrate the audit logs from your cluster's Kubernetes API server with Cloud Audit Logs, fill in this section. Otherwise, remove this section or leave it commented out.
cloudAuditLogging.projectID
Required for Cloud Audit Logs
Immutable
String
The ID of your fleet host project.
For new clusters, this project ID must be the same as the ID set in
gkeConnect.projecID
and stackdriver.projectID
. If the project IDs
aren't the same, cluster creation fails. This requirement isn't applied to
existing clusters.
If needed, you can configure a log router in this project to route logs into log buckets in another project. For information on how to configure the log router, see Supported destinations.
Example:
cloudAuditLogging: projectID: "my-fleet-host-project"
cloudAuditLogging.clusterLocation
Required for Cloud Audit Logs
Immutable
String
The Google Cloud region where you want to store audit logs. It is a good idea to choose a region that is near your on-premises data center
For new clusters, if you include the gkeOnPremAPI
and stackdriver
sections in the configuration file, the region that you set here must be
the same region that you set in gkeConnect.location
,
gkeOnPremAPI.location
, and stackdriver.clusterLocation
. If the regions
aren't the same, cluster creation fails.
Example:
cloudAuditLogging: clusterLocation: "us-central1"
cloudAuditLogging.serviceAccountKeyPath
Required for Cloud Audit Logs
Mutable
String
The path of the JSON key file for your audit-logging service account.
To update the value of this field, use gkectl update cluster
.
For information about changing your audit-logging service account key, see Rotating service account keys.
Example:
cloudAuditLogging: serviceAccountKeyPath: "my-key-folder/audit-log-key.json"
cloudAuditLogging.serviceAccountKey.secretRef.version
Mutable
String
Possible values: An integer string or "latest"
Default value: "latest"
The version of a prepared Secret in the admin cluster that holds a JSON key for your audit logging service account.
Example:
cloudAuditLogging: serviceAccountKey: secretRef: version: "1"
autoRepair.enabled
Optional
Mutable
Boolean
Prepopulated: true
Set this to true
to enable node auto repair. Otherwise, set it to false
.
Example:
autoRepair: enabled: true
secretsEncryption
If you want to encrypt Secrets without the need for an external KMS (Key Management Service), or any other dependencies, fill in this section. Otherwise, remove this section or leave it commented out.
secretsEncryption.mode
Required for Secrets encryption
Immutable
String
Possible value: "GeneratedKey"
Prepopulated: "GeneratedKey"
The Secret encryption mode.
secretsEncryption: mode: "GeneratedKey"
secretsEncryption.generatedKey.keyVersion
Required for Secrets encryption
Mutable
Integer
Prepopulated: 1
An integer of your choice to use for the key version number. We recommend that
you start with 1
.
For information about updating this field for an existing cluster, see Always-on Secrets encryption.
Example:
secretsEncryption: generatedKey: keyVersion: 1
secretsEncryption.generatedKey.disabled
Mutable
Boolean
Prepopulated: false
Set this to true
to disable Secrets encryption. Otherwise set it to false
.
For information about updating this field for an existing cluster, see Always-on Secrets encryption.
Example:
secretsEncryption: generatedKey: disabled: false