Version 1.19. This version is no longer supported. For information about how to upgrade to version 1.10, see Upgrading Anthos on bare metal in the 1.10 documentation. For more information about supported and unsupported versions, see the Version history page in the latest documentation.
This page describes the fields supported in the GKE on Bare Metal cluster
configuration file. For each field, the following table identifies whether the
field is required. The table also shows which fields are mutable, meaning which
fields can be changed after a cluster has been created.
Generating a template for your cluster configuration file
You can create a cluster configuration file with the bmctl create config
command. Although some fields have default values and others, such as
metadata.name can be auto-filled, this YAML format configuration file is a
template for specifying information about your cluster.
To create a new cluster configuration file, use the following command in the
/baremetal folder:
bmctl create config -c CLUSTER_NAME
Replace CLUSTER_NAME with the name for the cluster you want
to create. For more information about bmctl, see bmctl tool.
For an example of the generated cluster configuration file, see
Cluster configuration file sample.
Filling in your configuration file
In your configuration file, enter field values as described in the following
field reference table before you create or upgrade your cluster.
Cluster configuration fields
Field name
Resource type
Required?
Mutable?
anthosBareMetalVersion
Required. String. The cluster version. This value is set for cluster
creation and cluster upgrades.
Mutability: This value can't be modified for existing clusters.
The version can be updated only through the
cluster upgrade process.
Cluster resource
Yes
Yes
authentication
This section contains settings needed to use OpenID Connect (OIDC).
OIDC lets you use your existing identity provider to manage user and
group authentication in Anthos clusters on bare metal clusters.
Cluster resource
—
—
authentication.oidc.certificateAuthorityData
Optional. A base64-encoded
PEM-encoded certificate for the OIDC provider. To create the
string, encode the certificate, including headers, into
base64. Include the resulting string in
certificateAuthorityData as a single line.
Optional. String. The ID for the client application that makes
authentication requests to the OpenID provider.
Cluster resource
No
No
authentication.oidc.clientSecret
Optional. String. Shared secret between OIDC client application and
OIDC provider.
Cluster resource
No
No
authentication.oidc.deployCloudConsoleProxy
Optional. Boolean (true|false). Specifies whether a reverse proxy
is deployed in the cluster to connect Google Cloud console to an
on-premises identity provider that is not publicly accessible over the
internet. If your identity provider isn't reachable over the public
internet, set this field to true to authenticate with
Google Cloud console. By default this value is set to false.
Cluster resource
No
No
authentication.oidc.extraParams
Optional. Comma-delimited list. Additional key-value parameters to
send to the OpenID provider.
Cluster resource
No
No
authentication.oidc.groupPrefix
Optional. String. Prefix prepended to group claims to prevent clashes
with existing names. For example, given a group dev and a prefix
oidc:, oidc:dev.
Cluster resource
No
No
authentication.oidc.group
Optional. String.
JWT
claim that the provider uses to return your security groups.
Cluster resource
No
No
authentication.oidc.issuerURL
Optional. URL string. URL where authorization requests are sent to
your OpenID, such as https://example.com/adfs. The Kubernetes API
server uses this URL to discover public keys for verifying tokens. The
URL must use HTTPS.
Cluster resource
No
No
authentication.oidc.kubectlRedirectURL
Optional. URL string. The redirect URL that kubectl uses for
authorization. When you enable OIDC, you must specify a
kubectlRedirectURL value.
Cluster resource
No
No
authentication.oidc.proxy
Optional. URL string. Proxy server to use for the cluster to connect
to your OIDC provider, if applicable. The value should include a
hostname/IP address and optionally a port, username, and password. For
example: http://user:password@10.10.10.10:8888.
Cluster resource
No
No
authentication.oidc.scopes
Optional. Comma-delimited list. Additional scopes to send to the
OpenID provider. Microsoft Azure and Okta require the offline_access
scope.
Cluster resource
No
No
authentication.oidc.usernamePrefix
Optional. String. Prefix prepended to username claims.
Cluster resource
No
No
authentication.oidc.username
Optional. String.
JWT
claim to use as the username. If not specified, defaults to sub.
Cluster resource
No
No
bypassPreflightCheck
Optional. Boolean (true|false). When set to
true, the internal preflight checks are ignored when
applying resources to existing clusters. Defaults to false.
Mutability: This value can be modified for existing clusters
with the bmctl update command.
Cluster resource
No
Yes
clusterNetwork
This section contains network settings for your cluster.
Cluster resource
—
—
clusterNetwork.multipleNetworkInterfaces
Optional. Boolean. Set this field to true to enable
multiple network interfaces for your pods. This capability is in
Preview.
Required. Range of IPv4 addresses in CIDR block format. Pods specify
the IP ranges from which pod networks are allocated.
For example:
pods:
cidrBlocks:
- 192.168.0.0/16
Cluster resource
Yes
No
clusterNetwork.services.cidrBlocks
Required. Range of IPv4 addresses in CIDR block format. Specify the
range of IP addresses from which service virtual IP (VIP) addresses
are allocated. The ranges must not overlap with any subnets reachable
from your network. For more information about address allocation for
private internets, see
RFC 1918.
For example:
services:
cidrBlocks:
- 10.96.0.0/12
Cluster resource
Yes
No
clusterOperations
This section holds information for Cloud Logging and
Cloud Monitoring.
Cluster resource
—
—
clusterOperations.enableApplication
Boolean. Set to true to collect application logs/metrics
in addition to the default collection of system logs/metrics, which
correspond to system components such as the Kubernetes control plane
or cluster management agents. You can change this value at any time.
Cluster resource
No
Yes
clusterOperations.disableCloudAuditLogging
Boolean. Cloud Audit Logs is useful for investigating suspicious API
requests and for collecting statistics. Starting with
GKE on Bare Metal release 1.9.0, Cloud Audit Logs is enabled
(disableCloudAuditLogging: false) by default. Set to
true to disable Cloud Audit Logs.
String. A Google Cloud region where you want to store
Logging logs and Monitoring metrics.
It's a good idea to choose a region that is near your on-premises data
center. For more information, see
Global Locations.
For example:
location: us-central1
Cluster resource
Yes
No
clusterOperations.projectID
String. The project ID of the Google Cloud project where you want to view
logs and metrics.
Cluster resource
Yes
No
controlPlane
This section holds information about the control plane and its
components.
Cluster resource
—
—
controlPlane.nodePoolSpec
This section specifies the IP addresses for the node pool used by the
control plane and its components. The control plane node pool
specification (like the
load balancer node pool specification)
is special. This specification declares and controls critical cluster
resources. The canonical source for this resource is this section in
the cluster configuration file. Don't modify the top-level control
plane node pool resources directly. Modify the associated sections in
the cluster configuration file instead.
Cluster resource
—
—
controlPlane.nodePoolSpec.nodes
Required. An array of IP addresses. Typically, this array is either
an IP address for a single machine, or IP addresses for three machines
for a high-availability (HA) deployment.
This value can't be modified for existing clusters.
Cluster resource
Yes
No
kubevirt.useEmulation
Boolean. Determines whether or not software emulation is used to run
virtual machines. If the node supports hardware virtualization, set
useEmulation to false for better
performance. If hardware virtualization isn't supported or you aren't
sure, set it to true.
Cluster resource
No
Yes
loadBalancer
This section contains settings for cluster load balancing.
Cluster resource
—
—
loadBalancer.addressPools
Object. The name and an array of IP addresses for your cluster load
balancer pool. Address pool configuration is only valid for
bundled LB mode in non-admin clusters. You can add new
address pools at any time, but you can't modify or remove existing
address pools.
Cluster resource
No
No
loadBalancer.addressPools.addresses
Array of IP address ranges. Specify a list of non-overlapping IP
ranges for the data plane load balancer. All addresses must be in the
same subnet as the load balancer nodes.
String. The name you choose for your cluster load balancer pool.
Cluster resource
Yes
No
loadBalancer.addressPools.avoidBuggyIPs
Optional. Boolean (true | false). If true,
the pool omits IP addresses ending in .0 and .255.
Some network hardware drops traffic to these special addresses. You
can omit this field, its default value is false.
Cluster resource
No
No
loadBalancer.addressPools.manualAssign
Optional. Boolean (true | false). If true,
addresses in this pool are not automatically assigned to Kubernetes
Services. If true, an IP address in this pool is used
only when it is specified explicitly by a service. You can omit this
field, its default value is false.
Cluster resource
No
Yes
loadBalancer.mode
Required. String. Specifies the load-balancing mode. In
bundled mode, GKE on Bare Metal installs a load
balancer on load balancer nodes during cluster creation. In
manual mode, the cluster relies on a manually configured
external load balancer. For more information, see
Overview of load balancers.
Allowed values: bundled | manual
Cluster resource
Yes
No
loadBalancer.type
Optional. String. Specifies the type of bundled load-balancing used,
Layer 2 or Border Gateway Protocol (BGP). If you are using the
standard, bundled load
balancing, set type to layer2. If you
are using bundled load
balancing with BGP, set type to bgp. If
you do not set type, it defaults to layer2.
Allowed values: layer2 | bgp
Cluster resource
No
No
loadBalancer.nodePoolSpec
Optional. Use this section to configure a load balancer node pool. The
nodes you specify are part of the Kubernetes cluster and run regular
workloads and load balancers. If you don't specify a node pool, then
the control plane nodes are used for load balancing. This section
applies only when the load-balancing mode is set to bundled.
Cluster resource
—
—
loadBalancer.nodePoolSpec.nodes
This section contains an array of IP addresses for the nodes in your
load-balancer node pool.
Cluster resource
No
Yes
loadBalancer.nodePoolSpec.nodes.address
Optional. String (IPv4 address). IP address of a node.
Cluster resource
No
Yes
loadBalancer.ports.controlPlaneLBPort
Number. The destination port used for traffic sent to the Kubernetes
control plane (the Kubernetes API servers).
Cluster resource
Yes
No
loadBalancer.vips.controlPlaneVIP
Required. Specifies the virtual IP address (VIP) to connect to the
Kubernetes API server. This address must not fall within the range of
any IP addresses used for load balancer address pools,
loadBalancer.addressPools.addresses.
Cluster resource
Yes
No
loadBalancer.vips.ingressVIP
Optional. String (IPv4 address). The IP address that you have chosen
to configure on the load balancer for ingress traffic.
Optional. String. Specifies the autonomous system number (ASN) for the
cluster being created. This field is used when setting up the bundled
load-balancing solution that uses border gateway protocol (BGP).
For more information, see
Configure bundled load balancers with BGP.
Optional. Object (list of mappings). This section specifies one or
more border gateway protocol (BGP) peers from your (external to the
cluster) local network. You specify BGP peers when you set up control
plane load balancing part of the bundled load-balancing solution that
uses BGP. Each peer is specified with a mapping, consisting of an IP
address, an autonomous system number (ASN), and, optionally, a list of
one or more IP addresses for control plane nodes. The BGP-peering
configuration for control plane load balancing can't be updated after
the cluster has been created.
Optional. String (IPv4 address). The IP address of an external peering
device from your local network.
For more information, see
Configure bundled load balancers with BGP.
Optional. String. The autonomous system number (ASN) for the network
that contains the external peer device. Specify an ASN for every BGP
peer you set up for control plane load balancing, when you set up the
bundled load-balancing solution that uses BGP.
For more information, see
Configure bundled load balancers with BGP.
Optional. Array of IP (IPv4) addresses. One or more IP addresses for
control plane nodes that connect to the external BGP peer, when you
set up the bundled load-balancing solution that uses BGP. If you
don't specify any control plane nodes, all control plane nodes will
connect to the external peer. If you specify one or more IP addresses,
only the nodes specified participate in peering sessions.
For more information, see
Configure bundled load balancers with BGP.
Cluster resource
No
No
maintenanceBlocks.cidrBlocks
Optional. Single IPv4 address or a range of IPv4 addresses. Specify
the IP addresses for the node machines you want to put into
maintenance mode. For more information, see
Put nodes into
maintenance mode.
For example:
maintenanceBlocks:
cidrBlocks:
- 192.168.1.200 # Single machine
- 192.168.1.100-192.168.1.109 # Ten machines
Cluster resource
No
Yes
nodeAccess.loginUser
Optional. String. Specify the non-root username you want to use for
passwordless SUDO capability access to the node machines in your
cluster. Your SSH key,
sshPrivateKeyPath, must
work for the specified user. The cluster create and update operations
check that node machines can be accessed with the specified user and
SSH key.
Cluster resource
No
Yes
osEnvironmentConfig.addPackageRepo
Optional. Boolean (true|false). Specifies whether adding package repo when initializing bare metal machines.
Cluster resource
No
No
nodeConfig.containerRuntime
Optional. String. Specify which container runtime, docker
or containerd, to use for scheduling containers on nodes.
If you don't specify the container runtime, it defaults to
docker. For more information about setting the container
runtime, see Change your
container runtime.
Allowed values: containerd | docker
Cluster resource
No
No
nodeConfig.podDensity
This section specifies the pod density configuration.
Cluster resource
—
—
nodeConfig.podDensity.maxPodsPerNode
Optional. Integer. Specifies the maximum number of pods that can be
run on a single node. For self-managed clusters, allowable values for
maxPodsPerNode are 32–250 for
high-availability (HA) clusters and 64–250
for non-HA clusters. For user clusters, allowable values for
maxPodsPerNode are 32–250.
The default value if unspecified is 110. Once the cluster
is created, this value can't be updated.
Kubernetes assigns a
Classless Inter-Domain Routing (CIDR) block
to each node so that each pod can have a unique IP address. The size
of the CIDR block corresponds to the maximum number of pods per node.
For more information about setting the maximum number of pods per node,
see Pod networking.
Cluster resource
No
No
profile
Optional. String. When profile is set to edge
for a standalone cluster, it minimizes the resource consumption of the
cluster. The edge profile is available for standalone clusters only.
The edge profile has reduced system resource requirements and is
recommended for edge devices with restrictive resource constraints.
For hardware requirements associated with the edge profile, see
Resource
requirements for standalone clusters using the edge profile.
Cluster resource
No
No
proxy
If your network is behind a proxy server, fill in this section.
Otherwise, remove this section.
Cluster resource
—
—
proxy.noProxy
String. A comma-separated list of IP addresses, IP address ranges,
host names, and domain names that should not go through the proxy
server. When GKE on Bare Metal sends a request to one of these
addresses, hosts, or domains, the request is sent directly.
Cluster resource
No
No
proxy.url
String. The HTTP address of your proxy server. Include the port number
even if it's the same as the scheme's default port.
Required. String. Use the path field to specify the host
machine path where mounted disks can be discovered. A local
PersistentVolume (PV) is created for each mount. The default path is
/mnt/localpv-share. For instructions for configuring your node
mounts, see
Configure
LVP node mounts.
Cluster resource
Yes
No
storage.lvpNodeMounts
This section specifies the configuration (path) for local persistent
volumes backed by mounted disks. You must format and mount these disks
yourself. You can do this task before or after cluster creation. For
more information, see
LVP
node mounts.
Cluster resource
Yes
No
storage.lvpShare
This section specifies the configuration for local persistent volumes
backed by subdirectories in a shared file system. These subdirectories
are automatically created during cluster creation.
For more information, see
LVP
share.
Cluster resource
Yes
No
storage.lvpShare.path
Required. String. Use the path field to specify the host
machine path where subdirectories can be created. A local
PersistentVolume (PV) is created for each subdirectory. For
instructions to configure your LVP share, see
Configuring
an LVP share.
Cluster resource
Yes
No
storage.lvpShare.numPVUnderSharedPath
Required. String. Specify the number of subdirectories to create under
lvpShare.path. The default value is 5. For
instructions to configure your LVP share, see
Configuring
an LVP share.
Cluster resource
Yes
No
storage.lvpShare.storageClassName
Required. String. Specify the StorageClass to use to create persistent
volumes. The StorageClass is created during cluster creation. The
default value is local-shared. For instructions to
configure your LVP share, see
Configuring
an LVP share.
Cluster resource
No
No
type
Required. String. Specifies the type of cluster. The standard deployment model
consists of a single admin cluster and one or more user clusters, which are
managed by the admin cluster. GKE on Bare Metal supports the following
types of clusters:
Admin - cluster used to manage user clusters.
User - cluster used to run workloads.
Hybrid - single cluster for both admin and workloads, that can also manage
user clusters.
Standalone - single cluster that can administer itself, and that can also
run workloads, but can't create or manage other user clusters.
Cluster type is specified at cluster creation and can't be changed for updates
or upgrades. For more information about how to create a cluster, see
Creating clusters: overview.
Allowed values: admin | user | hybrid | standalone
This value can't be modified for existing clusters.
Cluster resource
Yes
No
name
Required. String. Typically, the namespace name uses a pattern of
cluster-CLUSTER_NAME, but the
cluster- prefix is not strictly required since
GKE on Bare Metal release 1.7.2.
This value can't be modified for existing clusters.
Namespace resource
Yes
No
clusterName
String. Required. The name of the cluster to which you are adding the
node pool. Create the node pool resource in the same namespace as the
associated cluster and reference the cluster name in this field. For
more information, see
Add and remove
node pools in a cluster.
Optional. Array of IP (IPv4) addresses. This defines the node pool for
your worker nodes.
NodePool resource
No
Yes
nodes.address
Optional. String (IPv4 address). One or more IP addresses for the
nodes that make your pool for worker nodes.
NodePool resource
No
Yes
taints
Optional. Object. A node taint lets you mark a node so that the
scheduler avoids or prevents using it for certain pods. A taint
consists of a key-value pair and an associated effect. The
key and value values are strings you use to
identify the taint and the effect value specifies how
pods are handled for the node. The taints object can have
multiple taints.
The effect field can take one of the following values:
NoSchedule - no pod is able to schedule onto the
node unless it has a matching toleration.
PreferNoSchedule - the system avoids placing a pod
that does not tolerate the taint on the node, but it is not
required.
NoExecute - pods that do not tolerate the taint
are evicted immediately, and pods that do tolerate the taint are
never evicted.
For GKE on Bare Metal, taints are reconciled to the nodes of the
node pool unless the baremetal.cluster.gke.io/label-taint-no-sync
annotation is applied to the cluster. For more information about
taints, see
Taints and Tolerations.
For example:
taints:
- key: status
value: testpool
effect: NoSchedule
NodePool resource
No
Yes
labels
Optional. Mapping (key-value pairs).
Labels are reconciled to the nodes of the node pool unless the
baremetal.cluster.gke.io/label-taint-no-sync annotation
is applied to the cluster. For more information about labels, see
Labels and Selectors.
NodePool resource
No
Yes
registryMirrors
Optional. Use this section to specify a registry mirror to use for
installing clusters, instead of Container Registry
(gcr.io). For more information about using a registry
mirror, see
Installing
GKE on Bare Metal using a registry mirror.
String. The endpoint of the mirror, consisting of the registry server
IP address and port number. Optionally, you can use your own namespace
in your registry server instead of the root namespace. Without a
namespace, the endpoint format is
REGISTRY_IP:PORT. When you use a
namespace, the endpoint format is
REGISTRY_IP:PORT/v2/NAMESPACE.
The /v2 is required when specifying a namespace.
String. Path of the CA cert file if your registry server uses a
private TLS certificate. If your registry doesn't require a private
TLS certificate, then you can leave the caCertPath field
blank.
Registry mirror
Optional
Mutable
registryMirrors.pullCredentialConfigPath
String. Path to the
Docker CLI configuration file, config.json. Docker saves authentication settings in the
configuration file. This field applies to the use of registry mirrors
only. If your registry server doesn't require a Docker configuration
file for authentication, then you can leave the
pullCredentialConfigPath field blank.
The cluster configuration file generated by bmctl for
GKE on Bare Metal includes fields for specifying paths to credentials
and keys files in the local file system. These credentials and keys
needed to connect your clusters to each other and to your
Google Cloud project.
String. The path to the Container Registry service account key. The
Container Registry service account
is a Google-managed service account that acts on behalf of
Container Registry when interacting with Google Cloud services.
Credentials
No
Yes
sshPrivateKeyPath
String. The path to the SSH private key. SSH is required for Node access.
Credentials
No
Yes
gkeConnectAgentServiceAccountKeyPath
String. The path to the agent service account key.
GKE on Bare Metal uses this service account to maintain a
connection between GKE on Bare Metal and Google Cloud.
String. The path to the registration service account key.
GKE on Bare Metal uses this service account to register your user
clusters with Google Cloud.
String. The path to the operations service account key.
GKE on Bare Metal uses the operations service account to
authenticate with Google Cloud Observability for access to the
Logging API and the Monitoring API.
