Update clusters

Version 1.14 

• Version 1.28 (latest)
• Version 1.16
• Version 1.15
• Earlier versions (unsupported)

After you create a cluster with bmctl, you can change some aspects of the cluster's configuration by performing the following sequence of actions:

1. Change the values of certain fields in the cluster's configuration file, which by default is located here: bmctl-workspace/CLUSTER-NAME/CLUSTER-NAME.yaml.

2. Update the cluster by running the bmctl update command.

In this way, you can, for example, add or remove nodes in a cluster or replace nodes in a cluster. This document describes how to perform these and other updates to a cluster.

It's important to note, however, that many aspects of your cluster configuration are immutable and can't be updated after you create your cluster. For a comprehensive list of mutable and immutable fields, see the Cluster configuration field reference. The field reference is a sortable table. Click the column headings to change the sort order. Click a field name to view its description.

Note: If you run the command kubectl apply to create a cluster, and the preflight check fails, you can use kubectl edit to change any field value, even immutable fields. In other words, any field in your cluster configuration can be edited before the first successful preflight check occurs. This is a one-time exception to the immutability rule.

Add or remove nodes in a cluster

A node pool is a group of nodes within a cluster that have the same configuration. Keep in mind that a node always belongs to a node pool. To add a new node to a cluster, you need to add it to a particular node pool. Removing a node from a node pool amounts to removing the node from the cluster altogether.

There are three kinds of node pools in GKE on Bare Metal: control plane, load balancer, and worker node pools.

You add or remove a node from a node pool by adding or removing the IP address of the node in a specific section of the cluster configuration file. The following list shows which section to edit for a given node pool:

• Worker node pool: add or remove the IP address of the node in the spec.nodes section of the NodePool spec.
• Control plane node pool: add or remove the IP address of the node in the spec.controlPlane.nodePoolSpec.nodes section of the Cluster spec.
• Load balancer node pool: add or remove the IP address of the node in the spec.loadBalancer.nodePoolSpec.nodes section of the Cluster spec.

Example: how to remove a worker node

Here's a sample cluster configuration file that shows the specifications of two worker nodes:

---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: nodepool1
  namespace: cluster-cluster1
spec:
  clusterName: cluster1
  nodes:
  - address: 172.18.0.5
  - address: 172.18.0.6

To remove a node:

1. (Optional) If the node that you're removing is running critical pods, first put the node into maintenance mode.

   You can monitor the node draining process for worker nodes by viewing the status.nodesDrained and status.nodesDraining fields on the NodePool resource.

2. Edit the cluster configuration file to delete the IP address entry for the node.

3. Update the cluster:

   bmctl update cluster -c CLUSTER_NAME \
       --kubeconfig=ADMIN_KUBECONFIG
   

   Replace the following:

   • CLUSTER_NAME: the name of the cluster you want to update.
   • ADMIN_KUBECONFIG: the path to the admin cluster kubeconfig file.

   After the bmctl update command has executed successfully, it takes some time to complete the machine-preflight and machine-init jobs. You can view the status of nodes and their respective node pools by running the commands described in the Verify your updates section of this document.

Forcing the removal of a node

If the bmctl update command is unable to remove a node, you may have to force its removal from the cluster. For details, see Force-removing broken nodes.

Replace HA control plane nodes

You can replace high availability (HA) control plane nodes in admin, user, standalone, and hybrid clusters.

You replace a node in a cluster by performing the following steps:

1. Remove the node's IP address from the cluster configuration file.
2. Update the cluster.
3. Check the status of nodes in the cluster.
4. Add a new node's IP address to the same cluster configuration file.
5. Update the cluster.

The rest of this section goes through an example.

Here's a sample cluster configuration file that shows three control plane nodes in a user cluster:

---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
 name: user-cluster
 namespace: cluster-user-cluster
spec:
  controlPlane:
   nodePoolSpec:
     nodes:
     - address: 10.200.0.11
     - address: 10.200.0.12
     - address: 10.200.0.13

To replace the last node listed in the spec.controlPlane.nodePoolSpec.nodes section, perform the following steps:

1. Remove the node by deleting its IP address entry in the cluster configuration file. After making this change, the cluster configuration file should look something like this:

   ---
   apiVersion: baremetal.cluster.gke.io/v1
   kind: Cluster
   metadata:
    name: user-cluster
    namespace: cluster-user-cluster
   spec:
     controlPlane:
      nodePoolSpec:
        nodes:
        - address: 10.200.0.11
        - address: 10.200.0.12
   

2. Update the cluster by running the following command:

   bmctl update cluster -c CLUSTER_NAME \
       --kubeconfig=KUBECONFIG
   

   Make the following changes:

   • Replace CLUSTER_NAME with the name of the cluster you want to update.
   • If the cluster is a self-managing cluster (such as admin or standalone cluster), replace KUBECONFIG with the path to the cluster's kubeconfig file. If the cluster is a user cluster, as it is in this example, replace KUBECONFIG with the path to the admin cluster's kubeconfig file.

3. After the bmctl update command has executed successfully, it takes some time to complete the machine-preflight and machine-init jobs. You can view the status of nodes and their respective node pools by running the commands described in the Verify your updates section of this document. Once the node pool and nodes are in a ready state, you can proceed to the next step.

4. Add a new control plane node to the node pool by adding the IP address of the new control plane node to the spec.controlPlane.nodePoolSpec.nodes section of the cluster configuration file. After making this change, the cluster configuration file should look something like this:

   ---
   apiVersion: baremetal.cluster.gke.io/v1
   kind: Cluster
   metadata:
    name: user-cluster
    namespace: cluster-user-cluster
   spec:
     controlPlane:
      nodePoolSpec:
        nodes:
        - address: 10.200.0.11
        - address: 10.200.0.12
        - address: 10.200.0.14
   

5. Update the cluster by running the following command:

   bmctl update cluster -c CLUSTER_NAME \
       --kubeconfig=KUBECONFIG
   

   Make the following changes:

   • Replace CLUSTER_NAME with the name of the cluster you want to update.
   • If the cluster is a self-managing cluster (such as admin or standalone cluster), replace KUBECONFIG with the path to the cluster's kubeconfig file. If the cluster is a user cluster, as it is in this example, replace KUBECONFIG with the path to the admin cluster's kubeconfig file.

Verify your updates

You can view the status of nodes and their respective node pools with the kubectl get command.

For example, the following command shows the status of the node pools in the cluster namespace cluster-my-cluster:

kubectl -n cluster-my-cluster get nodepools.baremetal.cluster.gke.io

The system returns results similar to the following:

NAME                    READY   RECONCILING   STALLED   UNDERMAINTENANCE   UNKNOWN
cluster-my-cluster      3       0             0         0                  0
cluster-my-cluster-lb   2       0             0         0                  0
np1                     3       0             0         0                  0

Reconciling=1 means that the reconciliation step is still in progress. You should wait until the status changes to Reconciling=0.

You can also check status of nodes in the a cluster by running following command:

kubectl get nodes --kubeconfig=KUBECONFIG

If you need more information about how to diagnose your clusters, see Create snapshots for diagnosing clusters.

Features you can change with an update

Besides adding, removing or replacing nodes, you can use the bmctl update command to modify certain mutable field values, Custom Resources (CRs), and annotations in the cluster configuration file.

To update a cluster resource, edit the cluster configuration file and use bmctl update to apply your changes.

The following sections outline some common examples for updating an existing cluster by changing either a field value, CR or annotation.

loadBalancer.addressPools

The addressPools section contains fields for specifying load-balancing pools for bundled load balancers. You can add more load-balancing address pools at any time, but you can't remove or modify any existing address pools.

addressPools:
- name: pool1
  addresses:
  - 192.168.1.0-192.168.1.4
  - 192.168.1.240/28
- name: pool2
  addresses:
  - 192.168.1.224/28

Prevent inadvertent cluster deletion

If you add the baremetal.cluster.gke.io/prevent-deletion: "true" annotation to your cluster configuration file, you are prevented from deleting the cluster. For example, running kubectl delete cluster or bmctl reset cluster produce an error.

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: ci-10c3c6f4d9c698e
  namespace: cluster-ci-10c3c6f4d9c698e
  annotations:
    baremetal.cluster.gke.io/prevent-deletion: "true"
spec:
  clusterNetwork:

bypassPreflightCheck

The default value of the bypassPreflightCheck field is false. If you set this field to true in the cluster configuration file, the internal preflight checks are ignored you apply resources to existing clusters.

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: cluster1
  namespace: cluster-cluster1
  annotations:
    baremetal.cluster.gke.io/private-mode: "true"
spec:
  bypassPreflightCheck: true

loginUser

You can set the loginUser field under the node access configuration. This field supports passwordless sudo capability for machine login.

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: cluster1
  namespace: cluster-cluster1
  annotations:
    baremetal.cluster.gke.io/private-mode: "true"
spec:
  nodeAccess:
    loginUser: abm

NetworkGatewayGroup

The NetworkGatewayGroup custom resource is used to provide floating IP addresses for advanced networking features, such as the egress NAT gateway or the bundled load-balancing feature with BGP. To use the NetworkGatewayGroup custom resource and related networking features, you must set clusterNetwork.advancedNetworking to true when you create your clusters.

apiVersion: networking.gke.io/v1
kind: NetworkGatewayGroup
  name: default
  namespace: cluster-bm
spec:
  floatingIPs:
  - 10.0.1.100
  - 10.0.2.100

BGPLoadBalancer

When you configure bundled load balancers with BGP, the data plane load balancing uses, by default, the same external peers that were specified for control plane peering. Alternatively, you can configure the data plane load balancing separately, using the BGPLoadBalancer custom resource (and the BGPPeer custom resource). For more information, see Configure bundled load balancers with BGP.

apiVersion: networking.gke.io/v1
kind: BGPLoadBalancer
metadata:
  name: default
  namespace: cluster-bm
spec:
  peerSelector:
    cluster.baremetal.gke.io/default-peer: "true"

BGPPeer

When you configure bundled load balancers with BGP, the data plane load balancing uses, by default, the same external peers that were specified for control plane peering. Alternatively, you can configure the data plane load balancing separately, using the BGPPeer custom resource (and the BGPLoadBalancer custom resource). For more information, see Configure bundled load balancers with BGP.

apiVersion: networking.gke.io/v1
kind: BGPPeer
metadata:
  name: bgppeer1
  namespace: cluster-bm
  labels:
    cluster.baremetal.gke.io/default-peer: "true"
spec:
  localASN: 65001
  peerASN: 65002
  peerIP: 10.0.3.254
  sessions: 2

NetworkAttachmentDefinition

You can use the bmctl update command to modify NetworkAttachmentDefinition custom resources that correspond to the network.

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: gke-network-1
  namespace: cluster-my-cluster
spec:
  config: '{
  "type": "ipvlan",
  "master": "enp2342",
  "mode": "l2",
  "ipam": {
    "type": "whereabouts",
    "range": "172.120.0.0/24"

After you modify the config file, update the cluster by running the following bmctl update command:

bmctl update cluster -c CLUSTER_NAME --kubeconfig=KUBECONFIG

Make the following changes:

• Replace CLUSTER_NAME with the name of the cluster you want to update.
• If the cluster is a self-managing cluster (such as admin or standalone cluster), replace KUBECONFIG with the path to the cluster's kubeconfig file. If the cluster is a user cluster, replace KUBECONFIG with the path to the admin cluster's kubeconfig file.

Disable kubelet read-only port

Starting with release 1.15.0, GKE on Bare Metal disables by default port 10255, the kubelet read-only port. Any customer workloads that are configured to read data from this insecure kubelet port 10255 should migrate to use the secure kubelet port 10250.

Only clusters created with version 1.15.0 or higher have this port disabled by default. The kubelet read-only port 10255 remains accessible for clusters created with a version lower than 1.15.0, even after a cluster upgrade to version 1.15.0 or higher.

This change was made because the kubelet leaks low sensitivity information over port 10255, which is unauthenticated. The information includes the full configuration information for all Pods running on a Node, which can be valuable to an attacker. It also exposes metrics and status information, which can provide business-sensitive insights.

Disabling the kubelet read-only port is recommended by the CIS Kubernetes Benchmark. To manually disable the port in version 1.14, complete the following steps:

1. Edit your cluster configuration file and add the following annotation:

   baremetal.cluster.gke.io/enable-kubelet-read-only-port: "false"
   

   The following example cluster configuration file shows that this annotation has been added:

   [...]
   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: cluster-my-cluster
   ---
   apiVersion: baremetal.cluster.gke.io/v1
   kind: Cluster
   metadata:
     annotations:
       baremetal.cluster.gke.io/enable-kubelet-read-only-port: "false"
     name: my-cluster
     namespace: cluster-my-cluster
   [...]
   

2. Edit the cluster webhook validation cluster resource:

   kubectl edit validatingwebhookconfigurations lcm-validating-webhook-configuration
   

3. Temporarily disable webhook validation by deleting the line with the UPDATE verb:

   - admissionReviewVersions:
   - v1
   clientConfig:
     caBundle: …
     service:
       name: lcm-webhook-service
       namespace: kube-system
       path: /validate-baremetal-cluster-gke-io-v1-cluster
       port: 443
   failurePolicy: Fail
   matchPolicy: Equivalent
   name: vcluster.kb.io
   namespaceSelector: {}
   objectSelector: {}
   rules:
   - apiGroups:
     - baremetal.cluster.gke.io
     apiVersions:
     - v1
     operations:
     - CREATE
     - UPDATE # <- DELETE THIS LINE
     - DELETE
     resources:
     - clusters
     scope: '*'
   sideEffects: None
   timeoutSeconds: 10
   

4. Add the annotation to the cluster custom resource:

   kubectl edit clusters CLUSTER_NAME --kubeconfig KUBECONFIG_PATH
   

   Make the following changes:

   • Replace CLUSTER_NAME with the name of the cluster you want to update.
   • Replace KUBECONFIG_PATH with the path to the cluster's kubeconfig file.

5. Add the annotation, as shown in the following example updated cluster custom resource:

   apiVersion: baremetal.cluster.gke.io/v1
   kind: Cluster
   metadata:
     annotations:
       baremetal.cluster.gke.io/enable-kubelet-read-only-port: "false"
   [...]
   

6. Save and close the cluster custom resource.

7. Re-enable webhook validation by adding the UPDATE verb back to the cluster webhook validation cluster resource:

   kubectl edit validatingwebhookconfigurations lcm-validating-webhook-configuration
   

   In the custom resource, add the UPDATE verb back in the operations section:

   - admissionReviewVersions:
   - v1
   clientConfig:
     caBundle: …
     service:
       name: lcm-webhook-service
       namespace: kube-system
       path: /validate-baremetal-cluster-gke-io-v1-cluster
       port: 443
   failurePolicy: Fail
   matchPolicy: Equivalent
   name: vcluster.kb.io
   namespaceSelector: {}
   objectSelector: {}
   rules:
   - apiGroups:
     - baremetal.cluster.gke.io
     apiVersions:
     - v1
     operations:
     - CREATE
     - UPDATE # <- ADD THIS LINE BACK
     - DELETE
     resources:
     - clusters
     scope: '*'
   sideEffects: None
   timeoutSeconds: 10
   

   Save and close the cluster custom resource.

These changes are retained when you upgrade to version 1.15.0 or later.