Force-removing broken nodes in Google Distributed Cloud

Sometimes, to remove a broken node for repair or replacement, you may have to force its removal from the cluster. Force removal only removes the broken node from the cluster management's perspective. Force removal bypasses clean up jobs for the installed components on the node itself. Upon recovery of the node, you run bmctl reset nodes to clean up the installed components on the node so that it can be reused.

Force-removing nodes

The following methods apply to both control plane nodes and worker nodes. For control plane nodes, controllers in Google Distributed Cloud also take care of the bookkeeping of etcd memberships.

Using bmctl

You can use bmctl to remove the node from the cluster. Normally, bmctl reset triggers a reset job to try to clean up installed components on the node. To remove the node from the cluster without being blocked on cleaning up installed packages, you can run the bmctl command with the --force flag:

bmctl reset nodes --addresses NODE_IP --force --kubeconfig ADMIN_KUBECONFIG --cluster CLUSTER_NAME

Replace the following:

  • NODE_IP: the IP address of the node to reset, such as 10.200.0.8.

  • ADMIN_KUBECONFIG: the path to the admin cluster kubeconfig file.

  • CLUSTER_NAME: the name of the target cluster that contains the nodes.

Using kubectl

In Google Distributed Cloud, you can add an annotation to mark a node for force removal.

After removing the node from the parent nodepool, run the following command to annotate the corresponding failing machine with the baremetal.cluster.gke.io/force-remove annotation. The value of the annotation itself does not matter:

kubectl --kubeconfig ADMIN_KUBECONFIG -n CLUSTER_NAMESPACE \
  annotate machine 10.200.0.8 baremetal.cluster.gke.io/force-remove=true

Google Distributed Cloud removes the node successfully.