Troubleshoot the bootstrap cluster

When Google Distributed Cloud creates or upgrades clusters, it deploys a Kubernetes in Docker (kind) cluster to temporarily host the Kubernetes controllers needed to create or upgrade clusters. This transient cluster is called a bootstrap cluster. If a kind cluster already exists in your deployment when you attempt to install, Google Distributed Cloud deletes the existing kind cluster. Deletion only happens after the installation or upgrade is successful. To preserve the existing kind cluster even after success, use the --keep-bootstrap-cluster flag of bmctl.

Google Distributed Cloud creates a configuration file for the bootstrap cluster under WORKSPACE_DIR/.kindkubeconfig. You can connect to the bootstrap cluster only during cluster creation and upgrade.

The bootstrap cluster needs to access a Docker repository to pull images. The registry defaults to Container Registry unless you are using a private registry. During cluster creation,bmctl creates the following files:

  • bmctl-workspace/config.json: Contains Google Cloud service account credentials for the registry access. The credentials are obtained from the gcrKeyPath field in the cluster configuration file.

  • bmctl-workspace/config.toml: Contains the containerd configuration in the kind cluster.

Debug the bootstrap cluster

To debug the bootstrap cluster you can take the following steps:

  • Connect to the bootstrap cluster during cluster creation and upgrade.
  • Get the logs of the bootstrap cluster.

You can find the logs in the machine you use to run bmctl in the following folders:

  • bmctl-workspace/CLUSTER_NAME/log/create-cluster-TIMESTAMP/bootstrap-cluster/
  • bmctl-workspace/CLUSTER_NAME/log/upgrade-cluster-TIMESTAMP/bootstrap-cluster/

Replace CLUSTER_NAME and TIMESTAMP with the name of your cluster and the corresponding system's time.

To get the logs from the bootstrap cluster directly, you can run the following command during cluster creation and upgrade:

docker exec -it bmctl-control-plane bash

The command opens a terminal inside the bmctl control plane container that runs in the bootstrap cluster.

To inspect the kubelet and containerd logs, use the following commands and look for errors or warnings in the output:

journalctl -u kubelet
journalctl -u containerd