Version 1.9. This is the most recent version. It's supported as outlined in the Anthos version support policy, offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on bare metal. For release details, see the release notes 1.9. For a complete list of each minor and patch release in chronological order, see the combined release notes.

Available supported versions: 1.9  |   1.8  |   1.7  |  

Anthos clusters on bare metal known issues


Unspecified containerRuntime doesn't default to containerd

In Anthos clusters on bare metal release 1.9.0, the containerRuntime default was updated from docker to containerd in the generated cluster configuration file. If the containerRuntime field isn't set or is removed from the cluster configuration file, containerRuntime is set to docker when you create clusters. The containerRuntime value should default to containerd, unless it is explicitly set to docker. This issue applies to releases 1.9.0 and 1.9.1 only.

To determine which container runtime your cluster is using, follow the steps in Retrieve cluster information. Check the value of containerRuntime in the cluster.spec.nodeConfig section.

The only way to change the container runtime is by upgrading your clusters. For more information, see Change your container runtime.

This issue is fixed in Anthos clusters on bare metal release 1.9.2.

Control group v2 incompatibility

Control group v2 (cgroup v2) is not officially supported in Anthos clusters on bare metal 1.9. The presence of /sys/fs/cgroup/cgroup.controllers indicates that your system uses cgroup v2.

The preflight checks verify that cgroup v2 is not in use on the cluster machine.

Benign error messages during installation

When examining cluster creation logs, you may notice transient failures about registering clusters or calling webhooks. These errors can be safely ignored, because the installation will retry these operations until they succeed.

Preflight checks and service account credentials

For installations triggered by admin or hybrid clusters (in other words, clusters not created with bmctl, like user clusters), the preflight check does not verify Google Cloud Platform service account credentials or their associated permissions.

Preflight checks and permission denied

During installation you may see errors about /bin/sh: /tmp/ Permission denied. These error messages are caused because /tmp is mounted with noexec option. For bmctl to work you need to remove noexec option from /tmp mount point.

Creating cloud monitoring workspace before viewing dashboards

You need to create a cloud monitoring workspace through the Google Cloud Console before you can view any Anthos clusters on bare metal monitoring dashboards,

Application default credentials and bmctl

bmctl uses Application Default Credentials (ADC) to validate the cluster operation's location value in the cluster spec when it is not set to global.

For ADC to work, you need to either point the GOOGLE_APPLICATION_CREDENTIALS environment variable to a service account credential file, or run gcloud auth application-default login.

Docker service

On cluster node machines, if the Docker executable is present in the PATH environment variable, but the Docker service is not active, preflight check will fail and report that the Docker service is not active. To fix this error, either remove Docker, or enable the Docker service.

Installing on vSphere

When installing Anthos clusters on bare metal on vSphere VMs, you must set the tx-udp_tnl-segmentation and tx-udp_tnl-csum-segmentation flags to off. These flags are related to the hardware segmentation offload done by the vSphere driver VMXNET3 and they don't work with the GENEVE tunnel of Anthos clusters on bare metal.

Run the following command on each node to check the current values for these flags. ethtool -k NET_INTFC |grep segm ... tx-udp_tnl-segmentation: on tx-udp_tnl-csum-segmentation: on ... Replace NET_INTFC with the network interface associated with the IP address of the node.

Sometimes in RHEL 8.4, ethtool shows these flags are off while they aren't. To explicitly set these flags to off, toggle the flags on and then off with the following commands.

ethtool -K ens192 tx-udp_tnl-segmentation on
ethtool -K ens192 tx-udp_tnl-csum-segmentation on

ethtool -K ens192 tx-udp_tnl-segmentation off
ethtool -K ens192 tx-udp_tnl-csum-segmentation off

This flag change does not persist across reboots. Configure the startup scripts to explicitly set these flags when the system boots.

Upgrading Anthos clusters on bare metal

User cluster patch upgrade limitation

User clusters that are managed by an admin cluster must be at the same Anthos clusters on bare metal version or lower and within one minor release. For example, a version 1.9.0 (anthosBareMetalVersion: 1.9.0) admin cluster managing version 1.8.4 user clusters is acceptable.

An upgrade limitation prevents you from upgrading your user clusters to a new security patch when the patch is released after the release version the admin cluster is using. For example, if your admin cluster is at version 1.7.2, which was released on June 2, 2021, you can't upgrade your user clusters to version 1.6.4, because it was released on August 13, 2021.

Node draining can't start when Node is out of reach

The draining process for Nodes won't start if the Node is out of reach from Anthos clusters on bare metal. For example, if a Node goes offline during a cluster upgrade process, it may cause the upgrade to stop responding. This is a rare occurrence. To minimize the likelyhood of encountering this problem, ensure your Nodes are operating properly before initiating an upgrade.


Namespace deletion

Deleting a namespace will prevent new resources from being created in that namespace, including jobs to reset machines. When deleting a user cluster, you must delete the cluster object first before deleting its namespace. Otherwise, the jobs to reset machines cannot get created, and the deletion process will skip the machine clean-up step.

containerd service

The bmctl reset command doesn't delete any containerd configuration files or binaries. The containerd systemd service is left up and running. The command deletes the containers running pods scheduled to the node.


The cluster CA/certificate will be rotated during upgrade. On-demand rotation support is a Preview feature.

Anthos clusters on bare metal rotates kubelet serving certificates automatically. Each kubelet node agent can send out a Certificate Signing Request (CSR) when a certificate nears expiration. A controller in your admin clusters validates and approves the CSR.

Cluster CA Rotation (Preview Feature)

After you perform a user cluster certificate authority (CA) rotation on a cluster, all user authentication flows fail. These failures occur because the ClientConfig custom resource used in authentication flows isn't being updated with the new CA data during CA rotation. If you have performed a cluster CA rotation on your cluster, check to see if the certificateAuthorityData field in default ClientConfig of the kube-public namespace contains the older cluster CA.

To resolve the issue manually, update the certificateAuthorityData field with the current cluster CA.


Modifying firewalld will erase Cilium iptable policy chains

When running Anthos clusters on bare metal with firewalld enabled on either CentOS or Red Had Enterprise Linux (RHEL), changes to firewalld can remove the Cilium iptables chains on the host network. The iptables chains are added by the anetd Pod when it is started. The loss of the Cilium iptables chains causes the Pod on the Node to lose network connectivity outside of the Node.

Changes to firewalld that will remove the iptables chains include, but aren't limited to:

  • Restarting firewalld, using systemctl
  • Reloading the firewalld with the command line client (firewall-cmd --reload)

You can fix this connectivity issue by restarting anetd on the Node. Locate and delete the anetd Pod with the following commands to restart anetd:

kubectl get pods -n kube-system
kubectl delete pods -n kube-system ANETD_XYZ

Replace ANETD_XYZ with the name of the anetd Pod.

Duplicate egressSourceIP addresses

When using the egress NAT gateway feature preview, it is possible to set traffic selection rules that specify an egressSourceIP address that is already in use for another EgressNATPolicy object. This may cause egress traffic routing conflicts. Coordinate with your development team to determine which floating IP addresses are available for use before specifying the egressSourceIP address in your EgressNATPolicy custom resource.

Pod connectivity failures due to I/O timeout and reverse path filtering

Anthos clusters on bare metal configures reverse path filtering on nodes to disable source validation (net.ipv4.conf.all.rp_filter=0). If therp_filter setting is changed to 1 or 2, pods fail due to out-of-node communication timeouts.

Observed connectivity failures communicating to Kubernetes Service IP addresses are a symptom this problem. Here are a couple of examples of the types of errors you might see:

  • If all pods for a given node fail to communicate to the Service IP addresses, the istiod Pod might report an error like the following:

        "message":"watch error in cluster Kubernetes: failed to list *v1.Node:
        Get \"  34239\":
        dial tcp i/o timeout"}
  • For the localpv daemon set that runs on every node, the log might show a timeout like the following:

     I1112 17:24:33.191654       1 main.go:128] Could not get node information
    (remaining retries: 2): Get
    dial tcp i/o timeout

Reverse path filtering is set with rp_filter files in the IPv4 configuration folder (net/ipv4/conf/all). The sysctl command stores reverse path filtering settings in a network security configuration file, such as /etc/sysctl.d/60-gce-network-security.conf.. The sysctl command can override the reverse path filtering setting.

To restore Pod connectivity, either set net.ipv4.conf.all.rp_filter back to 0 manually, or restart the anetd Pod to set net.ipv4.conf.all.rp_filter back to 0. To restart the anetd Pod, use the following commands to locate and delete the anetd Pod. A new anetd Pod start up in its place:

kubectl get pods -n kube-system
kubectl delete pods -n kube-system ANETD_XYZ

Replace ANETD_XYZ with the name of the anetd Pod.

To set net.ipv4.conf.all.rp_filter manually, run the following command:

sysctl -w net.ipv4.conf.all.rp_filter = 0

Bootstrap (kind) cluster IP addresses and cluster node IP addresses overlapping and are the default pod and service CIDRs used by the bootstrap (kind) cluster. Preflight checks will fail if they overlap with cluster node machine IP addresses. To avoid the conflict, you can pass the --bootstrap-cluster-pod-cidr and --bootstrap-cluster-service-cidr flags to bmctl to specify different values.

Overlapping IP addresses across different clusters

There is no validation for overlapping IP addresses across different clusters during update. The validation only applies at cluster/node pool creation time.

Operating system endpoint limitations

On RHEL and CentOS, there is a cluster level limitation of 100,000 endpoints. This number is the sum of all pods that are referenced by a Kubernetes service. If 2 services reference the same set of pods, this counts as 2 separate sets of endpoints. The underlying nftable implementation on RHEL and CentOS causes this limitation; it is not an intrinsic limitation of Anthos clusters on bare metal.


Control plane and load balancer specifications

The control plane and load balancer node pool specifications are special. These specifications declare and control critical cluster resources. The canonical source for these resources is their respective sections in the cluster config file:

  • spec.controlPlane.nodePoolSpec
  • spec.LoadBalancer.nodePoolSpec

Consequently, do not modify the top-level control plane and load balancer node pool resources directly. Modify the associated sections in the cluster config file instead.

Anthos VM Runtime

  • Restarting a pod causes the VMs on the pod to change IP addresses or lose their IP address altogether. If the IP address of a VM changes, this does not affect the reachability of VM applications exposed as a Kubernetes service. If the IP address is lost, you must run dhclient from the VM to acquire an IP address for the VM.


SELinux errors during pod creation

Pod creation sometimes fails when SELinux prevents the container runtime from setting labels on tmpfs mounts. This failure is rare, but can happen when SELinux is in Enforcing mode and in some kernels.

To verify that SELinux is the cause of pod creation failures, use the following command to check for errors in the kubelet logs:

journalctl -u kubelet

If SELinux is causing pod creation to fail, the command response contains an error similar to the following:

error setting label on mount source '/var/lib/kubelet/pods/
failed to set file label on /var/lib/kubelet/pods/
permission denied

To verify that this issue is related to SELinux enforcement, run the following command:

ausearch -m avc

This command searches the audit logs for access vector cache (AVC) permission errors. The avc: denied in the following sample response confirms that the pod creation failures are related to SELinux enforcement.

type=AVC msg=audit(1627410995.808:9534): avc:  denied  { associate } for
pid=20660 comm="dockerd" name="/" dev="tmpfs" ino=186492
tcontext=system_u:object_r:locale_t:s0 tclass=filesystem permissive=0

The root cause of this pod creation problem with SELinux is a kernel bug found in the following Linux images:

  • Red Hat Enterprise Linux (RHEL) releases prior to 8.3
  • CentOS releases prior to 8.3

Rebooting the machine helps recover from the issue.

To prevent pod creation errors from occurring, use RHEL 8.3 or later or CentOS 8.3 or later, because those versions have fixed the kernel bug.


Taking a snapshot as a non-root login user

If you use containerd as the container runtime, running snapshot as non-root user requires /usr/local/bin to be in the user's PATH. Otherwise it will fail with a crictl: command not found error.

When you aren't logged in as the root user, sudo is used to run the snapshot commands. The sudo PATH can differ from the root profile and may not contain /usr/local/bin.

You can fix this error by updating the secure_path in /etc/sudoers to include /usr/local/bin. Alternatively, create a symbolic link for crictl in another /bin directory.