Version 1.7. This version is supported as outlined in the Anthos version support policy, offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on bare metal. For more details, see the release notes 1.7. This is the most recent version. For a complete list of each minor and patch release in chronological order, see the combined release notes.

Available versions: 1.7  |   1.6

Anthos clusters on bare metal known issues

Installation

Control group v2 incompatibility

Control group v2 (cgroup v2) is incompatible with Anthos clusters on bare metal 1.6. Kubernetes 1.18 does not support cgroup v2. Also Docker only offers experimental support as of 20.10. systemd switched to cgroup v2 by default in version 247.2-2. The presence of /sys/fs/cgroup/cgroup.controllers indicates that your system uses cgroup v2.

Starting with Anthos clusters on bare metal 1.6.2, the preflight checks verify that cgroup v2 is not in use on the cluster machine.

Benign error messages during installation

During highly available (HA) cluster installation, you may see errors about etcdserver leader change. These error messages are benign and can be ignored.

When you use bmctl for cluster installation, you may see a Log streamer failed to get BareMetalMachine log message at the very end of the create-cluster.log. This error message is benign and can be ignored.

When examining cluster creation logs, you may notice transient failures about registering clusters or calling webhooks. These errors can be safely ignored, because the installation will retry these operations until they succeed.

Preflight checks and service account credentials

For installations triggered by admin or hybrid clusters (in other words, clusters not created with bmctl, like user clusters), the preflight check does not verify Google Cloud Platform service account credentials or their associated permissions.

Preflight checks and permission denied

During installation you may see errors about /bin/sh: /tmp/disks_check.sh: Permission denied. These error messages are caused because /tmp is mounted with noexec option. For bmctl to work you need to remove noexec option from /tmp mount point.

Creating cloud monitoring workspace before viewing dashboards

You need to create a cloud monitoring workspace through the Google Cloud Console before you can view any Anthos clusters on bare metal monitoring dashboards,

Application default credentials and bmctl

bmctl uses Application Default Credentials (ADC) to validate the cluster operation's location value in the cluster spec when it is not set to global.

For ADC to work, you need to either point the GOOGLE_APPLICATION_CREDENTIALS environment variable to a service account credential file, or run gcloud auth application-default login.

Docker service

On cluster node machines, if the Docker executable is present in the PATH environment variable, but the Docker service is not active, preflight check will fail and report that the Docker service is not active. To fix this error, either remove Docker, or enable the Docker service.

Upgrading Anthos clusters on bare metal

Upgrading is not available in the 1.6.0 release.

Reset/Deletion

User cluster support

You can't reset user clusters with the bmctl reset command.

Mount points and fstab

Reset does not unmount the mount points under /mnt/localpv-share/ and it does not clean up the corresponding entries in /etc/fstab.

Namespace deletion

Deleting a namespace will prevent new resources from being created in that namespace, including jobs to reset machines. When deleting a user cluster, you must delete the cluster object first before deleting its namespace. Otherwise, the jobs to reset machines cannot get created, and the deletion process will skip the machine clean-up step.

containerd service

The bmctl reset command doesn't delete any containerd configuration files or binaries. The containerd systemd service is left up and running. The command deletes the containers running pods scheduled to the node.

Security

The cluster CA/certificate will be rotated during upgrade. On-demand rotation support is not currently available.

Anthos clusters on bare metal rotates kubelet serving certificates automatically. Each kubelet node agent can send out a Certificate Signing Request (CSR) when a certificate nears expiration. A controller in your admin clusters validates and approves the CSR.

Logging and Monitoring

Node logs aren't exported to Cloud Logging

Node logs from nodes with a dot (".") in their name are not exported to Cloud Logging. As a workaround, use the following instructions to add a filter to the stackdriver-log-forwarder-config resource to enable the Stackdriver Operator to recognize and export these logs.

  1. Scale down the size of the Stackdriver Operator, stackdriver-operator:

    kubectl --kubeconfig ADMIN_KUBECONFIG -n kube-system scale \
        deploy stackdriver-operator --replicas=0
    
  2. Edit the Log Forwarder configmap, stackdriver-log-forwarder-config:

    kubectl --kubeconfig ADMIN_KUBECONFIG -n kube-system edit configmap \
        stackdriver-log-forwarder-config
    
  3. Add the following filter to the end of the input-systemd.conf section of the configmap:

       [FILTER]
           Name    lua
           Match_Regex   container-runtime|kubelet|node-problem-detector|node-journal
           script  replace_dot.lua
           call    replace
    
     replace_dot.lua: |
       function replace(tag, timestamp, record)
           new_record = record
    
           local local_resource_id_key = "logging.googleapis.com/local_resource_id"
    
           -- Locate the local_resource_id
           local local_resource_id = record[local_resource_id_key]
    
           local first = 1
           local new_local_resource_id = ""
           for s in string.gmatch(local_resource_id, "[^.]+") do
               new_local_resource_id = new_local_resource_id .. s
               if first == 1 then
                   new_local_resource_id = new_local_resource_id .. "."
                   first = 0
               else
                   new_local_resource_id = new_local_resource_id .. "_"
               end
           end
    
           -- Remove the trailing underscore
           new_local_resource_id = new_local_resource_id:sub(1, -2)
           new_record[local_resource_id_key] = new_local_resource_id
           return 1, timestamp, new_record
       end
    
  4. Delete all Log Forwarder pods:

    kubectl --kubeconfig ADMIN_KUBECONFIG -n kube-system patch daemonset \
        stackdriver-log-forwarder -p \
        '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'
    

    Verify that the stackdriver-log-forwarder pods are deleted before going to the next step.

  5. Deploy a daemonset to clean up all currupted, unprocessed data in buffers in fluent-bit:

    kubectl --kubeconfig ADMIN_KUBECONFIG -n kube-system apply -f - << EOF
        apiVersion: apps/v1
        kind: DaemonSet
        metadata:
          name: fluent-bit-cleanup
          namespace: kube-system
        spec:
          selector:
            matchLabels:
              app: fluent-bit-cleanup
          template:
            metadata:
              labels:
                app: fluent-bit-cleanup
            spec:
              containers:
              - name: fluent-bit-cleanup
                image: debian:10-slim
                command: ["bash", "-c"]
                args:
                - |
                  rm -rf /var/log/fluent-bit-buffers/
                  echo "Fluent Bit local buffer is cleaned up."
                  sleep 3600
                volumeMounts:
                - name: varlog
                  mountPath: /var/log
                securityContext:
                  privileged: true
              tolerations:
              - key: "CriticalAddonsOnly"
                operator: "Exists"
              - key: node-role.kubernetes.io/master
                effect: NoSchedule
              - key: node-role.gke.io/observability
                effect: NoSchedule
              volumes:
              - name: varlog
                hostPath:
                  path: /var/log
    EOF
    
  6. Verify that the daemonset has cleaned up all the nodes with the following commands.

    kubectl --kubeconfig ADMIN_KUBECONFIG logs -n kube-system \
        -l app=fluent-bit-cleanup | grep "cleaned up" | wc -l
    
    kubectl --kubeconfig ADMIN_KUBECONFIG -n kube-system get pods \
        -l app=fluent-bit-cleanup --no-headers | wc -l
    

    The output of the two commands should be equal to your node number in the cluster

  7. Delete the cleanup daemonset:

    kubectl --kubeconfig ADMIN_KUBECONFIG -n kube-system delete ds fluent-bit-cleanup
    
  8. Restart pods:

    kubectl --kubeconfig ADMIN_KUBECONFIG -n kube-system patch \
        daemonset stackdriver-log-forwarder --type json \
        -p='[{"op": "remove", "path": "/spec/template/spec/nodeSelector/non-existing"}]'
    

Networking

Modifying firewalld will erase Cilium iptable policy chains

When running Anthos clusters on bare metal with firewalld enabled on either CentOS or Red Had Enterprise Linux (RHEL), changes to firewalld can remove the Cilium iptables chains on the host network. The iptables chains are added by the anetd Pod when it is started. The loss of the Cilium iptables chains causes the Pod on the Node to lose network connectivity outside of the Node.

Changes to firewalld that will remove the iptables chains include, but aren't limited to:

  • Restarting firewalld, using systemctl
  • Reloading the firewalld with the command line client (firewall-cmd --reload)

You can fix this connectivity issue by restarting anetd on the Node. Locate and delete the anetd Pod with the following commands to restart anetd:

kubectl get pods -n kube-system
kubectl delete pods -n kube-system ANETD_XYZ

Replace ANETD_XYZ with the name of the anetd Pod.

Pod connectivity failures and reverse path filtering

Anthos clusters on bare metal configures reverse path filtering on nodes to disable source validation (rp_filter=0). If therp_filter setting is changed to 1 or 2, pods will fail due to out-of-node communication timeouts.

Reverse path filtering is set with rp_filter files in the IPv4 configuration folder (net/ipv4/conf). This value may also be overridden by sysctl, which stores reverse path filtering settings in a network security configuration file, such as /etc/sysctl.d/60-gce-network-security.conf. Setting rp_filter back to 0 should restore pod connectivity.

Bootstrap (kind) cluster IP addresses and cluster node IP addresses overlapping

192.168.122.0/24 and 10.96.0.0/27 are the default pod and service CIDRs used by the bootstrap (kind) cluster. Preflight checks will fail if they overlap with cluster node machine IP addresses. To avoid the conflict, you can pass the --bootstrap-cluster-pod-cidr and --bootstrap-cluster-service-cidr flags to bmctl to specify different values.

Overlapping IP addresses across different clusters

There is no preflight check to validate overlapping IP addresses across different clusters.

hostport feature in Anthos clusters on bare metal

The hostport feature in ContainerPort is not currently supported.

Operating system endpoint limitations

On RHEL and CentOS, there is a cluster level limitation of 100,000 endpoints. This number is the sum of all pods that are referenced by a Kubernetes service. If 2 services reference the same set of pods, this counts as 2 separate sets of endpoints. The underlying nftable implementation on RHEL and CentOS causes this limitation; it is not an intrinsic limitation of Anthos clusters on bare metal.

Configuration

Control plane and load balancer specifications

The control plane and load balancer node pool specifications are special. These specifications declare and control critical cluster resources. The canonical source for these resources is their respective sections in the cluster config file:

  • spec.controlPlane.nodePoolSpec
  • spec.LoadBalancer.nodePoolSpec

Consequently, do not modify the top-level control plane and load balancer node pool resources directly. Modify the associated sections in the cluster config file instead.

Mutable fields in the cluster and node pool specification

Currently, only the following cluster and node pool specification fields in the cluster config file can be updated after the cluster is created (they are mutable fields):

  • For the Cluster object (kind: Cluster), the following fields are mutable:

    • spec.anthosBareMetalVersion
    • spec.bypassPreflightCheck
    • spec.controlPlane.nodePoolSpec.nodes
    • spec.loadBalancer.nodePoolSpec.nodes
    • spec.maintenanceBlocks
    • spec.nodeAccess.loginUser
  • For the NodePool object (kind: NodePool), the following fields are mutable:

    • spec.nodes