Version 1.7. This version is supported as outlined in the Anthos version support policy, offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on bare metal. For more details, see the release notes 1.7. This is the most recent version. For a complete list of each minor and patch release in chronological order, see the combined release notes.

Available versions: 1.7  |   1.6

Release notes 1.7

This document lists production updates to Anthos clusters on bare metal. We recommend that Anthos clusters on bare metal developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, or browse and filter all release notes in the Google Cloud Console.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/anthos-bare-metal-release-notes.xml

June 02, 2021

Release 1.7.2

Anthos clusters on bare metal release 1.7.2 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.2 runs on Kubernetes 1.19.

Fixes:

  • Fixed CVE-2021-25735 that could allow node updates to bypass a Validating Admission Webhook. For more details, open the Anthos clusters on bare metal tab of the GCP-2021-003 security bulletin.
  • Resolved the bmctl snapshot command failure when the user creates a custom cluster namespace omitting "cluster-" prefix from the cluster config file. The prefix is no longer required for a custom cluster namespace.
  • Added webhook blocks to prevent users from modifying control plane node pool and load balancer node pool resources directly. Control plane and load balancer node pools for Anthos clusters on bare metal are specified in the cluster resource, using the spec.controlPlane.nodePoolSpec and spec.LoadBalancer.nodePoolSpec sections of the cluster config file respectively.
  • Fixed the cluster upgrade command, bmctl upgrade cluster, to prevent it from interfering with user-installed Anthos Service Mesh (ASM).

Functionality changes:

  • Updated the bmctl check snapshot command so that it includes certificate signing requests in the snapshot.
  • Changed the upgrade process to prevent node drain issues from blocking upgrades. The upgrade process triggers a node drain. Now, if the node drain takes longer than 20 minutes, the upgrade process carries on to completion even when the draining hasn't completed. In this case, the upgrade output reports the incomplete node drain. Excessive drain times signal a problematic with pods. You may need to restart problem pods.
  • Updated cluster creation process, bmctl create cluster, to display logged errors directly on the command line. Prior to this release, detailed error messages were only available in the log files.

Known issues:

  • Node logs from nodes with a dot (".") in their name are not exported to Cloud Logging. For workaround instructions, see Node logs aren't exported to Cloud Logging in Anthos clusters on bare metal known issues.

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 06, 2021

The Envoy and Istio projects recently announced several new security vulnerabilities (CVE-2021-28683, CVE-2021-28682, and CVE-2021-29258) that could allow an attacker to crash Envoy.

For more information, see the GCP-2021-004 security bulletin.

April 30, 2021

Anthos clusters on bare metal release 1.7.1 is now available. To upgrade, see Upgrading Anthos clusters on bare metal. Anthos clusters on bare metal 1.7.1 runs on Kubernetes 1.19.

Functionality changes:

  • Customers can now take cluster snapshots regardless of whether the admin cluster control plane is running. This is helpful for diagnosing installation issues.
  • Deploying Anthos clusters on bare metal with SELinux is now fully supported on supported versions of Redhat Enterprise Linux. This applies for new installations of Anthos clusters on bare metal cases only.
  • User cluster creation with bmctl supports credential inheritance from the admin cluster by default. Credential overrides for the user cluster can be specified in the config file during cluster creation.

Fixes:

  • (Updated May 12, 2021) Fixed CVE-2021-28683, CVE-2021-28682, CVE-2021-29258. For more details, see the GCP-2021-004 security bulletin.
  • Fixed potential stuck upgrade from 1.6.x to 1.7.0. The bug was caused by a rare race condition when the coredns configmap failed to be backed up and restored during the upgrade.
  • Fixed potential missing GKE connect agent during installation due to a rare race condition.
  • Fixed issue that prevented automatic updates to the control plane load balancer config when adding/removing node(s) from the control plane node pool.
  • Addressed problem with syncing NodePool taints and labels that resulted in deletion of pre-existing items. Syncs will now append, update, or delete items that are added by taints and labels themselves only.

Known issues:

  • Upgrading the container runtime from containerd to Docker will fail in Anthos clusters on bare metal release 1.7.1. This operation is not supported while the containerd runtime option is in preview.
  • bmctl snapshot command fails when the user creates a custom cluster namespace omitting cluster- prefix from the cluster config file. To avoid this issue, the cluster namespace should follow the cluster-$CLUSTER_NAME naming convention.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 20, 2021

The Kubernetes project recently announced a new security vulnerability, CVE-2021-25735, that could allow node updates to bypass a Validating Admission Webhook. For more details, see the GCP-2021-003 security bulletin.

March 25, 2021

Anthos on bare metal 1.7.0 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.7.0 runs on Kubernetes 1.19.

Extended installation support:

  • Added requirement for Anthos clusters on bare metal connectivity with Google Cloud for install and upgrade operations. As of 1.7.0 preflight checks will check for connectivity to Google Cloud, enabled APIs, and permissions for service accounts. Existing clusters need to be registered in Google Cloud before upgrading. The connectivity checks are not overridable by the --force flag. For details, see the cluster creation and cluster upgrade documentation.

  • Added support for installing Anthos clusters on bare metal on OpenStack. For configuration instructions, see Configure your clusters to use OpenStack.

  • Added support for installing Anthos clusters on bare metal, using a private package repository instead of the default Docker APT repository. For instructions and additional information, see Use a private package repository server.

  • Removed installation prerequisite for setting Security-Enhanced Linux (SELinux) operational mode to be permissive. The related preflight check has been removed, as well.

  • Removed installation prerequisite for disabling firewalld . The related preflight check has also been removed. For information on configuring ports to use firewalld with Anthos clusters on bare metal, see Configuring firewalld ports on the Network requirements page.

  • Updated requirements for installing behind a proxy server and removed restriction on system-wide proxy configurations. For a detailed list of prerequisites, see Installing behind a proxy.

Improved upgrade:

  • Updated cluster upgrade routines to ensure worker node failures do not block cluster upgrades, providing a more consistent user experience. Control plane node failures will still block cluster upgrades.

  • Added bmctl support for running upgrade preflight checks. bmctl check preflight will run upgrade preflight checks if users specify the --kubeconfig flag. For example:
    bmctl check preflight --kubeconfig bmctl-workspace/cluster1/cluster1-kubeconfig

Updated user cluster lifecycle management:

  • Added support in bmctl for user cluster creation and upgrade functions.

  • Improved resource handling. Anthos clusters on bare metal now reconciles node pool taints and labels to nodes unless the node has a baremetal.cluster.gke.io/label-taint-no-sync annotation.

Enhanced monitoring and logging:

  • Preview: Added out-of-the-box alerts for critical cluster metrics and events. For information on working with alerting policies and getting notified, see Creating alerting policies.

  • Added support for collecting ansible job logs in admin and hybrid clusters by default.

Expanded support for newer versions of operating systems:

  • Added support for installing Anthos clusters on bare metal on Red Hat Enterprise Linux (RHEL) 8.3 and CentOS 8.3.

Functionality changes:

  • Added support for configuring the number of pods per node. New clusters can be configured to run up to 250 pods per node. For more information about configuring nodes, see Pod networking. You can find additional information for configuring pods in the cluster creation documentation.
  • Preview: Added support to use containerd as the container runtime. Anthos clusters on bare metal 1.6.x supports only Docker for container runtime (dockershim). In 1.7.0, Kubelet can be configured to use either Docker or containerd, using the new containerRuntime cluster config field. You must upgrade existing clusters to 1.7.0 to add or update the containerRuntime field.
  • Added support for more load balancer addressPool entries under cluster.spec.loadBalancer.addressPools. For existing addressPools, users can use cluster.spec.loadBalancer.AddressPools[].manualAssign specify additional addressPool entries.

Known issues:

  • Under rare circumstances, bmctl upgrade may become stuck at the Moving resources to upgraded cluster stage after finishing upgrading all nodes in the cluster. The issue does not affect cluster operation, but the final step needs to be finished.

    If bmctl does not move forward after 30 minutes in this state, re-run the bmctl upgrade command to complete the upgrade.

    The issue is captured in the upgrade-cluster.log file located in .../bmctl-workspace/<cluster name>/log/upgrade-cluster-<timestamp>. The following log entry shows how the failure is reported:

    Operation failed, retrying with backoff. Cause: error creating "baremetal.cluster.gke.io/v1, Kind=Cluster" <cluster name>: Internal error occurred: failed calling webhook "vcluster.kb.io": Post "https://webhook-service.kube-system.svc:443/validate-baremetal-cluster-gke-io-v1-cluster? timeout=30s": net/http: TLS handshake timeout

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.