Version 1.7. This version is supported as outlined in the Anthos version support policy, offering the latest patches and updates for security vulnerabilities, exposures, and issues affecting Anthos clusters on bare metal. For more details, see the release notes 1.7. For a complete list of each minor and patch release in chronological order, see the combined release notes.

Available versions: 1.9  |   1.8  |   1.7

Combined release notes (1.6 and 1.7)

This document lists production updates to Anthos clusters on bare metal. We recommend that Anthos clusters on bare metal developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud Console, or you can programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/anthos-bare-metal-release-notes.xml

November 29, 2021

1.8

Release 1.8.6

Anthos clusters on bare metal 1.8.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.6 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 15, 2021

1.7

Release 1.7.6

Anthos clusters on bare metal 1.7.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.6 runs on Kubernetes 1.19.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.9

Release 1.9.2

Anthos clusters on bare metal 1.9.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.2 runs on Kubernetes 1.21.

Fixes:

  • Updated preflight checks to use strict mode for decoding YAML to catch issues with indentation and misplaced fields in the cluster configuration file.

  • Fixed an issue that caused containerRuntime to default to docker, instead of containerd in certain uncommon situations.

  • Fixed an issue where node_filesystem metrics report incorrect size in Cloud Monitoring for mount-points other than root.

  • Fixed an issue that caused communication failures between Cloud Logging metadata agent and the Cloud Monitoring API when the root certificate authority (CA) on the host node isn't set up properly.

  • The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 29, 2021

1.6 & 1.7 & 1.8 & 1.9

Security bulletin (all minor versions)

The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem.

For more information, see the GCP-2021-011 security bulletin.

October 26, 2021

1.9

Release 1.9.1

Anthos clusters on bare metal 1.9.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.1 runs on Kubernetes 1.21.

Fixes:

Functionality changes:

  • Updated the bmctl reset cluster command to prevent you from resetting an admin cluster if the admin cluster is managing user clusters.
  • Updated the bmctl create cluster command to block you from enabling the Anthos VM Runtime for admin clusters.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 21, 2021

1.8

Release 1.8.5

Anthos clusters on bare metal 1.8.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.5 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.6 & 1.7 & 1.8 & 1.9

Security bulletin (all minor versions)

A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742. Ingress-nginx custom snippets allows retrieval of ingress-nginx service account tokens and secrets across all namespaces. For more information, see the GCP-2021-024 security bulletin.

October 19, 2021

1.7

Release 1.7.5

Anthos clusters on bare metal 1.7.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.5 runs on Kubernetes 1.19.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 04, 2021

1.6 & 1.7 & 1.8 & 1.9

Security bulletin (all minor versions)

A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server.

For more information, see the GCP-2021-021 security bulletin.

September 28, 2021

1.9

Release 1.9.0

Anthos clusters on bare metal 1.9.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.0 runs on Kubernetes 1.21.

Improved cluster lifecycle functionalities:

  • Preview: Added ability to reset individual nodes with the bmctl reset node command. To give access to the needed cluster configuration file, use the command with the -c flag.

  • Preview: Added ability to recover from HA control plane quorum loss withbmctl restore --control-plane-node command.

  • Added bmctl create ksa command to create a Kubernetes Service Account (KSA) and generate a bearer token. To log in to the registered cluster, you can use the token in Cloud Console Kubernetes Engine > Clusters.

  • Preview: Added bmctl backup cluster and bmctl restore cluster commands to facilitate disaster recovery for clusters.

Introduced new troubleshooting capabilities:

  • Updated the bmctl check cluster --snapshot command to support uploading cluster diagnostic snapshots to a Cloud Storage bucket for review by Cloud Customer Care.

  • Provided access to bootstrap cluster logs to help troubleshoot cluster creation or upgrade problems.

  • Preview: Added support for Node Problem Detector service on nodes for quick detection of common node problems.

Enhanced monitoring and logging:

  • GA: Cloud Audit Logs capability is now generally available and enabled by default. Audit logs are useful for investigating suspicious API requests and for collecting statistics. For more information, see Use Audit Logging.

  • Switched to new open telemetry-based metrics agents to improve reliability, ability to scale, and resource usage.

Improved networking capabilities:

  • GA: The multi-NIC capability to provide additional interfaces to your pods is now generally available.

  • Preview: Added the single root I/O virtualization (SR-IOV) container network interface (CNI) plugin for multi-NIC.

  • Added support to configure cluster Domain Name System (DNS) provider options, such as upstream nameservers, with the new ClusterDNS custom resource definition.

Enhanced security:

  • SELinux is now always enabled in the container runtime for CentOS and RHEL.

  • Preview: Enhanced the capability to rotate cluster certificate authorities (CAs). Updates include support for all cluster types, rotation of front-proxy and etcd CAs, and changes to the bmctl command syntax.

  • Preview: Added Okta group support for authentication in Anthos Identity Service.

Functionality changes:

  • Changed default container runtime to containerd, containerRuntime: containerd for new clusters. Customers can still choose Docker as the container runtime.
  • Preview: Updated bmctl command, bmctl reset nodes --force, to support force removal of control plane nodes with etcd membership cleanup.

  • Added checks for cluster updates to verify access to cluster machines if changes to loginUser or sshKeyPrivatePath are detected. If the checks pass, Anthos clusters on bare metal saves the secret in the cluster.

  • Added new Anthos cluster control plane uptime dashboard in Cloud Monitoring with new metric kubernetes.io/anthos/container/uptime for component availability.

  • Added new alerts for control plane components availability with new metric kubernetes.io/anthos/container/uptime to replace deprecated alerts with metric kubernetes.io/anthos/up.

Fixes:

  • Added missing registry mirror package required for Cloud Audit Logs to the Registry Mirror.

  • Fixed issue with containerd not finding crictl due to /usr/local/bin not being in the SSH user's PATH.

  • Fixed flapping node readiness issues caused by an unhealthy Pod Lifecycle Event Generator (PLEG).

  • Fixed kernel support issue for Ubuntu 18.04 and 18.04.1 that prevented the anetd networking controller from working properly. Anthos clusters on bare metal release 1.9.0 works with all kernels supplied with supported distributions.

Known issues:

  • Control group v2 (cgroup v2) is not officially supported in Anthos clusters on bare metal release 1.9.0 and later. The presence of /sys/fs/cgroup/cgroup.controllers indicates that your system uses cgroup v2.

  • Anthos Service Mesh v1.10 is incompatible with Anthos clusters on bare metal release 1.9.0 running on Red Hat Enterprise Linux (RHEL) when SELinux is enabled. If you want to use Anthos Service Mesh, you must disable SELinux or set it to permissive mode on the host.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

September 21, 2021

1.8

Release 1.8.4

Anthos clusters on bare metal 1.8.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.4 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerabilities have been fixed: - CVE-2021-3711 - CVE-2021-3712 - CVE-2021-20305 - CVE-2021-33560

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

September 20, 2021

1.7

Release 1.7.4

Anthos clusters on bare metal release 1.7.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.4 runs on Kubernetes 1.19.

Fixes:

  • Fixed vulnerability CVE-2021-25741 that might allow users to create a container with subpath volume mounts to access files and directories outside of the volume, including on the host filesystem. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.7.0. For more information, see the GCP-2021-018 security bulletin.

  • Updated the Kubernetes patch version to address the following container image security vulnerabilities:

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.7 & 1.8

Security bulletin (1.7 and 1.8)

A security issue was discovered in Kubernetes, CVE-2021-25741, where a user may be able to create a container with subpath volume mounts to access files and directories outside of the volume, including on the host filesystem. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal 1.7.x and 1.8.x releases, specifically 1.7.3 and earlier and 1.8.2 and earlier.

To fix this vulnerability, upgrade your Anthos clusters to version 1.7.4 or 1.8.3. For more information, see the GCP-2021-018 security bulletin.

August 27, 2021

1.8

Release 1.8.3

Anthos clusters on bare metal 1.8.3 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.3 runs on Kubernetes 1.20.

Breaking changes:

In Anthos clusters on bare metal release 1.8.0, we added a kernel version requirement for Ubuntu 18.04. We required a Linux kernel version of 4.17.0 or later. Anthos clusters on bare metal release 1.8.3 again supports all Linux kernel versions that ship with Ubuntu 18.04 and 20.04 distributions. As a result of this change, however, the egress NAT gateway feature that was provided for Preview in release 1.8.0 does not work with Anthos clusters on bare metal release 1.8.3.

Features:

  • Preview: Anthos Identity Service now works with Anthos clusters on bare metal to support LDAP authentication methods in addition to OIDC. You can use AIS with Microsoft Active Directory without the need for provisioning Active Directory Federation Services. For more information, see Setting up Anthos Identity Service with LDAP.

  • Preview: Anthos Metadata Agent replaces Stackdriver Metadata Collector and collects more accurate and usable metadata for Kubernetes resources. When you configure logging and monitoring, you need to enable the Config Monitoring for Ops API and grant the opsconfigmonitoring.resourceMetadata.writer IAM role to your logging-monitoring service account. If Anthos clusters on bare metal is installed behind a proxy, your proxy server must also allow connections to opsconfigmonitoring.googleapis.com.

  • Added preflight checks to verify that specific APIs are enabled for your Google Cloud project. Preflight checks return an error if any of the following APIs aren't enabled for your project:

    • anthos.googleapis.com
    • anthosaudit.googleapis.com
    • anthosgke.googleapis.com
    • cloudresourcemanager.googleapis.com
    • gkeconnect.googleapis.com
    • gkehub.googleapis.com
    • iam.googleapis.com
    • opsconfigmonitoring.googleapis.com
    • logging.googleapis.com
    • monitoring.googleapis.com
    • stackdriver.googleapis.com

    To enable these APIs when you create a cluster configuration file, use the --enable-apis flag with the bmctl create config command. For an example that uses the --enable-apis flag, see Create an admin cluster config with bmctl.

  • Added preflight checks for the following machine requirements:

    • Minimum supported Linux kernel version
    • Minimum required CPU
    • Minimum required RAM

Fixes:

  • Fixed the following container image security vulnerabilities:
  • Fixed cluster creation and cluster update failures for nodes running CentOS or Red Hat Enterprise Linux (RHEL) with both SELinux and Cloud Audit Logs enabled.
  • Fixed Transmission Control Protocol (TCP) connection leakage issue.
  • Fixed an issue that prevented cert-manager from issuing ACME certificates over HTTP due to ImagePullBackOff errors.

Changes:

  • The Kubevirt version used for working with VM-based workloads is now v0.43.0-gke.3.
  • The bootstrap cluster is deleted when a cluster upgrade completes without errors.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 16, 2021

1.7

Release 1.7.3

Anthos clusters on bare metal 1.7.3 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.3 runs on Kubernetes 1.19.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 13, 2021

1.6

Release 1.6.4

Anthos clusters on bare metal 1.6.4 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.6.4 runs on Kubernetes 1.18.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

July 29, 2021

1.8

Release 1.8.2

Anthos clusters on bare metal 1.8.2 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.2 runs on Kubernetes 1.20.

Features:

  • Preview: Added capability to rotate cluster certificate authorities (CAs) for user clusters. For instructions on using the bmctl cluster credentials command to rotate cluster CAs, see Rotate user cluster certificate authority.

  • Preview: Added support for AppArmor with Anthos clusters on bare metal. You don't need to disable AppArmor on Ubuntu as a prerequisite for installation. When you create new 1.8.2 clusters or upgrade clusters to version 1.8.2, you can enable AppArmor either before or after you upgrade.

Fixes:

  • Fixed CVE-2021-3520 vulnerability related to a flaw in lz4, which provides support for LZ4 a lossless compression algorithm. The flaw impacts availability, but has potential to impact confidentiality and integrity as well.

  • Fixed bmctl operation failures that occur for some Ubuntu 20.04 LTS distributions with a more recent Linux kernel, including GCP Ubuntu 20.04 LTS images on the 5.8 kernel. For more information about this issue and a workaround, see Ubuntu 20.04 LTS and bmctl.

  • Fixed OpenStack support for user clusters. In prior releases, cluster creation fails for user type clusters when the baremetal.cluster.gke.io/external-cloud-provider: "true" annotation is added to the cluster configuration file.

  • Fixed PATH environment issues for executing commands as a non-root user. For more information, see Known Issues.

  • Fixed an issue that caused user cluster resets (bmctl reset cluster) to get stuck while deleting namespaces.

  • Fixed out-of-memory (OOM) conditions related to Connect Agent memory usage that resulted in pod failures.

  • Fixed issue that blocked snapshots for clusters configured for passwordless SUDO capability for machine login (nodeAccess.loginUser: <login user name>).

  • Fixed issue that blocked some 1.7.x version admin, hybrid, or standalone clusters from upgrading to the 1.8 minor release. This issue affected some clusters that were updated by applying changes from an updated cluster configuration file.

  • Fixed Address Resolution Protocol (ARP) table issue for high-availability (HA) deployments that blocked upgrades from completing.

Functionality changes:

  • Expanded snapshots to include resource usage metrics to improve troubleshooting and support. Added metrics include the output of ip neigh, kubectl top nodes, and kubectl top pods commands.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

July 02, 2021

1.8

Release 1.8.1

Anthos clusters on bare metal release 1.8.1 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.8.1 runs on Kubernetes 1.20.

Fixes:

Fixed CVE-2021-34824 that could expose private keys and certificates from Kubernetes secrets through the credentialName field when using Gateway or DestinationRule. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.8.0. For more information, see the GCP-2021-012 security bulletin.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 30, 2021

1.8

Security bulletin (1.8)

The Istio project recently announced a security vulnerability (CVE-2021-34824) where credentials specified in the credentialName field for Gateway or DestinationRule can be accessed from different namespaces. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.8.0. For more information, see the GCP-2021-012 security bulletin.

June 21, 2021

1.8

Release 1.8.0

Anthos clusters on bare metal release 1.8.0 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.0 runs on Kubernetes 1.20.

Extended installation support:

  • Provided support to use containerd as the container runtime as GA for Anthos clusters on bare metal release 1.8.0. Cluster upgrades to 1.8.0 are blocked for 1.7.x clusters that are configured to use the preview containerd capability. For more information, see Upgrading 1.7.x clusters that use containerd in Known Issues.
  • Preview: Improved virtual machine (VM) management capability. Anthos VM Runtime uses KubeVirt to orchestrate VMs on clusters, allowing you to work with your VM-based apps and workloads in a uniform development environment. Anthos VM Runtime has worked with Anthos clusters on bare metal as a preview feature since November 2020 and we have continued to enhance its capability. For more information, see Working with VM-based workloads.
  • Added edge profile support for standalone clusters. The edge profile is recommended for edge devices with limited resources. Add profile: edge to the cluster config file when you create a standalone cluster to produce a cluster that has significantly reduced system resource requirements. The edge profile is only available for standalone clusters, it is ignored for other cluster types. For more information, see Creating standalone clusters.
  • Added support to specify provider ID for Nodes (controlPlane.nodePoolSpec.nodes.providerID) to support deploying on OpenStack using Load Balancing as a Service (LBaaS) resources. For more information, see Configure your clusters to use OpenStack.
  • Preview: Added support for installing Anthos clusters on bare metal, using your own registry service, instead of gcr.io. For instructions and additional information, see Installing Anthos Bare Metal using registry mirror.

Improved upgrade:

  • Enabled support for upgrading non-SELinux clusters to SELinux. For more information, see Enable SELinux in Upgrading Anthos clusters on bare metal.
  • Cluster upgrades are not blocked by excessive Node draining durations. During a cluster upgrade, if the draining process takes longer than 20 minutes for any specific Node, the upgrade process will carry on without waiting for draining to complete.

Updated user cluster lifecycle management:

  • Added bmctl improvements for resetting user cluster and adding additional preflight checks to confirm machine and network readiness for cluster creation:

Enhanced monitoring and logging:

  • Preview: Added Cloud Audit Logging capability, which enables audit logs to be written to Cloud Audit Logs in your Google project. Audit logs are useful for investigating suspicious API requests and for collecting statistics. For more information, see Enable Audit Logging.

Introduced new networking capabilities in preview:

  • Preview: Added multi-NIC capability to provide additional interfaces to your Pods.
  • Preview: Added egress NAT gateway capability to provide persistent, deterministic routing for the egress traffic from your clusters. For more information, see Configure an egress NAT gateway for external communication.
  • Preview: Added option for BGP bundled load balancer for Layer-3 (L3) topologies. This feature can be used with user clusters and admin clusters.

Enhanced security:

  • Workload Identity is GA. The Connect Agent Service Account Key is no longer required during installation. Connect Agent uses Workload Identity to authenticate to GCP instead of an exported GCP Service Account Key.

Expanded support for newer versions of operating systems:

  • Added support for installing Anthos clusters on bare metal on Red Hat Enterprise Linux (RHEL) 8.4, and CentOS 8.4

Functionality changes:

  • Added --workspace-dir flag to bmctl to allow changing the path and name of the workspace directory from the default bmctl-workspace. The workspace directory contains the configuration and log files generated by bmctl. When using the bmctl command, pass in a --workspace-dir flag to specify a non-default workspace directory location. If the directory does not exist, bmctl will create it for you.
  • Moved away from iptables-based NodePort and masquerade handling to eBPF-based management. NodePort and masquerade handling are now applied to the Node IP and default gateway interfaces only.

Fixes:

  • Resolved, as part of the GA support for using containerd as the container runtime, incorrect cgroup driver use. Newly created 1.8.0 clusters that are configured to use containerd will use the correct systemd cgroup driver.
  • Fixed issue that prevented usage metrics for the containerd process from being collected by Cloud Logging. This fix applies to newly created 1.8.0 clusters only.

Known issues:

  • If a Node is out of reach, Anthos clusters on bare metal can't start the draining process, which may impact the cluster upgrade process. For more information, see Node draining can't start when Node is out of reach.
  • Upgrading from 1.7.x clusters that use containerd as the container runtime to 1.8.0 is blocked. For more information, see Upgrading 1.7.x clusters that use containerd.
  • When running Anthos clusters on bare metal with firewalld enabled on either CentOS or Red Hat Enterprise Linux (RHEL), changes to firewalld can remove the Cilium iptables chains on the host network. The loss of the Cilium iptables chains causes the Pod on the Node to lose network connectivity outside of the Node. for more information, see Modifying firewalld will erase Cilium iptable chains.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 02, 2021

1.7

Release 1.7.2

Anthos clusters on bare metal release 1.7.2 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.2 runs on Kubernetes 1.19.

Fixes:

  • Fixed CVE-2021-25735 that could allow node updates to bypass a Validating Admission Webhook. For more details, open the Anthos clusters on bare metal tab of the GCP-2021-003 security bulletin.
  • Resolved the bmctl snapshot command failure when the user creates a custom cluster namespace omitting "cluster-" prefix from the cluster config file. The prefix is no longer required for a custom cluster namespace.
  • Added webhook blocks to prevent users from modifying control plane node pool and load balancer node pool resources directly. Control plane and load balancer node pools for Anthos clusters on bare metal are specified in the cluster resource, using the spec.controlPlane.nodePoolSpec and spec.LoadBalancer.nodePoolSpec sections of the cluster config file respectively.
  • Fixed the cluster upgrade command, bmctl upgrade cluster, to prevent it from interfering with user-installed Anthos Service Mesh (ASM).

Functionality changes:

  • Updated the bmctl check snapshot command so that it includes certificate signing requests in the snapshot.
  • Changed the upgrade process to prevent node drain issues from blocking upgrades. The upgrade process triggers a node drain. Now, if the node drain takes longer than 20 minutes, the upgrade process carries on to completion even when the draining hasn't completed. In this case, the upgrade output reports the incomplete node drain. Excessive drain times signal a problematic with pods. You may need to restart problem pods.
  • Updated cluster creation process, bmctl create cluster, to display logged errors directly on the command line. Prior to this release, detailed error messages were only available in the log files.

Known issues:

  • Node logs from nodes with a dot (".") in their name are not exported to Cloud Logging. For workaround instructions, see Node logs aren't exported to Cloud Logging in Anthos clusters on bare metal known issues.

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 17, 2021

1.6

Release 1.6.3

Anthos clusters on bare metal release 1.6.3 is now available. To upgrade, see Upgrading Anthos clusters on bare metal. Anthos clusters on bare metal 1.6.3 runs on Kubernetes 1.18.

Fixed:

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 06, 2021

1.6 & 1.7

Security bulletin (1.6 and 1.7)

The Envoy and Istio projects recently announced several new security vulnerabilities (CVE-2021-28683, CVE-2021-28682, and CVE-2021-29258) that could allow an attacker to crash Envoy.

For more information, see the GCP-2021-004 security bulletin.

April 30, 2021

1.7

Release 1.7.1

Anthos clusters on bare metal release 1.7.1 is now available. To upgrade, see Upgrading Anthos clusters on bare metal. Anthos clusters on bare metal 1.7.1 runs on Kubernetes 1.19.

Functionality changes:

  • Customers can now take cluster snapshots regardless of whether the admin cluster control plane is running. This is helpful for diagnosing installation issues.
  • Deploying Anthos clusters on bare metal with SELinux is now fully supported on supported versions of Redhat Enterprise Linux. This applies for new installations of Anthos clusters on bare metal cases only.
  • User cluster creation with bmctl supports credential inheritance from the admin cluster by default. Credential overrides for the user cluster can be specified in the config file during cluster creation.

Fixes:

  • (Updated May 12, 2021) Fixed CVE-2021-28683, CVE-2021-28682, CVE-2021-29258. For more details, see the GCP-2021-004 security bulletin.
  • Fixed potential stuck upgrade from 1.6.x to 1.7.0. The bug was caused by a rare race condition when the coredns configmap failed to be backed up and restored during the upgrade.
  • Fixed potential missing GKE connect agent during installation due to a rare race condition.
  • Fixed issue that prevented automatic updates to the control plane load balancer config when adding/removing node(s) from the control plane node pool.
  • Addressed problem with syncing NodePool taints and labels that resulted in deletion of pre-existing items. Syncs will now append, update, or delete items that are added by taints and labels themselves only.

Known issues:

  • Upgrading the container runtime from containerd to Docker will fail in Anthos clusters on bare metal release 1.7.1. This operation is not supported while the containerd runtime option is in preview.
  • bmctl snapshot command fails when the user creates a custom cluster namespace omitting cluster- prefix from the cluster config file. To avoid this issue, the cluster namespace should follow the cluster-$CLUSTER_NAME naming convention.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 20, 2021

1.6 & 1.7

Security bulletin (1.6 and 1.7)

The Kubernetes project recently announced a new security vulnerability, CVE-2021-25735, that could allow node updates to bypass a Validating Admission Webhook. For more details, see the GCP-2021-003 security bulletin.

March 25, 2021

1.7

Release 1.7.0

Anthos clusters on bare metal release 1.7.0 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.7.0 runs on Kubernetes 1.19.

Extended installation support:

  • Added requirement for Anthos clusters on bare metal connectivity with Google Cloud for install and upgrade operations. As of 1.7.0 preflight checks will check for connectivity to Google Cloud, enabled APIs, and permissions for service accounts. Existing clusters need to be registered in Google Cloud before upgrading. The connectivity checks are not overridable by the --force flag. For details, see the cluster creation and cluster upgrade documentation.

  • Added support for installing Anthos clusters on bare metal on OpenStack. For configuration instructions, see Configure your clusters to use OpenStack.

  • Added support for installing Anthos clusters on bare metal, using a private package repository instead of the default Docker APT repository. For instructions and additional information, see Use a private package repository server.

  • Removed installation prerequisite for setting Security-Enhanced Linux (SELinux) operational mode to be permissive. The related preflight check has been removed, as well.

  • Removed installation prerequisite for disabling firewalld . The related preflight check has also been removed. For information on configuring ports to use firewalld with Anthos clusters on bare metal, see Configuring firewalld ports on the Network requirements page.

  • Updated requirements for installing behind a proxy server and removed restriction on system-wide proxy configurations. For a detailed list of prerequisites, see Installing behind a proxy.

Improved upgrade:

  • Updated cluster upgrade routines to ensure worker node failures do not block cluster upgrades, providing a more consistent user experience. Control plane node failures will still block cluster upgrades.

  • Added bmctl support for running upgrade preflight checks. bmctl check preflight will run upgrade preflight checks if users specify the --kubeconfig flag. For example:
    bmctl check preflight --kubeconfig bmctl-workspace/cluster1/cluster1-kubeconfig

Updated user cluster lifecycle management:

  • Added support in bmctl for user cluster creation and upgrade functions.

  • Improved resource handling. Anthos clusters on bare metal now reconciles node pool taints and labels to nodes unless the node has a baremetal.cluster.gke.io/label-taint-no-sync annotation.

Enhanced monitoring and logging:

  • Preview: Added out-of-the-box alerts for critical cluster metrics and events. For information on working with alerting policies and getting notified, see Creating alerting policies.

  • Added support for collecting ansible job logs in admin and hybrid clusters by default.

Expanded support for newer versions of operating systems:

  • Added support for installing Anthos clusters on bare metal on Red Hat Enterprise Linux (RHEL) 8.3 and CentOS 8.3.

Functionality changes:

  • Added support for configuring the number of pods per node. New clusters can be configured to run up to 250 pods per node. For more information about configuring nodes, see Pod networking. You can find additional information for configuring pods in the cluster creation documentation.
  • Preview: Added support to use containerd as the container runtime. Anthos clusters on bare metal 1.6.x supports only Docker for container runtime (dockershim). In 1.7.0, Kubelet can be configured to use either Docker or containerd, using the new containerRuntime cluster config field. You must upgrade existing clusters to 1.7.0 to add or update the containerRuntime field.
  • Added support for more load balancer addressPool entries under cluster.spec.loadBalancer.addressPools. For existing addressPools, users can use cluster.spec.loadBalancer.AddressPools[].manualAssign specify additional addressPool entries.

Known issues:

  • Under rare circumstances, bmctl upgrade may become stuck at the Moving resources to upgraded cluster stage after finishing upgrading all nodes in the cluster. The issue does not affect cluster operation, but the final step needs to be finished.

    If bmctl does not move forward after 30 minutes in this state, re-run the bmctl upgrade command to complete the upgrade.

    The issue is captured in the upgrade-cluster.log file located in .../bmctl-workspace/<cluster name>/log/upgrade-cluster-<timestamp>. The following log entry shows how the failure is reported:

    Operation failed, retrying with backoff. Cause: error creating "baremetal.cluster.gke.io/v1, Kind=Cluster" <cluster name>: Internal error occurred: failed calling webhook "vcluster.kb.io": Post "https://webhook-service.kube-system.svc:443/validate-baremetal-cluster-gke-io-v1-cluster? timeout=30s": net/http: TLS handshake timeout

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 26, 2021

1.6

Release 1.6.2

Anthos clusters on bare metal release 1.6.2 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.6.2 runs on Kubernetes 1.18.

Fixes:

  • Updated custom resource API to reject changes to Cluster and NodePool configuration fields that are not currently supported. For a list of supported mutable fields, see Configuration in Known Issues.
  • Updated bmctl to allow creating or upgrading Anthos clusters on bare metal to the current bmctl version (1.6.2) only. For more information about version restrictions, see Installation in Known Issues.
  • Fixed an issue that caused the automatic reset of bare metal machines to fail after deleting the user cluster.
  • Added preflight check to verify that control group v2, or cgroup v2 for short, is not in use on the cluster machine. Anthos on bare betal 1.6.x is incompatible with cgroup v2. For more information, see Control group v2 incompatibility in Known Issues.
  • Updated csi-snapshot-validation-webhook to support certification rotation. For more information about certificate rotation, see Security in Known Issues.
  • Fixed an issue to prevent constant patching for snapshot.storage.k8s.io CRDs.
  • Fixed a Certificate Signing Request (CSR) issue with kubelet to ensure fully qualified domain name(FQDN) hostnames are supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

January 29, 2021

1.6

Release 1.6.1

Anthos clusters on bare metal release 1.6.1 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.6.1 runs on Kubernetes 1.18.6-gke.6600.

Functionality changes:

  • Added upgrade support from 1.6.0. Users are able to upgrade existing Anthos bare metal cluster from 1.6.0 to 1.6.1.
  • Improved upgrade preflight check. Added preflight check before cluster upgrade to validate current cluster status, machine health and other issues before proceeding to upgrade.
  • Added support for deleting mounts and data from the anthos-system StorageClass during bmctl reset.
  • Relaxed the requirement for an odd number of control plane node pools to allow customers to add and remove nodes for maintenance or replacement.
  • Added support to force removing a broken worker node through annotation on the operator machine.
  • Added etcddefrag pod to control-plane nodes, which are responsible for monitoring etcd's database size and defragmenting the database as needed. This helps reclaim etcd database size and recover etcd when its disk space is exceeded.
  • Enabled kubelet server TLS certification auto-rotation. Kubelet on each node sends out CSR when nearing serving certificate expiration. A controller running inside the admin cluster validates and approves the CSR for user clusters.
  • Added proxy support to connect to the OIDC provider. This allows overriding the cluster proxy configuration with a different proxy.
  • Added bmctl update cluster for updating standalone clusters.

Fixes:

  • Fixed bug causing cluster deletion stall problem because of pods refusing to evacuate, or dead nodes.

November 30, 2020

1.6

Release 1.6.0

Anthos on bare metal is generally available

Anthos on bare metal is a deployment option to run Anthos on physical or virtual servers, deployed on an operating system provided by you, without a hypervisor layer. Anthos on bare metal ships with built-in networking, lifecycle management, diagnostics, health checks, logging, and monitoring. Anthos on bare metal supports CentOS, Red Hat Enterprise Linux (RHEL), and Ubuntu—all validated by Google. With Anthos on bare metal, you can use your company's standard hardware and operating system images, taking advantage of existing investments, which are automatically checked and validated against Anthos infrastructure requirements.

Anthos on bare metal is available today, with either subscription or pay-as-you-go pricing. Anthos on bare metal lets you leverage existing investments in hardware, OS, and networking infrastructure. The minimum system requirement to run Anthos on bare metal is 2 nodes with a minimum total of 4 cores, 32 GB RAM, and 128 GB of disk space with no specialized hardware. The setup lets you run Anthos on bare metal on almost any infrastructure.

Anthos on bare metal uses a "bring your own operating system" model. It runs atop physical or virtual instances, and supports Red Hat Enterprise Linux 8.1/8.2, CentOS 8.1/8.2, or Ubuntu 18.04/20.04 LTS. Anthos provides overlay networking and L4/L7 load balancing. You can also integrate with your own load balancer such as F5 and Citrix. For storage, you can deploy persistent workloads using CSI integration with your existing infrastructure.

You can deploy Anthos on bare metal using one of the following deployment models:

  • A standalone model lets you manage every cluster independently. This is a good choice when running in an edge location or if you want your clusters to be administered independent of one another.
  • The multiple-cluster model lets central IT teams manage a fleet of clusters from a centralized cluster, called the admin cluster. This is more suitable if you want to build automation or tooling, or if you want to delegate the lifecycle of clusters to individual teams without sharing sensitive credentials such as SSH keys or Google Cloud service account details.

Like with all Anthos environments, a bare metal cluster has a thin, secure connection back to Google Cloud called Connect. After it's installed in your clusters, you can centrally view, configure, and monitor your clusters from the Google Cloud Console.

Anthos on bare metal, which is part of the Anthos 1.6 release, provides the following features and capabilities:

  • Kubernetes 1.18
  • Ubuntu/RHEL/CentOS support
  • Standalone and multiple-cluster architecture
  • In-place upgrades (minor and major)
  • Overlay networking, Ingress (L7), integrated load balancing (L4, L2-Mode)
  • Manual load balancing (F5, Citrix)
  • Installs behind proxy support
  • Preflight and health checks
  • Node maintenance mode
  • Cloud Monitoring and Cloud Logging
  • ACM, ASM, identity, hub or connect, billing, and pay-as-you-go
  • NVIDIA GPU support
  • Scales to 500 nodes
  • Virtual machine management (Kubevirt) preview