Combined release notes (all minor versions)

This document lists production updates to Anthos clusters on bare metal. We recommend that Anthos clusters on bare metal developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or you can programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/anthos-bare-metal-release-notes.xml

March 24, 2023

1.14

Release 1.14.3

Anthos clusters on bare metal 1.14.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.3 runs on Kubernetes 1.25.

Fixes:

Improved maintenance mode operation by ignoring non-running pods on nodes.
Updated etcd version to version 3.4.21-0-gke.1 to resolve an issue that could lead to watch starvation and non-operational watch for resources.
Updated kubernetes version to 1.25.6-gke.1000 to honor exponential backoff in job controller.
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

March 21, 2023

1.13

Release 1.13.6

Anthos clusters on bare metal 1.13.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.6 runs on Kubernetes 1.24.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

March 09, 2023

1.13 & 1.14

Cluster lifecycle improvements 1.13.1 and later

Starting with Anthos clusters on bare metal release 1.13.1, you can use the Google Cloud console or the gcloud CLI to upgrade admin and user clusters managed by the Anthos On-Prem API. If your cluster is at version 1.13.0 or lower, you must use bmctl to upgrade the cluster.

For more information about using the console or the gcloud CLI for upgrades, see the documentation for your version of Anthos clusters on bare metal:

March 02, 2023

1.12

Release 1.12.8

Anthos clusters on bare metal 1.12.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.8 runs on Kubernetes 1.23.

Fixes:

Updated Anthos Identity Service to better handle concurrent authentication webhook requests.
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

March 01, 2023

1.14

Release 1.14.2

Anthos clusters on bare metal 1.14.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.2 runs on Kubernetes 1.25.

Fixes:

Updated Anthos Identity Service to better handle concurrent authentication webhook requests.
Updated stackdriver-operator to set CPU and memory resource limits.
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

February 23, 2023

1.13

Release 1.13.5

Anthos clusters on bare metal 1.13.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.5 runs on Kubernetes 1.24.

Fixes:

Updated Anthos Identity service to better handle concurrent authentication webhook requests.
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

February 07, 2023

1.12

Release 1.12.7

Anthos clusters on bare metal 1.12.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.7 runs on Kubernetes 1.23.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

January 27, 2023

1.14

1.14.0 Upgrade problem

Control plane nodes for Anthos clusters on bare metal use Kubernetes taints to prevent workload pods from being scheduled on them. When you upgrade version 1.13 Anthos clusters to version 1.14.0, the control plane nodes lose required taints. We recommend that you skip upgrading to version 1.14.0 and upgrade to version 1.14.1 directly.

This problem doesn't cause upgrade failures, but pods that aren't supposed to run on the control plane nodes may start doing so. These workload pods can overwhelm control plane nodes and lead to cluster instability. This issue has security implications, as well. We strongly recommend that you not upgrade your clusters to version 1.14.0, but upgrade instead to a subsequent release version with the fix.

For more information about the issue, including workaround instructions, see the Clusters upgraded to 1.14.0 lose master taints known issue.

Release 1.14.1

Anthos clusters on bare metal 1.14.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.1 runs on Kubernetes 1.25.

Fixes:

Fixed an issue with the anthos-cluster-operator that caused CertificateSigningRequest (CSR) events to be missed during reconciliation steps. The lack of signing resulted in Istio crashlooping.
Fixed an issue that prevented the Pod CIDR for nodes from being adjusted from the default /24 mask size to account for the maxPodsPerNode cluster setting.
Fixed an issue that removed taints from control plane nodes when upgrading clusters to version 1.14.0.
The following container image security vulnerabilities have been fixed:

Functionality changes:

Changed the behavior for periodic health checks during upgrades. Now, during the upgrade process, existing periodic health checks continue to run in the admin cluster. Once the cluster is upgraded to the next version, the previous version periodic health checks are replaced with periodic health checks for the new version.
Lowered the priority of health check jobs to minimize contention for resources.
Changed the etcd history compaction interval from the default of 5 minutes to 2.5 minutes. This value is set in the kube-apiserver.yaml file.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

January 26, 2023

1.13

Release 1.13.4

Anthos clusters on bare metal 1.13.4 is now available for download. To upgrade, see Upgrade clusters. Anthos clusters on bare metal 1.13.4 runs on Kubernetes 1.24.

Fixed an issue with the anthos-cluster-operator that caused CertificateSigningRequest (CSR) events to be missed during reconciliation steps. The lack of signing resulted in Istio crashlooping.

The following container image security vulnerabilities have been fixed:

CVE-2021-3759
CVE-2021-4037
CVE-2021-46848
CVE-2022-1184
CVE-2022-20421
CVE-2022-2978
CVE-2022-3169
CVE-2022-3176
CVE-2022-3524
CVE-2022-3564
CVE-2022-3565
CVE-2022-3586
CVE-2022-3594
CVE-2022-3621
CVE-2022-3640
CVE-2022-3643
CVE-2022-3646
CVE-2022-3903
CVE-2022-39188
CVE-2022-40303
CVE-2022-40304
CVE-2022-40307
CVE-2022-41849
CVE-2022-41850
CVE-2022-41916
CVE-2022-42010
CVE-2022-42011
CVE-2022-42012
CVE-2022-42328
CVE-2022-42329
CVE-2022-42895
CVE-2022-42896
CVE-2022-42898
CVE-2022-43680
CVE-2022-43750
CVE-2022-44638
CVE-2022-47518
CVE-2022-47519
CVE-2022-47520
CVE-2022-47521

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 21, 2022

1.13 & 1.14

Anthos clusters on bare metal release 1.14.0 is now available for download. Note that Anthos clusters on bare metal version 1.14.0 runs on Kubernetes 1.25. Multiple deprecated APIs are deleted in Kubernetes 1.25. Before you upgrade version 1.13 Anthos clusters to version 1.14, check to see if you are affected by the Kubernetes API deletions.

If you aren't affected by the API deletions, see Upgrade clusters in the 1.14 documentation for upgrade instructions.

December 19, 2022

1.13

Release 1.13.3

Anthos clusters on bare metal 1.13.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.3 runs on Kubernetes 1.24.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 14, 2022

1.12

Release 1.12.6

Anthos clusters on bare metal 1.12.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.6 runs on Kubernetes 1.23.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 13, 2022

1.14

Release 1.14.0

Anthos clusters on bare metal 1.14.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.0 runs on Kubernetes 1.25.

Improved cluster lifecycle functionalities:

Upgraded from Kubernetes version 1.24 to 1.25.
Enabled customers to run the latest health and preflight checks by running the command bmctl check cluster –check-image-version=latest. Setting the check-image-version flag to 'latest' ensures that clusters are examined for more recent issues, including issues discovered after a release.
Preview: Added support of Control group v2 (cgroup v2).
GA: Added automatic reservation of CPU and memory resources on cluster nodes so that system daemons have the resources they require.
Optimized the consumption of resources by components such as cluster-operator, cap-manager, preflight-check operator, and lifecycle-controllers-manager.
GA: Enabled automatic and periodic health checks on all clusters.

Networking:

Preview: Added support for turning on kube-proxy-free mode for cluster objects. WARNING: This operation is not reversible. Once enabled, it cannot be disabled.
Changed behavior of Dataplane V2 so that it drops a packet if no service backends are available. Previously, the packet was passed to the kernel stack.
Enabled automatic API rate limit adjustments in Dataplane V2.

Observability:

Added severity level to container logs.
Enabled collection of uptime and other Kubernetes resource metrics from the kubelet summary API.
Enabled Stackdriver log forwarder in the bootstrap cluster. This log forwarder publishes bootstrap container logs to Cloud Logging.

Security and Identity:

GA: Added feature enabling cluster administrators to configure RBAC policies based on Azure Active Directory (AD) groups. Groups information for users belonging to more than 200 groups can now be retrieved.
GA: Added secure computing mode (seccomp) support. Running containers with a seccomp profile improves the security of a cluster because it restricts the system calls that containers are allowed to make to the kernel.
Added annotation in the cluster configuration file which allows customers to disable the kubelet read-only port. After disabling the read-only port, customers have to change their cluster configurations so that workloads use the kubelet secure port.

VM Runtime:

GA: Added support for guest OS booting of UEFI. Previously, only BIOS was supported.
Preview: Enabled Terraform scripting to create VMs on an Anthos cluster. For more information, including usage instructions, an input reference, and examples, see the terraform-google-anthos-vm GitHub repository.
Preview: Add support for non-uniform memory access (NUMA) awareness. When enabled, all communication within the VM is local to the NUMA node, thus avoiding the performance cost of data transactions with remote memory locations.
Preview: Enabled multicast traffic for VMs.
Added Anthos VM Runtime preflight checks to validate hardware accelerator configuration.
Enabled configuration of storage's volume mode (block or filesystem) and access modes, such as RWO and RWX.
Enabled means to configure the storage class of a scratch space. A scratch space is sometimes required when importing or uploading a VM disk image.
Added support for configuring cloud-init, using virtctl.
Enabled ability to disable auto-installation of the guest agent binary. After the initial guest agent installation, yoiu can set the autoInstallGuestAgent flag to false so that the binary doesn't mount in subsequent restarts.
Enabled the support of multiple network interfaces, by default, for all clusters.
Improved security for creating a VM with kubectl virt create. If an initial password is specified, it is now stored in a secret and not as a VM annotation.
Reduced the permissions of the network controller.
Changed default to always use Asynchronous IO mode (AIO) in order to reduce QEMU memory pressure.
Added VM creation and disk provisioning times to Prometheus metrics.
Added support for the Tesla T4 GPU.
Enabled reset of GPU card to its original status when GPU functionality is disabled.
Enabled ability to disable Anthos VM Runtime when it's in the enabling state and custom resource definitions haven't yet been installed.
Added the following command, which allows you to display the VM screen: kubectll virt vnc --screenshot VM_NAME.
Fixed the IP address update for Windows guest VMs.
Resolved the MacVTap interface creation failure which occurred when the name of the interface was too long.
Fixed attaching VM disk using SATA driver.
Fixed issue so that setting disableCDIUploadProxyVIP to true correctly disables the cdi-uploadproxy service.
Fixed issue so that specifying a PersistentVolumeClaim (PVC) with an empty underlying PersistentVolume (PV) correctly creates the underlying empty disk format (raw or qcow2).
Enforced VM names to follow the standard RFC1123 format.
Fixed issue so that ISO image is correctly imported from a Cloud Storage bucket.
Fixed benign crash looping of the NVIDIA device plugin and the Multi-Instance GPU (MIG) manager when all GPU cards are allocated to a VM.
Fixed issue so that virt-launcher Pod can be created when advanced compute is enabled.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 09, 2022

1.12

Release 1.12.5

Anthos clusters on bare metal 1.12.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.5 runs on Kubernetes 1.23.

Fixes:

The following container image security vulnerabilities have been fixed:

CVE-2019-25013
CVE-2021-3326
CVE-2021-3999
CVE-2021-4037
CVE-2021-33574
CVE-2021-35942
CVE-2022-1184
CVE-2022-1586
CVE-2022-1587
CVE-2022-2663
CVE-2022-3061
CVE-2022-3176
CVE-2022-3303
CVE-2022-3586
CVE-2022-3621
CVE-2022-3646
CVE-2022-3649
CVE-2022-20421
CVE-2022-23218
CVE-2022-23219
CVE-2022-32221
CVE-2022-33745
CVE-2022-33746
CVE-2022-33748
CVE-2022-34903
CVE-2022-37434
CVE-2022-39188
CVE-2022-40307
CVE-2022-42309
CVE-2022-42310
CVE-2022-42311
CVE-2022-42312
CVE-2022-42313
CVE-2022-42314
CVE-2022-42315
CVE-2022-42316
CVE-2022-42317
CVE-2022-42318
CVE-2022-42319
CVE-2022-42320
CVE-2022-42321
CVE-2022-42322
CVE-2022-42323
CVE-2022-42324
CVE-2022-42325
CVE-2022-42326
CVE-2022-43680
CVE-2022-43750

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 22, 2022

1.13

Release 1.13.2

Anthos clusters on bare metal 1.13.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.2 runs on Kubernetes 1.24.

Fixes:

Ensured the kubeadmconfig Secret is deleted when a Cluster API node is removed.
Added preflight check command (bmctl check preflight) that you can use when upgrading version 1.13 and higher clusters.
Updated the commands bmctl check preflight and bmctl create cluster so that they fail if worker or control-plane nodes have docker credentials in /root/.docker/config.json. (Anthos clusters on bare metal version 1.13 and higher can no longer use Docker Engine as a container runtime. All clusters must use the default container runtime containerd).
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 18, 2022

1.11

Release 1.11.8

Anthos clusters on bare metal 1.11.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.8 runs on Kubernetes 1.22.

Fixes:

The following container image security vulnerabilities have been fixed:

CVE-2019-25013
CVE-2020-16156
CVE-2021-3326
CVE-2021-3999
CVE-2021-4037
CVE-2021-33574
CVE-2021-35942
CVE-2022-1184
CVE-2022-1586
CVE-2022-1587
CVE-2022-2663
CVE-2022-3061
CVE-2022-3116
CVE-2022-3176
CVE-2022-3303
CVE-2022-3586
CVE-2022-3621
CVE-2022-3646
CVE-2022-3649
CVE-2022-20421
CVE-2022-23218
CVE-2022-23219
CVE-2022-33745
CVE-2022-33746
CVE-2022-33748
CVE-2022-37434
CVE-2022-39188
CVE-2022-40307
CVE-2022-42309
CVE-2022-42310
CVE-2022-42311
CVE-2022-42312
CVE-2022-42313
CVE-2022-42314
CVE-2022-42315
CVE-2022-42316
CVE-2022-42317
CVE-2022-42318
CVE-2022-42319
CVE-2022-42320
CVE-2022-42321
CVE-2022-42322
CVE-2022-42323
CVE-2022-42324
CVE-2022-42325
CVE-2022-42326
CVE-2022-43750

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 08, 2022

1.12

Release 1.12.4

Anthos clusters on bare metal 1.12.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.4 runs on Kubernetes 1.23.

Fixes:

Increased the CPU limit for the metrics-server Pod to prevent it from frequently restarting.
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 07, 2022

1.11 & 1.12 & 1.13

Security bulletin (1.11, 1.12, and 1.13)

A security vulnerability, CVE-2022-39278, has been discovered in Istio, which is used in Anthos Service Mesh, that allows a malicious attacker to crash the control plane.

For instructions and more details, see the Anthos clusters on bare metal security bulletin.

November 01, 2022

1.13

Cluster lifecycle improvements in 1.13 and later

Preview: You can use the Google Cloud console to create user clusters, delete user clusters, and to add and remove node pools from a user cluster. To explore the new feature, try out the tutorial Create an Anthos on bare metal user cluster on Compute Engine VMs using the console.

October 31, 2022

1.13

Release 1.13.1

Anthos clusters on bare metal 1.13.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.1 runs on Kubernetes 1.24.

Fixes:

Increased the CPU limit for the metrics-server Pod to prevent it from frequently restarting.
Updated the command bmctl enable/disable vmruntime to block the CLI and periodically show the status of VMRuntime. The CLI remains blocked until the command has finished and VMRuntime's ready status is true, or until a default period of 10 mins has expired.
Removed deprecated IPv6DualStack featureGate field that blocks clusters from upgrading to Anthos clusters on bare metal version 1.13.0.
Fixed the no path ID mapping for prefix error that sometimes occurred when bgpd updated bgpsession.
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 28, 2022

1.11

Anthos clusters on bare metal 1.11.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.7 runs on Kubernetes 1.22.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues: For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 05, 2022

1.12

Release 1.12.3

Anthos clusters on bare metal 1.12.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.3 runs on Kubernetes 1.23.

Fixes:

Updated the container image to resolve a YAML text/template vulnerability.
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

September 29, 2022

1.13

Release 1.13.0

Anthos clusters on bare metal 1.13.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.0 runs on Kubernetes 1.24.

The dockershim component in Kubernetes enables cluster nodes to use the Docker Engine container runtime. However, Kubernetes 1.24 removed the dockershim component. Since Anthos clusters on bare metal version 1.13 runs on Kubernetes 1.24, version 1.13 and higher clusters can no longer use Docker Engine as a container runtime. All clusters must use the default container runtime containerd.

Improved cluster lifecycle functionalities:

Upgraded from Kubernetes version 1.23 to 1.24:
- Reverted some of the changes Kubernetes and the kubeadm tool made to certain labels and taints on control plane nodes. Changes were reverted so that older versions of Anthos clusters on bare metal remain supported. As a result, control plane nodes have the following labels and taints:
  - node-role.kubernetes.io/master label
  - node-role.kubernetes.io/control-plane label
  - node-role.kubernetes.io/master:NoSchedule taint
- Upgraded from kubeadm.k8s.io/v1beta2 to kubeadm.k8s.io/v1beta3 since the former is deprecated.
- Stopped automatic generation of Secret API objects containing service account tokens for every Service Account. For more information, see the LegacyServiceAccountTokenNoAutoGeneration section of the upgrade notes.
Breaking change: Version 1.12 clusters that use Docker Engine can upgrade to 1.13 only if the new container runtime is specified as containerd. Blocked the creation of new 1.13 clusters that use Docker Engine as the container runtime.
Preview: Added feature so that upgrades of an admin/hybrid/standalone cluster can proceed without a bootstrap cluster. Management of Anthos clusters on bare metal is now fully conformant to the Kubernetes Resource Model.
Added support of Red Hat Enterprise Linux (RHEL) 8.6.
Removed an erroneous CustomResourceDefinition (app.k8s.io.Application) from inclusion in the cluster creation process.
Fixed vulnerability to YAML injection by switching to safetext/yamltemplate.
GA: Added support for installing Anthos clusters on bare metal, using your own registry service, instead of gcr.io. For instructions and additional information, see Use a registry mirror to create clusters.
Eliminated false error messaging when the bmctl create cluster is run. The message erroneously reported an Invalid value in the spec.labels field of NodePool specifications.
Added feature so that resetting a user cluster doesn't require the cluster configuration file.
Reduced containerd disk usage by having containerd store just the uncompressed layers of an image rather than both the compressed and uncompressed layers.
Upgraded containerd to version 1.6.6.

Networking:

GA: Enabled Dynamic Flat IP with Border Gateway Protocol (BGP) support. This feature lets you configure flat mode using BGP in clusters by leveraging Network Gateway Group and BGP. In this mode the Pod's IP address is visible and routable without masquerading across multiple subdomains. Currently supports advertising IPv4 and IPv6 routes over IPv4 sessions.
GA: Added BGP-based Load Balancer support for IPv6. Added ability to disable the Bundled Ingress feature. Customers should disable this feature if they are using full Anthos Service Mesh (ASM) instead. (Bundled Ingress is unnecessary when full ASM is installed).

Observability:

Preview: Added support of multi-line parsing for Go and Java logs.
GA: Added support for Google Cloud Managed Service for Prometheus (GMP) for application metrics.
Refined kube-state-metrics so that only core metrics are collected by default.

Security:

GA: Added Google Groups support for Connect Gateway.
Switched distroless base image for Node Problem Detector.
Changed anet-operator/cilium-operator to run as non-root container.
Secured communication between metrics-server and api-server using the Transport Layer Security (TLS) protocol.

VM Runtime:

Fixed a memory leak in libvirt-go, which caused unbounded memory growth and risked crashing long-running VMs.
Provided guaranteed compute support so that customers can get Guaranteed Quality of Service (QoS)for the VM when needed.
Preview: Enabled Anthos VM to be allocated dedicated host cores. Each VM virtual core can be pinned to a dedicated host core.
Separated GPU installation and deletion logic. If only the container GPU workload is needed, customers can enable the GPU without having to enable VM Runtime.
Added support for the T4 GPU card.
Enabled automatic use of the VirtualMachineDisk name as the disk serial number. This change makes it easier for customers to identify the disk in the VM.
Enabled KubeVM cloud-init API and startup script API.
Added new CLI command (Virtctl) for resetting Windows VM password.
Fixed the following container image security vulnerability: CVE-2022-1798
Added feature that stops NVIDIA device plugins from crashing if a GPU card hasn't been allocated to a container.
Added support for automatic VM restarts after a configuration update. Previously, customers needed to stop the VM, apply the change, and then re-start the VM. To use the feature, set the autoRestartOnConfigurationChange flag to true in the VirtualMachine custom resource.
Improved the Kubernetes audit log of VM operations so that it contains detailed VM configuration and update information.
Fixed flooding of logs with cluster events that arise when a VM encounters disk I/O errors.
Added KubeVM roles. By binding with these roles, customers are granted permission to resources that manage VMs.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

September 26, 2022

1.11

Release 1.11.6

Anthos clusters on bare metal 1.11.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.6 runs on Kubernetes 1.22.

Fixes:

Updated the container image to resolve a yaml text/template vulnerability.
The following container image security vulnerabilities have been fixed:
- CVE-2022-2526
- CVE-2022-1798

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 30, 2022

1.11

Release 1.11.5

Anthos clusters on bare metal 1.11.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.5 runs on Kubernetes 1.22.

Fixes:

Increased the default storage size limit of etcd to 6 GiB.
The following container image security vulnerabilities have been fixed:
- CVE-2022-1664

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 25, 2022

1.12

Release 1.12.2

Anthos clusters on bare metal 1.12.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.2 runs on Kubernetes 1.23.

Features:

Added –use-disk flag to bmctl backup cluster command to use the disk instead of the in-memory buffer to back up a cluster. Use this option when available RAM is limited on your admin workstation.
Added --quiet flag to bmctl check cluster -- snapshot command to suppress logging to the console during the snapshot creation.

Fixes:

Added caching for the Cloud Audit Logging feature status to avoid unnecessary checks and improve performance.
Increased the etcd default DB size to 6GiB by default to address NO_SPACE_ALARM in high-scale clusters.
Fixed a libseccomp package incompatibility issue.
Fixed an issue with the machine-reset job getting stuck.
Fixed an issue that caused continuous, unneeded cluster reconciliation operations.
Fixed an issue that prevented the node problem detector from running after a cluster upgrade.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 23, 2022

1.10

Release 1.10.8

Anthos clusters on bare metal 1.10.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.8 runs on Kubernetes 1.21.

Fixes

The following container image security vulnerability has been fixed:

CVE-2022-1664

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.12

Anthos VM Runtime

Anthos VM Runtime is Generally Available (GA). Some features and capabilities are available for Preview only, as indicated in the following descriptions:

Upgraded Kubevirt to version 0.49.
Upgraded Containerized Data Importer (CDI) to version 1.43.0.
Added bmctl command to enable or disable Anthos VM Runtime on user clusters.
Added automatic upgrade of Anthos VM Runtime when upgrading Anthos clusters on bare metal.
Preview: Added ability to configure an eviction policy that controls how VMs automatically migrate to other hosts during maintenance events.
Preview: Added non-disruptive upgrading of VM runtime during live migration (that is, when VMs are unobtrusively migrated from one node to another).

VM APIs:

Simplified VM Compute API.
Added ability to create and manage disk resources for VMs that use Anthos VM Runtime.
Added ability to schedule VMs using standard Kubernetes scheduling primitives.
Preview: Added ability to use GPUs in VMs.
Added more access management capabilities to VM Guest Environment.
Preview: Added support for guest OS booting of UEFI. Previously, only BIOS was supported.

Observability:

Integrated VM telemetry and console logs into Google Cloud console. Telemetry information and log data are critical for monitoring the status of VMs and for troubleshooting problems with your cluster VMs.
Added VM CPU and memory metrics to Cloud Monitoring. These metrics can be viewed in the Anthos clusters VM status dashboard.
Added ability to view console logs for VMs that use Anthos VM Runtime.
Added logs that audit VM pods.

Guest OS support:

Added support for the following guest OS versions running on a Virtual Machine:

Windows Server 2019
Windows Server 2016
Windows 10
Red Hat Enterprise Linux (RHEL) 8
RHEL 7
CentOS 8
CentOS 7
Ubuntu 20.04
Ubuntu 18.04

VM networking features:

IPAMv4: Static IP Allocation for VM interfaces.
IP and MAC Stickiness for VM interfaces.
IPAMv4: DHCP for VM interfaces.
VLAN tagging support for VM Interfaces.
Multi-NIC for VM interfaces through native Dataplane V2 support (macvtap + Dataplane V2).
Static routes and DNS configurations at per-network basis.
NetworkPolicy enforcement at per-network basis.
Validating admission webhooks for Network and NetworkInterface object.
Network Mutation, allow the mutations of Gateway, DNS and the customized network routes in the network custom resource. The parent interface for the VM and the VLAN ID are not mutable. VMs that were already running before the network configuration change need to be restarted to pick up the change.
Added command to restart all VMs in a network.
Graceful IP release for VMs:
- During VM migration, the IP isn't released.
- IP addresses are released for VMs that are deleted or stopped.
For more information on networking, see Create and use virtual networks for Anthos VM Runtime.

VM Runtime issues:

When kubevirt is configured, customers should ensure that TOR switches have MAC learning enabled.
If you choose to manually run a DHCP ipconfig /renew command in a Windows VM, you should first perform a DHCP release, using theipconfig /release command. In other words, the sequence for manually performing a DHCP renewal in a Windows environment is the following:
```
ipconfig /release
ipconfig /renew
```

August 04, 2022

1.11

Release 1.11.4

Anthos clusters on bare metal 1.11.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.4 runs on Kubernetes 1.22.

Fixes:

Fixed issue in which cluster restores failed when /var/lib/etcd is a mount point.
Fixed issue in which attempts to skip minor versions when upgrading weren't blocked. For details about the upgrade policy, see Minor version upgrades.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 03, 2022

1.12

Release 1.12.1

Anthos clusters on bare metal 1.12.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.1 runs on Kubernetes 1.23.

Functionality changes:

Increased default memory limits for coredns, metallb-controller, metallb-speaker, metrics-server, anthos-cluster-operator, and cap-controller-manager.
Modified the dashboards Anthos cluster pod status and Anthos cluster node status. Specifically, the following changes were made:
- Replaced cadvisor resource metrics with summary API resource metrics.
- Added cpu, memory, and volume utilization metrics.
If you have already installed these dashboards in a project, you need to download the JSON files Anthos-cluster-pod-status.json and Anthos-cluster-node-status.json from the Dashboards for Anthos GitHub repository. You then need to import these JSON files into Cloud Monitoring. For details, see Install sample dashboards.

Fixes:

Fixed issue in which nodes drained or cordoned by kubectl were mistakenly marked as schedulable.
Fixed issue in which cluster controller and autoscaler conflicted with each other in the scaling of istiod, coredns, and istio-ingress Pods.
Fixed issue in which the wrong data type was used in health check log messages, resulting in panic messages.
Fixed issue in which cluster restores failed when /var/lib/etcd is a mount point.
Fixed issue in which attempts to skip minor versions when upgrading weren't blocked. For details about the upgrade policy, see Minor version upgrades.
Fixed issue in which an external VIP Service of type LoadBalancer would not respond when flat IP mode was enabled.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

Release 1.12.1 ships with containerd version 1.5.13, which requires libseccomp version 2.5 or higher. If your system doesn't have libseccomp version 2.5 or higher installed, update it in advance of upgrading existing clusters to version 1.12.1. Otherwise, you may see errors in cplb-update Pods for load balancer nodes such as:

runc did not terminate successfully: runc: symbol lookup error: runc:
undefined symbol: seccomp_notify_respond

To install the latest version of libseccomp in Ubuntu, run the following command:

sudo apt-get install  libseccomp-dev

To install the latest version of libseccomp in CentOS or RHEL, run the following command:

sudo dnf -y install libseccomp-devel

August 01, 2022

1.10

Release 1.10.7

Anthos clusters on bare metal 1.10.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.7 runs on Kubernetes 1.21.

Fixed a CrashLoopBackOff error generated by gke-metrics-agent when application metrics are enabled (that is, when enableStackdriverForApplications=true).
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

July 07, 2022

1.10

Release 1.10.6

Anthos clusters on bare metal 1.10.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.6 runs on Kubernetes 1.21.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 29, 2022

1.12

Release 1.12.0

Anthos clusters on bare metal 1.12.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.0 runs on Kubernetes 1.23.

The dockershim component in Kubernetes enables cluster nodes to use the Docker Engine container runtime. However, Kubernetes 1.24 removed the dockershim component. Starting from Anthos clusters on bare metal 1.12.0, you will not be able to create new clusters that use the Docker Engine container runtime. All new clusters should use the default container runtime containerd.

Improved cluster lifecycle functionalities:

Upgraded Anthos clusters on bare metal to use Kubernetes version 1.23.
Upgraded container runtime to containerd 1.5.
Updated preflight check to forward default SSH key if no key is provided.
Added support for new GCPAccounts field in the cluster configuration file. This field enables the assignment of a cluster-admin role to end-users.
Added labels to control plane, control plane load balancer, and load balancer node pools, so that these different node pools can be distinguished from each other.
Added nodepool reference label to nodes so that worker nodes can be listed in the UI.

Observability:

GA: Added Summary API metrics. These metrics are scraped from the Kubernetes Summary API and provide CPU, memory, and storage metrics for Pods, containers, and Nodes.
Added separate flags to enable logging and monitoring for user applications separately: EnableCloudLoggingForApplications and EnableGMPForApplications. The legacy flag EnableStackdriverForApplications will be deprecated and removed in future releases.
Preview: Added Google Cloud Managed Service for Prometheus to collect application metrics and monitor cluster health.
Upgraded GKE Metrics Agent (gke-metrics-agent) from version 1.1.0 to 1.8.3. This tool scrapes metrics from each cluster node and publishes them in Cloud Monitoring.
Added the following resource utilization metrics. For more information about these and other metrics, see View Anthos clusters on bare metal metrics:
- container/cpu/request_utilization
- container/cpu/limit_utilization
- container/memory/request_utilization
- container/memory/limit_utilization
- node/cpu/allocatable_utilization
- node/memory/allocatable_utilization
- pod/volume/utilization
Added sample dashboards for monitoring cluster health to Cloud Monitoring sample dashboards. Customers can install these dashboards with one click.  
Scoped down the RBAC permissions of stackdriver-operator, a component that performs logging and monitoring.

Security:

AIS CA deprecation. AIS certs are now signed by cluster CA.
Changed ca-rotation container image so that it uses a distroless rather than a Debian-based image.
RBAC permissions of the cluster-operator component have been eliminated or reduced to address elevated permissions.
GA: Anthos Identity Service LDAP authentication support.

Networking:

Preview: Enabled creation of IPv6 and Dual Stack LoadBalancer services. Border Gateway Protocol (BGP) is used for Dualstack clusters. Advertising IPv4 and IPv6 routes over IPv4 sessions is supported.
Preview: Added Network Connectivity Gateway feature support to provide HA VPN between Google Cloud and an on-premises Anthos cluster.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 23, 2022

1.11

Release 1.11.3

Anthos clusters on bare metal 1.11.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.3 runs on Kubernetes 1.22.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 09, 2022

1.9

Release 1.9.8

Anthos clusters on bare metal 1.9.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.8 runs on Kubernetes 1.21.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 02, 2022

1.10

Release 1.10.5

Anthos clusters on bare metal 1.10.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.5 runs on Kubernetes 1.21.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

May 26, 2022

1.11

Release 1.11.2

Anthos clusters on bare metal 1.11.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.2 runs on Kubernetes 1.22.

Starting with Anthos clusters on bare metal release 1.11.2, you can enable or disable Anthos VM Runtime by updating the VMRuntime custom resource only. The legacy spec.kubevirt settings in the cluster configuration are no longer supported. The VMRuntime custom resource is installed by default on version 1.10 and later hybrid, standalone, and user clusters. The VMRuntime custom resource can't be applied to admin clusters.

If you have Anthos VM Runtime enabled for your Anthos clusters on bare metal, you must disable it before upgrading clusters to version 1.11.2 or higher. If this step is not completed, your cluster upgrade will fail. You can re-enable Anthos VM Runtime after the upgrade is complete.

Starting with Anthos clusters on bare metal release 1.11.2, the Anthos VM Runtime API version has changed from v1alpha1 to v1. This version change doesn't affect the VMRuntime custom resource, but most other resources are affected.

Functionality changes:

The containerd runtime has been upgraded to 1.5.11-gke.0 to address CVE-2022-24769
Added a preflight check that disallows Ubuntu 18.04 distributions with 4.15.x Linux kernels.

Fixes:

Fixed cluster custom resource status reporting for pending reconciliations.
Fixed a bmctl check cluster command issue that caused the user cluster kubeconfig Secret to be overwritten.
Fixed an issue with manifest installation when last-applied-config is broken that caused upgrades to fail.
Fixed an issue to ensure that the 20-minute timeout for node draining is enforced during cluster upgrades. This timeout provides ample time for nodes to drain, but ensures that upgrades can always proceed.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

May 04, 2022

1.10

Release 1.10.4

Anthos clusters on bare metal 1.10.4 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.10.4 runs on Kubernetes 1.21.

Fixes:

The following container image security vulnerabilities have been fixed:
Role-based access control (RBAC) fixes:
- Set AutomountServiceAccountToken field for Node Problem Detector jobs and etcd-defrag Daemonsets to false.
- Set capi-kubeadm-bootstrap-controller-manager to use a dedicated service account.
- Scoped down configmap/(get, list, watch) permissions to metallb-config resource name.
- Scoped down configmap/get permission to core-dns-autoscaler resource name.
- Removed services.update permission for the MetalLB kube-system:controller role.
- anetd
  - Removed Cilium service account and replaced it with the account used by kubelet.
  - Removed pod and node access from Cilium cluster role.
  - Added Cilium cluster role to the kubelet service account.
  - Removed pods/(delete) role from cilium-operator cluster role.
  - Scoped down leases permissions in cilium-operator cluster role to cilium-operator-resource-lock resource name and kube-controller-manager resource name.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

May 02, 2022

1.11

Release 1.11.1

Anthos clusters on bare metal 1.11.1 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.11.1 runs on Kubernetes 1.22.

Fixes:

Resolved cluster installation issue in which cluster status is prematurely declared ready, resulting in a "Failed to wait for applied resources" error.
Added validation that a cluster's kubeconfig secret data is correct.
Added feature so that bmctl outputs line numbers of relevant yaml when a parsing error occurs.
Removed the misleading log "Waiting for pod to finish" on pods such as anetd that aren't meant to finish.
Added automatic inclusion of a control plane's virtual IP address to the cluster NO_PROXY list.
Role-based access control fixes:
- Set AutomountServiceAccountToken field for Node Problem Detector jobs to false.
- Set capi-kubeadm-bootstrap-controller-manager to use a dedicated service account.
- Scoped down deployment/(update,patch) permissions to the metrics-server resource name.
- Scoped down configmap/(get, list, watch) permissions to metallb-config resource name.
- anetd:
- Removed Cilium service account and replaced it with the account used by kubelet.
- Removed pod and node access from Cilium cluster role.
- Added Cilium cluster role to the kubelet service account.
- Removed pods/(delete) role from cilium-operator cluster role.
- Scoped down leases permissions in cilium-operator cluster role to cilium-operator-resource-lock resource name and kube-controller-manager resource name.
The following container image security vulnerabilities have been fixed:
- CVE-2021-3999

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 27, 2022

1.9

Release 1.9.7

Anthos clusters on bare metal 1.9.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.7 runs on Kubernetes 1.21.

Fixes:

The following container image security vulnerabilities have been fixed:
- CVE-2021-3999
- CVE-2022-24407

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 26, 2022

1.6 & 1.7 & 1.8 & 1.9 & 1.10 & 1.11

Security bulletin (all minor versions)

Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666 have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect Linux operating systems supported by Anthos clusters on bare metal. For instructions and more details, see the GCP-2022-014 security bulletin.

April 12, 2022

1.8 & 1.9 & 1.10

Security bulletin (1.8, 1.9, and 1.10)

A security vulnerability, CVE-2022-23648, has been discovered in containerd's handling of path traversal in the OCI image volume specification. Containers launched through containerd's CRI implementation with a specially-crafted image configuration could gain full read access to arbitrary files and directories on the host.

For more information, see the GCP-2022-013 security bulletin.

March 31, 2022

1.11

Release 1.11.0

Anthos clusters on bare metal 1.11.0 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.11.0 runs on Kubernetes 1.22.

Containerd is the default runtime in Anthos clusters on bare metal. Support for Docker as a container runtime on Kubernetes nodes will be removed from Anthos clusters on bare metal starting with version 1.13.0. If you use a node image based on Docker container runtime, please migrate your workloads to a Containerd node image as soon as possible. For more details, see Containerd node images.

The structure of the Anthos clusters on bare metal documentation is substantially different from previous versions. For details, see New documentation structure.

Kubernetes 1.22 has deprecated certain APIs, and a list of these deprecated APIs can be found in Kubernetes 1.22 deprecated APIs. In their manifests and API clients, customers need to replace references to the deprecated APIs with references to the newer API calls. For more information, see Deprecated API Migration Guide.

On January 31, 2022, CentOS 8 reached its end of life (EOL). As a result of the EOL, yum repositories stopped working for CentOS, which causes cluster creation and cluster upgrade operations to fail. For a workaround and more information, see Cluster creation or upgrades fail on CentOS.

Improved cluster lifecycle functionalities:

Upgraded Anthos clusters on bare metal to use Kubernetes version 1.22.
Updated cert-manager to version 1.5.4.
Added error messaging in the bmctl command line interface to better surface cluster installation or upgrade failure.
Incorporated audit logs into bmctl snapshots.
Added ability for registry mirror users to customize containerd configuration and have it automatically mirror public registry hosts other than gcr.io.
Changed bmctl update command so that it extracts manifests before updating a cluster.
Added feature so that a cluster kubeconfig file automatically renews when the cluster is upgraded and the kubeconfig Secret is renewed whenever cluster reconciliation takes place.
Added support for Red Hat Enterprise Linux (RHEL) and CentOS 8.5.
Added warning to bmctl command that docker containerRuntime will not be supported in version 1.13 of Anthos cluster on bare metal.
Added support for specifying CIDR blocks in the NoProxy section of the cluster's configuration file.
Added Service CIDR to NoProxy section of a cluster configuration file by default in order to fix a multinic in proxy environment issue.
Fixed a multi-NIC in proxy environment issue. Whenever the NO_PROXY environment variable is set, it includes the Service CIDR from the cluster specification.

Networking:

GA: Added egress Network Address Translation (NAT) gateway capability to provide persistent, deterministic routing for egress traffic from clusters. For more information, see Configure an egress NAT gateway for external communication.
GA: Added option for BGP bundled load balancer which advertises Load Balancer (LB) Virtual IP addresses (VIPs) to the network using the Border Gateway Protocol (BGP). This feature supports topologies across multiple subnets and can provide greater load-balancing bandwidth than bundled Layer 2 mode.
GA: Enabled SR-IOV. This feature allows you to configure Virtual Functions (VFs) on the supported devices on the nodes of their cluster. It also allows you to define the kernel module you want to bind to the VF.
GA: Enabled IPv4/IPv6 dual-stack support. Clusters can be deployed in a dual-stack network in which IPv4 and IPv6 addresses are assigned to both nodes and pods. By default, IPv4 is in island mode and IPv6 is in flat mode (a simplified network topology).
GA: Enabled static flat network (without BGP). This feature lets you configure a flat mode network for IPv4 addresses. A pod's IPv4 address is visible and routable within the same Layer 2 domain, without having to masquerade as the node's IP address.
Preview: Enabled Dynamic Flat IP with Border Gateway Protocol (BGP) support. This feature lets you configure flat mode using BGP in clusters with the help of Anthos Network Gateway and BGP. In this mode, the pod's IP address is visible and routable without masquerading across multiple subdomains. Currently supports advertising IPv4 and IPv6 routes over IPv4 sessions.
Fixed issue in which new MAC addresses of re-imaged nodes weren't updated.

Observability:

GA: Enabled collection of multiple network interfaces (multinic) logs from clusters. Logs are collected as system logs and are sent to Cloud Logging without charge to the customer.
Preview: Added Summary API metrics. These metrics provide CPU, memory, and storage statistics about pods, containers, and nodes.
Updated fluent-bit (stackdriver-log-forwarder) cri parser to avoid matching time fields multiple times.
Upgraded kube-state-metrics from version 1.9 to 2.4. This service generates metrics about Kubernetes API objects such as deployments, nodes, and pods.
Upgraded Metric Server from version 0.3.6 to 0.4.5. Metrics Server retrieves metrics from kubelets and exposes them through the Kubernetes Metrics API.

Security:

Preview: Added secure computing mode (seccomp) support. Running containers with a seccomp profile improves the security of a cluster because it restricts the system calls that containers are allowed to make to the kernel.
Added ability to disable rootless mode for system containers. Since version 1.10.0, Kubernetes control planes and Anthos clusters on bare metal system containers run as non-root containers by default.
Fixed CA rotation issues by increasing the ca-rotation timeout for admin clusters. While verifying that a static pod has been restarted after manifest update, the current hash is retrieved before the manifest changes are applied.

Known issues:

Deprecated metrics

Several Anthos metrics have been deprecated and, starting with this release, data is no longer collected for these deprecated metrics. If you use these metrics in any of your alerting policies, there won't be any data to trigger the alerting condition. For more information, including instructions to migrate to updated replacement metrics, see Deprecated metrics affects Cloud Monitoring dashboard in Known Issues.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.10

Release 1.10.3

Anthos clusters on bare metal 1.10.3 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.10.3 runs on Kubernetes 1.21.

Fixes:

The following container image security vulnerabilities have been fixed:
- CVE-2022-24407
- CVE-2022-23648

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

March 23, 2022

1.9

Release 1.9.6

Anthos clusters on bare metal 1.9.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.6 runs on Kubernetes 1.21.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend that you upgrade to the highest patch version to ensure that you have the latest security fixes. Always review the release notes before upgrading so that you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

March 14, 2022

1.8

Release 1.8.9

Anthos clusters on bare metal 1.8.9 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.9 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

When upgrading Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend that you upgrade to the highest patch version to ensure that you have the latest security fixes. Always review the release notes before upgrading so that you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 28, 2022

1.9

Release 1.9.5

Anthos clusters on bare metal 1.9.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.5 runs on Kubernetes 1.21.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 25, 2022

1.10

Release 1.10.2

Anthos clusters on bare metal 1.10.2 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.10.2 runs on Kubernetes 1.21.

Functionality changes:

A preflight check now verifies whether your node machine has enough disk space before starting an install.
Updated the bmctl check cluster --snapshot command so that snapshots now capture information about pods in cluster namespaces.
Updated the bmctl check cluster --snapshot command so that snapshots now capture information about cluster API machines and kubeadmin Secrets.

Fixes:

Fixed issue in which the edge profile's request to reserve resources is lost during the upgrade process.
Fixed bmctl upgrade command so that the log file upgrade-cluster.log is generated in the bmctl-workspace/cluster/logs directory.
Fixed issue in which the non-root login didn't have the proper permissions to perform bmctl backup or bmctl restore.
Fixed a Node Problem Detector service that sometimes failed to run on nodes after a cluster installation or upgrade.
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.8 & 1.9 & 1.10

Security bulletin (1.8, 1.9, and 1.10)

Envoy recently released multiple security vulnerability fixes. The vulnerabilities affect Anthos clusters on bare metal, because Envoy is used for Metrics Server.

For instructions and more details, see the GCP-2022-008 security bulletin.

February 04, 2022

1.6 & 1.7 & 1.8 & 1.9 & 1.10

Security bulletin (all minor versions)

A security vulnerability, CVE-2021-4034, has been discovered in pkexec, a part of the Linux policy kit package (polkit), that allows an authenticated user to perform a privilege escalation attack. PolicyKit is generally used only on Linux desktop systems to allow non-root users to perform actions, such as rebooting the system, installing packages, restarting services, as governed by a policy.

For instructions and more details, see the GCP-2022-004 security bulletin.

February 01, 2022

1.8

Release 1.8.8

Anthos clusters on bare metal 1.8.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.8 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

January 27, 2022

1.9

Release 1.9.4

Anthos clusters on bare metal 1.9.4 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.9.4 runs on Kubernetes 1.21.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.10

Release 1.10.1

Anthos clusters on bare metal 1.10.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.1 runs on Kubernetes 1.21.

Fixes:

Fixed PreflightCheck to allow the preflightCheck.Spec.ConfigYAML field to be empty.
Fixed PreflightCheck to allow an existing GKE Hub membership, if the cluster already exists.
Fixed issue that blocked access to external Virtual IP addresses of Services, such as a Load Balancer, when Flat IPv4 is enabled.
Fixed issue in which the use of –nodes/ and –node-ssh-key flags when taking an admin-less snapshot of a cluster resulted in an empty snapshot.
Fixed issue that caused installation of version 1.10.0 clusters to fail when the umask setting for the root user on the target machine wasn't 0022. For more information, see Failure on systems with restrictive umask setting.
Fixed issue in which BGP load balancer preflight checks failed if the Kubernetes interface had a period ('.') in the name. (For example, VLAN interfaces often have names such as eth0.1).
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 22, 2021

1.9

Release 1.9.3

Anthos clusters on bare metal 1.9.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.3 runs on Kubernetes 1.21.

Fixes:

Fixed an issue in which cluster creation fails if a cluster has more than one control plane node, and the HTTPS_PROXY environment variable has been defined on one or more of the control plane nodes.
Upgraded Kubernetes version from 1.21.4 to 1.21.5 to address an error in which pods become stuck in the ContainerCreating state because libcontainer mistakenly throws a "unit already exists" error.
The following container image security vulnerability has been fixed:
- CVE-2020-21913

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 14, 2021

1.8

Release 1.8.7

Anthos clusters on bare metal 1.8.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.7 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerability has been fixed:

CVE-2020-21913

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 10, 2021

1.10

Release 1.10.0

Anthos clusters on bare metal 1.10.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.0 runs on Kubernetes 1.21.

Improved cluster lifecycle functionalities:

GA: Enabled Node Problem Detector to run by default on all nodes. You can check if a problem was detected on a node by running the kubectl describe command for the node. Then look for NodeConditions or Events reported by Node Problem Detector.
GA: Added bmctl backup cluster and bmctl restore cluster commands to facilitate disaster recovery for clusters.
Preview: Added the ability to reset individual nodes using the SSH key.
Updated the bmctl check cluster command so that the snapshot of a cluster includes the cluster's YAML file and logs that are in the bmctl-workspace directory.
Added a new status field cluster.gkeHubRegistrationStatus. The command kubectl get cluster now shows information about the cluster's membership to GKE Hub.

Networking:

Preview: Enabled Anthos multi-cluster connectivity to provide Anthos clusters a way to connect to another Anthos cluster in the same data center (intra-site, cluster-to-cluster). Pods in connected clusters can reach each other over pod IP addresses without using native address translation (NAT) in between.
Preview: Enabled IPv4/IPv6 dual stack support. Customers can deploy clusters in a dual-stack network, where IPv4 and IPv6 addresses can be assigned to both nodes and pods.
Preview: Enabled "flat mode" (a simplified network topology) for IPv4 , where the pod's IPv4 address is visible and routable without masquerading as node IP within the same Layer 2 domain.
Preview: Enabled SR-IOV. This feature lets you configure Virtual Functions (VFs) on the supported devices on the nodes of their cluster. This feature also lets you define the kernel module you want to bind to the VF.

Observability:

GA: Added ability to show severity level of an issue in Cloud Logging. Severity level is extracted from containerd and kubelet node logs.
GA: Changed collection of application metrics to use a more scalable monitoring pipeline based on OpenTelemetry. This change significantly reduces the amount of resources required to collect metrics.

Security:

GA: Enhanced the capability to rotate cluster certificate authorities (CAs). Updates include support for all cluster types, rotation of front-proxy and etcd CAs, and changes to the bmctl command syntax.
Preview: Enabled installation of Anthos clusters on bare metal using a short-lived Google Service Account token instead of using Google Service Account keys.
Enabled Kubernetes control plane and most Anthos system containers to run as non-root users. For details, see Don't run containers as root user.

VM Runtime:

Preview: Supported enabling or disabling Anthos VM Runtime on user clusters.
Preview: Enabled Anthos VM Runtime to support QEMU Copy On Write (QCOW2) format, which is a storage format for virtual disks on virtual machines. Some benefits of virtual disk capabilities are independent thin provisioning, better compression, and encryption at rest.
Preview: Enabled VMRuntime custom resource and the Network custom resource, which let you create VMs on either the node network with a static IP address or the default pod network.
Preview: Enabled VM pods audit logs for VM runtime resources.
Preview: Expanded guest OS versions that can run on the virtual machine. We support Windows Server 2019, 2016, Windows 10, Red Hat Enterprise Linux (RHEL) 8, Centos 8, and Ubuntu 20.04 as guest OS.
Preview: Enabled virtual machine high availability to provide greater uptime for virtual machines instances (VMIs) by automatically detecting and recovering from a range of host machine failures.

Breaking changes:

The gateway capability used by the egress NAT gateway and Bundled load balancing with BGP Preview features have changed in this release. The NetworkGatewayGroup custom resource replaces AnthosNetworkGateway and the capability is enabled with a new advancedNetworking field in the cluster configuration file, instead of an annotation. These changes affect the ability to upgrade clusters that use earlier versions of the features.

Anthos clusters on bare metal blocks cluster upgrades from version 1.9 to version 1.10 for clusters that use either of these two advanced networking features. You can upgrade a version 1.9 admin cluster that is managing 1.9 user clusters that use these features to version 1.10, but object reconciliation breaks for the AnthosNetworkGateway custom resource. Object reconciliation is the mechanism whereby admin clusters automatically copy/restore objects on managed user clusters when the objects have been defined alongside the cluster configuration. Any AnthosNetworkGateway custom resources are still functional and can be modified with kubectl.

To bring a version 1.9 cluster that uses either advanced networking Preview feature up to version 1.10, reset or delete the cluster and create a new 1.10 cluster.

Preview features and products are subject to change and are provided for testing and evaluation purposes only. Do not use Preview features on your production clusters.

Functionality changes:

Enabled use of ADMIN_KUBECONFIG environment variable to reduce the number of bmctl command flags.
The cluster reconciliation process now checks for differences in the GKEHub membership before attempting to update it. If the GKEHub membership needs to be changed, the cluster is unregistered and then re-registered.
The advancedNetworking field in the cluster configuration file replaces the deprecated baremetal.cluster.gke.io/enable-anthos-network-gateway annotation for enabling advanced networking capabilities.
The NetworkGatewayGroup custom resource replaces the AnthosNetworkGateway custom resource.

Fixed cluster lifecycle functionalities:

Outputs from all bmctl commands except bmctl version are now written to log files.
Fixed strict mode for decoding the cluster YAML file. Extraneous information in the cluster YAML file now results in an error.
Fixed preflight check so that it no longer ignores the no_proxy setting.
Binaries in cluster provision no longer run from /tmp, which is often mounted with noexec options. This change fixes a preflight check "permission denied" error.
Switched the default server-side containerRuntime value from docker to containerd.

Observability:

Increased the priority of the kube-state-metrics service to keep it from being stuck in a pending state. This service generates metrics about Kubernetes API objects such as deployments, nodes, and pods.
Upgraded metrics-server to version 0.3.6 to fix a missing metrics issue that occurs when a duplicated pod name is present.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 09, 2021

1.7

Release 1.7.7

Anthos clusters on bare metal 1.7.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.7 runs on Kubernetes 1.19.

Fixes:

The 1.7.6 release has a known issue that blocks upgrades of 1.7.5 clusters. The 1.7.7 release allows you to upgrade from all earlier versions to get the latest security fixes.
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 29, 2021

1.8

Release 1.8.6

Anthos clusters on bare metal 1.8.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.6 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 15, 2021

1.7

Release 1.7.6

Anthos clusters on bare metal 1.7.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.6 runs on Kubernetes 1.19.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.9

Release 1.9.2

Anthos clusters on bare metal 1.9.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.2 runs on Kubernetes 1.21.

Fixes:

Updated preflight checks to use strict mode for decoding YAML to catch issues with indentation and misplaced fields in the cluster configuration file.
Fixed an issue that caused containerRuntime to default to docker, instead of containerd in certain uncommon situations.
Fixed an issue where node_filesystem metrics report incorrect size in Cloud Monitoring for mount-points other than root.
Fixed an issue that caused communication failures between Cloud Logging metadata agent and the Cloud Monitoring API when the root certificate authority (CA) on the host node isn't set up properly.
The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 29, 2021

1.6 & 1.7 & 1.8 & 1.9

Security bulletin (all minor versions)

The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem.

For more information, see the GCP-2021-011 security bulletin.

October 26, 2021

1.9

Release 1.9.1

Anthos clusters on bare metal 1.9.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.1 runs on Kubernetes 1.21.

Fixes:

Fixed bmctl to eliminate stack trace from error output.
The following container image security vulnerabilities have been fixed:

Functionality changes:

Updated the bmctl reset cluster command to prevent you from resetting an admin cluster if the admin cluster is managing user clusters.
Updated the bmctl create cluster command to block you from enabling the Anthos VM Runtime for admin clusters.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 21, 2021

1.8

Release 1.8.5

Anthos clusters on bare metal 1.8.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.5 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.6 & 1.7 & 1.8 & 1.9

Security bulletin (all minor versions)

A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742. Ingress-nginx custom snippets allows retrieval of ingress-nginx service account tokens and secrets across all namespaces. For more information, see the GCP-2021-024 security bulletin.

October 19, 2021

1.7

Release 1.7.5

Anthos clusters on bare metal 1.7.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.5 runs on Kubernetes 1.19.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 04, 2021

1.6 & 1.7 & 1.8 & 1.9

Security bulletin (all minor versions)

A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server.

For more information, see the GCP-2021-021 security bulletin.

September 28, 2021

1.9

Release 1.9.0

Anthos clusters on bare metal 1.9.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.0 runs on Kubernetes 1.21.

Improved cluster lifecycle functionalities:

Preview: Added ability to reset individual nodes with the bmctl reset node command. To give access to the needed cluster configuration file, use the command with the -c flag.
Preview: Added ability to recover from HA control plane quorum loss withbmctl restore --control-plane-node command.
Added bmctl create ksa command to create a Kubernetes Service Account (KSA) and generate a bearer token. To log in to the registered cluster, you can use the token in Cloud Console Kubernetes Engine > Clusters.
Preview: Added bmctl backup cluster and bmctl restore cluster commands to facilitate disaster recovery for clusters.

Introduced new troubleshooting capabilities:

Updated the bmctl check cluster --snapshot command to support uploading cluster diagnostic snapshots to a Cloud Storage bucket for review by Cloud Customer Care.
Provided access to bootstrap cluster logs to help troubleshoot cluster creation or upgrade problems.
Preview: Added support for Node Problem Detector service on nodes for quick detection of common node problems.

Enhanced monitoring and logging:

GA: Cloud Audit Logs capability is now generally available and enabled by default. Audit logs are useful for investigating suspicious API requests and for collecting statistics. For more information, see Use Audit Logging.
Switched to new open telemetry-based metrics agents to improve reliability, ability to scale, and resource usage.

Improved networking capabilities:

GA: The multi-NIC capability to provide additional interfaces to your pods is now generally available.
Preview: Added the single root I/O virtualization (SR-IOV) container network interface (CNI) plugin for multi-NIC.
Added support to configure cluster Domain Name System (DNS) provider options, such as upstream nameservers, with the new ClusterDNS custom resource definition.

Enhanced security:

SELinux is now always enabled in the container runtime for CentOS and RHEL.
Preview: Enhanced the capability to rotate cluster certificate authorities (CAs). Updates include support for all cluster types, rotation of front-proxy and etcd CAs, and changes to the bmctl command syntax.
Preview: Added Okta group support for authentication in Anthos Identity Service.

Functionality changes:

Changed default container runtime to containerd, containerRuntime: containerd for new clusters. Customers can still choose Docker as the container runtime.
Preview: Updated bmctl command, bmctl reset nodes --force, to support force removal of control plane nodes with etcd membership cleanup.
Added checks for cluster updates to verify access to cluster machines if changes to loginUser or sshKeyPrivatePath are detected. If the checks pass, Anthos clusters on bare metal saves the secret in the cluster.
Added new Anthos cluster control plane uptime dashboard in Cloud Monitoring with new metric kubernetes.io/anthos/container/uptime for component availability.
Added new alerts for control plane components availability with new metric kubernetes.io/anthos/container/uptime to replace deprecated alerts with metric kubernetes.io/anthos/up.

Fixes:

Added missing registry mirror package required for Cloud Audit Logs to the Registry Mirror.
Fixed issue with containerd not finding crictl due to /usr/local/bin not being in the SSH user's PATH.
Fixed flapping node readiness issues caused by an unhealthy Pod Lifecycle Event Generator (PLEG).
Fixed kernel support issue for Ubuntu 18.04 and 18.04.1 that prevented the anetd networking controller from working properly. Anthos clusters on bare metal release 1.9.0 works with all kernels supplied with supported distributions.

Known issues:

Control group v2 (cgroup v2) is not officially supported in Anthos clusters on bare metal release 1.9.0 and later. The presence of /sys/fs/cgroup/cgroup.controllers indicates that your system uses cgroup v2.
Anthos Service Mesh v1.10 is incompatible with Anthos clusters on bare metal release 1.9.0 running on Red Hat Enterprise Linux (RHEL) when SELinux is enabled. If you want to use Anthos Service Mesh, you must disable SELinux or set it to permissive mode on the host.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

September 21, 2021

1.8

Release 1.8.4

Anthos clusters on bare metal 1.8.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.4 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerabilities have been fixed: - CVE-2021-3711 - CVE-2021-3712 - CVE-2021-20305 - CVE-2021-33560

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

September 20, 2021

1.7

Release 1.7.4

Anthos clusters on bare metal release 1.7.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.4 runs on Kubernetes 1.19.

Fixes:

Fixed vulnerability CVE-2021-25741 that might allow users to create a container with subpath volume mounts to access files and directories outside of the volume, including on the host filesystem. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.7.0. For more information, see the GCP-2021-018 security bulletin.
Updated the Kubernetes patch version to address the following container image security vulnerabilities:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.7 & 1.8

Security bulletin (1.7 and 1.8)

A security issue was discovered in Kubernetes, CVE-2021-25741, where a user may be able to create a container with subpath volume mounts to access files and directories outside of the volume, including on the host filesystem. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal 1.7.x and 1.8.x releases, specifically 1.7.3 and earlier and 1.8.2 and earlier.

To fix this vulnerability, upgrade your Anthos clusters to version 1.7.4 or 1.8.3. For more information, see the GCP-2021-018 security bulletin.

August 27, 2021

1.8

Release 1.8.3

Anthos clusters on bare metal 1.8.3 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.3 runs on Kubernetes 1.20.

Breaking changes:

In Anthos clusters on bare metal release 1.8.0, we added a kernel version requirement for Ubuntu 18.04. We required a Linux kernel version of 4.17.0 or later. Anthos clusters on bare metal release 1.8.3 again supports all Linux kernel versions that ship with Ubuntu 18.04 and 20.04 distributions. As a result of this change, however, the egress NAT gateway feature that was provided for Preview in release 1.8.0 does not work with Anthos clusters on bare metal release 1.8.3.

Features:

Preview: Anthos Identity Service now works with Anthos clusters on bare metal to support LDAP authentication methods in addition to OIDC. You can use AIS with Microsoft Active Directory without the need for provisioning Active Directory Federation Services. For more information, see Setting up Anthos Identity Service with LDAP.
Preview: Anthos Metadata Agent replaces Stackdriver Metadata Collector and collects more accurate and usable metadata for Kubernetes resources. When you configure logging and monitoring, you need to enable the Config Monitoring for Ops API and grant the opsconfigmonitoring.resourceMetadata.writer IAM role to your logging-monitoring service account. If Anthos clusters on bare metal is installed behind a proxy, your proxy server must also allow connections to opsconfigmonitoring.googleapis.com.
Added preflight checks to verify that specific APIs are enabled for your Google Cloud project. Preflight checks return an error if any of the following APIs aren't enabled for your project:
- anthos.googleapis.com
- anthosaudit.googleapis.com
- anthosgke.googleapis.com
- cloudresourcemanager.googleapis.com
- gkeconnect.googleapis.com
- gkehub.googleapis.com
- iam.googleapis.com
- opsconfigmonitoring.googleapis.com
- logging.googleapis.com
- monitoring.googleapis.com
- stackdriver.googleapis.com
To enable these APIs when you create a cluster configuration file, use the --enable-apis flag with the bmctl create config command. For an example that uses the --enable-apis flag, see Create an admin cluster config with bmctl.
Added preflight checks for the following machine requirements:
- Minimum supported Linux kernel version
- Minimum required CPU
- Minimum required RAM

Fixes:

Fixed the following container image security vulnerabilities:
Fixed cluster creation and cluster update failures for nodes running CentOS or Red Hat Enterprise Linux (RHEL) with both SELinux and Cloud Audit Logs enabled.
Fixed Transmission Control Protocol (TCP) connection leakage issue.
Fixed an issue that prevented cert-manager from issuing ACME certificates over HTTP due to ImagePullBackOff errors.

Changes:

The Kubevirt version used for working with VM-based workloads is now v0.43.0-gke.3.
The bootstrap cluster is deleted when a cluster upgrade completes without errors.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 16, 2021

1.7

Release 1.7.3

Anthos clusters on bare metal 1.7.3 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.3 runs on Kubernetes 1.19.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 13, 2021

1.6

Release 1.6.4

Anthos clusters on bare metal 1.6.4 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.6.4 runs on Kubernetes 1.18.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

July 29, 2021

1.8

Release 1.8.2

Anthos clusters on bare metal 1.8.2 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.2 runs on Kubernetes 1.20.

Features:

Preview: Added capability to rotate cluster certificate authorities (CAs) for user clusters. For instructions on using the bmctl cluster credentials command to rotate cluster CAs, see Rotate user cluster certificate authority.
Preview: Added support for AppArmor with Anthos clusters on bare metal. You don't need to disable AppArmor on Ubuntu as a prerequisite for installation. When you create new 1.8.2 clusters or upgrade clusters to version 1.8.2, you can enable AppArmor either before or after you upgrade.

Fixes:

Fixed CVE-2021-3520 vulnerability related to a flaw in lz4, which provides support for LZ4 a lossless compression algorithm. The flaw impacts availability, but has potential to impact confidentiality and integrity as well.
Fixed bmctl operation failures that occur for some Ubuntu 20.04 LTS distributions with a more recent Linux kernel, including Ubuntu 20.04 LTS images on the 5.8 kernel. For more information about this issue and a workaround, see Ubuntu 20.04 LTS and bmctl.
Fixed OpenStack support for user clusters. In prior releases, cluster creation fails for user type clusters when the baremetal.cluster.gke.io/external-cloud-provider: "true" annotation is added to the cluster configuration file.
Fixed PATH environment issues for executing commands as a non-root user. For more information, see Known Issues.
Fixed an issue that caused user cluster resets (bmctl reset cluster) to get stuck while deleting namespaces.
Fixed out-of-memory (OOM) conditions related to Connect Agent memory usage that resulted in pod failures.
Fixed issue that blocked snapshots for clusters configured for passwordless SUDO capability for machine login (nodeAccess.loginUser: <login user name>).
Fixed issue that blocked some 1.7.x version admin, hybrid, or standalone clusters from upgrading to the 1.8 minor release. This issue affected some clusters that were updated by applying changes from an updated cluster configuration file.
Fixed Address Resolution Protocol (ARP) table issue for high-availability (HA) deployments that blocked upgrades from completing.

Functionality changes:

Expanded snapshots to include resource usage metrics to improve troubleshooting and support. Added metrics include the output of ip neigh, kubectl top nodes, and kubectl top pods commands.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

July 02, 2021

1.8

Release 1.8.1

Anthos clusters on bare metal release 1.8.1 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.8.1 runs on Kubernetes 1.20.

Fixes:

Fixed CVE-2021-34824 that could expose private keys and certificates from Kubernetes secrets through the credentialName field when using Gateway or DestinationRule. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.8.0. For more information, see the GCP-2021-012 security bulletin.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 30, 2021

1.8

Security bulletin (1.8)

The Istio project recently announced a security vulnerability (CVE-2021-34824) where credentials specified in the credentialName field for Gateway or DestinationRule can be accessed from different namespaces. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.8.0. For more information, see the GCP-2021-012 security bulletin.

June 21, 2021

1.8

Release 1.8.0

Anthos clusters on bare metal release 1.8.0 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.0 runs on Kubernetes 1.20.

Extended installation support:

Provided support to use containerd as the container runtime as GA for Anthos clusters on bare metal release 1.8.0. Cluster upgrades to 1.8.0 are blocked for 1.7.x clusters that are configured to use the preview containerd capability. For more information, see Upgrading 1.7.x clusters that use containerd in Known Issues.
Preview: Improved virtual machine (VM) management capability. Anthos VM Runtime uses KubeVirt to orchestrate VMs on clusters, allowing you to work with your VM-based apps and workloads in a uniform development environment. Anthos VM Runtime has worked with Anthos clusters on bare metal as a preview feature since November 2020 and we have continued to enhance its capability. For more information, see Working with VM-based workloads.
Added edge profile support for standalone clusters. The edge profile is recommended for edge devices with limited resources. Add profile: edge to the cluster config file when you create a standalone cluster to produce a cluster that has significantly reduced system resource requirements. The edge profile is only available for standalone clusters, it is ignored for other cluster types. For more information, see Creating standalone clusters.
Added support to specify provider ID for Nodes (controlPlane.nodePoolSpec.nodes.providerID) to support deploying on OpenStack using Load Balancing as a Service (LBaaS) resources. For more information, see Configure your clusters to use OpenStack.
Preview: Added support for installing Anthos clusters on bare metal, using your own registry service, instead of gcr.io. For instructions and additional information, see Installing Anthos Bare Metal using registry mirror.

Improved upgrade:

Enabled support for upgrading non-SELinux clusters to SELinux. For more information, see Enable SELinux in Upgrading Anthos clusters on bare metal.
Cluster upgrades are not blocked by excessive Node draining durations. During a cluster upgrade, if the draining process takes longer than 20 minutes for any specific Node, the upgrade process will carry on without waiting for draining to complete.

Updated user cluster lifecycle management:

Added bmctl improvements for resetting user cluster and adding additional preflight checks to confirm machine and network readiness for cluster creation:

Enhanced monitoring and logging:

Preview: Added Cloud Audit Logging capability, which enables audit logs to be written to Cloud Audit Logs in your Google project. Audit logs are useful for investigating suspicious API requests and for collecting statistics. For more information, see Enable Audit Logging.

Introduced new networking capabilities in preview:

Preview: Added multi-NIC capability to provide additional interfaces to your Pods.
Preview: Added egress NAT gateway capability to provide persistent, deterministic routing for the egress traffic from your clusters. For more information, see Configure an egress NAT gateway for external communication.
Preview: Added option for BGP bundled load balancer for Layer-3 (L3) topologies. This feature can be used with user clusters and admin clusters.

Enhanced security:

Workload Identity is GA. The Connect Agent Service Account Key is no longer required during installation. Connect Agent uses Workload Identity to authenticate to Google Cloud instead of an exported Google Cloud Service Account Key.

Expanded support for newer versions of operating systems:

Added support for installing Anthos clusters on bare metal on Red Hat Enterprise Linux (RHEL) 8.4, and CentOS 8.4

Functionality changes:

Added --workspace-dir flag to bmctl to allow changing the path and name of the workspace directory from the default bmctl-workspace. The workspace directory contains the configuration and log files generated by bmctl. When using the bmctl command, pass in a --workspace-dir flag to specify a non-default workspace directory location. If the directory does not exist, bmctl will create it for you.
Moved away from iptables-based NodePort and masquerade handling to eBPF-based management. NodePort and masquerade handling are now applied to the Node IP and default gateway interfaces only.

Fixes:

Resolved, as part of the GA support for using containerd as the container runtime, incorrect cgroup driver use. Newly created 1.8.0 clusters that are configured to use containerd will use the correct systemd cgroup driver.
Fixed issue that prevented usage metrics for the containerd process from being collected by Cloud Logging. This fix applies to newly created 1.8.0 clusters only.

Known issues:

If a Node is out of reach, Anthos clusters on bare metal can't start the draining process, which may impact the cluster upgrade process. For more information, see Node draining can't start when Node is out of reach.
Upgrading from 1.7.x clusters that use containerd as the container runtime to 1.8.0 is blocked. For more information, see Upgrading 1.7.x clusters that use containerd.
When running Anthos clusters on bare metal with firewalld enabled on either CentOS or Red Hat Enterprise Linux (RHEL), changes to firewalld can remove the Cilium iptables chains on the host network. The loss of the Cilium iptables chains causes the Pod on the Node to lose network connectivity outside of the Node. for more information, see Modifying firewalld will erase Cilium iptable chains.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 02, 2021

1.7

Release 1.7.2

Anthos clusters on bare metal release 1.7.2 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.7.2 runs on Kubernetes 1.19.

Fixes:

Fixed CVE-2021-25735 that could allow node updates to bypass a Validating Admission Webhook. For more details, open the Anthos clusters on bare metal tab of the GCP-2021-003 security bulletin.
Resolved the bmctl snapshot command failure when the user creates a custom cluster namespace omitting "cluster-" prefix from the cluster config file. The prefix is no longer required for a custom cluster namespace.
Added webhook blocks to prevent users from modifying control plane node pool and load balancer node pool resources directly. Control plane and load balancer node pools for Anthos clusters on bare metal are specified in the cluster resource, using the spec.controlPlane.nodePoolSpec and spec.LoadBalancer.nodePoolSpec sections of the cluster config file respectively.
Fixed the cluster upgrade command, bmctl upgrade cluster, to prevent it from interfering with user-installed Anthos Service Mesh (ASM).

Functionality changes:

Updated the bmctl check snapshot command so that it includes certificate signing requests in the snapshot.
Changed the upgrade process to prevent node drain issues from blocking upgrades. The upgrade process triggers a node drain. Now, if the node drain takes longer than 20 minutes, the upgrade process carries on to completion even when the draining hasn't completed. In this case, the upgrade output reports the incomplete node drain. Excessive drain times signal a problematic with pods. You may need to restart problem pods.
Updated cluster creation process, bmctl create cluster, to display logged errors directly on the command line. Prior to this release, detailed error messages were only available in the log files.

Known issues:

Node logs from nodes with a dot (".") in their name are not exported to Cloud Logging. For workaround instructions, see Node logs aren't exported to Cloud Logging in Anthos clusters on bare metal known issues.

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 17, 2021

1.6

Release 1.6.3

Anthos clusters on bare metal release 1.6.3 is now available. To upgrade, see Upgrading Anthos clusters on bare metal. Anthos clusters on bare metal 1.6.3 runs on Kubernetes 1.18.

Fixed:

Fixed CVE-2021-25735 that could allow node updates to bypass a Validating Admission Webhook. For more details, open the Anthos clusters on bare metal tab of the GCP-2021-003 security bulletin.
Fixed CVE-2021-28683, CVE-2021-28682, and CVE-2021-29258. For more details, open the Anthos clusters on bare metal tab of the GCP-2021-004 security bulletin.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 06, 2021

1.6 & 1.7

Security bulletin (1.6 and 1.7)

The Envoy and Istio projects recently announced several new security vulnerabilities (CVE-2021-28683, CVE-2021-28682, and CVE-2021-29258) that could allow an attacker to crash Envoy.

For more information, see the GCP-2021-004 security bulletin.

April 30, 2021

1.7

Release 1.7.1

Anthos clusters on bare metal release 1.7.1 is now available. To upgrade, see Upgrading Anthos clusters on bare metal. Anthos clusters on bare metal 1.7.1 runs on Kubernetes 1.19.

Functionality changes:

Customers can now take cluster snapshots regardless of whether the admin cluster control plane is running. This is helpful for diagnosing installation issues.
Deploying Anthos clusters on bare metal with SELinux is now fully supported on supported versions of Redhat Enterprise Linux. This applies for new installations of Anthos clusters on bare metal cases only.
User cluster creation with bmctl supports credential inheritance from the admin cluster by default. Credential overrides for the user cluster can be specified in the config file during cluster creation.

Fixes:

(Updated May 12, 2021) Fixed CVE-2021-28683, CVE-2021-28682, CVE-2021-29258. For more details, see the GCP-2021-004 security bulletin.
Fixed potential stuck upgrade from 1.6.x to 1.7.0. The bug was caused by a rare race condition when the coredns configmap failed to be backed up and restored during the upgrade.
Fixed potential missing GKE connect agent during installation due to a rare race condition.
Fixed issue that prevented automatic updates to the control plane load balancer config when adding/removing node(s) from the control plane node pool.
Addressed problem with syncing NodePool taints and labels that resulted in deletion of pre-existing items. Syncs will now append, update, or delete items that are added by taints and labels themselves only.

Known issues:

Upgrading the container runtime from containerd to Docker will fail in Anthos clusters on bare metal release 1.7.1. This operation is not supported while the containerd runtime option is in preview.
bmctl snapshot command fails when the user creates a custom cluster namespace omitting cluster- prefix from the cluster config file. To avoid this issue, the cluster namespace should follow the cluster-$CLUSTER_NAME naming convention.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 20, 2021

1.6 & 1.7

Security bulletin (1.6 and 1.7)

The Kubernetes project recently announced a new security vulnerability, CVE-2021-25735, that could allow node updates to bypass a Validating Admission Webhook. For more details, see the GCP-2021-003 security bulletin.

March 25, 2021

1.7

Release 1.7.0

Anthos clusters on bare metal release 1.7.0 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.7.0 runs on Kubernetes 1.19.

Extended installation support:

Added requirement for Anthos clusters on bare metal connectivity with Google Cloud for install and upgrade operations. As of 1.7.0 preflight checks will check for connectivity to Google Cloud, enabled APIs, and permissions for service accounts. Existing clusters need to be registered in Google Cloud before upgrading. The connectivity checks are not overridable by the --force flag. For details, see the cluster creation and cluster upgrade documentation.
Added support for installing Anthos clusters on bare metal on OpenStack. For configuration instructions, see Configure your clusters to use OpenStack.
Added support for installing Anthos clusters on bare metal, using a private package repository instead of the default Docker APT repository. For instructions and additional information, see Use a private package repository server.
Removed installation prerequisite for setting Security-Enhanced Linux (SELinux) operational mode to be permissive. The related preflight check has been removed, as well.
Removed installation prerequisite for disabling firewalld . The related preflight check has also been removed. For information on configuring ports to use firewalld with Anthos clusters on bare metal, see Configuring firewalld ports on the Network requirements page.
Updated requirements for installing behind a proxy server and removed restriction on system-wide proxy configurations. For a detailed list of prerequisites, see Installing behind a proxy.

Improved upgrade:

Updated cluster upgrade routines to ensure worker node failures do not block cluster upgrades, providing a more consistent user experience. Control plane node failures will still block cluster upgrades.
Added bmctl support for running upgrade preflight checks. bmctl check preflight will run upgrade preflight checks if users specify the --kubeconfig flag. For example:
bmctl check preflight --kubeconfig bmctl-workspace/cluster1/cluster1-kubeconfig

Updated user cluster lifecycle management:

Added support in bmctl for user cluster creation and upgrade functions.
Improved resource handling. Anthos clusters on bare metal now reconciles node pool taints and labels to nodes unless the node has a baremetal.cluster.gke.io/label-taint-no-sync annotation.

Enhanced monitoring and logging:

Preview: Added out-of-the-box alerts for critical cluster metrics and events. For information on working with alerting policies and getting notified, see Creating alerting policies.
Added support for collecting ansible job logs in admin and hybrid clusters by default.

Expanded support for newer versions of operating systems:

Added support for installing Anthos clusters on bare metal on Red Hat Enterprise Linux (RHEL) 8.3 and CentOS 8.3.

Functionality changes:

Added support for configuring the number of pods per node. New clusters can be configured to run up to 250 pods per node. For more information about configuring nodes, see Pod networking. You can find additional information for configuring pods in the cluster creation documentation.
Preview: Added support to use containerd as the container runtime. Anthos clusters on bare metal 1.6.x supports only Docker for container runtime (dockershim). In 1.7.0, Kubelet can be configured to use either Docker or containerd, using the new containerRuntime cluster config field. You must upgrade existing clusters to 1.7.0 to add or update the containerRuntime field.
Added support for more load balancer addressPool entries under cluster.spec.loadBalancer.addressPools. For existing addressPools, users can use cluster.spec.loadBalancer.AddressPools[].manualAssign specify additional addressPool entries.

Known issues:

Under rare circumstances, bmctl upgrade may become stuck at the Moving resources to upgraded cluster stage after finishing upgrading all nodes in the cluster. The issue does not affect cluster operation, but the final step needs to be finished.

If bmctl does not move forward after 30 minutes in this state, re-run the bmctl upgrade command to complete the upgrade.

The issue is captured in the upgrade-cluster.log file located in .../bmctl-workspace/<cluster name>/log/upgrade-cluster-<timestamp>. The following log entry shows how the failure is reported:

Operation failed, retrying with backoff. Cause: error creating "baremetal.cluster.gke.io/v1, Kind=Cluster" <cluster name>: Internal error occurred: failed calling webhook "vcluster.kb.io": Post "https://webhook-service.kube-system.svc:443/validate-baremetal-cluster-gke-io-v1-cluster? timeout=30s": net/http: TLS handshake timeout

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 26, 2021

1.6

Release 1.6.2

Anthos clusters on bare metal release 1.6.2 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.6.2 runs on Kubernetes 1.18.

Fixes:

Updated custom resource API to reject changes to Cluster and NodePool configuration fields that are not currently supported. For a list of supported mutable fields, see Configuration in Known Issues.
Updated bmctl to allow creating or upgrading Anthos clusters on bare metal to the current bmctl version (1.6.2) only. For more information about version restrictions, see Installation in Known Issues.
Fixed an issue that caused the automatic reset of bare metal machines to fail after deleting the user cluster.
Added preflight check to verify that control group v2, or cgroup v2 for short, is not in use on the cluster machine. Anthos on bare betal 1.6.x is incompatible with cgroup v2. For more information, see Control group v2 incompatibility in Known Issues.
Updated csi-snapshot-validation-webhook to support certification rotation. For more information about certificate rotation, see Security in Known Issues.
Fixed an issue to prevent constant patching for snapshot.storage.k8s.io CRDs.
Fixed a Certificate Signing Request (CSR) issue with kubelet to ensure fully qualified domain name(FQDN) hostnames are supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

January 29, 2021

1.6

Release 1.6.1

Anthos clusters on bare metal release 1.6.1 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.6.1 runs on Kubernetes 1.18.6-gke.6600.

Functionality changes:

Added upgrade support from 1.6.0. Users are able to upgrade existing Anthos bare metal cluster from 1.6.0 to 1.6.1.
Improved upgrade preflight check. Added preflight check before cluster upgrade to validate current cluster status, machine health and other issues before proceeding to upgrade.
Added support for deleting mounts and data from the anthos-system StorageClass during bmctl reset.
Relaxed the requirement for an odd number of control plane node pools to allow customers to add and remove nodes for maintenance or replacement.
Added support to force removing a broken worker node through annotation on the operator machine.
Added etcddefrag pod to control-plane nodes, which are responsible for monitoring etcd's database size and defragmenting the database as needed. This helps reclaim etcd database size and recover etcd when its disk space is exceeded.
Enabled kubelet server TLS certification auto-rotation. Kubelet on each node sends out CSR when nearing serving certificate expiration. A controller running inside the admin cluster validates and approves the CSR for user clusters.
Added proxy support to connect to the OIDC provider. This allows overriding the cluster proxy configuration with a different proxy.
Added bmctl update cluster for updating standalone clusters.

Fixes:

Fixed bug causing cluster deletion stall problem because of pods refusing to evacuate, or dead nodes.

November 30, 2020

1.6

Release 1.6.0

Anthos on bare metal is generally available

Anthos on bare metal is a deployment option to run Anthos on physical or virtual servers, deployed on an operating system provided by you, without a hypervisor layer. Anthos on bare metal ships with built-in networking, lifecycle management, diagnostics, health checks, logging, and monitoring. Anthos on bare metal supports CentOS, Red Hat Enterprise Linux (RHEL), and Ubuntu—all validated by Google. With Anthos on bare metal, you can use your company's standard hardware and operating system images, taking advantage of existing investments, which are automatically checked and validated against Anthos infrastructure requirements.

Anthos on bare metal is available today, with either subscription or pay-as-you-go pricing. Anthos on bare metal lets you leverage existing investments in hardware, OS, and networking infrastructure. The minimum system requirement to run Anthos on bare metal is 2 nodes with a minimum total of 4 cores, 32 GB RAM, and 128 GB of disk space with no specialized hardware. The setup lets you run Anthos on bare metal on almost any infrastructure.

Anthos on bare metal uses a "bring your own operating system" model. It runs atop physical or virtual instances, and supports Red Hat Enterprise Linux 8.1/8.2, CentOS 8.1/8.2, or Ubuntu 18.04/20.04 LTS. Anthos provides overlay networking and L4/L7 load balancing. You can also integrate with your own load balancer such as F5 and Citrix. For storage, you can deploy persistent workloads using CSI integration with your existing infrastructure.

You can deploy Anthos on bare metal using one of the following deployment models:

A standalone model lets you manage every cluster independently. This is a good choice when running in an edge location or if you want your clusters to be administered independent of one another.
The multiple-cluster model lets central IT teams manage a fleet of clusters from a centralized cluster, called the admin cluster. This is more suitable if you want to build automation or tooling, or if you want to delegate the lifecycle of clusters to individual teams without sharing sensitive credentials such as SSH keys or Google Cloud service account details.

Like with all Anthos environments, a bare metal cluster has a thin, secure connection back to Google Cloud called Connect. After it's installed in your clusters, you can centrally view, configure, and monitor your clusters from the Google Cloud Console.

Anthos on bare metal, which is part of the Anthos 1.6 release, provides the following features and capabilities:

Kubernetes 1.18
Ubuntu/RHEL/CentOS support
Standalone and multiple-cluster architecture
In-place upgrades (minor and major)
Overlay networking, Ingress (L7), integrated load balancing (L4, L2-Mode)
Manual load balancing (F5, Citrix)
Installs behind proxy support
Preflight and health checks
Node maintenance mode
Cloud Monitoring and Cloud Logging
ACM, ASM, identity, hub or connect, billing, and pay-as-you-go
NVIDIA GPU support
Scales to 500 nodes
Virtual machine management (Kubevirt) preview