GKE on Bare Metal 1.12 release notes

This document lists production updates to GKE on Bare Metal. We recommend that GKE on Bare Metal developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/anthos-bare-metal-release-notes.xml

January 31, 2024

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

June 27, 2023

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

June 16, 2023

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

May 10, 2023

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

April 12, 2023

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

March 28, 2023

Release 1.12.9

Anthos clusters on bare metal 1.12.9 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.9 runs on Kubernetes 1.23.

FIxes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

March 02, 2023

Release 1.12.8

Anthos clusters on bare metal 1.12.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.8 runs on Kubernetes 1.23.

Fixes:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

February 07, 2023

Release 1.12.7

Anthos clusters on bare metal 1.12.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.7 runs on Kubernetes 1.23.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

December 14, 2022

Release 1.12.6

Anthos clusters on bare metal 1.12.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.6 runs on Kubernetes 1.23.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 09, 2022

Release 1.12.5

Anthos clusters on bare metal 1.12.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.5 runs on Kubernetes 1.23.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 08, 2022

Release 1.12.4

Anthos clusters on bare metal 1.12.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.4 runs on Kubernetes 1.23.

Fixes:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 07, 2022

Security bulletin (1.11, 1.12, and 1.13)

A security vulnerability, CVE-2022-39278, has been discovered in Istio, which is used in Anthos Service Mesh, that allows a malicious attacker to crash the control plane.

For instructions and more details, see the Anthos clusters on bare metal security bulletin.

October 05, 2022

Release 1.12.3

Anthos clusters on bare metal 1.12.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.3 runs on Kubernetes 1.23.

Fixes:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 25, 2022

Release 1.12.2

Anthos clusters on bare metal 1.12.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.2 runs on Kubernetes 1.23.

Features:

  • Added –use-disk flag to bmctl backup cluster command to use the disk instead of the in-memory buffer to back up a cluster. Use this option when available RAM is limited on your admin workstation.
  • Added --quiet flag to bmctl check cluster -- snapshot command to suppress logging to the console during the snapshot creation.

Fixes:

  • Added caching for the Cloud Audit Logging feature status to avoid unnecessary checks and improve performance.
  • Increased the etcd default DB size to 6GiB by default to address NO_SPACE_ALARM in high-scale clusters.
  • Fixed a libseccomp package incompatibility issue.
  • Fixed an issue with the machine-reset job getting stuck.
  • Fixed an issue that caused continuous, unneeded cluster reconciliation operations.
  • Fixed an issue that prevented the node problem detector from running after a cluster upgrade.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 23, 2022

Anthos VM Runtime

Anthos VM Runtime is Generally Available (GA). Some features and capabilities are available for Preview only, as indicated in the following descriptions:

  • Upgraded Kubevirt to version 0.49.
  • Upgraded Containerized Data Importer (CDI) to version 1.43.0.
  • Added bmctl command to enable or disable Anthos VM Runtime on user clusters.
  • Added automatic upgrade of Anthos VM Runtime when upgrading Anthos clusters on bare metal.
  • Preview: Added ability to configure an eviction policy that controls how VMs automatically migrate to other hosts during maintenance events.
  • Preview: Added non-disruptive upgrading of VM runtime during live migration (that is, when VMs are unobtrusively migrated from one node to another).

VM APIs:

Observability:

Guest OS support:

Added support for the following guest OS versions running on a Virtual Machine:

  • Windows Server 2019
  • Windows Server 2016
  • Windows 10
  • Red Hat Enterprise Linux (RHEL) 8
  • RHEL 7
  • CentOS 8
  • CentOS 7
  • Ubuntu 20.04
  • Ubuntu 18.04

VM networking features:

  • IPAMv4: Static IP Allocation for VM interfaces.
  • IP and MAC Stickiness for VM interfaces.
  • IPAMv4: DHCP for VM interfaces.
  • VLAN tagging support for VM Interfaces.
  • Multi-NIC for VM interfaces through native Dataplane V2 support (macvtap + Dataplane V2).
  • Static routes and DNS configurations at per-network basis.
  • NetworkPolicy enforcement at per-network basis.
  • Validating admission webhooks for Network and NetworkInterface object.
  • Network Mutation, allow the mutations of Gateway, DNS and the customized network routes in the network custom resource. The parent interface for the VM and the VLAN ID are not mutable. VMs that were already running before the network configuration change need to be restarted to pick up the change.
  • Added command to restart all VMs in a network.

  • Graceful IP release for VMs:

    • During VM migration, the IP isn't released.
    • IP addresses are released for VMs that are deleted or stopped.

    For more information on networking, see Create and use virtual networks for Anthos VM Runtime.

VM Runtime issues:

  • When kubevirt is configured, customers should ensure that TOR switches have MAC learning enabled.

  • If you choose to manually run a DHCP ipconfig /renew command in a Windows VM, you should first perform a DHCP release, using theipconfig /release command. In other words, the sequence for manually performing a DHCP renewal in a Windows environment is the following:

    ipconfig /release
ipconfig /renew

August 03, 2022

Release 1.12.1

Anthos clusters on bare metal 1.12.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.1 runs on Kubernetes 1.23.

Functionality changes:

  • Increased default memory limits for coredns, metallb-controller, metallb-speaker, metrics-server, anthos-cluster-operator, and cap-controller-manager.

  • Modified the dashboards Anthos cluster pod status and Anthos cluster node status. Specifically, the following changes were made:

    • Replaced cadvisor resource metrics with summary API resource metrics.
    • Added cpu, memory, and volume utilization metrics.

    If you have already installed these dashboards in a project, you need to download the JSON files Anthos-cluster-pod-status.json and Anthos-cluster-node-status.json from the Dashboards for Anthos GitHub repository. You then need to import these JSON files into Cloud Monitoring. For details, see Install sample dashboards.

Fixes:

  • Fixed issue in which nodes drained or cordoned by kubectl were mistakenly marked as schedulable.
  • Fixed issue in which cluster controller and autoscaler conflicted with each other in the scaling of istiod, coredns, and istio-ingress Pods.
  • Fixed issue in which the wrong data type was used in health check log messages, resulting in panic messages.
  • Fixed issue in which cluster restores failed when /var/lib/etcd is a mount point.
  • Fixed issue in which attempts to skip minor versions when upgrading weren't blocked. For details about the upgrade policy, see Minor version upgrades.
  • Fixed issue in which an external VIP Service of type LoadBalancer would not respond when flat IP mode was enabled.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

Release 1.12.1 ships with containerd version 1.5.13, which requires libseccomp version 2.5 or higher. If your system doesn't have libseccomp version 2.5 or higher installed, update it in advance of upgrading existing clusters to version 1.12.1. Otherwise, you may see errors in cplb-update Pods for load balancer nodes such as:

runc did not terminate successfully: runc: symbol lookup error: runc:
undefined symbol: seccomp_notify_respond

To install the latest version of libseccomp in Ubuntu, run the following command:

sudo apt-get install  libseccomp-dev

To install the latest version of libseccomp in CentOS or RHEL, run the following command:

sudo dnf -y install libseccomp-devel

June 29, 2022

Release 1.12.0

Anthos clusters on bare metal 1.12.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.0 runs on Kubernetes 1.23.

The dockershim component in Kubernetes enables cluster nodes to use the Docker Engine container runtime. However, Kubernetes 1.24 removed the dockershim component. Starting from Anthos clusters on bare metal 1.12.0, you will not be able to create new clusters that use the Docker Engine container runtime. All new clusters should use the default container runtime containerd.

Improved cluster lifecycle functionalities:

  • Upgraded Anthos clusters on bare metal to use Kubernetes version 1.23. 

  • Upgraded container runtime to containerd 1.5.

  • Updated preflight check to forward default SSH key if no key is provided.

  • Added support for new GCPAccounts field in the cluster configuration file. This field enables the assignment of a cluster-admin role to end-users.

  • Added labels to control plane, control plane load balancer, and load balancer node pools, so that these different node pools can be distinguished from each other.

  • Added nodepool reference label to nodes so that worker nodes can be listed in the UI.

Observability:

  • GA: Added Summary API metrics. These metrics are scraped from the Kubernetes Summary API and provide CPU, memory, and storage metrics for Pods, containers, and Nodes.

  • Added separate flags to enable logging and monitoring for user applications separately: EnableCloudLoggingForApplications and EnableGMPForApplications. The legacy flag EnableStackdriverForApplications will be deprecated and removed in future releases.

  • Preview: Added Google Cloud Managed Service for Prometheus to collect application metrics and monitor cluster health.

  • Upgraded GKE Metrics Agent (gke-metrics-agent) from version 1.1.0 to 1.8.3. This tool scrapes metrics from each cluster node and publishes them in Cloud Monitoring.

  • Added the following resource utilization metrics. For more information about these and other metrics, see View Anthos clusters on bare metal metrics:

    • container/cpu/request_utilization
    • container/cpu/limit_utilization
    • container/memory/request_utilization
    • container/memory/limit_utilization
    • node/cpu/allocatable_utilization
    • node/memory/allocatable_utilization
    • pod/volume/utilization

  • Added sample dashboards for monitoring cluster health to Cloud Monitoring sample dashboards. Customers can install these dashboards with one click.



  • Scoped down the RBAC permissions of stackdriver-operator, a component that performs logging and monitoring.

Security:

  • AIS CA deprecation. AIS certs are now signed by cluster CA.

  • Changed ca-rotation container image so that it uses a distroless rather than a Debian-based image.

  • RBAC permissions of the cluster-operator component have been eliminated or reduced to address elevated permissions.

  • GA: Anthos Identity Service LDAP authentication support.

Networking:

  • Preview: Enabled creation of IPv6 and Dual Stack LoadBalancer services. Border Gateway Protocol (BGP) is used for Dualstack clusters. Advertising IPv4 and IPv6 routes over IPv4 sessions is supported.

  • Preview: Added Network Connectivity Gateway feature support to provide HA VPN between Google Cloud and an on-premises Anthos cluster.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.