GDCV for Bare Metal 1.28 release notes

This document lists production updates to GDCV for Bare Metal. We recommend that GKE on Bare Metal developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/anthos-bare-metal-release-notes.xml

April 08, 2024

Release 1.28.400-gke.77

GKE on Bare Metal 1.28.400-gke.77 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.400-gke.77 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Functionality changes:

  • Updated preflight checks to add a check for networking kernel modules.
  • Updated preflight checks to remove the check for iptables package availability.

Fixes:

  • Fixed a cluster upgrade issue where the lifecycle-controller-deployer Pod was unable to migrate existing GKE on Bare Metal resources to the latest API version. This issue blocked upgrades to earlier version 1.28 releases.

Fixes:

The following container image security vulnerabilities have been fixed in 1.28.400-gke.77:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

April 03, 2024

A Denial-of-Service (DoS) vulnerability (CVE-2023-45288) was recently discovered in multiple implementations of the HTTP/2 protocol, including the golang HTTP server used by Kubernetes. The vulnerability could lead to a DoS of the Google Kubernetes Engine (GKE) control plane. For more information, see the GCP-2024-022 security bulletin.

March 21, 2024

Release 1.28.300-gke.131

GKE on Bare Metal 1.28.300-gke.131 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.300-gke.131 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Functionality changes:

  • Updated preflight checks to add a check for networking kernel modules.

  • Updated preflight checks to remove the check for iptables package availability.

  • Increased the default memory limit for node-exporter.

Fixes:

  • Fixed an issue with configuring a proxy for your cluster that required you to manually set HTTPS_PROXY and NO_PROXY environment variables on the admin workstation.

The following container image security vulnerabilities have been fixed in 1.28.300-gke.131:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

March 04, 2024

Release 1.28.200-gke.118

GKE on Bare Metal 1.28.200-gke.118 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.200-gke.118 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Fixes:

  • Fixed an issue where upgrades are blocked because cluster-operator can't delete stale, failing preflight check resources.

Fixes:

The following container image security vulnerabilities have been fixed in 1.28.200-gke.118:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

January 31, 2024

Release 1.28.100-gke.146

GKE on Bare Metal 1.28.100-gke.146 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.100-gke.146 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

Fixes:

Fixed a rootless permission issue on file /var/lib/audit.log in 1.28.100, which might block control plane node upgrades.

The following container image security vulnerabilities have been fixed in 1.28.100-gke.146:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

December 15, 2023

Release 1.28.0-gke.435

GKE on Bare Metal 1.28.0-gke.435 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.0-gke.435 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Version alignment

For easier identification of the Kubernetes version for a given release, we are aligning Anthos clusters on bare metal version numbering with GKE version numbering. This change starts with this minor release, which is version 1.28. The version alignment is for major and minor versions only, patch versions are product specific. In addition to this version alignment, the Anthos clusters on bare metal release versions will follow the GKE semantic versioning scheme (x.y.z-gke.N), including the addition of a GKE patch version (-gke.N). Unlike GKE, however, the patch version (z) increments by 100.

Example version numbers for Anthos clusters on bare metal:

  • Minor release: 1.28.0-gke.435
  • Initial patch release: 1.28.100-gke.27
  • Second patch release: 1.28.200-gke.19

This change affects numbering only. Upgrades from 1.16 to 1.28 follow the same process as upgrades between prior minor releases. However, downloads, upgrades, and cluster creation for 1.28 and higher versions require the fully qualified version number, including the GKE patch version.

Version 1.14 end of life: In accordance with the Anthos Version Support Policy, version 1.14 (all patch releases) of Anthos clusters on bare metal has reached its end of life and is no longer supported.

  • Preview: Added support for skews of up to two minor versions for selective node pool upgrades.

  • Preview: Added capability to pause and resume cluster upgrades.

  • GA: Added support for using custom cluster certificate authorities (CAs) to enable secure authentication and encryption between cluster components.

  • GA: Added support for using gkeConnect.location to specify regional membership for fleets.

  • GA: Added support for using controlPlane.apiServerCertExtraSANs to specify extra subject alternative name (SAN) entries for the Kubernetes API server certificate.

  • GA: Added support for enabling Direct Server Return (DSR) load balancing for clusters. In GA, DSR load balancing is enabled with the clusterNetwork.forwardMode field in the cluster configuration file.

  • GA: Added support for multiple BGP load balancer (BGPLoadBalancer) resources and BGP Community. Multiple BGP load balancer resources provide more flexibility to define which peers advertise specific load balancer nodes and Services. BGP Community support helps you to distinguish routes coming from BGP load balancers from other routes in your network.

  • Preview: Added GKE Identity Service v2 capability for an improved security flow when you authenticate with third-party identity solutions.

Functionality changes:

  • Configured the local volume provisioner DaemonSet to tolerate all taints.

  • Updated the SRIOV operator.

  • To improve logging system integration, updated audit logging to always write a local Kubernetes audit log file, even when Cloud Audit Logging is enabled.

  • Changed upgrade preflight checks behavior to skip kubeadm job creation check to improve upgrade reliability.

  • Updated Dataplane V2 to use Cilium v1.13.

  • Added preflight check for control planes running RHEL 9.2 or Ubuntu 22.04 to check the fs.inotify kernel settings.

  • Removed hardcoded timeout value for bmctl backup operation.

  • Updated certificate management to propagate private-registry-certs Secret changes to all machines.

  • Added support for SSH client certificates in bmctl backup and bmctl restore commands.

  • Added the optional userClaim field to the ClientConfig custom resource definition bundled with Anthos clusters on bare metal. This change improves support for Azure AD integrations with Anthos Identity Service.

  • Updated constraint on NodePool spec.upgradeStrategy.concurrentNodes to be the smaller of either 15 nodes or 50% of the size of the node pool.

Supported node pool versions:

If you use selective worker node pool upgrades to upgrade a cluster to version 1.28.0-gke.435, see Node pool versioning rules for a list of the versions that are supported for the worker node pools.

Fixes:

  • Fixed an issue where the node-problem-detector systemd service doesn't restart after the node reboots.

  • Fixed an issue where CoreDNS Pods can get stuck in an unready state.

  • Fixed an issue that caused application metrics to be unavailable in Anthos clusters on bare metal versions 1.16.0 and 1.16.1.

  • Fixed a memory leak in Dataplane V2.

  • Fixed an issue that caused file and directory permissions to be set incorrectly after backing up and restoring a cluster.

  • Added direct dependencies on systemd, containerd, and kubelet over their mount point folders in /var/lib/.

  • Fixed an issue that blocked upgrades to version 1.16 for clusters that have secure computing mode (seccomp) disabled.

  • Fixed an issue where etcd blocked upgrades due to an incorrect initial-cluster-state.

  • Fixed an issue that sometimes resulted in the upgrade process starting before either all pods have been drained or the draining period has elapsed.

  • Fixed an issue that resulted in the etcd-events memory request (resources.requests.memory) being set incorrectly.

The following container image security vulnerabilities have been fixed in version 1.28.0-gke-435:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.