Google Distributed Cloud (software only) for bare metal 1.28 release notes

This document lists production updates to Google Distributed Cloud (software only) for bare metal (formerly known as Google Distributed Cloud Virtual, previously known as Anthos clusters on bare metal). Check this page periodically for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

December 11, 2024

Release 1.28.1300-gke.59

Google Distributed Cloud for bare metal 1.28.1300-gke.59 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.28.1300-gke.59 runs on Kubernetes 1.28.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

The following container image security vulnerabilities have been fixed in 1.28.1300-gke.59:

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

November 07, 2024

Release 1.28.1200-gke.83

Google Distributed Cloud for bare metal 1.28.1200-gke.83 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.28.1200-gke.83 runs on Kubernetes 1.28.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Fixes:

  • Fixed an issue where the registry mirror reachability check fails for a single unreachable registry mirror. Now the reachability check applies to configured registry mirrors only, instead of all registry mirrors.

  • Fixed the issue where non-root users can't run bmctl restore to restore quorum.

The following container image security vulnerabilities have been fixed in 1.28.1200-gke.83:

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

October 17, 2024

Release 1.28.1100-gke.94

Google Distributed Cloud for bare metal 1.28.1100-gke.94 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.28.1100-gke.94 runs on Kubernetes 1.28.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Fixed an issue where the control plane VIP might become unavailable because Keepalived didn't check correctly that the VIP is on a node with a responsive HAProxy.

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

September 23, 2024

Release 1.28.1000-gke.60

Google Distributed Cloud for bare metal 1.28.1000-gke.60 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.28.1000-gke.60 runs on Kubernetes 1.28.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Fixes:

  • Fixed Cloud Audit Logging failure due to allowlisting issue with multiple project IDs.

The following container image security vulnerabilities have been fixed in 1.28.1000-gke.60:

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

August 29, 2024

Release 1.28.900-gke.112

Google Distributed Cloud for bare metal 1.28.900-gke.112 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.28.900-gke.112 runs on Kubernetes 1.28.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

August 01, 2024

Release 1.28.800-gke.111

Google Distributed Cloud for bare metal 1.28.800-gke.111 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.28.800-gke.111 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Functionality changes:

  • Updated Kubernetes audit logging to include request and response payloads from the Kubernetes API server for bare metal custom resources, including the following: Cluster, NodePool, BareMetalMachine and BareMetalCluster.

Fixes:

The following container image security vulnerabilities have been fixed in 1.28.800-gke.111:

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

July 09, 2024

Release 1.28.700-gke.150

Google Distributed Cloud for bare metal 1.28.700-gke.150 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.28.700-gke.150 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Fixes:

  • Fixed an issue where upgraded clusters didn't get label updates that match the labels applied for newly created clusters, for a given version.

The following container image security vulnerabilities have been fixed in 1.28.700-gke.150:

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

July 03, 2024

Security bulletin (all minor versions)

A remote code execution vulnerability, CVE-2024-6387, was recently discovered in OpenSSH. The vulnerability exploits a race condition that can be used to obtain access to a remote shell, enabling attackers to gain root access. At the time of publication, exploitation is believed to be difficult and take several hours per machine being attacked. We are not aware of any exploitation attempts. This vulnerability has a Critical severity.

For mitigation steps and more details, see the GCP-2024-040 security bulletin.

June 06, 2024

Release 1.28.600-gke.163

Google Distributed Cloud for bare metal 1.28.600-gke.163 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.28.600-gke.163 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the Ready storage partners page to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Functionality changes:

  • Updated preflight checks add a check for networking kernel modules (ip_tables or np_tables) and remove the iptables package check.

  • Added checks to validate the SSH client certificate file type before saving the certificate as a Secret.

  • Added support for Red Hat Enterprise Linux 8.10 for Google Distributed Cloud software version 1.28.600-gke.163 and higher.

  • Removed support for Red Hat Enterprise Linux 8.9 as it is beyond the Red Hat support window.

Fixed an issue where the kubelet doesn't honor shortened, 1-second grace period for pod deletion during eviction-based draining.

The following container image security vulnerabilities have been fixed in 1.28.600-gke.163:

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

May 28, 2024

Security bulletin (all minor versions)

A new vulnerability (CVE-2024-4323) has been discovered in Fluent Bit that could result in remote code execution. Fluent Bit versions 2.0.7 through 3.0.3 are affected.

Google Distributed Cloud software doesn't use a vulnerable version of Fluent Bit and is unaffected.

For more information, see the GCP-2024-031 security bulletin.

May 02, 2024

Release 1.28.500-gke.120

GKE on Bare Metal 1.28.500-gke.120 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.500-gke.120 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

April 08, 2024

Release 1.28.400-gke.77

GKE on Bare Metal 1.28.400-gke.77 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.400-gke.77 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Functionality changes:

  • Updated preflight checks to add a check for networking kernel modules.
  • Updated preflight checks to remove the check for iptables package availability.

Fixes:

  • Fixed a cluster upgrade issue where the lifecycle-controller-deployer Pod was unable to migrate existing GKE on Bare Metal resources to the latest API version. This issue blocked upgrades to earlier version 1.28 releases.

Fixes:

The following container image security vulnerabilities have been fixed in 1.28.400-gke.77:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

April 03, 2024

A Denial-of-Service (DoS) vulnerability (CVE-2023-45288) was recently discovered in multiple implementations of the HTTP/2 protocol, including the golang HTTP server used by Kubernetes. The vulnerability could lead to a DoS of the Google Kubernetes Engine (GKE) control plane. For more information, see the GCP-2024-022 security bulletin.

March 21, 2024

Release 1.28.300-gke.131

GKE on Bare Metal 1.28.300-gke.131 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.300-gke.131 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Functionality changes:

  • Updated preflight checks to add a check for networking kernel modules.

  • Updated preflight checks to remove the check for iptables package availability.

  • Increased the default memory limit for node-exporter.

Fixes:

  • Fixed an issue with configuring a proxy for your cluster that required you to manually set HTTPS_PROXY and NO_PROXY environment variables on the admin workstation.

The following container image security vulnerabilities have been fixed in 1.28.300-gke.131:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

March 04, 2024

Release 1.28.200-gke.118

GKE on Bare Metal 1.28.200-gke.118 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.200-gke.118 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Fixes:

  • Fixed an issue where upgrades are blocked because cluster-operator can't delete stale, failing preflight check resources.

Fixes:

The following container image security vulnerabilities have been fixed in 1.28.200-gke.118:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

January 31, 2024

Release 1.28.100-gke.146

GKE on Bare Metal 1.28.100-gke.146 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.100-gke.146 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

Fixes:

Fixed a rootless permission issue on file /var/lib/audit.log in 1.28.100, which might block control plane node upgrades.

The following container image security vulnerabilities have been fixed in 1.28.100-gke.146:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

December 15, 2023

Release 1.28.0-gke.435

GKE on Bare Metal 1.28.0-gke.435 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.0-gke.435 runs on Kubernetes 1.28.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Version alignment

For easier identification of the Kubernetes version for a given release, we are aligning Anthos clusters on bare metal version numbering with GKE version numbering. This change starts with this minor release, which is version 1.28. The version alignment is for major and minor versions only, patch versions are product specific. In addition to this version alignment, the Anthos clusters on bare metal release versions will follow the GKE semantic versioning scheme (x.y.z-gke.N), including the addition of a GKE patch version (-gke.N). Unlike GKE, however, the patch version (z) increments by 100.

Example version numbers for Anthos clusters on bare metal:

  • Minor release: 1.28.0-gke.435
  • Initial patch release: 1.28.100-gke.27
  • Second patch release: 1.28.200-gke.19

This change affects numbering only. Upgrades from 1.16 to 1.28 follow the same process as upgrades between prior minor releases. However, downloads, upgrades, and cluster creation for 1.28 and higher versions require the fully qualified version number, including the GKE patch version.

Version 1.14 end of life: In accordance with the Anthos Version Support Policy, version 1.14 (all patch releases) of Anthos clusters on bare metal has reached its end of life and is no longer supported.

  • Preview: Added support for Red Hat Enterprise Linux (RHEL) version 9.2. For more information, see Select your operating system.

  • Preview: Added support for skews of up to two minor versions for selective node pool upgrades.

  • Preview: Added capability to pause and resume cluster upgrades.

  • GA: Added support for using custom cluster certificate authorities (CAs) to enable secure authentication and encryption between cluster components.

  • GA: Added support for using gkeConnect.location to specify regional membership for fleets.

  • GA: Added support for using controlPlane.apiServerCertExtraSANs to specify extra subject alternative name (SAN) entries for the Kubernetes API server certificate.

  • GA: Added support for enabling Direct Server Return (DSR) load balancing for clusters. In GA, DSR load balancing is enabled with the clusterNetwork.forwardMode field in the cluster configuration file.

  • GA: Added support for multiple BGP load balancer (BGPLoadBalancer) resources and BGP Community. Multiple BGP load balancer resources provide more flexibility to define which peers advertise specific load balancer nodes and Services. BGP Community support helps you to distinguish routes coming from BGP load balancers from other routes in your network.

  • Preview: Added GKE Identity Service v2 capability for an improved security flow when you authenticate with third-party identity solutions.

Functionality changes:

  • Configured the local volume provisioner DaemonSet to tolerate all taints.

  • Updated the SRIOV operator.

  • To improve logging system integration, updated audit logging to always write a local Kubernetes audit log file, even when Cloud Audit Logging is enabled.

  • Changed upgrade preflight checks behavior to skip kubeadm job creation check to improve upgrade reliability.

  • Updated Dataplane V2 to use Cilium v1.13.

  • Added preflight check for control planes running RHEL 9.2 or Ubuntu 22.04 to check the fs.inotify kernel settings.

  • Removed hardcoded timeout value for bmctl backup operation.

  • Updated certificate management to propagate private-registry-certs Secret changes to all machines.

  • Added support for SSH client certificates in bmctl backup and bmctl restore commands.

  • Added the optional userClaim field to the ClientConfig custom resource definition bundled with Anthos clusters on bare metal. This change improves support for Azure AD integrations with Anthos Identity Service.

  • Updated constraint on NodePool spec.upgradeStrategy.concurrentNodes to be the smaller of either 15 nodes or 50% of the size of the node pool.

Supported node pool versions:

If you use selective worker node pool upgrades to upgrade a cluster to version 1.28.0-gke.435, see Node pool versioning rules for a list of the versions that are supported for the worker node pools.

Fixes:

  • Fixed an issue where the node-problem-detector systemd service doesn't restart after the node reboots.

  • Fixed an issue where CoreDNS Pods can get stuck in an unready state.

  • Fixed an issue that caused application metrics to be unavailable in Anthos clusters on bare metal versions 1.16.0 and 1.16.1.

  • Fixed a memory leak in Dataplane V2.

  • Fixed an issue that caused file and directory permissions to be set incorrectly after backing up and restoring a cluster.

  • Added direct dependencies on systemd, containerd, and kubelet over their mount point folders in /var/lib/.

  • Fixed an issue that blocked upgrades to version 1.16 for clusters that have secure computing mode (seccomp) disabled.

  • Fixed an issue where etcd blocked upgrades due to an incorrect initial-cluster-state.

  • Fixed an issue that sometimes resulted in the upgrade process starting before either all pods have been drained or the draining period has elapsed.

  • Fixed an issue that resulted in the etcd-events memory request (resources.requests.memory) being set incorrectly.

The following container image security vulnerabilities have been fixed in version 1.28.0-gke-435:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.