Distributed Cloud 1.15 release notes

This document lists production updates to Distributed Cloud. We recommend that Google Distributed Cloud developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

April 03, 2024

A Denial-of-Service (DoS) vulnerability (CVE-2023-45288) was recently discovered in multiple implementations of the HTTP/2 protocol, including the golang HTTP server used by Kubernetes. The vulnerability could lead to a DoS of the Google Kubernetes Engine (GKE) control plane. For more information, see the GCP-2024-022 security bulletin.

March 21, 2024

Release 1.15.11

GKE on Bare Metal 1.15.11 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.11 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

The following container image security vulnerabilities have been fixed in 1.15.11:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

March 04, 2024

Release 1.15.10

GKE on Bare Metal 1.15.10 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.10 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

GKE on Bare Metal version 1.15.10 and later has been qualified on and supports Red Hat Enterprise Linux (RHEL) version 8.9.

Fixes:

The following container image security vulnerabilities have been fixed in 1.15.10:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

February 01, 2024

Release 1.15.9

GKE on Bare Metal 1.15.9 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.9 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

January 31, 2024

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

December 13, 2023

Release 1.15.8

GKE on Bare Metal 1.15.8 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.8 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Functionality changes:

  • Changed upgrade preflight checks behavior to skip kubeadm job creation check to improve upgrade reliability.

Fixes:

  • Fixed an issue where the network check ConfigMap wasn't being updated when nodes were added or removed.

Fixes:

The following container image security vulnerabilities have been fixed in 1.15.8:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

November 20, 2023

Release 1.15.7

Anthos clusters on bare metal 1.15.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.7 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Fixed an issue where CoreDNS Pods can get stuck in an unready state.

The following container image security vulnerabilities have been fixed in 1.15.7:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

October 30, 2023

Release 1.15.6

GKE on Bare Metal 1.15.6 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.6 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Functionality changes:

  • Removed hardcoded timeout value for the bmctl backup operation.

Fixes:

  • Fixed a memory leak in Dataplane V2.

  • Added direct dependencies on systemd, containerd, and kubelet over their mount point folders in /var/lib/.

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

September 25, 2023

Release 1.15.5

Anthos clusters on bare metal 1.15.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.5 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Fixed an issue to prevent cluster upgrades from starting on a node before either all Pods have been drained or the Pod draining timeout has been reached.

The following container image security vulnerabilities have been fixed in 1.15.5:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

August 16, 2023

Release 1.15.4

Anthos clusters on bare metal 1.15.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.4 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Functionality changes:

  • Audit logs are compressed on the wire for Cloud Audit Logs consumption, reducing egress bandwidth by approximately 60%.

  • Upgraded local volume provisioner to v2.5.0.

  • Upgraded snapshot controller to v5.0.1.

  • Deprecated v1beta1 volume snapshot custom resources. Anthos clusters on bare metal will stop serving v1beta1 resources in a future release.

Fixes:

  • Fixed an issue for clusters configured with manual load balancing where CA rotation reported that there were no (0) control plane nodes.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

July 20, 2023

Release 1.15.3

Anthos clusters on bare metal 1.15.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.3 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Anthos clusters on bare metal 1.15.3 supports adding the gkeOnPremAPI section to your admin and user cluster configuration files to enroll the clusters in the Anthos On-Prem API. Enrolling the clusters in the Anthos On-Prem API lets you upgrade admin and user clusters using the Google Cloud console or the Google Cloud CLI.

Fixes:

  • Fixed an issue where the apiserver could become responsive during a cluster upgrade for clusters with a single control plane node.

  • Fixed an issue where cluster installations or upgrades fail when the cluster name has more than 45 characters.

  • Fixed an issue where node-specific labels set on the node pool were sometimes overwritten.

  • Fixed an issue where audit logs were duplicated into the offline buffer even when they are sent to Cloud Audit Logs successfully.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

June 27, 2023

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

June 22, 2023

Release 1.15.2

Anthos clusters on bare metal 1.15.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.2 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Functionality changes:

  • Added preflight check to make sure control plane and load balancer nodes aren't in maintenance mode before an upgrade.

  • Upgraded etcd version to v3.4.26-0-gke.0.

Fixes:

  • Fixed an issue where containerd didn't restart when there was a version mismatch. This issue caused an inconsistent containerd version within the cluster.

  • Fixed an issue where the spec.proxy.noProxy value wasn't used in the Google Cloud connectivity preflight check (bmctl check gcp).

  • Fixed an issue that caused the logging agent to use continuously increasing amounts of memory. The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

June 16, 2023

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

May 31, 2023

Release 1.15.1

Anthos clusters on bare metal 1.15.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.1 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Functionality changes:

  • Updated the cluster snapshot capability so that information can be captured for the target cluster even when the cluster custom resource is missing or unavailable.

  • Improved bmctl error reporting for failures during the creation of a bootstrap cluster.

  • Added support for using the baremetal.cluster.gke.io/maintenance-mode-deadline-seconds cluster annotation to specify the maximum node draining duration, in seconds. By default, a 20-minute (1200 seconds) timeout is enforced. When the timeout elapses, all pods are stopped and the node is put into maintenance mode. For example to change the timeout to 10 minutes, add the annotation baremetal.cluster.gke.io/maintenance-mode-deadline-seconds: "600" to your cluster.

  • Added node_pool_name to the anthos_baremetal_node_os_count metric.

Fixes:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 10, 2023

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

April 27, 2023

Release 1.15.0

Anthos clusters on bare metal 1.15.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.0 runs on Kubernetes 1.26.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Version 1.12 end of life: In accordance with the Anthos Version Support Policy, version 1.12 (all patch releases) of Anthos clusters on bare metal has reached its end of life and is no longer supported.

Cluster lifecycle:

  • Upgraded from Kubernetes version 1.25 to version 1.26.
  • GA: Set in-place upgrade (without bootstrap cluster) as the default upgrade method for self-managed clusters.
  • GA: Added support for configuring worker node pools for parallel node upgrades to significantly reduce upgrade times. Added a minimumAvailableNodes field to specify a minimum number of nodes to keep available for workloads throughout the upgrade.
  • Preview: Added support for parallel upgrades of worker node pools.
  • Added support for Red Hat Enterprise Linux (RHEL) version 8.7.
  • Added support for Ubuntu 22.04 LTS.
  • GA: Added support for increasing the number of IP addresses for Services after cluster creation. For more information, see Increase service network range.
  • Preview: Added ability to configure kubelet image pull settings for node pools. For more information, see Configure kubelet image pull settings.
  • Streamlined the snapshot uploading and sharing process.
  • GA: Added support of Control group v2 (cgroup v2).
  • Preview: Added a separate instance of etcd for the etcd-events object.
  • Updated cert-manager to version 1.17.2.
  • Updated automated API enablement when you run bmctl create config with the --enable-apis flag. The following APIs are added to the enablement list:
    • Enable storage.googleapis.com as a required API.
    • Enable gkeonprem.googleapis.com as a recommended API.
  • Added a new field status.failures to the NodePool custom resource to aggregate failures across machines in the NodePool.
  • Added a new condition type PreflightCheckSuccessful to the NodePool custom resource. This condition type summarizes the preflight check status across machines in the NodePool.

Networking:

  • Added support for ClusterDNS to specify order for upstreamNameServers with an orderPolicy. Allowed values for orderPolicy are random, round_robin, or sequential. The default value is random.

Observability:

  • Added support for filtering application logs. This feature can reduce application logging billing and network traffic from the cluster to Cloud Logging. For more information, see Filter application logs.

  • GA: Fully managed Cloud Monitoring Integration dashboards:

    • In the next Anthos release (version 1.16), the following dashboards in Cloud Monitoring Sample Library are unavailable:
      • Anthos cluster control plane uptime
      • Anthos cluster node status
      • Anthos cluster pod status
      • Anthos utilization metering
      • GKE on-prem node status
      • GKE on-prem control plane uptime
      • GKE on-prem pod status
      • GKE on-prem vSphere vm health status
    • In the next Anthos release (version 1.16), the following customized dashboards aren't created when you create a new cluster:
      • Anthos cluster control plane uptime
      • Anthos cluster pod status
      • Anthos cluster node status
      • Anthos cluster VM status
    • An added Anthos integration page is available from the Cloud Monitoring Integration page. The Anthos integration includes descriptions and previews for the predefined Anthos dashboards:
      • Anthos Cluster Control Plane Uptime
      • Anthos Cluster Node Status
      • Anthos Cluster Pod Status
      • Anthos Cluster KubeVirt VM Status
      • Anthos Cluster Utilization Metering

    For more information, see Use predefined dashboards.

  • Preview: Added support for system metrics when you use Google Cloud Managed Service for Prometheus.

Security and Identity:

  • Preview: Added support for Binary Authorization, a service on Google Cloud that provides software supply-chain security for container-based applications. For more information, see Binary Authorization for Anthos clusters overview.
  • Preview: Added support for VPC Service Controls, which provides additional security for your clusters to help mitigate the risk of data exfiltration.
  • Improved security by disabling port 10255, the kubelet read-only port, by default. For more information, see Disable kubelet read-only port in Hardening your cluster's security.

Functionality changes:

  • Replacing taints and labels. Clusters created and upgraded to Anthos clusters on bare metal version 1.15.0 and higher have node-role.kubernetes.io/control-plane:* taints and node-role.kubernetes.io/control-plane labels. These new taints and labels replace the node-role.kubernetes.io/master label and node-role.kubernetes.io/master:* taints on new and upgraded control plane nodes.

Networking changes:

  • Replaced the anetd CNI plugin for the bootstrap cluster with kindnet.
  • Increased eBPF map limit to 512 K to allow for more load balancer Services.
  • Upgraded CoreDNS to version 1.9.4.

Anthos VM Runtime:

  • Moved the Anthos VM Runtime release notes to a separate page in the Anthos VM Runtime documentation section.

Fixes:

  • Fixed an issue that caused the bmctl reset nodes command to fail if the bmctl-workspace directory was empty.
  • Fixed an intermittent issue that caused the bmctl upgrade cluster command to indicate that the operation was complete before the cluster was in a ready state.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.