Combined release notes (all minor versions)

This document lists production updates to GKE on Bare Metal. We recommend that GKE on Bare Metal developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/anthos-bare-metal-release-notes.xml

February 20, 2024

1.16

Release 1.16.6

GKE on Bare Metal 1.16.6 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.16.6 runs on Kubernetes 1.27.

Fixes:

  • Fixed an issue where upgrades are blocked because cluster-operator can't delete stale, failing preflight check resources.

  • Cleaned up stale etcd-events membership to enhance control plane initialization reliability in the event of a node join failure.

Fixes:

The following container image security vulnerabilities have been fixed in 1.16.6:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

February 01, 2024

1.15

Release 1.15.9

GKE on Bare Metal 1.15.9 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.9 runs on Kubernetes 1.26.

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

January 31, 2024

1.28

Release 1.28.100-gke.146

GKE on Bare Metal 1.28.100-gke.146 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.100-gke.146 runs on Kubernetes 1.28.

Fixes:

Fixed a rootless permission issue on file /var/lib/audit.log in 1.28.100, which might block control plane node upgrades.

The following container image security vulnerabilities have been fixed in 1.28.100-gke.146:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

1.6 & 1.7 & 1.8 & 1.9 & 1.10 & 1.11 & 1.12 & 1.13 & 1.14 & 1.15 & 1.16 & 1.28

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

January 30, 2024

1.16

Release 1.16.5

GKE on Bare Metal 1.16.5 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.16.5 runs on Kubernetes 1.27.

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

December 15, 2023

1.28

Release 1.28.0-gke.435

GKE on Bare Metal 1.28.0-gke.435 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.28.0-gke.435 runs on Kubernetes 1.28.

Version alignment

For easier identification of the Kubernetes version for a given release, we are aligning Anthos clusters on bare metal version numbering with GKE version numbering. This change starts with this minor release, which is version 1.28. The version alignment is for major and minor versions only, patch versions are product specific. In addition to this version alignment, the Anthos clusters on bare metal release versions will follow the GKE semantic versioning scheme (x.y.z-gke.N), including the addition of a GKE patch version (-gke.N). Unlike GKE, however, the patch version (z) increments by 100.

Example version numbers for Anthos clusters on bare metal:

  • Minor release: 1.28.0-gke.435
  • Initial patch release: 1.28.100-gke.27
  • Second patch release: 1.28.200-gke.19

This change affects numbering only. Upgrades from 1.16 to 1.28 follow the same process as upgrades between prior minor releases. However, downloads, upgrades, and cluster creation for 1.28 and higher versions require the fully qualified version number, including the GKE patch version.

Version 1.14 end of life: In accordance with the Anthos Version Support Policy, version 1.14 (all patch releases) of Anthos clusters on bare metal has reached its end of life and is no longer supported.

  • Preview: Added support for skews of up to two minor versions for selective node pool upgrades.

  • Preview: Added capability to pause and resume cluster upgrades.

  • GA: Added support for using custom cluster certificate authorities (CAs) to enable secure authentication and encryption between cluster components.

  • GA: Added support for using gkeConnect.location to specify regional membership for fleets.

  • GA: Added support for using controlPlane.apiServerCertExtraSANs to specify extra subject alternative name (SAN) entries for the Kubernetes API server certificate.

  • GA: Added support for enabling Direct Server Return (DSR) load balancing for clusters. In GA, DSR load balancing is enabled with the clusterNetwork.forwardMode field in the cluster configuration file.

  • GA: Added support for multiple BGP load balancer (BGPLoadBalancer) resources and BGP Community. Multiple BGP load balancer resources provide more flexibility to define which peers advertise specific load balancer nodes and Services. BGP Community support helps you to distinguish routes coming from BGP load balancers from other routes in your network.

  • Preview: Added GKE Identity Service v2 capability for an improved security flow when you authenticate with third-party identity solutions.

Functionality changes:

  • Configured the local volume provisioner DaemonSet to tolerate all taints.

  • Updated the SRIOV operator.

  • To improve logging system integration, updated audit logging to always write a local Kubernetes audit log file, even when Cloud Audit Logging is enabled.

  • Changed upgrade preflight checks behavior to skip kubeadm job creation check to improve upgrade reliability.

  • Updated Dataplane V2 to use Cilium v1.13.

  • Added preflight check for control planes running RHEL 9.2 or Ubuntu 22.04 to check the fs.inotify kernel settings.

  • Removed hardcoded timeout value for bmctl backup operation.

  • Updated certificate management to propagate private-registry-certs Secret changes to all machines.

  • Added support for SSH client certificates in bmctl backup and bmctl restore commands.

  • Added the optional userClaim field to the ClientConfig custom resource definition bundled with Anthos clusters on bare metal. This change improves support for Azure AD integrations with Anthos Identity Service.

  • Updated constraint on NodePool spec.upgradeStrategy.concurrentNodes to be the smaller of either 15 nodes or 50% of the size of the node pool.

Supported node pool versions:

If you use selective worker node pool upgrades to upgrade a cluster to version 1.28.0-gke.435, see Node pool versioning rules for a list of the versions that are supported for the worker node pools.

Fixes:

  • Fixed an issue where the node-problem-detector systemd service doesn't restart after the node reboots.

  • Fixed an issue where CoreDNS Pods can get stuck in an unready state.

  • Fixed an issue that caused application metrics to be unavailable in Anthos clusters on bare metal versions 1.16.0 and 1.16.1.

  • Fixed a memory leak in Dataplane V2.

  • Fixed an issue that caused file and directory permissions to be set incorrectly after backing up and restoring a cluster.

  • Added direct dependencies on systemd, containerd, and kubelet over their mount point folders in /var/lib/.

  • Fixed an issue that blocked upgrades to version 1.16 for clusters that have secure computing mode (seccomp) disabled.

  • Fixed an issue where etcd blocked upgrades due to an incorrect initial-cluster-state.

  • Fixed an issue that sometimes resulted in the upgrade process starting before either all pods have been drained or the draining period has elapsed.

  • Fixed an issue that resulted in the etcd-events memory request (resources.requests.memory) being set incorrectly.

The following container image security vulnerabilities have been fixed in version 1.28.0-gke-435:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

1.16

Release 1.16.4

GKE on Bare Metal 1.16.4 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.16.4 runs on Kubernetes 1.27.

Functionality changes:

  • Changed upgrade preflight checks behavior to skip kubeadm job creation check to improve upgrade reliability.

Supported node pool versions:

If you use selective worker node pool upgrades to upgrade a cluster to version 1.16.4, see Node pool versioning rules for a list of the versions that are supported for the worker node pools.

Fixes:

  • Fixed an issue where the network check ConfigMap wasn't being updated when nodes were added or removed.

  • Fixed an issue where excessive stackdriver-operator reconciliations resulted in high CPU usage.

Fixes:

The following container image security vulnerabilities have been fixed in 1.16.4:

Known issues:

For information about the latest known issues, see GKE on Bare Metal issues in the Troubleshooting section.

December 13, 2023

1.15

Release 1.15.8

GKE on Bare Metal 1.15.8 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.8 runs on Kubernetes 1.26.

Functionality changes:

  • Changed upgrade preflight checks behavior to skip kubeadm job creation check to improve upgrade reliability.

Fixes:

  • Fixed an issue where the network check ConfigMap wasn't being updated when nodes were added or removed.

Fixes:

The following container image security vulnerabilities have been fixed in 1.15.8:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

November 28, 2023

1.16

Release 1.16.3

GKE on Bare Metal 1.16.3 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.16.3 runs on Kubernetes 1.27.

Functionality changes:

  • Increased the certificate time to live (TTL) for metrics-providers-ca and stackdriver-prometheus-scrape for third-party monitoring.

Supported node pool versions:

If you use selective worker node pool upgrades to upgrade a cluster to version 1.16.4, see Node pool versioning rules for a list of the versions that are supported for the worker node pools.

Fixes:

  • Fixed an issue where CoreDNS Pods can get stuck in an unready state.

  • Fixed an issue that caused application metrics to be unavailable in Anthos clusters on bare metal versions 1.16.0 and 1.16.1.

Fixes:

The following container image security vulnerabilities have been fixed in 1.16.3:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

November 21, 2023

1.14

Release 1.14.11

Anthos clusters on bare metal 1.14.11 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.11 runs on Kubernetes 1.25.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

November 20, 2023

1.15

Release 1.15.7

Anthos clusters on bare metal 1.15.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.7 runs on Kubernetes 1.26.

Fixed an issue where CoreDNS Pods can get stuck in an unready state.

The following container image security vulnerabilities have been fixed in 1.15.7:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

November 06, 2023

1.14

Release 1.14.10

Anthos clusters on bare metal 1.14.10 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.10 runs on Kubernetes 1.25.

Functionality changes:

  • Added NODEPOOL-NAME, NODEPOOL-NAMESPACE, and STATUS columns for the InventoryMachine resource to improve troubleshooting.

  • Removed hardcoded timeout value for the bmctl backup operation.

Fixes:

  • Fixed an issue where CoreDNS Pods can get stuck in an unready state.

  • Fixed a memory leak in Dataplane V2.

Fixes:

The following container image security vulnerabilities have been fixed in version 1.14.10:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

October 30, 2023

1.16

Release 1.16.2

Anthos clusters on bare metal 1.16.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.16.2 runs on Kubernetes 1.27.

Functionality changes:

  • Increased the certificate time to live (TTL) for metrics-providers-ca and stackdriver-prometheus-scrape for third-party monitoring.

  • Removed hardcoded timeout value for the bmctl backup operation.

Supported node pool versions:

If you use selective worker node pool upgrades to upgrade a cluster to version 1.16.4, see Node pool versioning rules for a list of the versions that are supported for the worker node pools.

Fixes:

  • Fixed the spec.featureGates.annotationBasedApplicationMetrics feature gate in the stackdriver custom resource to enable collection of annotation-based workload metrics. This function is broken in Anthos clusters on bare metal versions 1.16.0 and 1.16.1.

  • Fixed a memory leak in Dataplane V2.

  • Fixed an issue where garbage collection deleted Source Network Address Translation (SNAT) entries for long-lived egress NAT connections, causing connection resets.

  • Fixed an issue that caused file and directory permissions to be set incorrectly after backing up and restoring a cluster.

  • Added direct dependencies on systemd, containerd, and kubelet over their mount point folders in /var/lib/.

  • Fixed an issue where etcd blocked upgrades due to an incorrect initial-cluster-state.

  • Fixed an issue that blocked upgrades to version 1.16 for clusters that have secure computing mode (seccomp) disabled.

The following container image security vulnerabilities have been fixed in release 1.16.2:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

1.15

Release 1.15.6

GKE on Bare Metal 1.15.6 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.6 runs on Kubernetes 1.26.

Functionality changes:

  • Removed hardcoded timeout value for the bmctl backup operation.

Fixes:

  • Fixed a memory leak in Dataplane V2.

  • Added direct dependencies on systemd, containerd, and kubelet over their mount point folders in /var/lib/.

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

September 29, 2023

1.14

Release 1.14.9

Anthos clusters on bare metal 1.14.9 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.9 runs on Kubernetes 1.25.

Fixes:

Fixed an issue to prevent cluster upgrades from starting on a node before either all Pods have been drained or the Pod draining timeout has been reached.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

September 25, 2023

1.15

Release 1.15.5

Anthos clusters on bare metal 1.15.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.5 runs on Kubernetes 1.26.

Fixed an issue to prevent cluster upgrades from starting on a node before either all Pods have been drained or the Pod draining timeout has been reached.

The following container image security vulnerabilities have been fixed in 1.15.5:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

September 21, 2023

1.16

Release 1.16.1

Anthos clusters on bare metal 1.16.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.16 runs on Kubernetes 1.27.

Supported node pool versions:

If you use selective worker node pool upgrades to upgrade a cluster to version 1.16.4, see Node pool versioning rules for a list of the versions that are supported for the worker node pools.

Functionality changes:

  • Added the optional userClaim field to the ClientConfig custom resource definition bundled with Anthos clusters on bare metal. This change improves support for Azure AD integrations with Anthos Identity Service.

  • Updated constraint on NodePool spec.upgradeStrategy.concurrentNodes to be the smaller of either 15 nodes or 50% of the size of the node pool.

Fixes:

  • Fixed an issue where etcd blocked upgrades due to an incorrect initial-cluster-state.

  • Fixed an issue that blocked upgrades to version 1.16 for clusters that have secure computing mode (seccomp) disabled.

  • Fixed an issue to prevent cluster upgrades from starting on a node before either all Pods have been drained or the Pod draining timeout has been reached.

  • Fixed an issue where the memory resource requests value wasn't set properly for etcd-events.

Fixes:

The following container image security vulnerabilities have been fixed in 1.16.1:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

August 25, 2023

1.16

Release 1.16.0

Anthos clusters on bare metal 1.16.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.16.0 runs on Kubernetes 1.27.

Version 1.13 end of life: In accordance with the Anthos Version Support Policy, version 1.13 (all patch releases) of Anthos clusters on bare metal has reached its end of life and is no longer supported.

Red Hat Enterprise Linux (RHEL) 8 minor versions 8.2, 8.3, 8.4, and 8.5 have reached their end of life. Please ensure you're using a supported version of your operating system.

Cluster lifecycle:

  • Upgraded to Kubernetes version 1.27.4.

  • Added support for Red Hat Enterprise Linux (RHEL) version 8.8.

  • GA: Added support for parallel upgrades of worker node pools.

  • GA: Added support to upgrade specific worker node pools separately from the rest of the cluster.

  • GA: Added a separate instance of etcd for the etcd-events object. This new etcd instance is always on and requires ports 2382 and 2383 to be open on control plane nodes for inbound TCP traffic. If these ports aren't opened, cluster creation and cluster upgrades are blocked.

  • GA: Updated preflight checks for cluster installation and upgrades to use changes from the latest Anthos clusters on bare metal patch version to address known issues and provide more useful checks.

  • GA: Support enrolling admin and user clusters in the Anthos On-Prem API automatically to enable cluster lifecycle management from the Google Cloud CLI, the Google Cloud console, and Terraform when the Anthos On-Prem API is enabled. If needed, you have the option to disable enrollment. For more information, see the description for the gkeOnPremAPI field in the cluster configuration file.

  • GA: Added ability to configure kubelet image pull settings for node pools. For more information, see Configure kubelet image pull settings.

  • Added new health check to detect any unsupported drift in the custom resources managed by Anthos clusters on bare metal. Unsupported resource changes can lead to cluster problems.

  • Added a new flag, --target-cluster-name, that is supported by the bmctl register bootstrap command.

Networking:

  • GA: Added support for Services of type LoadBalancer to use externalTrafficPolicy=Local with bundled load balancing with BGP.

  • Preview: Added support for enabling Direct Server Return (DSR) load balancing for clusters configured with flat-mode networking. DSR load balancing is enabled with an annotation, preview.baremetal.cluster.gke.io/dpv2-lbmode-dsr: enable.

  • Preview: Upgraded wherabouts to v0.6.1-gke.1 to support dual-stack networking.

  • Added support for multiple BGP load balancer (BGPLoadBalancer) resources and BGP Community. Multiple BGP load balancer resources provide more flexibility to define which peers advertise specific load balancer nodes and Services. BGP Community support helps you to distinguish routes coming from BGP load balancers from other routes in your network.

Observability:

Security and Identity:

  • GA: Added support for Binary Authorization, a service on Google Cloud that provides software supply-chain security for container-based applications. For more information, see Set up Binary Authorization policy enforcement.

  • GA: Added support for VPC Service Controls, which provides additional security for your clusters to help mitigate the risk of data exfiltration.

  • Preview: Added support for using custom cluster certificate authorities (CAs) to enable secure authentication and encryption between cluster components.

  • Preview: Added support for configuring the Subject Alternative Names (SANs) of the kubeadm generated certificate for the kube-apiserver.

  • Added support to run keepalived as a non-root user.

Supported node pool versions:

If you use selective worker node pool upgrades to upgrade a cluster to version 1.16.4, see Node pool versioning rules for a list of the versions that are supported for the worker node pools.

Functionality changes:

  • Updated constraint on NodePool spec.upgradeStrategy.concurrentNodes to be the smaller of 15 nodes or 50% of the size of the node pool.

  • Replaced legacy method of enabling application logging in the cluster configuration file with two fields, enableCloudLoggingForApplications and enableGMPForApplications, in the stackdriver custom resource.

    The spec.clusterOperations.enableApplication field in the cluster configuration file has no effect on version 1.16.0 and higher clusters. This field populated the enableStackdriverForApplications field in the stackdriver custom resource, which enabled annotation based workload metric collection. If you need this capability, use the annotationBasedApplicationMetrics feature gate in the stackdriver custom resource as shown in the following sample to keep the same behavior:

    kind:stackdriver
    spec:
      enableCloudLoggingForApplications: true
      featureGates:
         annotationBasedApplicationMetrics: true
    
  • Added optional ksmNodePodMetricsOnly feature gate in the stackdriver custom resource to reduce the number of metrics from kube-state-metrics. Reducing the number of metrics makes monitoring pipeline more stable in large scale clusters.

  • Audit logs are compressed on the wire for Cloud Audit Logs consumption, reducing egress bandwidth by approximately 60%.

  • Upgraded local volume provisioner to v2.5.0.

  • Upgraded snapshot controller to v5.0.1.

  • Deprecated v1beta1 volume snapshot custom resources. Anthos clusters on bare metal will stop serving v1beta1 resources in a future release.

  • Removed resource request limits on edge profile workloads.

  • Added preflight check to make sure control plane and load balancer nodes aren't under maintenance before an upgrade.

  • Updated the cluster snapshot capability so that information can be captured for the target cluster even when the cluster custom resource is missing or unavailable.

  • Improved bmctl error reporting for failures during the creation of a bootstrap cluster.

  • Added support for using the baremetal.cluster.gke.io/maintenance-mode-deadline-seconds cluster annotation to specify the maximum node draining duration, in seconds. By default, a 20-minute (1200 seconds) timeout is enforced. When the timeout elapses, all pods are stopped and the node is put into maintenance mode. For example to change the timeout to 10 minutes, add the annotation baremetal.cluster.gke.io/maintenance-mode-deadline-seconds: "600" to your cluster.

  • Updated bmctl check cluster to create a HealthCheck custom resource in the admin cluster if it's healthy.

Fixes:

  • Fixed an issue where the apiserver could become unresponsive during a cluster upgrade for clusters with a single control plane node.

  • Fixed an issue where cluster installations or upgrades fail when the cluster name has more than 45 characters.

  • Fixed an issue where the control plane VIP wasn't reachable during cluster installation on Red Hat Enterprise Linux.

  • Fixed an issue where audit logs were duplicated into the offline buffer even when they are sent to Cloud Audit Logs successfully.

  • Fixed an issue where node-specific labels set on the node pool were sometimes overwritten.

  • Updated avoidBuggyIPs and manualAssign fields in load balancer address pools (spec.loadBalancers.addressPools) to allow changes at any time.

  • Fixed an issue where containerd didn't restart when there was a version mismatch. This issue caused an inconsistent containerd version within the cluster.

  • Fixed an issue that caused the logging agent to use continuously increasing amounts of memory.

  • Fixed preflight check so that it no longer ignores the no_proxy setting.

  • Fixed Anthos Identity Service annotation needed for exporting metrics.

  • Fixed an issue that caused the bmctl restore command to stop responding for clusters with manually configured load balancers.

  • Fixed an issue that prevented Anthos clusters on bare metal from restoring a high-availability quorum for nodes that use /var/lib/etcd as a mountpoint.

  • Fixed an issue that caused health checks to report failure when they find a Pod with a status of TaintToleration even when the replicaset for the Pod has sufficient Pods running.

  • Fixed an issue that caused conflicts with third-party Ansible automation.

  • Fixed a cluster upgrade issue that prevented some control plane nodes from rejoining a cluster configured for high availability.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

August 22, 2023

1.14

Release 1.14.8

Anthos clusters on bare metal 1.14.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.8 runs on Kubernetes 1.25.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

August 16, 2023

1.15

Release 1.15.4

Anthos clusters on bare metal 1.15.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.4 runs on Kubernetes 1.26.

Functionality changes:

  • Audit logs are compressed on the wire for Cloud Audit Logs consumption, reducing egress bandwidth by approximately 60%.

  • Upgraded local volume provisioner to v2.5.0.

  • Upgraded snapshot controller to v5.0.1.

  • Deprecated v1beta1 volume snapshot custom resources. Anthos clusters on bare metal will stop serving v1beta1 resources in a future release.

Fixes:

  • Fixed an issue for clusters configured with manual load balancing where CA rotation reported that there were no (0) control plane nodes.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

August 01, 2023

1.13

Release 1.13.10

Anthos clusters on bare metal 1.13.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.10 runs on Kubernetes 1.24.

Functionality changes:

  • Upgraded local volume provisioner to v2.5.0.

  • Upgraded snapshot controller to v5.0.1.

  • Deprecated v1beta1 volume snapshot custom resources. Anthos clusters on bare metal will stop serving v1beta1 resources in a future release.

Fixes:

  • Fixed an issue where the apiserver could become unresponsive during a cluster upgrade for clusters with a single control plane node.

  • Fixed an issue where audit logs were duplicated into the offline buffer even when they are sent to Cloud Audit Logs successfully.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

July 25, 2023

1.14

FEATURE

Release 1.14.7

Anthos clusters on bare metal 1.14.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.7 runs on Kubernetes 1.25.

Functionality changes:

  • Audit logs are compressed on the wire for Cloud Audit Logs consumption, reducing egress bandwidth by approximately 60%.

  • Upgraded local volume provisioner to v2.5.0.

  • Upgraded snapshot controller to v5.0.1.

  • Deprecated v1beta1 volume snapshot custom resources. Anthos clusters on bare metal will stop serving v1beta1 resources in a future release.

Fixes:

  • Fixed an issue where the smart default didn't work for gke-metrics-agent.

  • Fixed an issue where the apiserver could become responsive during a cluster upgrade for clusters with a single control plane node.

  • Fixed an issue where audit logs were duplicated into the offline buffer even when they are sent to Cloud Audit Logs successfully.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

July 20, 2023

1.15

Release 1.15.3

Anthos clusters on bare metal 1.15.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.3 runs on Kubernetes 1.26.

Anthos clusters on bare metal 1.15.3 supports adding the gkeOnPremAPI section to your admin and user cluster configuration files to enroll the clusters in the Anthos On-Prem API. Enrolling the clusters in the Anthos On-Prem API lets you upgrade admin and user clusters using the Google Cloud console or the Google Cloud CLI.

Fixes:

  • Fixed an issue where the apiserver could become responsive during a cluster upgrade for clusters with a single control plane node.

  • Fixed an issue where cluster installations or upgrades fail when the cluster name has more than 45 characters.

  • Fixed an issue where node-specific labels set on the node pool were sometimes overwritten.

  • Fixed an issue where audit logs were duplicated into the offline buffer even when they are sent to Cloud Audit Logs successfully.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

June 29, 2023

1.13

Release 1.13.9

Anthos clusters on bare metal 1.13.9 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.9 runs on Kubernetes 1.24.

ISSUE Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

June 27, 2023

1.6 & 1.7 & 1.8 & 1.9 & 1.10 & 1.11 & 1.12 & 1.13 & 1.14 & 1.15

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

June 23, 2023

1.14

Release 1.14.6

Anthos clusters on bare metal 1.14.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.6 runs on Kubernetes 1.25.

Functionality changes:

  • Upgraded etcd version to v3.4.26-0-gke.0.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

June 22, 2023

1.15

Release 1.15.2

Anthos clusters on bare metal 1.15.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.2 runs on Kubernetes 1.26.

Functionality changes:

  • Added preflight check to make sure control plane and load balancer nodes aren't in maintenance mode before an upgrade.

  • Upgraded etcd version to v3.4.26-0-gke.0.

Fixes:

  • Fixed an issue where containerd didn't restart when there was a version mismatch. This issue caused an inconsistent containerd version within the cluster.

  • Fixed an issue where the spec.proxy.noProxy value wasn't used in the Google Cloud connectivity preflight check (bmctl check gcp).

  • Fixed an issue that caused the logging agent to use continuously increasing amounts of memory. The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

June 16, 2023

1.6 & 1.7 & 1.8 & 1.9 & 1.10 & 1.11 & 1.12 & 1.13 & 1.14 & 1.15

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

June 01, 2023

1.13

Release 1.13.8

Anthos clusters on bare metal 1.13.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.8 runs on Kubernetes 1.24.

Fixes:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 31, 2023

1.15

Release 1.15.1

Anthos clusters on bare metal 1.15.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.1 runs on Kubernetes 1.26.

Functionality changes:

  • Updated the cluster snapshot capability so that information can be captured for the target cluster even when the cluster custom resource is missing or unavailable.

  • Improved bmctl error reporting for failures during the creation of a bootstrap cluster.

  • Added support for using the baremetal.cluster.gke.io/maintenance-mode-deadline-seconds cluster annotation to specify the maximum node draining duration, in seconds. By default, a 20-minute (1200 seconds) timeout is enforced. When the timeout elapses, all pods are stopped and the node is put into maintenance mode. For example to change the timeout to 10 minutes, add the annotation baremetal.cluster.gke.io/maintenance-mode-deadline-seconds: "600" to your cluster.

  • Added node_pool_name to the anthos_baremetal_node_os_count metric.

Fixes:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 24, 2023

1.14

Release 1.14.5

Anthos clusters on bare metal 1.14.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.5 runs on Kubernetes 1.25.

Fixes:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 10, 2023

1.6 & 1.7 & 1.8 & 1.9 & 1.10 & 1.11 & 1.12 & 1.13 & 1.14 & 1.15

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

April 27, 2023

1.15

Release 1.15.0

Anthos clusters on bare metal 1.15.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.0 runs on Kubernetes 1.26.

Version 1.12 end of life: In accordance with the Anthos Version Support Policy, version 1.12 (all patch releases) of Anthos clusters on bare metal has reached its end of life and is no longer supported.

Cluster lifecycle:

  • Upgraded from Kubernetes version 1.25 to version 1.26.
  • GA: Set in-place upgrade (without bootstrap cluster) as the default upgrade method for self-managed clusters.
  • GA: Added support for configuring worker node pools for parallel node upgrades to significantly reduce upgrade times. Added a minimumAvailableNodes field to specify a minimum number of nodes to keep available for workloads throughout the upgrade.
  • Preview: Added support for parallel upgrades of worker node pools.
  • Added support for Red Hat Enterprise Linux (RHEL) version 8.7.
  • Added support for Ubuntu 22.04 LTS.
  • GA: Added support for increasing the number of IP addresses for Services after cluster creation. For more information, see Increase service network range.
  • Preview: Added ability to configure kubelet image pull settings for node pools. For more information, see Configure kubelet image pull settings.
  • Streamlined the snapshot uploading and sharing process.
  • GA: Added support of Control group v2 (cgroup v2).
  • Preview: Added a separate instance of etcd for the etcd-events object.
  • Updated cert-manager to version 1.17.2.
  • Updated automated API enablement when you run bmctl create config with the --enable-apis flag. The following APIs are added to the enablement list:
    • Enable storage.googleapis.com as a required API.
    • Enable gkeonprem.googleapis.com as a recommended API.
  • Added a new field status.failures to the NodePool custom resource to aggregate failures across machines in the NodePool.
  • Added a new condition type PreflightCheckSuccessful to the NodePool custom resource. This condition type summarizes the preflight check status across machines in the NodePool.

Networking:

  • Added support for ClusterDNS to specify order for upstreamNameServers with an orderPolicy. Allowed values for orderPolicy are random, round_robin, or sequential. The default value is random.

Observability:

  • Added support for filtering application logs. This feature can reduce application logging billing and network traffic from the cluster to Cloud Logging. For more information, see Filter application logs.
  • GA: Fully managed Cloud Monitoring Integration dashboards:

    • In the next Anthos release (version 1.16), the following dashboards in Cloud Monitoring Sample Library are unavailable:
      • Anthos cluster control plane uptime
      • Anthos cluster node status
      • Anthos cluster pod status
      • Anthos utilization metering
      • GKE on-prem node status
      • GKE on-prem control plane uptime
      • GKE on-prem pod status
      • GKE on-prem vSphere vm health status
    • In the next Anthos release (version 1.16), the following customized dashboards aren't created when you create a new cluster:
      • Anthos cluster control plane uptime
      • Anthos cluster pod status
      • Anthos cluster node status
      • Anthos cluster VM status
    • An added Anthos integration page is available from the Cloud Monitoring Integration page. The Anthos integration includes descriptions and previews for the predefined Anthos dashboards:
      • Anthos Cluster Control Plane Uptime
      • Anthos Cluster Node Status
      • Anthos Cluster Pod Status
      • Anthos Cluster KubeVirt VM Status
      • Anthos Cluster Utilization Metering

    For more information, see Use predefined dashboards.

  • Preview: Added support for system metrics when you use Google Cloud Managed Service for Prometheus.

Security and Identity:

  • Preview: Added support for Binary Authorization, a service on Google Cloud that provides software supply-chain security for container-based applications. For more information, see Binary Authorization for Anthos clusters overview.
  • Preview: Added support for VPC Service Controls, which provides additional security for your clusters to help mitigate the risk of data exfiltration.
  • Improved security by disabling port 10255, the kubelet read-only port, by default. For more information, see Disable kubelet read-only port in Hardening your cluster's security.

Functionality changes:

  • Replacing taints and labels. Clusters created and upgraded to Anthos clusters on bare metal version 1.15.0 and higher have node-role.kubernetes.io/control-plane:* taints and node-role.kubernetes.io/control-plane labels. These new taints and labels replace the node-role.kubernetes.io/master label and node-role.kubernetes.io/master:* taints on new and upgraded control plane nodes.

Networking changes:

  • Replaced the anetd CNI plugin for the bootstrap cluster with kindnet.
  • Increased eBPF map limit to 512 K to allow for more load balancer Services.
  • Upgraded CoreDNS to version 1.9.4.

Anthos VM Runtime:

  • Moved the Anthos VM Runtime release notes to a separate page in the Anthos VM Runtime documentation section.

Fixes:

  • Fixed an issue that caused the bmctl reset nodes command to fail if the bmctl-workspace directory was empty.
  • Fixed an intermittent issue that caused the bmctl upgrade cluster command to indicate that the operation was complete before the cluster was in a ready state.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

April 25, 2023

1.13

Release 1.13.7

Anthos clusters on bare metal 1.13.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.7 runs on Kubernetes 1.24.

Fixes:

The following container image security vulnerability has been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

April 19, 2023

1.14

Release 1.14.4

Anthos clusters on bare metal 1.14.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.4 runs on Kubernetes 1.25.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

April 12, 2023

1.6 & 1.7 & 1.8 & 1.9 & 1.10 & 1.11 & 1.12 & 1.13 & 1.14

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

March 31, 2023

1.13 & 1.14

Cluster lifecycle improvements 1.13.1 and later

Starting with Anthos clusters on bare metal release 1.13.1, you can use the Google Cloud console or the gcloud CLI to create admin clusters. For more information, see the documentation for your version of Anthos clusters on bare metal:

March 28, 2023

1.12

Release 1.12.9

Anthos clusters on bare metal 1.12.9 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.9 runs on Kubernetes 1.23.

FIxes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

March 24, 2023

1.14

Release 1.14.3

Anthos clusters on bare metal 1.14.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.3 runs on Kubernetes 1.25.

Fixes:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

March 21, 2023

1.13

Release 1.13.6

Anthos clusters on bare metal 1.13.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.6 runs on Kubernetes 1.24.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

March 09, 2023

1.13 & 1.14

Cluster lifecycle improvements 1.13.1 and later

Starting with Anthos clusters on bare metal release 1.13.1, you can use the Google Cloud console or the gcloud CLI to upgrade admin and user clusters managed by the Anthos On-Prem API. If your cluster is at version 1.13.0 or lower, you must use bmctl to upgrade the cluster.

For more information about using the console or the gcloud CLI for upgrades, see the documentation for your version of Anthos clusters on bare metal:

March 02, 2023

1.12

Release 1.12.8

Anthos clusters on bare metal 1.12.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.8 runs on Kubernetes 1.23.

Fixes:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

March 01, 2023

1.14

Release 1.14.2

Anthos clusters on bare metal 1.14.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.2 runs on Kubernetes 1.25.

Fixes:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

February 23, 2023

1.13

Release 1.13.5

Anthos clusters on bare metal 1.13.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.5 runs on Kubernetes 1.24.

Fixes:

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

February 07, 2023

1.12

Release 1.12.7

Anthos clusters on bare metal 1.12.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.7 runs on Kubernetes 1.23.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

January 27, 2023

1.14

1.14.0 Upgrade problem

Control plane nodes for Anthos clusters on bare metal use Kubernetes taints to prevent workload pods from being scheduled on them. When you upgrade version 1.13 Anthos clusters to version 1.14.0, the control plane nodes lose required taints. We recommend that you skip upgrading to version 1.14.0 and upgrade to version 1.14.1 directly.

This problem doesn't cause upgrade failures, but pods that aren't supposed to run on the control plane nodes may start doing so. These workload pods can overwhelm control plane nodes and lead to cluster instability. This issue has security implications, as well. We strongly recommend that you not upgrade your clusters to version 1.14.0, but upgrade instead to a subsequent release version with the fix.

For more information about the issue, including workaround instructions, see the Clusters upgraded to 1.14.0 lose master taints known issue.

Release 1.14.1

Anthos clusters on bare metal 1.14.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.1 runs on Kubernetes 1.25.

Fixes:

Functionality changes:

  • Changed the behavior for periodic health checks during upgrades. Now, during the upgrade process, existing periodic health checks continue to run in the admin cluster. Once the cluster is upgraded to the next version, the previous version periodic health checks are replaced with periodic health checks for the new version.
  • Lowered the priority of health check jobs to minimize contention for resources.
  • Changed the etcd history compaction interval from the default of 5 minutes to 2.5 minutes. This value is set in the kube-apiserver.yaml file.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

January 26, 2023

1.13

Release 1.13.4

Anthos clusters on bare metal 1.13.4 is now available for download. To upgrade, see Upgrade clusters. Anthos clusters on bare metal 1.13.4 runs on Kubernetes 1.24.

Fixed an issue with the anthos-cluster-operator that caused CertificateSigningRequest (CSR) events to be missed during reconciliation steps. The lack of signing resulted in Istio crashlooping.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 21, 2022

1.13 & 1.14

Anthos clusters on bare metal release 1.14.0 is now available for download. Note that Anthos clusters on bare metal version 1.14.0 runs on Kubernetes 1.25. Multiple deprecated APIs are deleted in Kubernetes 1.25. Before you upgrade version 1.13 Anthos clusters to version 1.14, check to see if you are affected by the Kubernetes API deletions.

If you aren't affected by the API deletions, see Upgrade clusters in the 1.14 documentation for upgrade instructions.

December 19, 2022

1.13

Release 1.13.3

Anthos clusters on bare metal 1.13.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.3 runs on Kubernetes 1.24.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 14, 2022

1.12

Release 1.12.6

Anthos clusters on bare metal 1.12.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.6 runs on Kubernetes 1.23.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 13, 2022

1.14

Release 1.14.0

Anthos clusters on bare metal 1.14.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.0 runs on Kubernetes 1.25.

Improved cluster lifecycle functionalities:

  • Upgraded from Kubernetes version 1.24 to 1.25.

  • Enabled customers to run the latest health and preflight checks by running the command bmctl check cluster –check-image-version=latest. Setting the check-image-version flag to 'latest' ensures that clusters are examined for more recent issues, including issues discovered after a release.

  • Preview: Added support of Control group v2 (cgroup v2).

  • GA: Added automatic reservation of CPU and memory resources on cluster nodes so that system daemons have the resources they require.

  • Optimized the consumption of resources by components such as cluster-operator, cap-manager, preflight-check operator, and lifecycle-controllers-manager.

  • GA: Enabled automatic and periodic health checks on all clusters.

Networking:

  • Preview: Added support for turning on kube-proxy-free mode for cluster objects. WARNING: This operation is not reversible. Once enabled, it cannot be disabled.

  • Changed behavior of Dataplane V2 so that it drops a packet if no service backends are available. Previously, the packet was passed to the kernel stack.

  • Enabled automatic API rate limit adjustments in Dataplane V2.

Observability:

  • Added severity level to container logs.

  • Enabled collection of uptime and other Kubernetes resource metrics from the kubelet summary API.

  • Enabled Stackdriver log forwarder in the bootstrap cluster. This log forwarder publishes bootstrap container logs to Cloud Logging.

Security and Identity:

  • GA: Added feature enabling cluster administrators to configure RBAC policies based on Azure Active Directory (AD) groups. Groups information for users belonging to more than 200 groups can now be retrieved.

  • GA: Added secure computing mode (seccomp) support. Running containers with a seccomp profile improves the security of a cluster because it restricts the system calls that containers are allowed to make to the kernel.

  • Added annotation in the cluster configuration file which allows customers to disable the kubelet read-only port. After disabling the read-only port, customers have to change their cluster configurations so that workloads use the kubelet secure port.

VM Runtime:

  • GA: Added support for guest OS booting of UEFI. Previously, only BIOS was supported.

  • Preview: Enabled Terraform scripting to create VMs on an Anthos cluster. For more information, including usage instructions, an input reference, and examples, see the terraform-google-anthos-vm GitHub repository.

  • Preview: Add support for non-uniform memory access (NUMA) awareness. When enabled, all communication within the VM is local to the NUMA node, thus avoiding the performance cost of data transactions with remote memory locations.

  • Preview: Enabled multicast traffic for VMs.

  • Added Anthos VM Runtime preflight checks to validate hardware accelerator configuration.

  • Enabled configuration of storage's volume mode (block or filesystem) and access modes, such as RWO and RWX.

  • Enabled means to configure the storage class of a scratch space. A scratch space is sometimes required when importing or uploading a VM disk image.

  • Added support for configuring cloud-init, using virtctl.

  • Enabled ability to disable auto-installation of the guest agent binary. After the initial guest agent installation, you can set the autoInstallGuestAgent flag to false so that the binary doesn't mount in subsequent restarts.

  • Enabled the support of multiple network interfaces, by default, for all clusters.

  • Improved security for creating a VM with kubectl virt create. If an initial password is specified, it is now stored in a secret and not as a VM annotation.

  • Reduced the permissions of the network controller.

  • Changed default to always use Asynchronous IO mode (AIO) in order to reduce QEMU memory pressure.

  • Added VM creation and disk provisioning times to Prometheus metrics.

  • Added support for the Tesla T4 GPU.

  • Enabled reset of GPU card to its original status when GPU functionality is disabled.

  • Enabled ability to disable Anthos VM Runtime when it's in the enabling state and custom resource definitions haven't yet been installed.

  • Added the following command, which allows you to display the VM screen: kubectll virt vnc --screenshot VM_NAME.

  • Fixed the IP address update for Windows guest VMs.

  • Resolved the MacVTap interface creation failure which occurred when the name of the interface was too long.

  • Fixed attaching VM disk using SATA driver.

  • Fixed issue so that setting disableCDIUploadProxyVIP to true correctly disables the cdi-uploadproxy service.

  • Fixed issue so that specifying a PersistentVolumeClaim (PVC) with an empty underlying PersistentVolume (PV) correctly creates the underlying empty disk format (raw or qcow2).

  • Enforced VM names to follow the standard RFC1123 format.

  • Fixed issue so that ISO image is correctly imported from a Cloud Storage bucket.

  • Fixed benign crash looping of the NVIDIA device plugin and the Multi-Instance GPU (MIG) manager when all GPU cards are allocated to a VM.

  • Fixed issue so that virt-launcher Pod can be created when advanced compute is enabled.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 09, 2022

1.12

Release 1.12.5

Anthos clusters on bare metal 1.12.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.5 runs on Kubernetes 1.23.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 22, 2022

1.13

Release 1.13.2

Anthos clusters on bare metal 1.13.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.2 runs on Kubernetes 1.24.

Fixes:

  • Ensured the kubeadmconfig Secret is deleted when a Cluster API node is removed.
  • Added preflight check command (bmctl check preflight) that you can use when upgrading version 1.13 and higher clusters.
  • Updated the commands bmctl check preflight and bmctl create cluster so that they fail if worker or control-plane nodes have docker credentials in /root/.docker/config.json. (Anthos clusters on bare metal version 1.13 and higher can no longer use Docker Engine as a container runtime. All clusters must use the default container runtime containerd).
  • The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 18, 2022

1.11

Release 1.11.8

Anthos clusters on bare metal 1.11.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.8 runs on Kubernetes 1.22.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 08, 2022

1.12

Release 1.12.4

Anthos clusters on bare metal 1.12.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.4 runs on Kubernetes 1.23.

Fixes:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 07, 2022

1.11 & 1.12 & 1.13

Security bulletin (1.11, 1.12, and 1.13)

A security vulnerability, CVE-2022-39278, has been discovered in Istio, which is used in Anthos Service Mesh, that allows a malicious attacker to crash the control plane.

For instructions and more details, see the Anthos clusters on bare metal security bulletin.

November 01, 2022

1.13

Cluster lifecycle improvements in 1.13 and later

Preview: You can use the Google Cloud console to create user clusters, delete user clusters, and to add and remove node pools from a user cluster. To explore the new feature, try out the tutorial Create an Anthos on bare metal user cluster on Compute Engine VMs using the console.

October 31, 2022

1.13

Release 1.13.1

Anthos clusters on bare metal 1.13.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.1 runs on Kubernetes 1.24.

Fixes:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 28, 2022

1.11

Anthos clusters on bare metal 1.11.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.7 runs on Kubernetes 1.22.

Known issues: For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 05, 2022

1.12

Release 1.12.3

Anthos clusters on bare metal 1.12.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.3 runs on Kubernetes 1.23.

Fixes:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

September 29, 2022

1.13

Release 1.13.0

Anthos clusters on bare metal 1.13.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.0 runs on Kubernetes 1.24.

The dockershim component in Kubernetes enables cluster nodes to use the Docker Engine container runtime. However, Kubernetes 1.24 removed the dockershim component. Since Anthos clusters on bare metal version 1.13 runs on Kubernetes 1.24, version 1.13 and higher clusters can no longer use Docker Engine as a container runtime. All clusters must use the default container runtime containerd.

Improved cluster lifecycle functionalities:

  • Upgraded from Kubernetes version 1.23 to 1.24:

    • Reverted some of the changes Kubernetes and the kubeadm tool made to certain labels and taints on control plane nodes. Changes were reverted so that older versions of Anthos clusters on bare metal remain supported. As a result, control plane nodes have the following labels and taints:

      • node-role.kubernetes.io/master label
      • node-role.kubernetes.io/control-plane label
      • node-role.kubernetes.io/master:NoSchedule taint
    • Upgraded from kubeadm.k8s.io/v1beta2 to kubeadm.k8s.io/v1beta3 since the former is deprecated.

    • Stopped automatic generation of Secret API objects containing service account tokens for every Service Account. For more information, see the LegacyServiceAccountTokenNoAutoGeneration section of the upgrade notes.

  • Breaking change: Version 1.12 clusters that use Docker Engine can upgrade to 1.13 only if the new container runtime is specified as containerd. Blocked the creation of new 1.13 clusters that use Docker Engine as the container runtime.

  • Preview: Added feature so that upgrades of an admin/hybrid/standalone cluster can proceed without a bootstrap cluster. Management of Anthos clusters on bare metal is now fully conformant to the Kubernetes Resource Model.

  • Added support of Red Hat Enterprise Linux (RHEL) 8.6.

  • Removed an erroneous CustomResourceDefinition (app.k8s.io.Application) from inclusion in the cluster creation process.

  • Fixed vulnerability to YAML injection by switching to safetext/yamltemplate.

  • GA: Added support for installing Anthos clusters on bare metal, using your own registry service, instead of gcr.io. For instructions and additional information, see Use a registry mirror to create clusters.

  • Eliminated false error messaging when the bmctl create cluster is run. The message erroneously reported an Invalid value in the spec.labels field of NodePool specifications.

  • Added a webhook check to prevent worker node pools from being added to an Admin cluster inadvertently.

  • Added feature so that resetting a user cluster doesn't require the cluster configuration file.

  • Reduced containerd disk usage by having containerd store just the uncompressed layers of an image rather than both the compressed and uncompressed layers.

  • Upgraded containerd to version 1.6.6.

Networking:

  • GA: Enabled Dynamic Flat IP with Border Gateway Protocol (BGP) support. This feature lets you configure flat mode using BGP in clusters by leveraging Network Gateway Group and BGP. In this mode the Pod's IP address is visible and routable without masquerading across multiple subdomains. Currently supports advertising IPv4 and IPv6 routes over IPv4 sessions.

  • GA: Added BGP-based Load Balancer support for IPv6. Added ability to disable the Bundled Ingress feature. Customers should disable this feature if they are using full Anthos Service Mesh (ASM) instead. (Bundled Ingress is unnecessary when full ASM is installed).

Observability:

  • Preview: Added support of multi-line parsing for Go and Java logs.

  • GA: Added support for Google Cloud Managed Service for Prometheus (GMP) for application metrics.

  • Refined kube-state-metrics so that only core metrics are collected by default.

Security:

  • GA: Added Google Groups support for Connect Gateway.

  • Switched distroless base image for Node Problem Detector.

  • Changed anet-operator/cilium-operator to run as non-root container.

  • Secured communication between metrics-server and api-server using the Transport Layer Security (TLS) protocol.

VM Runtime:

  • Fixed a memory leak in libvirt-go, which caused unbounded memory growth and risked crashing long-running VMs.

  • Provided guaranteed compute support so that customers can get Guaranteed Quality of Service (QoS)for the VM when needed.

  • Preview: Enabled Anthos VM to be allocated dedicated host cores. Each VM virtual core can be pinned to a dedicated host core.

  • Separated GPU installation and deletion logic. If only the container GPU workload is needed, customers can enable the GPU without having to enable VM Runtime.

  • Added support for the T4 GPU card.

  • Enabled automatic use of the VirtualMachineDisk name as the disk serial number. This change makes it easier for customers to identify the disk in the VM.

  • Enabled KubeVM cloud-init API and startup script API.

  • Added new CLI command (Virtctl) for resetting Windows VM password.

  • Fixed the following container image security vulnerability: CVE-2022-1798

  • Added feature that stops NVIDIA device plugins from crashing if a GPU card hasn't been allocated to a container.

  • Added support for automatic VM restarts after a configuration update. Previously, customers needed to stop the VM, apply the change, and then re-start the VM. To use the feature, set the autoRestartOnConfigurationChange flag to true in the VirtualMachine custom resource.

  • Improved the Kubernetes audit log of VM operations so that it contains detailed VM configuration and update information.

  • Fixed flooding of logs with cluster events that arise when a VM encounters disk I/O errors.

  • Added KubeVM roles. By binding with these roles, customers are granted permission to resources that manage VMs.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

September 26, 2022

1.11

Release 1.11.6

Anthos clusters on bare metal 1.11.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.6 runs on Kubernetes 1.22.

Fixes:

  • Updated the container image to resolve a yaml text/template vulnerability.

  • The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 30, 2022

1.11

Release 1.11.5

Anthos clusters on bare metal 1.11.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.5 runs on Kubernetes 1.22.

Fixes:

  • Increased the default storage size limit of etcd to 6 GiB.

  • The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 25, 2022

1.12

Release 1.12.2

Anthos clusters on bare metal 1.12.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.2 runs on Kubernetes 1.23.

Features:

  • Added –use-disk flag to bmctl backup cluster command to use the disk instead of the in-memory buffer to back up a cluster. Use this option when available RAM is limited on your admin workstation.
  • Added --quiet flag to bmctl check cluster -- snapshot command to suppress logging to the console during the snapshot creation.

Fixes:

  • Added caching for the Cloud Audit Logging feature status to avoid unnecessary checks and improve performance.
  • Increased the etcd default DB size to 6GiB by default to address NO_SPACE_ALARM in high-scale clusters.
  • Fixed a libseccomp package incompatibility issue.
  • Fixed an issue with the machine-reset job getting stuck.
  • Fixed an issue that caused continuous, unneeded cluster reconciliation operations.
  • Fixed an issue that prevented the node problem detector from running after a cluster upgrade.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 23, 2022

1.10

Release 1.10.8

Anthos clusters on bare metal 1.10.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.8 runs on Kubernetes 1.21.

Fixes

The following container image security vulnerability has been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

1.12

Anthos VM Runtime

Anthos VM Runtime is Generally Available (GA). Some features and capabilities are available for Preview only, as indicated in the following descriptions:

  • Upgraded Kubevirt to version 0.49.
  • Upgraded Containerized Data Importer (CDI) to version 1.43.0.
  • Added bmctl command to enable or disable Anthos VM Runtime on user clusters.
  • Added automatic upgrade of Anthos VM Runtime when upgrading Anthos clusters on bare metal.
  • Preview: Added ability to configure an eviction policy that controls how VMs automatically migrate to other hosts during maintenance events.
  • Preview: Added non-disruptive upgrading of VM runtime during live migration (that is, when VMs are unobtrusively migrated from one node to another).

VM APIs:

Observability:

Guest OS support:

Added support for the following guest OS versions running on a Virtual Machine:

  • Windows Server 2019
  • Windows Server 2016
  • Windows 10
  • Red Hat Enterprise Linux (RHEL) 8
  • RHEL 7
  • CentOS 8
  • CentOS 7
  • Ubuntu 20.04
  • Ubuntu 18.04

VM networking features:

  • IPAMv4: Static IP Allocation for VM interfaces.
  • IP and MAC Stickiness for VM interfaces.
  • IPAMv4: DHCP for VM interfaces.
  • VLAN tagging support for VM Interfaces.
  • Multi-NIC for VM interfaces through native Dataplane V2 support (macvtap + Dataplane V2).
  • Static routes and DNS configurations at per-network basis.
  • NetworkPolicy enforcement at per-network basis.
  • Validating admission webhooks for Network and NetworkInterface object.
  • Network Mutation, allow the mutations of Gateway, DNS and the customized network routes in the network custom resource. The parent interface for the VM and the VLAN ID are not mutable. VMs that were already running before the network configuration change need to be restarted to pick up the change.
  • Added command to restart all VMs in a network.
  • Graceful IP release for VMs:

    • During VM migration, the IP isn't released.
    • IP addresses are released for VMs that are deleted or stopped.

    For more information on networking, see Create and use virtual networks for Anthos VM Runtime.

VM Runtime issues:

  • When kubevirt is configured, customers should ensure that TOR switches have MAC learning enabled.

  • If you choose to manually run a DHCP ipconfig /renew command in a Windows VM, you should first perform a DHCP release, using theipconfig /release command. In other words, the sequence for manually performing a DHCP renewal in a Windows environment is the following:

    ipconfig /release
    ipconfig /renew
    

August 04, 2022

1.11

Release 1.11.4

Anthos clusters on bare metal 1.11.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.4 runs on Kubernetes 1.22.

Fixes:

  • Fixed issue in which cluster restores failed when /var/lib/etcd is a mount point.
  • Fixed issue in which attempts to skip minor versions when upgrading weren't blocked. For details about the upgrade policy, see Minor version upgrades.

The following container image security vulnerabilities have been fixed: