GKE on Bare Metal 1.13 release notes

Version 1.13 

• Version 1.28 (latest)
• Version 1.16
• Version 1.15
• Earlier versions (unsupported)

This document lists production updates to GKE on Bare Metal. We recommend that GKE on Bare Metal developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/anthos-bare-metal-release-notes.xml

January 31, 2024

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

August 01, 2023

Release 1.13.10

Anthos clusters on bare metal 1.13.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.10 runs on Kubernetes 1.24.

Functionality changes:

• Upgraded local volume provisioner to v2.5.0.

• Upgraded snapshot controller to v5.0.1.

• Deprecated v1beta1 volume snapshot custom resources. Anthos clusters on bare metal will stop serving v1beta1 resources in a future release.

Fixes:

• Fixed an issue where the apiserver could become unresponsive during a cluster upgrade for clusters with a single control plane node.

• Fixed an issue where audit logs were duplicated into the offline buffer even when they are sent to Cloud Audit Logs successfully.

Fixes:

The following container image security vulnerabilities have been fixed:

• CVE-2009-5155

• CVE-2015-8985

• CVE-2019-9511

• CVE-2019-9513

• CVE-2020-11080

• CVE-2020-13844

• CVE-2022-23524

• CVE-2022-23525

• CVE-2022-23526

• CVE-2022-36055

• CVE-2023-0464

• CVE-2023-0465

• CVE-2023-0466

• CVE-2023-1667

• CVE-2023-2283

• CVE-2023-2454

• CVE-2023-2455

• CVE-2023-2650

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

June 29, 2023

Release 1.13.9

Anthos clusters on bare metal 1.13.9 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.9 runs on Kubernetes 1.24.

Fixes:

The following container image security vulnerabilities have been fixed:

• CVE-2019-17594

• CVE-2019-17595

• CVE-2021-20206

• CVE-2021-36222

• CVE-2021-37750

• CVE-2022-2097

• CVE-2022-3821

• CVE-2022-4304

• CVE-2022-4415

• CVE-2022-4450

• CVE-2022-29178

• CVE-2022-29179

• CVE-2022-29458

• CVE-2023-0215

• CVE-2023-0286

• CVE-2023-0361

ISSUE Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

June 27, 2023

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

June 16, 2023

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

June 01, 2023

Release 1.13.8

Anthos clusters on bare metal 1.13.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.8 runs on Kubernetes 1.24.

Fixes:

• Fixed an issue that prevented Anthos clusters on bare metal from restoring a high-availability quorum for nodes that use /var/lib/etcd as a mountpoint.

• Fixed an upgrade race condition between a node and the CNI, which could trigger two worker nodes to upgrade simultaneously.

• The following container image security vulnerabilities have been fixed:

  • CVE-2022-4415

  • CVE-2022-29458

  • CVE-2022-41723

  • CVE-2022-41725

  • CVE-2023-0045

  • CVE-2023-0386

  • CVE-2023-0461

  • CVE-2023-1077

  • CVE-2023-1078

  • CVE-2023-1118

  • CVE-2023-1281

  • CVE-2023-1670

  • CVE-2023-1829

  • CVE-2023-1989

  • CVE-2023-23559

  • CVE-2023-28466

  • CVE-2023-31436

  • CVE-2023-32233

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 10, 2023

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

April 25, 2023

Release 1.13.7

Anthos clusters on bare metal 1.13.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.7 runs on Kubernetes 1.24.

Fixes:

The following container image security vulnerability has been fixed:

• CVE-2022-23824
• CVE-2022-42331
• CVE-2022-42332
• CVE-2022-42333
• CVE-2022-42334

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

April 12, 2023

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

March 31, 2023

Cluster lifecycle improvements 1.13.1 and later

Starting with Anthos clusters on bare metal release 1.13.1, you can use the Google Cloud console or the gcloud CLI to create admin clusters. For more information, see the documentation for your version of Anthos clusters on bare metal:

• 1.13: Create an admin cluster using Anthos On-Prem API clients

• 1.14: Create an admin cluster using Anthos On-Prem API clients

March 21, 2023

Release 1.13.6

Anthos clusters on bare metal 1.13.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.6 runs on Kubernetes 1.24.

Fixes:

The following container image security vulnerabilities have been fixed:

• CVE-2021-3449
• CVE-2021-3711
• CVE-2021-3712
• CVE-2021-40528
• CVE-2022-0778
• CVE-2022-1292
• CVE-2022-2068
• CVE-2022-45142
• CVE-2023-0215
• CVE-2023-0286

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

March 09, 2023

Cluster lifecycle improvements 1.13.1 and later

Starting with Anthos clusters on bare metal release 1.13.1, you can use the Google Cloud console or the gcloud CLI to upgrade admin and user clusters managed by the Anthos On-Prem API. If your cluster is at version 1.13.0 or lower, you must use bmctl to upgrade the cluster.

For more information about using the console or the gcloud CLI for upgrades, see the documentation for your version of Anthos clusters on bare metal:

• 1.13: Upgrade an admin or user cluster using Anthos On-Prem API clients

• 1.14: Upgrade an admin or user cluster using Anthos On-Prem API clients

February 23, 2023

Release 1.13.5

Anthos clusters on bare metal 1.13.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.5 runs on Kubernetes 1.24.

Fixes:

• Updated Anthos Identity service to better handle concurrent authentication webhook requests.

• The following container image security vulnerabilities have been fixed:

  • CVE-2018-20217
  • CVE-2022-2873
  • CVE-2022-3545
  • CVE-2022-32221
  • CVE-2022-36280
  • CVE-2022-41218
  • CVE-2022-43552
  • CVE-2022-45934
  • CVE-2022-47929
  • CVE-2023-23454
  • CVE-2023-23455

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

January 26, 2023

Release 1.13.4

Anthos clusters on bare metal 1.13.4 is now available for download. To upgrade, see Upgrade clusters. Anthos clusters on bare metal 1.13.4 runs on Kubernetes 1.24.

Fixed an issue with the anthos-cluster-operator that caused CertificateSigningRequest (CSR) events to be missed during reconciliation steps. The lack of signing resulted in Istio crashlooping.

The following container image security vulnerabilities have been fixed:

• CVE-2021-3759
• CVE-2021-4037
• CVE-2021-46848
• CVE-2022-1184
• CVE-2022-20421
• CVE-2022-2978
• CVE-2022-3169
• CVE-2022-3176
• CVE-2022-3524
• CVE-2022-3564
• CVE-2022-3565
• CVE-2022-3586
• CVE-2022-3594
• CVE-2022-3621
• CVE-2022-3640
• CVE-2022-3643
• CVE-2022-3646
• CVE-2022-3903
• CVE-2022-39188
• CVE-2022-40303
• CVE-2022-40304
• CVE-2022-40307
• CVE-2022-41849
• CVE-2022-41850
• CVE-2022-41916
• CVE-2022-42010
• CVE-2022-42011
• CVE-2022-42012
• CVE-2022-42328
• CVE-2022-42329
• CVE-2022-42895
• CVE-2022-42896
• CVE-2022-42898
• CVE-2022-43680
• CVE-2022-43750
• CVE-2022-44638
• CVE-2022-47518
• CVE-2022-47519
• CVE-2022-47520
• CVE-2022-47521

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 21, 2022

Anthos clusters on bare metal release 1.14.0 is now available for download. Note that Anthos clusters on bare metal version 1.14.0 runs on Kubernetes 1.25. Multiple deprecated APIs are deleted in Kubernetes 1.25. Before you upgrade version 1.13 Anthos clusters to version 1.14, check to see if you are affected by the Kubernetes API deletions.

If you aren't affected by the API deletions, see Upgrade clusters in the 1.14 documentation for upgrade instructions.

December 19, 2022

Release 1.13.3

Anthos clusters on bare metal 1.13.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.3 runs on Kubernetes 1.24.

The following container image security vulnerabilities have been fixed:

• CVE-2022-35737
• CVE-2022-42311
• CVE-2022-33745
• CVE-2022-42309
• CVE-2022-42320
• CVE-2022-42323
• CVE-2022-33748
• CVE-2022-42321
• CVE-2022-33746
• CVE-2022-42310
• CVE-2022-42316
• CVE-2022-42322
• CVE-2022-42319
• CVE-2022-42325
• CVE-2022-42315
• CVE-2022-42324
• CVE-2022-42314
• CVE-2022-42317
• CVE-2022-42312
• CVE-2022-42318
• CVE-2022-42313
• CVE-2022-42326

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 22, 2022

Release 1.13.2

Anthos clusters on bare metal 1.13.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.2 runs on Kubernetes 1.24.

Fixes:

• Ensured the kubeadmconfig Secret is deleted when a Cluster API node is removed.
• Added preflight check command (bmctl check preflight) that you can use when upgrading version 1.13 and higher clusters.
• Updated the commands bmctl check preflight and bmctl create cluster so that they fail if worker or control-plane nodes have docker credentials in /root/.docker/config.json. (Anthos clusters on bare metal version 1.13 and higher can no longer use Docker Engine as a container runtime. All clusters must use the default container runtime containerd).
• The following container image security vulnerabilities have been fixed:
  • CVE-2022-1586
  • CVE-2022-1587
  • CVE-2022-3116
  • CVE-2021-3999
  • CVE-2022-37434
  • CVE-2022-41715

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 07, 2022

Security bulletin (1.11, 1.12, and 1.13)

A security vulnerability, CVE-2022-39278, has been discovered in Istio, which is used in Anthos Service Mesh, that allows a malicious attacker to crash the control plane.

For instructions and more details, see the Anthos clusters on bare metal security bulletin.

November 01, 2022

Cluster lifecycle improvements in 1.13 and later

Preview: You can use the Google Cloud console to create user clusters, delete user clusters, and to add and remove node pools from a user cluster. To explore the new feature, try out the tutorial Create an Anthos on bare metal user cluster on Compute Engine VMs using the console.

October 31, 2022

Release 1.13.1

Anthos clusters on bare metal 1.13.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.1 runs on Kubernetes 1.24.

Fixes:

• Increased the CPU limit for the metrics-server Pod to prevent it from frequently restarting.
• Updated the command bmctl enable/disable vmruntime to block the CLI and periodically show the status of VMRuntime. The CLI remains blocked until the command has finished and VMRuntime's ready status is true, or until a default period of 10 mins has expired.
• Removed deprecated IPv6DualStack featureGate field that blocks clusters from upgrading to Anthos clusters on bare metal version 1.13.0.

• Fixed the no path ID mapping for prefix error that sometimes occurred when bgpd updated bgpsession.

• The following container image security vulnerabilities have been fixed:

  • CVE-2018-25032
  • CVE-2021-22924
  • CVE-2021-22946
  • CVE-2021-22947
  • CVE-2021-23840
  • CVE-2021-23841
  • CVE-2021-3999
  • CVE-2021-43618
  • CVE-2021-46828
  • CVE-2022-0778
  • CVE-2022-1292
  • CVE-2022-1586
  • CVE-2022-1587
  • CVE-2022-1664
  • CVE-2022-1679
  • CVE-2022-2097
  • CVE-2022-2257
  • CVE-2022-2509
  • CVE-2022-2509
  • CVE-2022-2526
  • CVE-2022-27776
  • CVE-2022-27781
  • CVE-2022-27782
  • CVE-2022-3172
  • CVE-2022-32206
  • CVE-2022-32208
  • CVE-2022-34903
  • CVE-2022-37434
  • CVE-2022-39278
  • CVE-2022-40674

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

September 29, 2022

Release 1.13.0

Anthos clusters on bare metal 1.13.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.13.0 runs on Kubernetes 1.24.

The dockershim component in Kubernetes enables cluster nodes to use the Docker Engine container runtime. However, Kubernetes 1.24 removed the dockershim component. Since Anthos clusters on bare metal version 1.13 runs on Kubernetes 1.24, version 1.13 and higher clusters can no longer use Docker Engine as a container runtime. All clusters must use the default container runtime containerd.

Improved cluster lifecycle functionalities:

• Upgraded from Kubernetes version 1.23 to 1.24:

  • Reverted some of the changes Kubernetes and the kubeadm tool made to certain labels and taints on control plane nodes. Changes were reverted so that older versions of Anthos clusters on bare metal remain supported. As a result, control plane nodes have the following labels and taints:

    • node-role.kubernetes.io/master label
    • node-role.kubernetes.io/control-plane label
    • node-role.kubernetes.io/master:NoSchedule taint

  • Upgraded from kubeadm.k8s.io/v1beta2 to kubeadm.k8s.io/v1beta3 since the former is deprecated.

  • Stopped automatic generation of Secret API objects containing service account tokens for every Service Account. For more information, see the LegacyServiceAccountTokenNoAutoGeneration section of the upgrade notes.

• Breaking change: Version 1.12 clusters that use Docker Engine can upgrade to 1.13 only if the new container runtime is specified as containerd. Blocked the creation of new 1.13 clusters that use Docker Engine as the container runtime.

• Preview: Added feature so that upgrades of an admin/hybrid/standalone cluster can proceed without a bootstrap cluster. Management of Anthos clusters on bare metal is now fully conformant to the Kubernetes Resource Model.

• Added support of Red Hat Enterprise Linux (RHEL) 8.6.

• Removed an erroneous CustomResourceDefinition (app.k8s.io.Application) from inclusion in the cluster creation process.

• Fixed vulnerability to YAML injection by switching to safetext/yamltemplate.

• GA: Added support for installing Anthos clusters on bare metal, using your own registry service, instead of gcr.io. For instructions and additional information, see Use a registry mirror to create clusters.

• Eliminated false error messaging when the bmctl create cluster is run. The message erroneously reported an Invalid value in the spec.labels field of NodePool specifications.

• Added a webhook check to prevent worker node pools from being added to an Admin cluster inadvertently.

• Added feature so that resetting a user cluster doesn't require the cluster configuration file.

• Reduced containerd disk usage by having containerd store just the uncompressed layers of an image rather than both the compressed and uncompressed layers.

• Upgraded containerd to version 1.6.6.

Networking:

• GA: Enabled Dynamic Flat IP with Border Gateway Protocol (BGP) support. This feature lets you configure flat mode using BGP in clusters by leveraging Network Gateway Group and BGP. In this mode the Pod's IP address is visible and routable without masquerading across multiple subdomains. Currently supports advertising IPv4 and IPv6 routes over IPv4 sessions.

• GA: Added BGP-based Load Balancer support for IPv6. Added ability to disable the Bundled Ingress feature. Customers should disable this feature if they are using full Anthos Service Mesh (ASM) instead. (Bundled Ingress is unnecessary when full ASM is installed).

Observability:

• Preview: Added support of multi-line parsing for Go and Java logs.

• GA: Added support for Google Cloud Managed Service for Prometheus (GMP) for application metrics.

• Refined kube-state-metrics so that only core metrics are collected by default.

Security:

• GA: Added Google Groups support for Connect Gateway.

• Switched distroless base image for Node Problem Detector.

• Changed anet-operator/cilium-operator to run as non-root container.

• Secured communication between metrics-server and api-server using the Transport Layer Security (TLS) protocol.

VM Runtime:

• Fixed a memory leak in libvirt-go, which caused unbounded memory growth and risked crashing long-running VMs.

• Provided guaranteed compute support so that customers can get Guaranteed Quality of Service (QoS)for the VM when needed.

• Preview: Enabled Anthos VM to be allocated dedicated host cores. Each VM virtual core can be pinned to a dedicated host core.

• Separated GPU installation and deletion logic. If only the container GPU workload is needed, customers can enable the GPU without having to enable VM Runtime.

• Added support for the T4 GPU card.

• Enabled automatic use of the VirtualMachineDisk name as the disk serial number. This change makes it easier for customers to identify the disk in the VM.

• Enabled KubeVM cloud-init API and startup script API.

• Added new CLI command (Virtctl) for resetting Windows VM password.

• Fixed the following container image security vulnerability: CVE-2022-1798

• Added feature that stops NVIDIA device plugins from crashing if a GPU card hasn't been allocated to a container.

• Added support for automatic VM restarts after a configuration update. Previously, customers needed to stop the VM, apply the change, and then re-start the VM. To use the feature, set the autoRestartOnConfigurationChange flag to true in the VirtualMachine custom resource.

• Improved the Kubernetes audit log of VM operations so that it contains detailed VM configuration and update information.

• Fixed flooding of logs with cluster events that arise when a VM encounters disk I/O errors.

• Added KubeVM roles. By binding with these roles, customers are granted permission to resources that manage VMs.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.