GKE on Bare Metal 1.14 release notes

Version 1.14 

• Version 1.28 (latest)
• Version 1.16
• Version 1.15
• Earlier versions (unsupported)

This document lists production updates to GKE on Bare Metal. We recommend that GKE on Bare Metal developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/anthos-bare-metal-release-notes.xml

January 31, 2024

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

November 21, 2023

Release 1.14.11

Anthos clusters on bare metal 1.14.11 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.11 runs on Kubernetes 1.25.

Fixes:

The following container image security vulnerabilities have been fixed in 1.14.11:

• Critical container vulnerabilities:

  • CVE-2023-45871

• High-severity container vulnerabilities:

  • CVE-2023-1989

  • CVE-2023-4244

  • CVE-2023-4622

  • CVE-2023-4623

  • CVE-2023-4921

  • CVE-2023-42753

• Medium-severity container vulnerabilities:

  • CVE-2021-3507

  • CVE-2021-3930

  • CVE-2021-20196

  • CVE-2022-0216

  • CVE-2023-0330

  • CVE-2023-3180

  • CVE-2023-3772

  • CVE-2023-34969

  • CVE-2023-37453

  • CVE-2023-39189

  • CVE-2023-39192

  • CVE-2023-39193

  • CVE-2023-39194

  • CVE-2023-42754

  • CVE-2023-42755

• Low-severity container vulnerabilities:

  • CVE-2017-16516

  • CVE-2020-14394

  • CVE-2021-20203

  • CVE-2022-24795

  • CVE-2023-1544

  • CVE-2023-3301

  • CVE-2023-3354

  • CVE-2023-3773

  • CVE-2023-6176

  • CVE-2023-33460

  • CVE-2023-42756

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

November 06, 2023

Release 1.14.10

Anthos clusters on bare metal 1.14.10 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.10 runs on Kubernetes 1.25.

Functionality changes:

• Added NODEPOOL-NAME, NODEPOOL-NAMESPACE, and STATUS columns for the InventoryMachine resource to improve troubleshooting.

• Removed hardcoded timeout value for the bmctl backup operation.

Fixes:

• Fixed an issue where CoreDNS Pods can get stuck in an unready state.

• Fixed a memory leak in Dataplane V2.

Fixes:

The following container image security vulnerabilities have been fixed in version 1.14.10:

• Critical container vulnerabilities:

  • CVE-2023-38408

• High-severity container vulnerabilities:

  • CVE-2021-20206

  • CVE-2021-25741

  • CVE-2022-3064

  • CVE-2023-28840

• Medium-severity container vulnerabilities:

  • CVE-2019-11250

  • CVE-2020-8554

  • CVE-2020-8561

  • CVE-2020-21047

  • CVE-2021-4235

  • CVE-2021-25736

  • CVE-2022-46146

  • CVE-2023-28841

  • CVE-2023-28842

  • CVE-2023-39347

  • CVE-2023-41333

• Low-severity container vulnerabilities:

  • CVE-2021-25740

  • CVE-2021-32292

  • CVE-2023-34242

  • CVE-2023-41332

  • GHSA-qq97-vm5h-rrhg

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

September 29, 2023

Release 1.14.9

Anthos clusters on bare metal 1.14.9 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.9 runs on Kubernetes 1.25.

Fixes:

Fixed an issue to prevent cluster upgrades from starting on a node before either all Pods have been drained or the Pod draining timeout has been reached.

Fixes:

The following container image security vulnerabilities have been fixed in version 1.14.9:

• High-severity container vulnerabilities:

  • CVE-2019-13509

  • CVE-2020-8558

  • CVE-2022-39189

  • CVE-2023-1380

  • CVE-2023-2007

  • CVE-2023-2124

  • CVE-2023-3090

  • CVE-2023-3111

  • CVE-2023-3268

  • CVE-2023-3390

  • CVE-2023-3609

  • CVE-2023-3611

  • CVE-2023-3776

  • CVE-2023-4128

  • CVE-2023-4206

  • CVE-2023-4207

  • CVE-2023-4208

  • CVE-2023-21255

  • CVE-2023-27561

  • CVE-2023-29002

  • CVE-2023-34319

  • CVE-2023-35001

  • CVE-2023-35788

  • CVE-2023-40283

• Medium-severity container vulnerabilities:

  • CVE-2015-3627

  • CVE-2019-11251

  • CVE-2020-8555

  • CVE-2020-8564

  • CVE-2020-8569

  • CVE-2021-25735

  • CVE-2022-4269

  • CVE-2022-40982

  • CVE-2023-1206

  • CVE-2023-2002

  • CVE-2023-2269

  • CVE-2023-3212

  • CVE-2023-3338

  • CVE-2023-3863

  • CVE-2023-4132

  • CVE-2023-4194

  • CVE-2023-4273

  • CVE-2023-20569

  • CVE-2023-20593

  • CVE-2023-27593

  • CVE-2023-27594

  • CVE-2023-27595

  • CVE-2023-30851

  • CVE-2023-31084

• Low-severity container vulnerabilities:

  • CVE-2020-8562

  • CVE-2022-48554

  • CVE-2023-2156

  • CVE-2023-2898

  • CVE-2023-3141

  • CVE-2023-3389

  • CVE-2023-3610

  • CVE-2023-3777

  • CVE-2023-4004

  • CVE-2023-4147

  • CVE-2023-21400

  • CVE-2023-31248

  • CVE-2023-34256

  • CVE-2023-35823

  • CVE-2023-35824

  • CVE-2023-35828

  • CVE-2023-35829

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

August 22, 2023

Release 1.14.8

Anthos clusters on bare metal 1.14.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.8 runs on Kubernetes 1.25.

Fixes:

The following container image security vulnerabilities have been fixed:

• High-severity container vulnerabilities:

  • CVE-2017-11468

• Medium-severity container vulnerabilities:

  • CVE-2019-9511

  • CVE-2019-9513

  • CVE-2020-11080

  • CVE-2020-13844

• Low-severity container vulnerabilities:

  • CVE-2009-5155

  • CVE-2015-8985

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

July 25, 2023

FEATURE

Release 1.14.7

Anthos clusters on bare metal 1.14.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.7 runs on Kubernetes 1.25.

Functionality changes:

• Audit logs are compressed on the wire for Cloud Audit Logs consumption, reducing egress bandwidth by approximately 60%.

• Upgraded local volume provisioner to v2.5.0.

• Upgraded snapshot controller to v5.0.1.

• Deprecated v1beta1 volume snapshot custom resources. Anthos clusters on bare metal will stop serving v1beta1 resources in a future release.

Fixes:

• Fixed an issue where the smart default didn't work for gke-metrics-agent.

• Fixed an issue where the apiserver could become responsive during a cluster upgrade for clusters with a single control plane node.

• Fixed an issue where audit logs were duplicated into the offline buffer even when they are sent to Cloud Audit Logs successfully.

Fixes:

The following container image security vulnerabilities have been fixed:

• CVE-2022-23524

• CVE-2022-23525

• CVE-2022-23526

• CVE-2022-36055

• CVE-2023-0464

• CVE-2023-0465

• CVE-2023-0466

• CVE-2023-1667

• CVE-2023-2283

• CVE-2023-2650

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

June 27, 2023

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

June 23, 2023

Release 1.14.6

Anthos clusters on bare metal 1.14.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.6 runs on Kubernetes 1.25.

Functionality changes:

• Upgraded etcd version to v3.4.26-0-gke.0.

Fixes:

The following container image security vulnerabilities have been fixed:

• CVE-2019-17594

• CVE-2019-17595

• CVE-2021-20206

• CVE-2022-3821

• CVE-2022-4415

• CVE-2022-29458

• CVE-2022-32190

• CVE-2023-2454

• CVE-2023-2455

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

June 16, 2023

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

May 24, 2023

Release 1.14.5

Anthos clusters on bare metal 1.14.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.5 runs on Kubernetes 1.25.

Fixes:

• Fixed an issue that caused a continuous increase in memory usage for stackdriver-log-forwarder Pods.
• Fixed an issue that caused the bmctl restore command to stop responding for clusters with manually configured load balancers.
• Fixed an issue that caused preflight checks to fail for clusters configured with spec.proxy.noProxy settings.
• Fixed an upgrade issue where adding upgradeStrategy.parallelUpgrade.concurrentNodes to the NodePool spec (for a parallel upgrade) caused the upgrade operation to fail.
• Fixed an issue that caused conflicts with third-party Ansible automation.
• Fixed an issue that prevented Anthos clusters on bare metal from restoring a high-availability quorum for nodes that use /var/lib/etcd as a mountpoint.
• Fixed a cluster upgrade issue that prevented some control plane nodes from rejoining a cluster configured for high availability.
• Fixed an upgrade race condition between a node and the CNI, which could result in two worker nodes to upgrade simultaneously.
• The following container image security vulnerabilities have been fixed:
  • CVE-2022-3821
  • CVE-2022-4415
  • CVE-2022-4450
  • CVE-2022-29458
  • CVE-2022-41723
  • CVE-2022-41725
  • CVE-2023-0045
  • CVE-2023-0215
  • CVE-2023-0286
  • CVE-2023-0386
  • CVE-2023-0461
  • CVE-2023-1077
  • CVE-2023-1078
  • CVE-2023-1118
  • CVE-2023-1281
  • CVE-2023-1670
  • CVE-2023-1829
  • CVE-2023-1989
  • CVE-2023-23559
  • CVE-2023-27487
  • CVE-2023-27488
  • CVE-2023-27491
  • CVE-2023-27492
  • CVE-2023-27493
  • CVE-2023-27496
  • CVE-2023-28466
  • CVE-2023-31436
  • CVE-2023-32233

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

May 10, 2023

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

April 19, 2023

Release 1.14.4

Anthos clusters on bare metal 1.14.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.4 runs on Kubernetes 1.25.

Fixes:

The following container image security vulnerabilities have been fixed:

• CVE-2021-3711
• CVE-2021-3712
• CVE-2021-40528
• CVE-2022-23824
• CVE-2022-42331
• CVE-2022-42332
• CVE-2022-42333
• CVE-2022-42334
• CVE-2022-48303
• CVE-2023-0361
• CVE-2023-23916

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

April 12, 2023

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

March 31, 2023

Cluster lifecycle improvements 1.13.1 and later

Starting with Anthos clusters on bare metal release 1.13.1, you can use the Google Cloud console or the gcloud CLI to create admin clusters. For more information, see the documentation for your version of Anthos clusters on bare metal:

• 1.13: Create an admin cluster using Anthos On-Prem API clients

• 1.14: Create an admin cluster using Anthos On-Prem API clients

March 24, 2023

Release 1.14.3

Anthos clusters on bare metal 1.14.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.3 runs on Kubernetes 1.25.

Fixes:

• Improved maintenance mode operation by ignoring non-running pods on nodes.
• Updated etcd version to version 3.4.21-0-gke.1 to resolve an issue that could lead to watch starvation and non-operational watch for resources.
• Updated kubernetes version to 1.25.6-gke.1000 to honor exponential backoff in job controller.
• The following container image security vulnerabilities have been fixed:
  • CVE-2021-3449
  • CVE-2021-41990
  • CVE-2021-41991
  • CVE-2021-45079
  • CVE-2022-0778
  • CVE-2022-1292
  • CVE-2022-2068
  • CVE-2022-40617

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

March 09, 2023

Cluster lifecycle improvements 1.13.1 and later

Starting with Anthos clusters on bare metal release 1.13.1, you can use the Google Cloud console or the gcloud CLI to upgrade admin and user clusters managed by the Anthos On-Prem API. If your cluster is at version 1.13.0 or lower, you must use bmctl to upgrade the cluster.

For more information about using the console or the gcloud CLI for upgrades, see the documentation for your version of Anthos clusters on bare metal:

• 1.13: Upgrade an admin or user cluster using Anthos On-Prem API clients

• 1.14: Upgrade an admin or user cluster using Anthos On-Prem API clients

March 01, 2023

Release 1.14.2

Anthos clusters on bare metal 1.14.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.2 runs on Kubernetes 1.25.

Fixes:

• Updated Anthos Identity Service to better handle concurrent authentication webhook requests.
• Updated stackdriver-operator to set CPU and memory resource limits.
• The following container image security vulnerabilities have been fixed:
  • CVE-2018-20217
  • CVE-2022-2873
  • CVE-2022-3545
  • CVE-2022-32221
  • CVE-2022-36280
  • CVE-2022-41218
  • CVE-2022-41717
  • CVE-2022-42898
  • CVE-2022-43552
  • CVE-2022-45934
  • CVE-2022-47629
  • CVE-2022-47929
  • CVE-2023-23454
  • CVE-2023-23455

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

January 27, 2023

1.14.0 Upgrade problem

Control plane nodes for Anthos clusters on bare metal use Kubernetes taints to prevent workload pods from being scheduled on them. When you upgrade version 1.13 Anthos clusters to version 1.14.0, the control plane nodes lose required taints. We recommend that you skip upgrading to version 1.14.0 and upgrade to version 1.14.1 directly.

This problem doesn't cause upgrade failures, but pods that aren't supposed to run on the control plane nodes may start doing so. These workload pods can overwhelm control plane nodes and lead to cluster instability. This issue has security implications, as well. We strongly recommend that you not upgrade your clusters to version 1.14.0, but upgrade instead to a subsequent release version with the fix.

For more information about the issue, including workaround instructions, see the Clusters upgraded to 1.14.0 lose master taints known issue.

Release 1.14.1

Anthos clusters on bare metal 1.14.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.1 runs on Kubernetes 1.25.

Fixes:

• Fixed an issue with the anthos-cluster-operator that caused CertificateSigningRequest (CSR) events to be missed during reconciliation steps. The lack of signing resulted in Istio crashlooping.
• Fixed an issue that prevented the Pod CIDR for nodes from being adjusted from the default /24 mask size to account for the maxPodsPerNode cluster setting.
• Fixed an issue that removed taints from control plane nodes when upgrading clusters to version 1.14.0.
• The following container image security vulnerabilities have been fixed:
  • CVE-2021-3759
  • CVE-2021-46848
  • CVE-2022-3169
  • CVE-2022-3524
  • CVE-2022-3564
  • CVE-2022-3565
  • CVE-2022-3594
  • CVE-2022-3640
  • CVE-2022-3643
  • CVE-2022-40303
  • CVE-2022-40304
  • CVE-2022-41849
  • CVE-2022-41850
  • CVE-2022-42328
  • CVE-2022-42329
  • CVE-2022-42895
  • CVE-2022-42896
  • CVE-2022-42898
  • CVE-2022-44638
  • CVE-2022-47518
  • CVE-2022-47519
  • CVE-2022-47520
  • CVE-2022-47521

Functionality changes:

• Changed the behavior for periodic health checks during upgrades. Now, during the upgrade process, existing periodic health checks continue to run in the admin cluster. Once the cluster is upgraded to the next version, the previous version periodic health checks are replaced with periodic health checks for the new version.
• Lowered the priority of health check jobs to minimize contention for resources.
• Changed the etcd history compaction interval from the default of 5 minutes to 2.5 minutes. This value is set in the kube-apiserver.yaml file.

Known issues:

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

December 21, 2022

Anthos clusters on bare metal release 1.14.0 is now available for download. Note that Anthos clusters on bare metal version 1.14.0 runs on Kubernetes 1.25. Multiple deprecated APIs are deleted in Kubernetes 1.25. Before you upgrade version 1.13 Anthos clusters to version 1.14, check to see if you are affected by the Kubernetes API deletions.

If you aren't affected by the API deletions, see Upgrade clusters in the 1.14 documentation for upgrade instructions.

December 13, 2022

Release 1.14.0

Anthos clusters on bare metal 1.14.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.14.0 runs on Kubernetes 1.25.

Improved cluster lifecycle functionalities:

• Upgraded from Kubernetes version 1.24 to 1.25.

• Enabled customers to run the latest health and preflight checks by running the command bmctl check cluster –check-image-version=latest. Setting the check-image-version flag to 'latest' ensures that clusters are examined for more recent issues, including issues discovered after a release.

• Preview: Added support of Control group v2 (cgroup v2).

• GA: Added automatic reservation of CPU and memory resources on cluster nodes so that system daemons have the resources they require.

• Optimized the consumption of resources by components such as cluster-operator, cap-manager, preflight-check operator, and lifecycle-controllers-manager.

• GA: Enabled automatic and periodic health checks on all clusters.

Networking:

• Preview: Added support for turning on kube-proxy-free mode for cluster objects. WARNING: This operation is not reversible. Once enabled, it cannot be disabled.

• Changed behavior of Dataplane V2 so that it drops a packet if no service backends are available. Previously, the packet was passed to the kernel stack.

• Enabled automatic API rate limit adjustments in Dataplane V2.

Observability:

• Added severity level to container logs.

• Enabled collection of uptime and other Kubernetes resource metrics from the kubelet summary API.

• Enabled Stackdriver log forwarder in the bootstrap cluster. This log forwarder publishes bootstrap container logs to Cloud Logging.

Security and Identity:

• GA: Added feature enabling cluster administrators to configure RBAC policies based on Azure Active Directory (AD) groups. Groups information for users belonging to more than 200 groups can now be retrieved.

• GA: Added secure computing mode (seccomp) support. Running containers with a seccomp profile improves the security of a cluster because it restricts the system calls that containers are allowed to make to the kernel.

• Added annotation in the cluster configuration file which allows customers to disable the kubelet read-only port. After disabling the read-only port, customers have to change their cluster configurations so that workloads use the kubelet secure port.

VM Runtime:

• GA: Added support for guest OS booting of UEFI. Previously, only BIOS was supported.

• Preview: Enabled Terraform scripting to create VMs on an Anthos cluster. For more information, including usage instructions, an input reference, and examples, see the terraform-google-anthos-vm GitHub repository.

• Preview: Add support for non-uniform memory access (NUMA) awareness. When enabled, all communication within the VM is local to the NUMA node, thus avoiding the performance cost of data transactions with remote memory locations.

• Preview: Enabled multicast traffic for VMs.

• Added Anthos VM Runtime preflight checks to validate hardware accelerator configuration.

• Enabled configuration of storage's volume mode (block or filesystem) and access modes, such as RWO and RWX.

• Enabled means to configure the storage class of a scratch space. A scratch space is sometimes required when importing or uploading a VM disk image.

• Added support for configuring cloud-init, using virtctl.

• Enabled ability to disable auto-installation of the guest agent binary. After the initial guest agent installation, you can set the autoInstallGuestAgent flag to false so that the binary doesn't mount in subsequent restarts.

• Enabled the support of multiple network interfaces, by default, for all clusters.

• Improved security for creating a VM with kubectl virt create. If an initial password is specified, it is now stored in a secret and not as a VM annotation.

• Reduced the permissions of the network controller.

• Changed default to always use Asynchronous IO mode (AIO) in order to reduce QEMU memory pressure.

• Added VM creation and disk provisioning times to Prometheus metrics.

• Added support for the Tesla T4 GPU.

• Enabled reset of GPU card to its original status when GPU functionality is disabled.

• Enabled ability to disable Anthos VM Runtime when it's in the enabling state and custom resource definitions haven't yet been installed.

• Added the following command, which allows you to display the VM screen: kubectll virt vnc --screenshot VM_NAME.

• Fixed the IP address update for Windows guest VMs.

• Resolved the MacVTap interface creation failure which occurred when the name of the interface was too long.

• Fixed attaching VM disk using SATA driver.

• Fixed issue so that setting disableCDIUploadProxyVIP to true correctly disables the cdi-uploadproxy service.

• Fixed issue so that specifying a PersistentVolumeClaim (PVC) with an empty underlying PersistentVolume (PV) correctly creates the underlying empty disk format (raw or qcow2).

• Enforced VM names to follow the standard RFC1123 format.

• Fixed issue so that ISO image is correctly imported from a Cloud Storage bucket.

• Fixed benign crash looping of the NVIDIA device plugin and the Multi-Instance GPU (MIG) manager when all GPU cards are allocated to a VM.

• Fixed issue so that virt-launcher Pod can be created when advanced compute is enabled.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.