Google Distributed Cloud (software only) for bare metal 1.29 release notes

This document lists production updates to Google Distributed Cloud. We recommend that Google Distributed Cloud developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/anthos-bare-metal-release-notes.xml

May 15, 2024

Release 1.29.100-gke.251

GKE on Bare Metal 1.29.100-gke.251 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.29.100-gke.251 runs on Kubernetes 1.29.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

• Added new API and IAM role requirements for Cloud Monitoring:

  • You must enable the kubernetesmetadata.googleapis.com API for your project and grant the roles/kubernetesmetadata.publisher IAM role to the Logging and Monitoring service account (anthos-baremetal-cloud-ops, when created automatically). Clusters use this API as an endpoint to send Kubernetes metadata to Google Cloud. The metadata is vital for cluster monitoring, debugging, and recovery. If you install your clusters behind a proxy, add kubernetesmetadata.googleapis.com to the list of allowed connections.

  • Due to changes in the way service accounts are checked, you must also grant the following IAM roles to the Logging and Monitoring service account:

    • roles/monitoring.viewer

    • roles/serviceusage.serviceUsageViewer

  These API and IAM role requirements apply to both creating new 1.29 clusters and upgrading existing clusters to 1.29.

Functionality changes:

• Added checks to validate the SSH client certificate file type before saving the certificate as a Secret.

• Deprecated the spec.gkeVersion field in Machine and BareMetalMachine custom resources. After GKE on Bare Metal release 1.30, the value of gkeVersion isn't guaranteed to be reliable.

• Added preflight checks for available disk space in specific directories:

  • During cluster creation, the following directories are checked:

  • / (the root directory) has at least 4 GiB of free space

  • /var/log/fluent-bit-buffers has at least 12 GiB of free space

  • /var/opt/buffered-metrics has at least 10016 MiB of free space

  • During a cluster upgrade, the following directory is checked:

  • / (the root directory) has at least 2 GiB of free space

Fixes:

• Fixed an issue where the kubelet doesn't honor shortened, 1-second grace period for pod deletion during eviction-based draining.

The following container image security vulnerabilities have been fixed in 1.29.100-gke.251:

• Medium-severity container vulnerabilities:

  • CVE-2024-2961

  • CVE-2024-28182

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

April 29, 2024

Release 1.29.0-gke.1449

GKE on Bare Metal 1.29.0-gke.1449 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.29.0-gke.1449 runs on Kubernetes 1.29.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Version 1.15 end of life: In accordance with the Version Support Policy, version 1.15 (all patch releases) of GKE on Bare Metal has reached its end of life and is no longer supported.

• Added new API and IAM role requirements for Cloud Monitoring:

  • You must enable the kubernetesmetadata.googleapis.com API for your project and grant the roles/kubernetesmetadata.publisher IAM role to the Logging and Monitoring service account (anthos-baremetal-cloud-ops, when created automatically). Clusters use this API as an endpoint to send Kubernetes metadata to Google Cloud. The metadata is vital for cluster monitoring, debugging, and recovery. If you install your clusters behind a proxy, add kubernetesmetadata.googleapis.com to the list of allowed connections.

  • Due to changes in the way service accounts are checked, you must also grant the following IAM roles to the Logging and Monitoring service account:

    • roles/monitoring.viewer

    • roles/serviceusage.serviceUsageViewer

  These API and IAM role requirements apply to both creating new 1.29 clusters and upgrading existing clusters to 1.29.

• GA: Support GKE Identity Service v2 capability for an improved security flow when you authenticate with third-party identity solutions.

  The GA offering of GKE Identity Service v2 has the following requirements and restrictions:

  • GKE Identity Service v2 now requires ports 11001 and 11002 on the control plane load balancer nodes, instead of 8443 and 8444. Ensure these ports are open and available before you upgrade a cluster to version 1.29.0-gke.1449 and higher. If the ports aren't open, upgrade preflight checks fail.

  • GKE Identity Service v2 requires version 1.5.1 or higher of the Anthos Auth gcloud CLI component. If necessary, update the Anthos Auth component (gcloud components update anthos-auth). If you use the Google Cloud SDK, updating the SDK (gcloud components update) to version 474.0.0 or later also updates the Anthos Auth component to the required version.

  • GKE Identity Service v2 doesn't work with GKE on Bare Metal clusters with the following configurations:

    • Clusters with a single control plane node only.

    • Clusters that use control plane nodes for load balancing. That is, clusters that aren't configured with either a separate load balancing node pool or manual load balancing.

• GA: Added support for skews of up to two minor versions for selective node pool upgrades.

• GA: Added capability to pause and resume cluster upgrades.

• GA: Maintenance mode now uses eviction-based draining for nodes, instead of taint-based draining. Eviction-based draining uses the Eviction API, which honors Pod Disruption Budgets (PDBs). Draining nodes this way provides better protection against workload disruptions.

• Preview: Added support for node-level private registry configuration for workload images.

• Preview: Added support for rolling back select node pool upgrades.

• Preview: Added support for admin and hybrid clusters to manage multiple versions user clusters concurrently.

• Preview: Added support for using an intermediate Certificate Authority (CA) as the cluster root CA.

• Preview: Added support to route workload logs to a third-party custom Kafka destination. This capability isn't enabled by default. You enable this capability in the cluster stackdriver resource spec by adding the unmanagedKafkaOutputConfig section. This section lets you specify the IP addresses of Kafka message brokers (brokers), topic names (topics), and keys to map the topics to partitions (topicKeys).

• Improved command-line interface errors and error documentation.

Functionality changes:

• GKE Identity Service v2 now sends extra parameters (extraParams) to your OIDC provider.

• Extra node viewing permissions are added for accounts specified with the spec.clusterSecurity.authorization.clusterViewer.gcpAccounts field in the Cluster resource.

• Added Status.Available field to BareMetalMachine resources to indicate whether the machine is available.

• Updated preflight checks add a check for networking kernel modules (ip_tables or np_tables) and remove the iptables package check.

• The Google plugin for the GKE Identity Service now caches the public keys based on max-age in cache-control header.

Fixes:

• Fixed an issue where the kubelet doesn't honor shortened, 1-second grace period for pod deletion during eviction-based draining.

• Fixed a cluster upgrade issue where the lifecycle-controller-deployer Pod was unable to migrate existing GKE on Bare Metal resources to the latest API version. This issue blocked upgrades to earlier version 1.28 releases.

• Fixed an issue with configuring a proxy for your cluster that required you to manually set HTTPS_PROXY and NO_PROXY environment variables on the admin workstation.

• Fixed an issue where upgrades are blocked because cluster-operator can't delete stale, failing preflight check resources.

• Fixed an issue where the network check ConfigMap wasn't updated when nodes were added or removed.

The following container image security vulnerabilities have been fixed in version 1.29.0-gke.1449:

• Critical container vulnerabilities:

  • CVE-2021-38297

  • CVE-2022-23806

  • CVE-2023-24538

  • CVE-2023-24540

  • CVE-2023-25775

  • CVE-2023-29402

  • CVE-2023-29404

  • CVE-2023-29405

• High-severity container vulnerabilities:

  • CVE-2020-29652

  • CVE-2021-29923

  • CVE-2021-33195

  • CVE-2021-33196

  • CVE-2021-33198

  • CVE-2021-39293

  • CVE-2021-41771

  • CVE-2021-41772

  • CVE-2021-44716

  • CVE-2022-2879

  • CVE-2022-2880

  • CVE-2022-21698

  • CVE-2022-23772

  • CVE-2022-23773

  • CVE-2022-24675

  • CVE-2022-24921

  • CVE-2022-28131

  • CVE-2022-28327

  • CVE-2022-28948

  • CVE-2022-30580

  • CVE-2022-30630

  • CVE-2022-30631

  • CVE-2022-30632

  • CVE-2022-30633

  • CVE-2022-30635

  • CVE-2022-32189

  • CVE-2022-41715

  • CVE-2022-41724

  • CVE-2022-41725

  • CVE-2023-5717

  • CVE-2023-6040

  • CVE-2023-6356

  • CVE-2023-6536

  • CVE-2023-6606

  • CVE-2023-6931

  • CVE-2023-6932

  • CVE-2023-24534

  • CVE-2023-24536

  • CVE-2023-24537

  • CVE-2023-24539

  • CVE-2023-29400

  • CVE-2023-29403

  • CVE-2023-29499

  • CVE-2023-35827

  • CVE-2023-46838

  • CVE-2023-51780

  • CVE-2023-51781

  • CVE-2023-51782

  • CVE-2023-52436

  • CVE-2023-52439

  • CVE-2023-52444

  • CVE-2023-52445

  • CVE-2023-52451

  • CVE-2023-52464

  • CVE-2023-52469

  • CVE-2024-1086

  • CVE-2024-26586

  • CVE-2024-26597

  • CVE-2024-26598

• Medium-severity container vulnerabilities:

  • CVE-2020-29509

  • CVE-2020-29511

  • CVE-2021-33197

  • CVE-2021-34558

  • CVE-2021-36221

  • CVE-2021-44879

  • CVE-2022-1705

  • CVE-2022-1962

  • CVE-2022-32148

  • CVE-2022-41717

  • CVE-2023-3446

  • CVE-2023-3817

  • CVE-2023-6004

  • CVE-2023-6121

  • CVE-2023-6915

  • CVE-2023-6918

  • CVE-2023-24532

  • CVE-2023-29406

  • CVE-2023-29409

  • CVE-2023-32611

  • CVE-2023-32665

  • CVE-2023-34324

  • CVE-2023-39198

  • CVE-2023-39804

  • CVE-2023-45863

  • CVE-2023-46218

  • CVE-2023-46343

  • CVE-2023-49290

  • CVE-2023-52443

  • CVE-2023-52449

  • CVE-2023-52470

  • CVE-2024-21664

  • CVE-2024-28085

  • GHSA-2c7c-3mj9-8fqh

• Low-severity container vulnerabilities:

  • CVE-2021-25743

  • CVE-2022-30629

  • CVE-2023-26604

  • CVE-2023-2975

  • CVE-2023-5178

  • CVE-2023-5197

  • CVE-2023-6531

  • CVE-2023-6817

  • CVE-2023-46813

  • CVE-2023-46862

  • CVE-2023-52438

  • CVE-2023-52448

  • CVE-2023-52454

  • CVE-2023-52456

  • CVE-2023-52457

  • CVE-2023-52462

  • CVE-2023-52463

  • CVE-2023-52467

  • CVE-2023-52503

  • CVE-2023-52513

  • CVE-2023-52524

  • CVE-2023-52564

  • CVE-2023-52573

  • CVE-2023-52575

  • CVE-2024-0193

  • CVE-2024-0641

  • CVE-2024-0646

  • CVE-2024-24860

Known issues:

• Clusters that use bundled load balancing with BGP might have performance degradation as the total number of Services of type LoadBalancer approaches 2,000.

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.