Google Distributed Cloud 1.9 release notes

This document lists production updates to Google Distributed Cloud. We recommend that Google Distributed Cloud developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

January 31, 2024

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

June 27, 2023

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

June 16, 2023

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

May 10, 2023

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

April 12, 2023

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

June 09, 2022

Release 1.9.8

Anthos clusters on bare metal 1.9.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.8 runs on Kubernetes 1.21.

Fixes:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 27, 2022

Release 1.9.7

Anthos clusters on bare metal 1.9.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.7 runs on Kubernetes 1.21.

Fixes:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 26, 2022

Security bulletin (all minor versions)

Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666 have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect Linux operating systems supported by Anthos clusters on bare metal. For instructions and more details, see the GCP-2022-014 security bulletin.

April 12, 2022

Security bulletin (1.8, 1.9, and 1.10)

A security vulnerability, CVE-2022-23648, has been discovered in containerd's handling of path traversal in the OCI image volume specification. Containers launched through containerd's CRI implementation with a specially-crafted image configuration could gain full read access to arbitrary files and directories on the host.

For more information, see the GCP-2022-013 security bulletin.

March 23, 2022

Release 1.9.6

Anthos clusters on bare metal 1.9.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.6 runs on Kubernetes 1.21.

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend that you upgrade to the highest patch version to ensure that you have the latest security fixes. Always review the release notes before upgrading so that you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 28, 2022

Release 1.9.5

Anthos clusters on bare metal 1.9.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.5 runs on Kubernetes 1.21.

Fixes:

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend that you upgrade to the highest patch version to ensure that you have the latest security fixes. Always review the release notes before upgrading so that you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 25, 2022

Security bulletin (1.8, 1.9, and 1.10)

Envoy recently released multiple security vulnerability fixes. The vulnerabilities affect Anthos clusters on bare metal, because Envoy is used for Metrics Server.

For instructions and more details, see the GCP-2022-008 security bulletin.

February 04, 2022

Security bulletin (all minor versions)

A security vulnerability, CVE-2021-4034, has been discovered in pkexec, a part of the Linux policy kit package (polkit), that allows an authenticated user to perform a privilege escalation attack. PolicyKit is generally used only on Linux desktop systems to allow non-root users to perform actions, such as rebooting the system, installing packages, restarting services, as governed by a policy.

For instructions and more details, see the GCP-2022-004 security bulletin.

January 27, 2022

Release 1.9.4

Anthos clusters on bare metal 1.9.4 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.9.4 runs on Kubernetes 1.21.

Fixes:

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 22, 2021

Release 1.9.3

Anthos clusters on bare metal 1.9.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.3 runs on Kubernetes 1.21.

Fixes:

  • Fixed an issue in which cluster creation fails if a cluster has more than one control plane node, and the HTTPS_PROXY environment variable has been defined on one or more of the control plane nodes.

  • Upgraded Kubernetes version from 1.21.4 to 1.21.5 to address an error in which pods become stuck in the ContainerCreating state because libcontainer mistakenly throws a "unit already exists" error.

  • The following container image security vulnerability has been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 15, 2021

Release 1.9.2

Anthos clusters on bare metal 1.9.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.2 runs on Kubernetes 1.21.

Fixes:

  • Updated preflight checks to use strict mode for decoding YAML to catch issues with indentation and misplaced fields in the cluster configuration file.

  • Fixed an issue that caused containerRuntime to default to docker, instead of containerd in certain uncommon situations.

  • Fixed an issue where node_filesystem metrics report incorrect size in Cloud Monitoring for mount-points other than root.

  • Fixed an issue that caused communication failures between Cloud Logging metadata agent and the Cloud Monitoring API when the root certificate authority (CA) on the host node isn't set up properly.

  • The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 29, 2021

Security bulletin (all minor versions)

The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem.

For more information, see the GCP-2021-011 security bulletin.

October 26, 2021

Release 1.9.1

Anthos clusters on bare metal 1.9.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.1 runs on Kubernetes 1.21.

Fixes:

Functionality changes:

  • Updated the bmctl reset cluster command to prevent you from resetting an admin cluster if the admin cluster is managing user clusters.
  • Updated the bmctl create cluster command to block you from enabling the Anthos VM Runtime for admin clusters.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 21, 2021

Security bulletin (all minor versions)

A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742. Ingress-nginx custom snippets allows retrieval of ingress-nginx service account tokens and secrets across all namespaces. For more information, see the GCP-2021-024 security bulletin.

October 04, 2021

Security bulletin (all minor versions)

A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server.

For more information, see the GCP-2021-021 security bulletin.

September 28, 2021

Release 1.9.0

Anthos clusters on bare metal 1.9.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.9.0 runs on Kubernetes 1.21.

Improved cluster lifecycle functionalities:

  • Preview: Added ability to reset individual nodes with the bmctl reset node command. To give access to the needed cluster configuration file, use the command with the -c flag.

  • Preview: Added ability to recover from HA control plane quorum loss withbmctl restore --control-plane-node command.

  • Added bmctl create ksa command to create a Kubernetes Service Account (KSA) and generate a bearer token. To log in to the registered cluster, you can use the token in Cloud Console Kubernetes Engine > Clusters.

  • Preview: Added bmctl backup cluster and bmctl restore cluster commands to facilitate disaster recovery for clusters.

Introduced new troubleshooting capabilities:

  • Updated the bmctl check cluster --snapshot command to support uploading cluster diagnostic snapshots to a Cloud Storage bucket for review by Cloud Customer Care.

  • Provided access to bootstrap cluster logs to help troubleshoot cluster creation or upgrade problems.

  • Preview: Added support for Node Problem Detector service on nodes for quick detection of common node problems.

Enhanced monitoring and logging:

  • GA: Cloud Audit Logs capability is now generally available and enabled by default. Audit logs are useful for investigating suspicious API requests and for collecting statistics. For more information, see Use Audit Logging.

  • Switched to new open telemetry-based metrics agents to improve reliability, ability to scale, and resource usage.

Improved networking capabilities:

  • GA: The multi-NIC capability to provide additional interfaces to your pods is now generally available.

  • Preview: Added the single root I/O virtualization (SR-IOV) container network interface (CNI) plugin for multi-NIC.

  • Added support to configure cluster Domain Name System (DNS) provider options, such as upstream nameservers, with the new ClusterDNS custom resource definition.

Enhanced security:

  • SELinux is now always enabled in the container runtime for CentOS and RHEL.

  • Preview: Enhanced the capability to rotate cluster certificate authorities (CAs). Updates include support for all cluster types, rotation of front-proxy and etcd CAs, and changes to the bmctl command syntax.

  • Preview: Added Okta group support for authentication in Anthos Identity Service.

Functionality changes:

  • Changed default container runtime to containerd, containerRuntime: containerd for new clusters. Customers can still choose Docker as the container runtime.

  • Preview: Updated bmctl command, bmctl reset nodes --force, to support force removal of control plane nodes with etcd membership cleanup.

  • Added checks for cluster updates to verify access to cluster machines if changes to loginUser or sshKeyPrivatePath are detected. If the checks pass, Anthos clusters on bare metal saves the secret in the cluster.

  • Added new Anthos cluster control plane uptime dashboard in Cloud Monitoring with new metric kubernetes.io/anthos/container/uptime for component availability.

  • Added new alerts for control plane components availability with new metric kubernetes.io/anthos/container/uptime to replace deprecated alerts with metric kubernetes.io/anthos/up.

Fixes:

  • Added missing registry mirror package required for Cloud Audit Logs to the Registry Mirror.

  • Fixed issue with containerd not finding crictl due to /usr/local/bin not being in the SSH user's PATH.

  • Fixed flapping node readiness issues caused by an unhealthy Pod Lifecycle Event Generator (PLEG).

  • Fixed kernel support issue for Ubuntu 18.04 and 18.04.1 that prevented the anetd networking controller from working properly. Anthos clusters on bare metal release 1.9.0 works with all kernels supplied with supported distributions.

Known issues:

  • Control group v2 (cgroup v2) is not officially supported in Anthos clusters on bare metal release 1.9.0 and later. The presence of /sys/fs/cgroup/cgroup.controllers indicates that your system uses cgroup v2.

  • Anthos Service Mesh v1.10 is incompatible with Anthos clusters on bare metal release 1.9.0 running on Red Hat Enterprise Linux (RHEL) when SELinux is enabled. If you want to use Anthos Service Mesh, you must disable SELinux or set it to permissive mode on the host.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.