Google Distributed Cloud 1.8 release notes

This document lists production updates to Google Distributed Cloud. We recommend that Google Distributed Cloud developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

January 31, 2024

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

June 27, 2023

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

June 16, 2023

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

May 10, 2023

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

April 12, 2023

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

April 26, 2022

Security bulletin (all minor versions)

Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666 have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect Linux operating systems supported by Anthos clusters on bare metal. For instructions and more details, see the GCP-2022-014 security bulletin.

April 12, 2022

Security bulletin (1.8, 1.9, and 1.10)

A security vulnerability, CVE-2022-23648, has been discovered in containerd's handling of path traversal in the OCI image volume specification. Containers launched through containerd's CRI implementation with a specially-crafted image configuration could gain full read access to arbitrary files and directories on the host.

For more information, see the GCP-2022-013 security bulletin.

March 14, 2022

Release 1.8.9

Anthos clusters on bare metal 1.8.9 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.9 runs on Kubernetes 1.20.

Fixes:

Known issues:

When upgrading Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend that you upgrade to the highest patch version to ensure that you have the latest security fixes. Always review the release notes before upgrading so that you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 25, 2022

Security bulletin (1.8, 1.9, and 1.10)

Envoy recently released multiple security vulnerability fixes. The vulnerabilities affect Anthos clusters on bare metal, because Envoy is used for Metrics Server.

For instructions and more details, see the GCP-2022-008 security bulletin.

February 04, 2022

Security bulletin (all minor versions)

A security vulnerability, CVE-2021-4034, has been discovered in pkexec, a part of the Linux policy kit package (polkit), that allows an authenticated user to perform a privilege escalation attack. PolicyKit is generally used only on Linux desktop systems to allow non-root users to perform actions, such as rebooting the system, installing packages, restarting services, as governed by a policy.

For instructions and more details, see the GCP-2022-004 security bulletin.

February 01, 2022

Release 1.8.8

Anthos clusters on bare metal 1.8.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.8 runs on Kubernetes 1.20.

Fixes:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 14, 2021

Release 1.8.7

Anthos clusters on bare metal 1.8.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.7 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerability has been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 29, 2021

Release 1.8.6

Anthos clusters on bare metal 1.8.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.6 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 29, 2021

Security bulletin (all minor versions)

The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem.

For more information, see the GCP-2021-011 security bulletin.

October 21, 2021

Release 1.8.5

Anthos clusters on bare metal 1.8.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.5 runs on Kubernetes 1.20.

Security bulletin (all minor versions)

A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742. Ingress-nginx custom snippets allows retrieval of ingress-nginx service account tokens and secrets across all namespaces. For more information, see the GCP-2021-024 security bulletin.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

October 04, 2021

Security bulletin (all minor versions)

A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server.

For more information, see the GCP-2021-021 security bulletin.

September 21, 2021

Release 1.8.4

Anthos clusters on bare metal 1.8.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.4 runs on Kubernetes 1.20.

Fixes:

The following container image security vulnerabilities have been fixed: - CVE-2021-3711 - CVE-2021-3712 - CVE-2021-20305 - CVE-2021-33560

Known issues:

When you upgrade Anthos clusters on bare metal from a version with a security patch to the next minor release, we recommend you upgrade to the highest patch version to ensure you have the latest security fixes. Always review the release notes before upgrading so you're aware of what has changed, including security fixes and known issues. Upgrading to a lower release version isn't supported.

For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

September 20, 2021

Security bulletin (1.7 and 1.8)

A security issue was discovered in Kubernetes, CVE-2021-25741, where a user may be able to create a container with subpath volume mounts to access files and directories outside of the volume, including on the host filesystem. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal 1.7.x and 1.8.x releases, specifically 1.7.3 and earlier and 1.8.2 and earlier.

To fix this vulnerability, upgrade your Anthos clusters to version 1.7.4 or 1.8.3. For more information, see the GCP-2021-018 security bulletin.

August 27, 2021

Release 1.8.3

Anthos clusters on bare metal 1.8.3 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.3 runs on Kubernetes 1.20.

Breaking changes:

In Anthos clusters on bare metal release 1.8.0, we added a kernel version requirement for Ubuntu 18.04. We required a Linux kernel version of 4.17.0 or later. Anthos clusters on bare metal release 1.8.3 again supports all Linux kernel versions that ship with Ubuntu 18.04 and 20.04 distributions. As a result of this change, however, the egress NAT gateway feature that was provided for Preview in release 1.8.0 does not work with Anthos clusters on bare metal release 1.8.3.

Features:

  • Preview: Anthos Identity Service now works with Anthos clusters on bare metal to support LDAP authentication methods in addition to OIDC. You can use AIS with Microsoft Active Directory without the need for provisioning Active Directory Federation Services. For more information, see Setting up Anthos Identity Service with LDAP.

  • Preview: Anthos Metadata Agent replaces Stackdriver Metadata Collector and collects more accurate and usable metadata for Kubernetes resources. When you configure logging and monitoring, you need to enable the Config Monitoring for Ops API and grant the opsconfigmonitoring.resourceMetadata.writer IAM role to your logging-monitoring service account. If Anthos clusters on bare metal is installed behind a proxy, your proxy server must also allow connections to opsconfigmonitoring.googleapis.com.

  • Added preflight checks to verify that specific APIs are enabled for your Google Cloud project. Preflight checks return an error if any of the following APIs aren't enabled for your project:

    • anthos.googleapis.com
    • anthosaudit.googleapis.com
    • anthosgke.googleapis.com
    • cloudresourcemanager.googleapis.com
    • gkeconnect.googleapis.com
    • gkehub.googleapis.com
    • iam.googleapis.com
    • opsconfigmonitoring.googleapis.com
    • logging.googleapis.com
    • monitoring.googleapis.com
    • stackdriver.googleapis.com

    To enable these APIs when you create a cluster configuration file, use the --enable-apis flag with the bmctl create config command. For an example that uses the --enable-apis flag, see Create an admin cluster config with bmctl.

  • Added preflight checks for the following machine requirements:

    • Minimum supported Linux kernel version
    • Minimum required CPU
    • Minimum required RAM

Fixes:

  • Fixed the following container image security vulnerabilities:
  • Fixed cluster creation and cluster update failures for nodes running CentOS or Red Hat Enterprise Linux (RHEL) with both SELinux and Cloud Audit Logs enabled.
  • Fixed Transmission Control Protocol (TCP) connection leakage issue.
  • Fixed an issue that prevented cert-manager from issuing ACME certificates over HTTP due to ImagePullBackOff errors.

Changes:

  • The Kubevirt version used for working with VM-based workloads is now v0.43.0-gke.3.
  • The bootstrap cluster is deleted when a cluster upgrade completes without errors.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

July 29, 2021

Release 1.8.2

Anthos clusters on bare metal 1.8.2 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.2 runs on Kubernetes 1.20.

Features:

  • Preview: Added capability to rotate cluster certificate authorities (CAs) for user clusters. For instructions on using the bmctl cluster credentials command to rotate cluster CAs, see Rotate user cluster certificate authority.

  • Preview: Added support for AppArmor with Anthos clusters on bare metal. You don't need to disable AppArmor on Ubuntu as a prerequisite for installation. When you create new 1.8.2 clusters or upgrade clusters to version 1.8.2, you can enable AppArmor either before or after you upgrade.

Fixes:

  • Fixed CVE-2021-3520 vulnerability related to a flaw in lz4, which provides support for LZ4 a lossless compression algorithm. The flaw impacts availability, but has potential to impact confidentiality and integrity as well.

  • Fixed bmctl operation failures that occur for some Ubuntu 20.04 LTS distributions with a more recent Linux kernel, including Ubuntu 20.04 LTS images on the 5.8 kernel. For more information about this issue and a workaround, see Ubuntu 20.04 LTS and bmctl.

  • Fixed OpenStack support for user clusters. In prior releases, cluster creation fails for user type clusters when the baremetal.cluster.gke.io/external-cloud-provider: "true" annotation is added to the cluster configuration file.

  • Fixed PATH environment issues for executing commands as a non-root user. For more information, see Known Issues.

  • Fixed an issue that caused user cluster resets (bmctl reset cluster) to get stuck while deleting namespaces.

  • Fixed out-of-memory (OOM) conditions related to Connect Agent memory usage that resulted in pod failures.

  • Fixed issue that blocked snapshots for clusters configured for passwordless SUDO capability for machine login (nodeAccess.loginUser: <login user name>).

  • Fixed issue that blocked some 1.7.x version admin, hybrid, or standalone clusters from upgrading to the 1.8 minor release. This issue affected some clusters that were updated by applying changes from an updated cluster configuration file.

  • Fixed Address Resolution Protocol (ARP) table issue for high-availability (HA) deployments that blocked upgrades from completing.

Functionality changes:

  • Expanded snapshots to include resource usage metrics to improve troubleshooting and support. Added metrics include the output of ip neigh, kubectl top nodes, and kubectl top pods commands.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

July 02, 2021

Release 1.8.1

Anthos clusters on bare metal release 1.8.1 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos on bare metal 1.8.1 runs on Kubernetes 1.20.

Fixes:

Fixed CVE-2021-34824 that could expose private keys and certificates from Kubernetes secrets through the credentialName field when using Gateway or DestinationRule. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.8.0. For more information, see the GCP-2021-012 security bulletin.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 30, 2021

Security bulletin (1.8)

The Istio project recently announced a security vulnerability (CVE-2021-34824) where credentials specified in the credentialName field for Gateway or DestinationRule can be accessed from different namespaces. This vulnerability affects all clusters created or upgraded with Anthos clusters on bare metal release 1.8.0. For more information, see the GCP-2021-012 security bulletin.

June 21, 2021

Release 1.8.0

Anthos clusters on bare metal release 1.8.0 is now available. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.8.0 runs on Kubernetes 1.20.

Extended installation support:

  • Provided support to use containerd as the container runtime as GA for Anthos clusters on bare metal release 1.8.0. Cluster upgrades to 1.8.0 are blocked for 1.7.x clusters that are configured to use the preview containerd capability. For more information, see Upgrading 1.7.x clusters that use containerd in Known Issues.
  • Preview: Improved virtual machine (VM) management capability. Anthos VM Runtime uses KubeVirt to orchestrate VMs on clusters, allowing you to work with your VM-based apps and workloads in a uniform development environment. Anthos VM Runtime has worked with Anthos clusters on bare metal as a preview feature since November 2020 and we have continued to enhance its capability. For more information, see Working with VM-based workloads.
  • Added edge profile support for standalone clusters. The edge profile is recommended for edge devices with limited resources. Add profile: edge to the cluster config file when you create a standalone cluster to produce a cluster that has significantly reduced system resource requirements. The edge profile is only available for standalone clusters, it is ignored for other cluster types. For more information, see Creating standalone clusters.
  • Added support to specify provider ID for Nodes (controlPlane.nodePoolSpec.nodes.providerID) to support deploying on OpenStack using Load Balancing as a Service (LBaaS) resources. For more information, see Configure your clusters to use OpenStack.
  • Preview: Added support for installing Anthos clusters on bare metal, using your own registry service, instead of gcr.io. For instructions and additional information, see Installing Anthos Bare Metal using registry mirror.

Improved upgrade:

  • Enabled support for upgrading non-SELinux clusters to SELinux. For more information, see Enable SELinux in Upgrading Anthos clusters on bare metal.
  • Cluster upgrades are not blocked by excessive Node draining durations. During a cluster upgrade, if the draining process takes longer than 20 minutes for any specific Node, the upgrade process will carry on without waiting for draining to complete.

Updated user cluster lifecycle management:

  • Added bmctl improvements for resetting user cluster and adding additional preflight checks to confirm machine and network readiness for cluster creation:

Enhanced monitoring and logging:

  • Preview: Added Cloud Audit Logging capability, which enables audit logs to be written to Cloud Audit Logs in your Google project. Audit logs are useful for investigating suspicious API requests and for collecting statistics. For more information, see Enable Audit Logging.

Introduced new networking capabilities in preview:

  • Preview: Added multi-NIC capability to provide additional interfaces to your Pods.
  • Preview: Added egress NAT gateway capability to provide persistent, deterministic routing for the egress traffic from your clusters. For more information, see Configure an egress NAT gateway for external communication.
  • Preview: Added option for BGP bundled load balancer for Layer-3 (L3) topologies. This feature can be used with user clusters and admin clusters.

Enhanced security:

  • Workload Identity is GA. The Connect Agent Service Account Key is no longer required during installation. Connect Agent uses Workload Identity to authenticate to Google Cloud instead of an exported Google Cloud Service Account Key.

Expanded support for newer versions of operating systems:

  • Added support for installing Anthos clusters on bare metal on Red Hat Enterprise Linux (RHEL) 8.4, and CentOS 8.4

Functionality changes:

  • Added --workspace-dir flag to bmctl to allow changing the path and name of the workspace directory from the default bmctl-workspace. The workspace directory contains the configuration and log files generated by bmctl. When using the bmctl command, pass in a --workspace-dir flag to specify a non-default workspace directory location. If the directory does not exist, bmctl will create it for you.
  • Moved away from iptables-based NodePort and masquerade handling to eBPF-based management. NodePort and masquerade handling are now applied to the Node IP and default gateway interfaces only.

Fixes:

  • Resolved, as part of the GA support for using containerd as the container runtime, incorrect cgroup driver use. Newly created 1.8.0 clusters that are configured to use containerd will use the correct systemd cgroup driver.
  • Fixed issue that prevented usage metrics for the containerd process from being collected by Cloud Logging. This fix applies to newly created 1.8.0 clusters only.

Known issues:

  • If a Node is out of reach, Anthos clusters on bare metal can't start the draining process, which may impact the cluster upgrade process. For more information, see Node draining can't start when Node is out of reach.
  • Upgrading from 1.7.x clusters that use containerd as the container runtime to 1.8.0 is blocked. For more information, see Upgrading 1.7.x clusters that use containerd.
  • When running Anthos clusters on bare metal with firewalld enabled on either CentOS or Red Hat Enterprise Linux (RHEL), changes to firewalld can remove the Cilium iptables chains on the host network. The loss of the Cilium iptables chains causes the Pod on the Node to lose network connectivity outside of the Node. for more information, see Modifying firewalld will erase Cilium iptable chains.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.