Google Distributed Cloud 1.11 release notes

This document lists production updates to Google Distributed Cloud. We recommend that Google Distributed Cloud developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

January 31, 2024

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

June 27, 2023

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

June 16, 2023

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

May 10, 2023

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

April 12, 2023

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

November 18, 2022

Release 1.11.8

Anthos clusters on bare metal 1.11.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.8 runs on Kubernetes 1.22.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

November 07, 2022

Security bulletin (1.11, 1.12, and 1.13)

A security vulnerability, CVE-2022-39278, has been discovered in Istio, which is used in Anthos Service Mesh, that allows a malicious attacker to crash the control plane.

For instructions and more details, see the Anthos clusters on bare metal security bulletin.

October 28, 2022

Anthos clusters on bare metal 1.11.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.7 runs on Kubernetes 1.22.

Known issues: For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

September 26, 2022

Release 1.11.6

Anthos clusters on bare metal 1.11.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.6 runs on Kubernetes 1.22.

Fixes:

  • Updated the container image to resolve a yaml text/template vulnerability.

  • The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 30, 2022

Release 1.11.5

Anthos clusters on bare metal 1.11.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.5 runs on Kubernetes 1.22.

Fixes:

  • Increased the default storage size limit of etcd to 6 GiB.

  • The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 04, 2022

Release 1.11.4

Anthos clusters on bare metal 1.11.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.4 runs on Kubernetes 1.22.

Fixes:

  • Fixed issue in which cluster restores failed when /var/lib/etcd is a mount point.
  • Fixed issue in which attempts to skip minor versions when upgrading weren't blocked. For details about the upgrade policy, see Minor version upgrades.

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 23, 2022

Release 1.11.3

Anthos clusters on bare metal 1.11.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.3 runs on Kubernetes 1.22.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

May 26, 2022

Release 1.11.2

Anthos clusters on bare metal 1.11.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.11.2 runs on Kubernetes 1.22.

Starting with Anthos clusters on bare metal release 1.11.2, you can enable or disable Anthos VM Runtime by updating the VMRuntime custom resource only. The legacy spec.kubevirt settings in the cluster configuration are no longer supported. The VMRuntime custom resource is installed by default on version 1.10 and later hybrid, standalone, and user clusters. The VMRuntime custom resource can't be applied to admin clusters.

If you have Anthos VM Runtime enabled for your Anthos clusters on bare metal, you must disable it before upgrading clusters to version 1.11.2 or higher. If this step is not completed, your cluster upgrade will fail. You can re-enable Anthos VM Runtime after the upgrade is complete.

Starting with Anthos clusters on bare metal release 1.11.2, the Anthos VM Runtime API version has changed from v1alpha1 to v1. This version change doesn't affect the VMRuntime custom resource, but most other resources are affected.

Functionality changes:

  • The containerd runtime has been upgraded to 1.5.11-gke.0 to address CVE-2022-24769

  • Added a preflight check that disallows Ubuntu 18.04 distributions with 4.15.x Linux kernels.

Fixes:

  • Fixed cluster custom resource status reporting for pending reconciliations.

  • Fixed a bmctl check cluster command issue that caused the user cluster kubeconfig Secret to be overwritten.

  • Fixed an issue with manifest installation when last-applied-config is broken that caused upgrades to fail.

  • Fixed an issue to ensure that the 20-minute timeout for node draining is enforced during cluster upgrades. This timeout provides ample time for nodes to drain, but ensures that upgrades can always proceed.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

May 02, 2022

Release 1.11.1

Anthos clusters on bare metal 1.11.1 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.11.1 runs on Kubernetes 1.22.

Fixes:

  • Resolved cluster installation issue in which cluster status is prematurely declared ready, resulting in a "Failed to wait for applied resources" error.

  • Added validation that a cluster's kubeconfig secret data is correct.

  • Added feature so that bmctl outputs line numbers of relevant yaml when a parsing error occurs.

  • Removed the misleading log "Waiting for pod to finish" on pods such as anetd that aren't meant to finish.

  • Added automatic inclusion of a control plane's virtual IP address to the cluster NO_PROXY list.

  • Role-based access control fixes:

    • Set AutomountServiceAccountToken field for Node Problem Detector jobs to false.

    • Set capi-kubeadm-bootstrap-controller-manager to use a dedicated service account.

    • Scoped down deployment/(update,patch) permissions to the metrics-server resource name.

    • Scoped down configmap/(get, list, watch) permissions to metallb-config resource name.

    • anetd:

    • Removed Cilium service account and replaced it with the account used by kubelet.

    • Removed pod and node access from Cilium cluster role.

    • Added Cilium cluster role to the kubelet service account.

    • Removed pods/(delete) role from cilium-operator cluster role.

    • Scoped down leases permissions in cilium-operator cluster role to cilium-operator-resource-lock resource name and kube-controller-manager resource name.

  • The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 26, 2022

Security bulletin (all minor versions)

Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666 have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect Linux operating systems supported by Anthos clusters on bare metal. For instructions and more details, see the GCP-2022-014 security bulletin.

March 31, 2022

Release 1.11.0

Anthos clusters on bare metal 1.11.0 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.11.0 runs on Kubernetes 1.22.

Containerd is the default runtime in Anthos clusters on bare metal. Support for Docker as a container runtime on Kubernetes nodes will be removed from Anthos clusters on bare metal starting with version 1.13.0. If you use a node image based on Docker container runtime, please migrate your workloads to a Containerd node image as soon as possible. For more details, see Containerd node images.

The structure of the Anthos clusters on bare metal documentation is substantially different from previous versions. For details, see New documentation structure.

Kubernetes 1.22 has deprecated certain APIs, and a list of these deprecated APIs can be found in Kubernetes 1.22 deprecated APIs. In their manifests and API clients, customers need to replace references to the deprecated APIs with references to the newer API calls. For more information, see Deprecated API Migration Guide.

On January 31, 2022, CentOS 8 reached its end of life (EOL). As a result of the EOL, yum repositories stopped working for CentOS, which causes cluster creation and cluster upgrade operations to fail. For a workaround and more information, see Cluster creation or upgrades fail on CentOS.

Improved cluster lifecycle functionalities:

  • Upgraded Anthos clusters on bare metal to use Kubernetes version 1.22.

  • Updated cert-manager to version 1.5.4.

  • Added error messaging in the bmctl command line interface to better surface cluster installation or upgrade failure.

  • Incorporated audit logs into bmctl snapshots.

  • Added ability for registry mirror users to customize containerd configuration and have it automatically mirror public registry hosts other than gcr.io.

  • Changed bmctl update command so that it extracts manifests before updating a cluster.

  • Added feature so that a cluster kubeconfig file automatically renews when the cluster is upgraded and the kubeconfig Secret is renewed whenever cluster reconciliation takes place.

  • Added support for Red Hat Enterprise Linux (RHEL) and CentOS 8.5.

  • Added warning to bmctl command that docker containerRuntime will not be supported in version 1.13 of Anthos cluster on bare metal.

  • Added support for specifying CIDR blocks in the NoProxy section of the cluster's configuration file.

  • Added Service CIDR to NoProxy section of a cluster configuration file by default in order to fix a multinic in proxy environment issue.

  • Fixed a multi-NIC in proxy environment issue. Whenever the NO_PROXY environment variable is set, it includes the Service CIDR from the cluster specification.

Networking:

  • GA: Added egress Network Address Translation (NAT) gateway capability to provide persistent, deterministic routing for egress traffic from clusters. For more information, see Configure an egress NAT gateway for external communication.

  • GA: Added option for BGP bundled load balancer which advertises Load Balancer (LB) Virtual IP addresses (VIPs) to the network using the Border Gateway Protocol (BGP). This feature supports topologies across multiple subnets and can provide greater load-balancing bandwidth than bundled Layer 2 mode.

  • GA: Enabled SR-IOV. This feature allows you to configure Virtual Functions (VFs) on the supported devices on the nodes of their cluster. It also allows you to define the kernel module you want to bind to the VF.

  • GA: Enabled IPv4/IPv6 dual-stack support. Clusters can be deployed in a dual-stack network in which IPv4 and IPv6 addresses are assigned to both nodes and pods. By default, IPv4 is in island mode and IPv6 is in flat mode (a simplified network topology).

  • GA: Enabled static flat network (without BGP). This feature lets you configure a flat mode network for IPv4 addresses. A pod's IPv4 address is visible and routable within the same Layer 2 domain, without having to masquerade as the node's IP address.

  • Preview: Enabled Dynamic Flat IP with Border Gateway Protocol (BGP) support. This feature lets you configure flat mode using BGP in clusters with the help of Anthos Network Gateway and BGP. In this mode, the pod's IP address is visible and routable without masquerading across multiple subdomains. Currently supports advertising IPv4 and IPv6 routes over IPv4 sessions.

  • Fixed issue in which new MAC addresses of re-imaged nodes weren't updated.

Observability:

  • GA: Enabled collection of multiple network interfaces (multinic) logs from clusters. Logs are collected as system logs and are sent to Cloud Logging without charge to the customer.

  • Preview: Added Summary API metrics. These metrics provide CPU, memory, and storage statistics about pods, containers, and nodes.

  • Updated fluent-bit (stackdriver-log-forwarder) cri parser to avoid matching time fields multiple times.

  • Upgraded kube-state-metrics from version 1.9 to 2.4. This service generates metrics about Kubernetes API objects such as deployments, nodes, and pods.

  • Upgraded Metric Server from version 0.3.6 to 0.4.5. Metrics Server retrieves metrics from kubelets and exposes them through the Kubernetes Metrics API.

Security:

  • Preview: Added secure computing mode (seccomp) support. Running containers with a seccomp profile improves the security of a cluster because it restricts the system calls that containers are allowed to make to the kernel.

  • Added ability to disable rootless mode for system containers. Since version 1.10.0, Kubernetes control planes and Anthos clusters on bare metal system containers run as non-root containers by default.

  • Fixed CA rotation issues by increasing the ca-rotation timeout for admin clusters. While verifying that a static pod has been restarted after manifest update, the current hash is retrieved before the manifest changes are applied.

Known issues:

  • Deprecated metrics

    Several Anthos metrics have been deprecated and, starting with this release, data is no longer collected for these deprecated metrics. If you use these metrics in any of your alerting policies, there won't be any data to trigger the alerting condition. For more information, including instructions to migrate to updated replacement metrics, see Deprecated metrics affects Cloud Monitoring dashboard in Known Issues.

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.