Google Distributed Cloud 1.10 release notes

This document lists production updates to Google Distributed Cloud. We recommend that Google Distributed Cloud developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

January 31, 2024

Security bulletin (all minor versions)

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

June 27, 2023

Security bulletin (all minor versions)

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

June 16, 2023

Security bulletin (all minor versions)

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin.

May 10, 2023

CentOS Linux 8 Support Deprecated

CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.

April 12, 2023

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

August 23, 2022

Release 1.10.8

Anthos clusters on bare metal 1.10.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.8 runs on Kubernetes 1.21.

Fixes

The following container image security vulnerability has been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

August 01, 2022

Release 1.10.7

Anthos clusters on bare metal 1.10.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.7 runs on Kubernetes 1.21.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

July 07, 2022

Release 1.10.6

Anthos clusters on bare metal 1.10.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.6 runs on Kubernetes 1.21.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

June 02, 2022

Release 1.10.5

Anthos clusters on bare metal 1.10.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.5 runs on Kubernetes 1.21.

Fixes:

The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

May 04, 2022

Release 1.10.4

Anthos clusters on bare metal 1.10.4 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.10.4 runs on Kubernetes 1.21.

Fixes:

  • The following container image security vulnerabilities have been fixed:

  • Role-based access control (RBAC) fixes:

    • Set AutomountServiceAccountToken field for Node Problem Detector jobs and etcd-defrag Daemonsets to false.

    • Set capi-kubeadm-bootstrap-controller-manager to use a dedicated service account.

    • Scoped down configmap/(get, list, watch) permissions to metallb-config resource name.

    • Scoped down configmap/get permission to core-dns-autoscaler resource name.

    • Removed services.update permission for the MetalLB kube-system:controller role.

    • anetd

      • Removed Cilium service account and replaced it with the account used by kubelet.

      • Removed pod and node access from Cilium cluster role.

      • Added Cilium cluster role to the kubelet service account.

      • Removed pods/(delete) role from cilium-operator cluster role.

      • Scoped down leases permissions in cilium-operator cluster role to cilium-operator-resource-lock resource name and kube-controller-manager resource name.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

April 26, 2022

Security bulletin (all minor versions)

Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666 have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect Linux operating systems supported by Anthos clusters on bare metal. For instructions and more details, see the GCP-2022-014 security bulletin.

April 12, 2022

Security bulletin (1.8, 1.9, and 1.10)

A security vulnerability, CVE-2022-23648, has been discovered in containerd's handling of path traversal in the OCI image volume specification. Containers launched through containerd's CRI implementation with a specially-crafted image configuration could gain full read access to arbitrary files and directories on the host.

For more information, see the GCP-2022-013 security bulletin.

March 31, 2022

Release 1.10.3

Anthos clusters on bare metal 1.10.3 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.10.3 runs on Kubernetes 1.21.

Fixes:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 25, 2022

Release 1.10.2

Anthos clusters on bare metal 1.10.2 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.10.2 runs on Kubernetes 1.21.

Security bulletin (1.8, 1.9, and 1.10)

Envoy recently released multiple security vulnerability fixes. The vulnerabilities affect Anthos clusters on bare metal, because Envoy is used for Metrics Server.

For instructions and more details, see the GCP-2022-008 security bulletin.

Functionality changes:

  • A preflight check now verifies whether your node machine has enough disk space before starting an install.

  • Updated the bmctl check cluster --snapshot command so that snapshots now capture information about pods in cluster namespaces.

  • Updated the bmctl check cluster --snapshot command so that snapshots now capture information about cluster API machines and kubeadmin Secrets.

Fixes:

  • Fixed issue in which the edge profile's request to reserve resources is lost during the upgrade process.

  • Fixed bmctl upgrade command so that the log file upgrade-cluster.log is generated in the bmctl-workspace/cluster/logs directory.

  • Fixed issue in which the non-root login didn't have the proper permissions to perform bmctl backup or bmctl restore.

  • Fixed a Node Problem Detector service that sometimes failed to run on nodes after a cluster installation or upgrade.

  • The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

February 04, 2022

Security bulletin (all minor versions)

A security vulnerability, CVE-2021-4034, has been discovered in pkexec, a part of the Linux policy kit package (polkit), that allows an authenticated user to perform a privilege escalation attack. PolicyKit is generally used only on Linux desktop systems to allow non-root users to perform actions, such as rebooting the system, installing packages, restarting services, as governed by a policy.

For instructions and more details, see the GCP-2022-004 security bulletin.

January 27, 2022

Release 1.10.1

Anthos clusters on bare metal 1.10.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.1 runs on Kubernetes 1.21.

Fixes:

  • Fixed PreflightCheck to allow the preflightCheck.Spec.ConfigYAML field to be empty.

  • Fixed PreflightCheck to allow an existing GKE Hub membership, if the cluster already exists.

  • Fixed issue that blocked access to external Virtual IP addresses of Services, such as a Load Balancer, when Flat IPv4 is enabled.

  • Fixed issue in which the use of –nodes/ and –node-ssh-key flags when taking an admin-less snapshot of a cluster resulted in an empty snapshot.

  • Fixed issue that caused installation of version 1.10.0 clusters to fail when the umask setting for the root user on the target machine wasn't 0022. For more information, see Failure on systems with restrictive umask setting.

  • Fixed issue in which BGP load balancer preflight checks failed if the Kubernetes interface had a period ('.') in the name. (For example, VLAN interfaces often have names such as eth0.1).

  • The following container image security vulnerabilities have been fixed:

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

December 10, 2021

Release 1.10.0

Anthos clusters on bare metal 1.10.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.0 runs on Kubernetes 1.21.

Improved cluster lifecycle functionalities:

  • GA: Enabled Node Problem Detector to run by default on all nodes. You can check if a problem was detected on a node by running the kubectl describe command for the node. Then look for NodeConditions or Events reported by Node Problem Detector.

  • GA: Added bmctl backup cluster and bmctl restore cluster commands to facilitate disaster recovery for clusters.

  • Preview: Added the ability to reset individual nodes using the SSH key.

  • Updated the bmctl check cluster command so that the snapshot of a cluster includes the cluster's YAML file and logs that are in the bmctl-workspace directory.

  • Added a new status field cluster.gkeHubRegistrationStatus. The command kubectl get cluster now shows information about the cluster's membership to GKE Hub.

Networking:

  • Preview: Enabled Anthos multi-cluster connectivity to provide Anthos clusters a way to connect to another Anthos cluster in the same data center (intra-site, cluster-to-cluster). Pods in connected clusters can reach each other over pod IP addresses without using native address translation (NAT) in between.

  • Preview: Enabled IPv4/IPv6 dual stack support. Customers can deploy clusters in a dual-stack network, where IPv4 and IPv6 addresses can be assigned to both nodes and pods.

  • Preview: Enabled "flat mode" (a simplified network topology) for IPv4 , where the pod's IPv4 address is visible and routable without masquerading as node IP within the same Layer 2 domain.

  • Preview: Enabled SR-IOV. This feature lets you configure Virtual Functions (VFs) on the supported devices on the nodes of their cluster. This feature also lets you define the kernel module you want to bind to the VF.

Observability:

  • GA: Added ability to show severity level of an issue in Cloud Logging. Severity level is extracted from containerd and kubelet node logs.

  • GA: Changed collection of application metrics to use a more scalable monitoring pipeline based on OpenTelemetry. This change significantly reduces the amount of resources required to collect metrics.

Security:

  • GA: Enhanced the capability to rotate cluster certificate authorities (CAs). Updates include support for all cluster types, rotation of front-proxy and etcd CAs, and changes to the bmctl command syntax.

  • Preview: Enabled installation of Anthos clusters on bare metal using a short-lived Google Service Account token instead of using Google Service Account keys.

  • Enabled Kubernetes control plane and most Anthos system containers to run as non-root users. For details, see Don't run containers as root user.

VM Runtime:

  • Preview: Supported enabling or disabling Anthos VM Runtime on user clusters.

  • Preview: Enabled Anthos VM Runtime to support QEMU Copy On Write (QCOW2) format, which is a storage format for virtual disks on virtual machines. Some benefits of virtual disk capabilities are independent thin provisioning, better compression, and encryption at rest.

  • Preview: Enabled VMRuntime custom resource and the Network custom resource, which let you create VMs on either the node network with a static IP address or the default pod network.

  • Preview: Enabled VM pods audit logs for VM runtime resources.

  • Preview: Expanded guest OS versions that can run on the virtual machine. We support Windows Server 2019, 2016, Windows 10, Red Hat Enterprise Linux (RHEL) 8, Centos 8, and Ubuntu 20.04 as guest OS.

  • Preview: Enabled virtual machine high availability to provide greater uptime for virtual machines instances (VMIs) by automatically detecting and recovering from a range of host machine failures.

Breaking changes:

The gateway capability used by the egress NAT gateway and Bundled load balancing with BGP Preview features have changed in this release. The NetworkGatewayGroup custom resource replaces AnthosNetworkGateway and the capability is enabled with a new advancedNetworking field in the cluster configuration file, instead of an annotation. These changes affect the ability to upgrade clusters that use earlier versions of the features.

Anthos clusters on bare metal blocks cluster upgrades from version 1.9 to version 1.10 for clusters that use either of these two advanced networking features. You can upgrade a version 1.9 admin cluster that is managing 1.9 user clusters that use these features to version 1.10, but object reconciliation breaks for the AnthosNetworkGateway custom resource. Object reconciliation is the mechanism whereby admin clusters automatically copy/restore objects on managed user clusters when the objects have been defined alongside the cluster configuration. Any AnthosNetworkGateway custom resources are still functional and can be modified with kubectl.

To bring a version 1.9 cluster that uses either advanced networking Preview feature up to version 1.10, reset or delete the cluster and create a new 1.10 cluster.

Preview features and products are subject to change and are provided for testing and evaluation purposes only. Do not use Preview features on your production clusters.

Functionality changes:

  • Enabled use of ADMIN_KUBECONFIG environment variable to reduce the number of bmctl command flags.

  • The cluster reconciliation process now checks for differences in the GKEHub membership before attempting to update it. If the GKEHub membership needs to be changed, the cluster is unregistered and then re-registered.

  • The advancedNetworking field in the cluster configuration file replaces the deprecated baremetal.cluster.gke.io/enable-anthos-network-gateway annotation for enabling advanced networking capabilities.

  • The NetworkGatewayGroup custom resource replaces the AnthosNetworkGateway custom resource.

Fixed cluster lifecycle functionalities:

  • Outputs from all bmctl commands except bmctl version are now written to log files.

  • Fixed strict mode for decoding the cluster YAML file. Extraneous information in the cluster YAML file now results in an error.

  • Fixed preflight check so that it no longer ignores the no_proxy setting.

  • Binaries in cluster provision no longer run from /tmp, which is often mounted with noexec options. This change fixes a preflight check "permission denied" error.

  • Switched the default server-side containerRuntime value from docker to containerd.

Observability:

  • Increased the priority of the kube-state-metrics service to keep it from being stuck in a pending state. This service generates metrics about Kubernetes API objects such as deployments, nodes, and pods.

  • Upgraded metrics-server to version 0.3.6 to fix a missing metrics issue that occurs when a duplicated pod name is present.

Known issues:

For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.