This document lists production updates to Google Distributed Cloud. We recommend that Google Distributed Cloud developers periodically check this list for any new announcements.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.
January 31, 2024
Security bulletin (all minor versions)
A security vulnerability, CVE-2024-21626, has been discovered in runc
where a user with permission to create Pods might be able to gain full access to the node filesystem.
For instructions and more details, see the GCP-2024-005 security bulletin.
June 27, 2023
Security bulletin (all minor versions)
A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.
For more information, see the GCP-2023-016 security bulletin.
June 16, 2023
Security bulletin (all minor versions)
Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).
For more information, see the GCP-2023-014 security bulletin.
May 10, 2023
CentOS Linux 8 Support Deprecated
CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.
April 12, 2023
Kubernetes image registry redirect
As of March 21, 2023, traffic to k8s.gcr.io
is redirected to registry.k8s.io
, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.
To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.
August 23, 2022
Release 1.10.8
Anthos clusters on bare metal 1.10.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.8 runs on Kubernetes 1.21.
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
August 01, 2022
Release 1.10.7
Anthos clusters on bare metal 1.10.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.7 runs on Kubernetes 1.21.
Fixed a
CrashLoopBackOff
error generated bygke-metrics-agent
when application metrics are enabled (that is, whenenableStackdriverForApplications=true
).The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
July 07, 2022
Release 1.10.6
Anthos clusters on bare metal 1.10.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.6 runs on Kubernetes 1.21.
Fixes:
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
June 02, 2022
Release 1.10.5
Anthos clusters on bare metal 1.10.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.5 runs on Kubernetes 1.21.
Fixes:
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
May 04, 2022
Release 1.10.4
Anthos clusters on bare metal 1.10.4 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.10.4 runs on Kubernetes 1.21.
Fixes:
The following container image security vulnerabilities have been fixed:
Role-based access control (RBAC) fixes:
Set
AutomountServiceAccountToken
field for Node Problem Detector jobs andetcd-defrag
Daemonsets to false.Set
capi-kubeadm-bootstrap-controller-manager
to use a dedicated service account.Scoped down
configmap/(get, list, watch)
permissions tometallb-config
resource name.Scoped down
configmap/get
permission tocore-dns-autoscaler
resource name.Removed
services.update
permission for the MetalLBkube-system:controller
role.anetd
Removed Cilium service account and replaced it with the account used by
kubelet
.Removed pod and node access from Cilium cluster role.
Added Cilium cluster role to the
kubelet
service account.Removed
pods/(delete)
role fromcilium-operator
cluster role.Scoped down leases permissions in
cilium-operator
cluster role tocilium-operator-resource-lock
resource name andkube-controller-manager
resource name.
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
April 26, 2022
Security bulletin (all minor versions)
Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666 have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect Linux operating systems supported by Anthos clusters on bare metal. For instructions and more details, see the GCP-2022-014 security bulletin.
April 12, 2022
Security bulletin (1.8, 1.9, and 1.10)
A security vulnerability, CVE-2022-23648, has been discovered in containerd's handling of path traversal in the OCI image volume specification. Containers launched through containerd's CRI implementation with a specially-crafted image configuration could gain full read access to arbitrary files and directories on the host.
For more information, see the GCP-2022-013 security bulletin.
March 31, 2022
Release 1.10.3
Anthos clusters on bare metal 1.10.3 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.10.3 runs on Kubernetes 1.21.
Fixes:
- The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
February 25, 2022
Release 1.10.2
Anthos clusters on bare metal 1.10.2 is now available for download. To upgrade, see Upgrade Anthos on bare metal. Anthos clusters on bare metal 1.10.2 runs on Kubernetes 1.21.
Security bulletin (1.8, 1.9, and 1.10)
Envoy recently released multiple security vulnerability fixes. The vulnerabilities affect Anthos clusters on bare metal, because Envoy is used for Metrics Server.
For instructions and more details, see the GCP-2022-008 security bulletin.
Functionality changes:
A preflight check now verifies whether your node machine has enough disk space before starting an install.
Updated the
bmctl check cluster --snapshot
command so that snapshots now capture information about pods in cluster namespaces.Updated the
bmctl check cluster --snapshot
command so that snapshots now capture information about cluster API machines andkubeadmin
Secrets.
Fixes:
Fixed issue in which the edge profile's request to reserve resources is lost during the upgrade process.
Fixed
bmctl upgrade
command so that the log fileupgrade-cluster.log
is generated in thebmctl-workspace/cluster/logs
directory.Fixed issue in which the non-root login didn't have the proper permissions to perform
bmctl backup
orbmctl restore
.Fixed a Node Problem Detector service that sometimes failed to run on nodes after a cluster installation or upgrade.
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
February 04, 2022
Security bulletin (all minor versions)
A security vulnerability, CVE-2021-4034, has been discovered in pkexec
, a part of the Linux policy kit package (polkit), that allows an authenticated user to perform a privilege escalation attack. PolicyKit is generally used only on Linux desktop systems to allow non-root users to perform actions, such as rebooting the system, installing packages, restarting services, as governed by a policy.
For instructions and more details, see the GCP-2022-004 security bulletin.
January 27, 2022
Release 1.10.1
Anthos clusters on bare metal 1.10.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.1 runs on Kubernetes 1.21.
Fixes:
Fixed PreflightCheck to allow the
preflightCheck.Spec.ConfigYAML
field to be empty.Fixed PreflightCheck to allow an existing GKE Hub membership, if the cluster already exists.
Fixed issue that blocked access to external Virtual IP addresses of Services, such as a Load Balancer, when Flat IPv4 is enabled.
Fixed issue in which the use of
–nodes/
and–node-ssh-key
flags when taking an admin-less snapshot of a cluster resulted in an empty snapshot.Fixed issue that caused installation of version 1.10.0 clusters to fail when the
umask
setting for theroot
user on the target machine wasn't0022
. For more information, see Failure on systems with restrictive umask setting.Fixed issue in which BGP load balancer preflight checks failed if the Kubernetes interface had a period ('.') in the name. (For example, VLAN interfaces often have names such as
eth0.1
).The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
December 10, 2021
Release 1.10.0
Anthos clusters on bare metal 1.10.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.10.0 runs on Kubernetes 1.21.
Improved cluster lifecycle functionalities:
GA: Enabled Node Problem Detector to run by default on all nodes. You can check if a problem was detected on a node by running the
kubectl describe
command for the node. Then look forNodeConditions
orEvents
reported by Node Problem Detector.GA: Added
bmctl backup cluster
andbmctl restore cluster
commands to facilitate disaster recovery for clusters.Preview: Added the ability to reset individual nodes using the SSH key.
Updated the
bmctl check cluster
command so that the snapshot of a cluster includes the cluster's YAML file and logs that are in thebmctl-workspace
directory.Added a new status field
cluster.gkeHubRegistrationStatus
. The commandkubectl get cluster
now shows information about the cluster's membership to GKE Hub.
Networking:
Preview: Enabled Anthos multi-cluster connectivity to provide Anthos clusters a way to connect to another Anthos cluster in the same data center (intra-site, cluster-to-cluster). Pods in connected clusters can reach each other over pod IP addresses without using native address translation (NAT) in between.
Preview: Enabled IPv4/IPv6 dual stack support. Customers can deploy clusters in a dual-stack network, where IPv4 and IPv6 addresses can be assigned to both nodes and pods.
Preview: Enabled "flat mode" (a simplified network topology) for IPv4 , where the pod's IPv4 address is visible and routable without masquerading as node IP within the same Layer 2 domain.
Preview: Enabled SR-IOV. This feature lets you configure Virtual Functions (VFs) on the supported devices on the nodes of their cluster. This feature also lets you define the kernel module you want to bind to the VF.
Observability:
GA: Added ability to show severity level of an issue in Cloud Logging. Severity level is extracted from
containerd
andkubelet
node logs.GA: Changed collection of application metrics to use a more scalable monitoring pipeline based on OpenTelemetry. This change significantly reduces the amount of resources required to collect metrics.
Security:
GA: Enhanced the capability to rotate cluster certificate authorities (CAs). Updates include support for all cluster types, rotation of front-proxy and
etcd
CAs, and changes to thebmctl
command syntax.Preview: Enabled installation of Anthos clusters on bare metal using a short-lived Google Service Account token instead of using Google Service Account keys.
Enabled Kubernetes control plane and most Anthos system containers to run as non-root users. For details, see Don't run containers as root user.
VM Runtime:
Preview: Supported enabling or disabling Anthos VM Runtime on user clusters.
Preview: Enabled Anthos VM Runtime to support QEMU Copy On Write (QCOW2) format, which is a storage format for virtual disks on virtual machines. Some benefits of virtual disk capabilities are independent thin provisioning, better compression, and encryption at rest.
Preview: Enabled
VMRuntime
custom resource and theNetwork
custom resource, which let you create VMs on either the node network with a static IP address or the default pod network.Preview: Enabled VM pods audit logs for VM runtime resources.
Preview: Expanded guest OS versions that can run on the virtual machine. We support Windows Server 2019, 2016, Windows 10, Red Hat Enterprise Linux (RHEL) 8, Centos 8, and Ubuntu 20.04 as guest OS.
Preview: Enabled virtual machine high availability to provide greater uptime for virtual machines instances (VMIs) by automatically detecting and recovering from a range of host machine failures.
Breaking changes:
The gateway capability used by the egress NAT gateway and Bundled load balancing with BGP Preview features have changed in this release. The NetworkGatewayGroup
custom resource replaces AnthosNetworkGateway
and the capability is enabled with a new advancedNetworking
field in the cluster configuration file, instead of an annotation. These changes affect the ability to upgrade clusters that use earlier versions of the features.
Anthos clusters on bare metal blocks cluster upgrades from version 1.9 to version 1.10 for clusters that use either of these two advanced networking features. You can upgrade a version 1.9 admin cluster that is managing 1.9 user clusters that use these features to version 1.10, but object reconciliation breaks for the AnthosNetworkGateway
custom resource. Object reconciliation is the mechanism whereby admin clusters automatically copy/restore objects on managed user clusters when the objects have been defined alongside the cluster configuration. Any AnthosNetworkGateway
custom resources are still functional and can be modified with kubectl
.
To bring a version 1.9 cluster that uses either advanced networking Preview feature up to version 1.10, reset or delete the cluster and create a new 1.10 cluster.
Preview features and products are subject to change and are provided for testing and evaluation purposes only. Do not use Preview features on your production clusters.
Functionality changes:
Enabled use of
ADMIN_KUBECONFIG
environment variable to reduce the number ofbmctl
command flags.The cluster reconciliation process now checks for differences in the GKEHub membership before attempting to update it. If the GKEHub membership needs to be changed, the cluster is unregistered and then re-registered.
The
advancedNetworking
field in the cluster configuration file replaces the deprecatedbaremetal.cluster.gke.io/enable-anthos-network-gateway
annotation for enabling advanced networking capabilities.The
NetworkGatewayGroup
custom resource replaces theAnthosNetworkGateway
custom resource.
Fixed cluster lifecycle functionalities:
Outputs from all
bmctl
commands exceptbmctl version
are now written to log files.Fixed strict mode for decoding the cluster YAML file. Extraneous information in the cluster YAML file now results in an error.
Fixed preflight check so that it no longer ignores the
no_proxy
setting.Binaries in cluster provision no longer run from
/tmp
, which is often mounted withnoexec
options. This change fixes a preflight check "permission denied" error.Switched the default server-side
containerRuntime
value fromdocker
tocontainerd
.
Observability:
Increased the priority of the
kube-state-metrics
service to keep it from being stuck in a pending state. This service generates metrics about Kubernetes API objects such as deployments, nodes, and pods.Upgraded metrics-server to version 0.3.6 to fix a missing metrics issue that occurs when a duplicated pod name is present.
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.