This document lists production updates to Google Distributed Cloud. We recommend that Google Distributed Cloud developers periodically check this list for any new announcements.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.
January 31, 2024
Security bulletin (all minor versions)
A security vulnerability, CVE-2024-21626, has been discovered in runc
where a user with permission to create Pods might be able to gain full access to the node filesystem.
For instructions and more details, see the GCP-2024-005 security bulletin.
June 27, 2023
Security bulletin (all minor versions)
A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.
For more information, see the GCP-2023-016 security bulletin.
June 16, 2023
Security bulletin (all minor versions)
Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).
For more information, see the GCP-2023-014 security bulletin.
May 10, 2023
CentOS Linux 8 Support Deprecated
CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.
April 12, 2023
Kubernetes image registry redirect
As of March 21, 2023, traffic to k8s.gcr.io
is redirected to registry.k8s.io
, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.
To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.
March 28, 2023
Release 1.12.9
Anthos clusters on bare metal 1.12.9 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.9 runs on Kubernetes 1.23.
FIxes:
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.
March 02, 2023
Release 1.12.8
Anthos clusters on bare metal 1.12.8 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.8 runs on Kubernetes 1.23.
Fixes:
Updated Anthos Identity Service to better handle concurrent authentication webhook requests.
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.
February 07, 2023
Release 1.12.7
Anthos clusters on bare metal 1.12.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.7 runs on Kubernetes 1.23.
Fixes:
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.
December 14, 2022
Release 1.12.6
Anthos clusters on bare metal 1.12.6 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.6 runs on Kubernetes 1.23.
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
December 09, 2022
Release 1.12.5
Anthos clusters on bare metal 1.12.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.5 runs on Kubernetes 1.23.
Fixes:
The following container image security vulnerabilities have been fixed:
- CVE-2019-25013
- CVE-2021-3326
- CVE-2021-3999
- CVE-2021-4037
- CVE-2021-33574
- CVE-2021-35942
- CVE-2022-1184
- CVE-2022-1586
- CVE-2022-1587
- CVE-2022-2663
- CVE-2022-3061
- CVE-2022-3176
- CVE-2022-3303
- CVE-2022-3586
- CVE-2022-3621
- CVE-2022-3646
- CVE-2022-3649
- CVE-2022-20421
- CVE-2022-23218
- CVE-2022-23219
- CVE-2022-32221
- CVE-2022-33745
- CVE-2022-33746
- CVE-2022-33748
- CVE-2022-34903
- CVE-2022-37434
- CVE-2022-39188
- CVE-2022-40307
- CVE-2022-42309
- CVE-2022-42310
- CVE-2022-42311
- CVE-2022-42312
- CVE-2022-42313
- CVE-2022-42314
- CVE-2022-42315
- CVE-2022-42316
- CVE-2022-42317
- CVE-2022-42318
- CVE-2022-42319
- CVE-2022-42320
- CVE-2022-42321
- CVE-2022-42322
- CVE-2022-42323
- CVE-2022-42324
- CVE-2022-42325
- CVE-2022-42326
- CVE-2022-43680
- CVE-2022-43750
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
November 08, 2022
Release 1.12.4
Anthos clusters on bare metal 1.12.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.4 runs on Kubernetes 1.23.
Fixes:
Increased the CPU limit for the
metrics-server
Pod to prevent it from frequently restarting.The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
November 07, 2022
Security bulletin (1.11, 1.12, and 1.13)
A security vulnerability, CVE-2022-39278, has been discovered in Istio, which is used in Anthos Service Mesh, that allows a malicious attacker to crash the control plane.
For instructions and more details, see the Anthos clusters on bare metal security bulletin.
October 05, 2022
Release 1.12.3
Anthos clusters on bare metal 1.12.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.3 runs on Kubernetes 1.23.
Fixes:
Updated the container image to resolve a YAML text/template vulnerability.
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
August 25, 2022
Release 1.12.2
Anthos clusters on bare metal 1.12.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.2 runs on Kubernetes 1.23.
Features:
- Added
–use-disk
flag tobmctl backup cluster
command to use the disk instead of the in-memory buffer to back up a cluster. Use this option when available RAM is limited on your admin workstation. - Added
--quiet
flag tobmctl check cluster -- snapshot
command to suppress logging to the console during the snapshot creation.
Fixes:
- Added caching for the Cloud Audit Logging feature status to avoid unnecessary checks and improve performance.
- Increased the etcd default DB size to 6GiB by default to address
NO_SPACE_ALARM
in high-scale clusters. - Fixed a
libseccomp
package incompatibility issue. - Fixed an issue with the machine-reset job getting stuck.
- Fixed an issue that caused continuous, unneeded cluster reconciliation operations.
- Fixed an issue that prevented the node problem detector from running after a cluster upgrade.
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
August 23, 2022
Anthos VM Runtime
Anthos VM Runtime is Generally Available (GA). Some features and capabilities are available for Preview only, as indicated in the following descriptions:
- Upgraded Kubevirt to version 0.49.
- Upgraded Containerized Data Importer (CDI) to version 1.43.0.
- Added
bmctl
command to enable or disable Anthos VM Runtime on user clusters. - Added automatic upgrade of Anthos VM Runtime when upgrading Anthos clusters on bare metal.
- Preview: Added ability to configure an eviction policy that controls how VMs automatically migrate to other hosts during maintenance events.
- Preview: Added non-disruptive upgrading of VM runtime during live migration (that is, when VMs are unobtrusively migrated from one node to another).
VM APIs:
- Simplified VM Compute API.
- Added ability to create and manage disk resources for VMs that use Anthos VM Runtime.
- Added ability to schedule VMs using standard Kubernetes scheduling primitives.
- Preview: Added ability to use GPUs in VMs.
- Added more access management capabilities to VM Guest Environment.
- Preview: Added support for guest OS booting of UEFI. Previously, only BIOS was supported.
Observability:
- Integrated VM telemetry and console logs into Google Cloud console. Telemetry information and log data are critical for monitoring the status of VMs and for troubleshooting problems with your cluster VMs.
- Added VM CPU and memory metrics to Cloud Monitoring. These metrics can be viewed in the Anthos clusters VM status dashboard.
- Added ability to view console logs for VMs that use Anthos VM Runtime.
- Added logs that audit VM pods.
Guest OS support:
Added support for the following guest OS versions running on a Virtual Machine:
- Windows Server 2019
- Windows Server 2016
- Windows 10
- Red Hat Enterprise Linux (RHEL) 8
- RHEL 7
- CentOS 8
- CentOS 7
- Ubuntu 20.04
- Ubuntu 18.04
VM networking features:
- IPAMv4: Static IP Allocation for VM interfaces.
- IP and MAC Stickiness for VM interfaces.
- IPAMv4: DHCP for VM interfaces.
- VLAN tagging support for VM Interfaces.
- Multi-NIC for VM interfaces through native Dataplane V2 support (macvtap + Dataplane V2).
- Static routes and DNS configurations at per-network basis.
- NetworkPolicy enforcement at per-network basis.
- Validating admission webhooks for Network and NetworkInterface object.
- Network Mutation, allow the mutations of Gateway, DNS and the customized network routes in the network custom resource. The parent interface for the VM and the VLAN ID are not mutable. VMs that were already running before the network configuration change need to be restarted to pick up the change.
- Added command to restart all VMs in a network.
Graceful IP release for VMs:
- During VM migration, the IP isn't released.
- IP addresses are released for VMs that are deleted or stopped.
For more information on networking, see Create and use virtual networks for Anthos VM Runtime.
VM Runtime issues:
When kubevirt is configured, customers should ensure that TOR switches have MAC learning enabled.
If you choose to manually run a DHCP
ipconfig /renew
command in a Windows VM, you should first perform a DHCP release, using theipconfig /release
command. In other words, the sequence for manually performing a DHCP renewal in a Windows environment is the following:ipconfig /release ipconfig /renew
August 03, 2022
Release 1.12.1
Anthos clusters on bare metal 1.12.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.1 runs on Kubernetes 1.23.
Functionality changes:
Increased default memory limits for
coredns
,metallb-controller
,metallb-speaker
,metrics-server
,anthos-cluster-operator
, andcap-controller-manager
.Modified the dashboards
Anthos cluster pod status
andAnthos cluster node status
. Specifically, the following changes were made:- Replaced
cadvisor
resource metrics with summary API resource metrics. - Added
cpu
,memory
, andvolume
utilization metrics.
If you have already installed these dashboards in a project, you need to download the JSON files
Anthos-cluster-pod-status.json
andAnthos-cluster-node-status.json
from the Dashboards for Anthos GitHub repository. You then need to import these JSON files into Cloud Monitoring. For details, see Install sample dashboards.- Replaced
Fixes:
- Fixed issue in which nodes drained or cordoned by
kubectl
were mistakenly marked as schedulable. - Fixed issue in which cluster controller and autoscaler conflicted with each other in the scaling of
istiod
,coredns
, andistio-ingress
Pods. - Fixed issue in which the wrong data type was used in health check log messages, resulting in panic messages.
- Fixed issue in which cluster restores failed when
/var/lib/etcd
is a mount point. - Fixed issue in which attempts to skip minor versions when upgrading weren't blocked. For details about the upgrade policy, see Minor version upgrades.
- Fixed issue in which an external VIP Service of type LoadBalancer would not respond when flat IP mode was enabled.
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.
Release 1.12.1 ships with containerd
version 1.5.13, which requires libseccomp
version 2.5 or higher. If your system doesn't have libseccomp
version 2.5 or higher installed, update it in advance of upgrading existing clusters to version 1.12.1. Otherwise, you may see errors in cplb-update
Pods for load balancer nodes such as:
runc did not terminate successfully: runc: symbol lookup error: runc:
undefined symbol: seccomp_notify_respond
To install the latest version of libseccomp
in Ubuntu, run the following command:
sudo apt-get install libseccomp-dev
To install the latest version of libseccomp
in CentOS or RHEL, run the following command:
sudo dnf -y install libseccomp-devel
June 29, 2022
Release 1.12.0
Anthos clusters on bare metal 1.12.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.12.0 runs on Kubernetes 1.23.
The dockershim component in Kubernetes enables cluster nodes to use the Docker Engine container runtime. However, Kubernetes 1.24 removed the dockershim component. Starting from Anthos clusters on bare metal 1.12.0, you will not be able to create new clusters that use the Docker Engine container runtime. All new clusters should use the default container runtime containerd
.
Improved cluster lifecycle functionalities:
Upgraded Anthos clusters on bare metal to use Kubernetes version 1.23.
Upgraded container runtime to
containerd
1.5.Updated preflight check to forward default SSH key if no key is provided.
Added support for new
GCPAccounts
field in the cluster configuration file. This field enables the assignment of acluster-admin
role to end-users.Added labels to control plane, control plane load balancer, and load balancer node pools, so that these different node pools can be distinguished from each other.
Added nodepool reference label to nodes so that worker nodes can be listed in the UI.
Observability:
GA: Added Summary API metrics. These metrics are scraped from the Kubernetes Summary API and provide CPU, memory, and storage metrics for Pods, containers, and Nodes.
Added separate flags to enable logging and monitoring for user applications separately:
EnableCloudLoggingForApplications
andEnableGMPForApplications
. The legacy flagEnableStackdriverForApplications
will be deprecated and removed in future releases.Preview: Added Google Cloud Managed Service for Prometheus to collect application metrics and monitor cluster health.
Upgraded GKE Metrics Agent (gke-metrics-agent) from version 1.1.0 to 1.8.3. This tool scrapes metrics from each cluster node and publishes them in Cloud Monitoring.
Added the following resource utilization metrics. For more information about these and other metrics, see View Anthos clusters on bare metal metrics:
container/cpu/request_utilization
container/cpu/limit_utilization
container/memory/request_utilization
container/memory/limit_utilization
node/cpu/allocatable_utilization
node/memory/allocatable_utilization
pod/volume/utilization
Added sample dashboards for monitoring cluster health to Cloud Monitoring sample dashboards. Customers can install these dashboards with one click.
Scoped down the RBAC permissions of
stackdriver-operator
, a component that performs logging and monitoring.
Security:
AIS CA deprecation. AIS certs are now signed by cluster CA.
Changed
ca-rotation
container image so that it uses a distroless rather than a Debian-based image.RBAC permissions of the
cluster-operator
component have been eliminated or reduced to address elevated permissions.GA: Anthos Identity Service LDAP authentication support.
Networking:
Preview: Enabled creation of IPv6 and Dual Stack LoadBalancer services. Border Gateway Protocol (BGP) is used for Dualstack clusters. Advertising IPv4 and IPv6 routes over IPv4 sessions is supported.
Preview: Added Network Connectivity Gateway feature support to provide HA VPN between Google Cloud and an on-premises Anthos cluster.
Known issues:
For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.