GKE on VMware release notes

This page documents production updates to GKE on VMware. You can periodically check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.

See also:

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/gkeonprem-release-notes.xml

February 16, 2024

The following vulnerability was discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes:

  • CVE-2023-6932

For more information, see the GCP-2024-011 security bulletin.

February 14, 2024

The following vulnerability was discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes:

  • CVE-2023-6931

For more information, see the GCP-2024-010 security bulletin.

February 01, 2024

GKE on VMware 1.15.8-gke.41 is now available. To upgrade, see Upgrading Anthos clusters on VMware. GKE on VMware 1.15.8-gke.41 runs on Kubernetes v1.26.10-gke.2000.

Upgraded etcd to v3.4.27-0-gke.1.

The following issues are fixed in 1.15.8-gke.41:

  • Fixed Seesaw crashing on duplicated service IP.
  • Fixed a warning in the storage preflight check.

The following vulnerabilities are fixed in 1.15.8-gke.41:

January 31, 2024

A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods on Container-Optimized OS and Ubuntu nodes might be able to gain full access to the node filesystem.

For instructions and more details, see the GCP-2024-005 security bulletin.

January 25, 2024

GKE for VMware 1.28.100-gke.131 is now available. To upgrade, see Upgrading GKE on VMware. GDCV for VMware 1.28.100-gke.131 runs on Kubernetes v1.28.3-gke.1600.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The following issues are fixed in 1.28.100-gke.131:

  • Fixed an issue where duplicate Service IP addresses caused the Seesaw load balancer to fail.

  • Fixed an issue where egress NAT erroneously broke long-lived connections.

The following vulnerabilities are fixed in 1.28.100-gke.131:

GKE for VMware 1.16.5-gke.28 is now available. To upgrade, see Upgrading GKE on VMware. GDCV for VMware 1.16.5-gke.28 runs on Kubernetes 1.27.6-gke.2500.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The following issues are fixed in 1.16.5-gke.28:

  • Fixed an issue where duplicate Service IP addresses caused the Seesaw load balancer to fail.

The following vulnerabilities are fixed in 1.16.5-gke.28:

There is an issue that affects upgrading from 1.16.x to 1.28.100. If the 1.16.x cluster relies on an NFS volume, the upgrade will fail. Clusters that don't use an NFS volume are not affected.

December 18, 2023

GKE on VMware, formerly Anthos clusters on VMware, is a component of Google Distributed Cloud Virtual, software that brings Google Kubernetes Engine (GKE) to on-premises data centers. We are in the process of updating documentation and the Google Cloud Console UI with the new name.

GKE on VMware 1.28.0-gke.651 is now available. GKE on VMware 1.28.0-gke.651 runs on Kubernetes v1.28.3-gke.700. To upgrade, see Upgrading GKE on VMware clusters.

For easier identification of the Kubernetes version for a given release, we are
aligning GKE on VMware version numbering with GKE version numbering. This change starts with the December 2023 minor release, which is version 1.28. Additionally, GKE on VMware patch versions (z in the semantic version numbering scheme x.y.z-gke.N) will increment by 100.

Example version numbers for GKE on VMware:

  • Minor release: 1.28.0-gke.651
  • First patch release (example): 1.28.100-gke.27
  • Second patch release (example): 1.28.200-gke.19

This change affects numbering only. Upgrades from 1.16 to 1.28 follow the same process as upgrades between prior minor releases.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

New features in GKE on VMware 1.28.0-gke.651:

Breaking change in GKE on VMware 1.28.0-gke.651:

Cloud Monitoring now requires projects to enable the kubernetesmetadata.googleapis.com API and grant the kubernetesmetadata.publisher IAM role to the logging-monitoring service account. This applies to both creating new 1.28 clusters and upgrading existing clusters to 1.28. If your organization has set up an allowlist that lets traffic from Google APIs and other addresses pass through your proxy server, add [kubernetesmetadata.googleapis.com](kubernetesmetadata.googleapis.com) to the allowlist.

Version changes in GKE on VMware 1.28.0-gke.651:

  • Bumped etcd to version v3.4.27-0-gke.1.
  • Bumped istio-ingress to version 1.19.3.
  • Bumped the AIS version to hybrid_identity_charon_20230830_0730_RC00.

Other changes in GKE on VMware 1.28.0-gke.651:

  • HA admin clusters now have a long running controller to perform reconciliation periodically.
  • The command gkectl repair admin-master --restore-from-backup now supports restoration of etcd data for HA admin clusters.
  • When upgrading user clusters to version 1.28, we validate all changes made in the configuration file, and return an error for unsupported changes. See Remove unsupported changes to unblock upgrade.
  • The vSphere cloud controller manager is enabled in Controlplane V2 user clusters.
  • We now always write the local k8s audit log file, even when Cloud audit logging is enabled. This allows for easier third party logging system integration.
  • MetalLB will be the default load balancer for 1.29 user and admin clusters. The ability to use Seesaw as a load balancer will be removed with 1.29. We recommend migrating to the MetalLB load-balancer. Upgrades from existing Seesaw clusters will continue to work for a few more releases.
  • The loadBalancer.manualLB.addonsNodePort field is deprecated. The field was used for the in-cluster Prometheus and Grafana add-ons, which was deprecated in version 1.16.
  • The loadBalancer.vips.addonsVIP field is deprecated. The field was used for the in-cluster Prometheus and Grafana add-ons, which was deprecated in version 1.16.
  • yq is no longer pre-installed on the admin workstation.
  • Control-plane nodes now have the node-role.kubernetes.io/control-plane taint.
  • In-tree GlusterFS is removed from Kuberentes 1.27. Add storage validation to detect in-tree GlusterFS volumes.
  • Metrics data are now gzip compressed when they are sent to Cloud monitoring.

The following issues are fixed in 1.28.0-gke.651:

  • Fixed an issue where disable_bundled_ingress failed user cluster load balancer validation.
  • Fixed an issue where the cluster-health-controller sometimes leaked vSphere sessions.
  • Fixed an etcd hostname mismatch issue when using FQDN.
  • Fixed a known issue where admin cluster update or upgrade failed if the projects or locations of add-on services didn't match each other.
  • Fixed a known issue where the CSI workload preflight check failed due to a Pod startup failure.
  • Fixed an issue where deleting a user cluster with a volume attached might get stuck.
  • Fixed a known issue where deleting a Controlplane V2 user cluster might get stuck.
  • Fixed a logrotate error on the ubuntu_containerd image.
  • Fixed a disk full issue on Seesaw VMs due to no log rotation for fluent-bit.
  • Fixed a known issue where Seesaw didn't set the target IP in GARP replies.
  • Fixed a flaky SSH error on non-HA admin control-plane nodes after update/upgrade.

The following vulnerabilities are fixed in 1.28.0-gke.651:

There is an issue that affects upgrading from 1.16.x to 1.28.0. If the 1.16.x cluster relies on an NFS volume, the upgrade will fail. Clusters that don't use an NFS volume are not affected.

Anthos clusters on VMware 1.16.4-gke.37 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.16.4-gke.37 runs on Kubernetes 1.27.6-gke.2500.

The following issues are fixed in 1.16.4-gke.37:

  • Fixed a warning in the storage preflight check.
  • Fixed an issue where control plane creation failed for a user cluster when using a FQDN hostname for a HA admin cluster.
  • Fixed an issue where the cluster-health-controller might leak vSphere sessions.
  • Fixed an issue where disable_bundled_ingress failed user cluster load balancer validation.

The following vulnerabilities are fixed in 1.16.4-gke.37:

December 12, 2023

Anthos clusters on VMware 1.15.7-gke.40 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 11.15.7-gke.40 runs on Kubernetes 1.26.9-gke.700.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The following issues are fixed in 1.15.7-gke.40:

  • Fixed the etcd hostname mismatch issue when using a FQDN.
  • Fixed an issue where the cluster-health-controller might leak vSphere sessions.
    Fixed the known issue where the CSI workload preflight check fails due to Pod startup failure.

The following vulnerabilities are fixed in 1.15.7-gke.40:

December 04, 2023

The StatefulSet CSI Migration Tool is now available. To learn how to migrate stateful workloads from an in-tree vSphere volume plugin to the vSphere CSI Driver, see Using the StatefulSet CSI Migration Tool.

November 22, 2023

A vulnerability (CVE-2023-5717) has been discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes.

For more information, see the GCP-2023-046 security bulletin.

November 20, 2023

Anthos clusters on VMware 1.14.10-gke.35 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.8-gke.37 runs on Kubernetes v1.25.13-gke.200.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The following issues are fixed in 1.14.10-gke.35:

  • Fixed the etcd hostname mismatch issue when using FQDN
  • Fixed the issue where deleting a user cluster with a volume attached stalls, in which case the cluster can't be deleted and can't be used.

The following vulnerabilities are fixed in 1.14.10-gke.35:

November 16, 2023

Anthos clusters on VMware 1.16.3-gke.45 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.16.1-gke.44 runs on Kubernetes 1.27.4-gke.1600.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The Prometheus and Grafana add-ons field, loadBalancer.vips.addonsVIP, is deprecated. This change is because Google Managed Service for Prometheus replaced the Prometheus and Grafana add-ons.

The following issues are fixed in 1.16.3-gke.45:

  • Fixed a Cilium issue causing egress NAT to erroneously break long-lived connections.
  • Fixed the etcd hostname mismatch issue when using a FQDN.
  • Fixed the known issue that caused admin cluster updates or upgrades to fail if the projects or locations of add-on services don't match each other.
  • Fixed the issue that external cluster snapshot won't be taken after gkectl update admin fails.
  • Fixed an issue that caused the CSI workload preflight to fail when Istio is enabled.
  • Fixed the issue that deleting a user cluster with a volume attached may be stuck forever.
  • Fixed the known issue that caused user cluster deletion to fail when using a user-managed admin workstation.

The following vulnerabilities are fixed in 1.16.3-gke.45:

November 13, 2023

The following vulnerabilities were discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes.

  • CVE-2023-4147

For more information, see the GCP-2023-042 security bulletin.

November 08, 2023

A vulnerability (CVE-2023-4004) has been discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes. For more information, see the GCP-2023-041 security bulletin.

October 31, 2023

Anthos clusters on VMware 1.15.6-gke.25 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.15.6-gke.25 runs on Kubernetes 1.26.9-gke.700.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The following vulnerabilities are fixed in 1.15.6-gke.25:

October 19, 2023

Anthos clusters on VMware 1.16.2-gke.28 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.16.2-gke.28 runs on Kubernetes 1.27.4-gke.1600.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The following issue is fixed in 1.16.2-gke.28:

  • Fixed the known issue where a non-HA Controlplane V2 cluster is stuck at node deletion until it timesout.

The following vulnerabilities are fixed in 1.16.2-gke.28:

Anthos clusters on VMware 1.14.9-gke.21 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.9-gke.21 runs on Kubernetes 1.25.13-gke.200.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The following issues are fixed in 1.14.9-gke.21:

  • Fixed the known issue where a non-HA Controlplane V2 cluster is stuck at node deletion until it timesout.

The following vulnerabilities are fixed in 1.14.9-gke.21:

October 12, 2023

Anthos clusters on VMware 1.15.5-gke.41 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.15.5-gke.41 runs on Kubernetes 1.26.7-gke.2500.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The following issues are fixed in 1.15.5-gke.41:

  • Fixed the issue that server-side preflight checks fail to validate container registry access on clusters with a private network and no private registry.
  • Fixed the known issue where a non-HA Controlplane V2 cluster is stuck at node deletion until it timesout.
  • Fixed the known issue where upgrading or updating an admin cluster with a CA version greater than 1 fails.
  • Fixed the issue where the Controlplane V1 stackdriver operator has --is-kubeception-less=true specified by mistake.
  • Fixed the known issue that causes the secrets encryption key to be regenerated when upgrading the admin cluster from 1.14 to 1.15, resulting in the upgrade being blocked.

The following vulnerabilities are fixed in 1.15.5-gke.41:

October 02, 2023

Upgrading an admin cluster with always-on secrets encryption enabled might fail.

An admin cluster upgrade from 1.14.x to 1.15.0 - 1.15.4 with always-on secrets encryption enabled might fail depending on whether the feature was enabled during cluster creation or during cluster update.

We recommend that you don't upgrade your admin cluster until a fix is available in 1.15.5. If you must upgrade to 1.15.0-1.15.4, do the steps in Preventing the upgrade failure before upgrading the cluster.

For information on working around an admin cluster failure because of this issue, see Upgrading an admin cluster with always-on secrets encryption enabled fails. Note that the workaround relies on you having the old encryption key backed up. If the old key is no longer available, you will have to recreate the admin cluster and all user clusters.

September 29, 2023

Anthos clusters on VMware 1.16.1-gke.45 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.16.1-gke.44 runs on Kubernetes 1.27.4-gke.1600.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The Prometheus and Grafana add-ons field, loadBalancer.vips.addonsVIP is deprecated in 1.16 and later. This change is because Google Managed Service for Prometheus replaced the Prometheus and Grafana add-ons in 1.16.

The following issues are fixed in 1.16.1-gke.45:

  • Fixed the known issue that gkectl repair admin-master returns kubeconfig unmarshall error.
  • Fixed the known issue that GARP reply sent by Seesaw doesn't set target IP
  • Fixed the known issue that Seesaw VM may be broken due to low disk space
  • Fixed the known issue that false warnings might be generated against persistent volume claims.
  • Fixed the known issue that caused CNS attachvolume tasks to appear every minute for in-tree PVC/PV after upgrading to Anthos 1.15+.

The following vulnerabilities are fixed in 1.16.1-gke.44:

Anthos clusters on VMware 1.14.8-gke.37 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.8-gke.37 runs on Kubernetes 1.25.12-gke.2400.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

The following issues are fixed in 1.14.8-gke.37:

  • Fixed the disk full known issue on Seesaw VM due to no log rotation for fluent-bit.

The following vulnerabilities are fixed in 1.14.8-gke.37:

September 14, 2023

A standalone tool that you run before upgrading an admin or user cluster is now available. The pre-upgrade tool is supported for Anthos clusters on VMware version 1.9 through 1.13. The tool runs the applicable preflight checks for the version that you are upgrading to and also checks for specific known issues. Before upgrading a 1.9 - 1.13 cluster, we recommend that you run the pre-upgrade tool.

For details on running the tool, see the documentation for the version that you are upgrading to:

September 01, 2023

Anthos clusters on VMware 1.15.4-gke.37 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.15.4-gke.37 runs on Kubernetes 1.26.7-gke.2500.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

Upgrading an admin cluster with always-on secrets encryption enabled might fail.

An admin cluster upgrade from 1.14.x to 1.15.0 - 1.15.4 with always-on secrets encryption enabled might fail depending on whether the feature was enabled during cluster creation or during cluster update.

We recommend that you don't upgrade your admin cluster until a fix is available in 1.15.5. If you must upgrade to 1.15.0-1.15.4, do the steps in Preventing the upgrade failure before upgrading the cluster.

For information on working around an admin cluster failure because of this issue, see Upgrading an admin cluster with always-on secrets encryption enabled fails. Note that the workaround relies on you having the old encryption key backed up. If the old key is no longer available, you will have to recreate the admin cluster and all user clusters.

The following issues are fixed in 1.15.4-gke.37:

  • Fixed a known issue where incorrect log rotation configuration for fluent-bit caused low disk space on the Seesaw VM.

  • Fixed a known issue that GARP reply sent by Seesaw doesn't set target IP.

  • Fixed an issue where /etc/vsphere/certificate/ca.crt wasn't updated after vsphere CA rotation on the Controlplane v2 user cluster control plane machines.

  • Fixed a known issue where the admin SSH public key has error after admin cluster upgrade or update.

The following vulnerabilities are fixed in 1.15.4-gke.37:

August 23, 2023

Anthos clusters on VMware 1.16.0-gke.669 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.16.0-gke.669 runs on Kubernetes 1.27.4-gke.1600.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

Version changes:

  • Upgraded VMware vSphere Container Storage Plug-in from 3.0 to 3.0.2.
  • The crictl command-line tool was updated to 1.27.
  • The containerd config was updated to version 2.

Other changes:

  • The output of the gkectl diagnose cluster command has been updated to provide a summary that customers can copy and paste when opening support cases.
  • In-tree GlusterFS is removed from Kuberentes 1.27. Add storage validation to detect in-tree glusterFS volumes.

  • Metrics data are now gzip compressed when sending to Cloud Monitoring.

  • The stackdriver-log-forwarder (fluent-bit) now sends logs to Cloud Logging with gzip compression to reduce egress bandwidth needed.

  • Prometheus and Grafana are no longer bundled for in-cluster monitoring and they are replaced with Google Cloud Managed Service for Prometheus.

  • The following flags in the stackdriver custom resource are deprecated and changes to their values aren't honored:

    • scalableMonitoring
    • enableStackdriverForApplications (replaced by enableGMPForApplications and enableCloudLoggingForApplications)
    • enableCustomMetricsAdapter
  • Deploying the vSphere cloud controller manager in both admin and user clusters, and enabling it for admin and kubeception user clusters is now supported.

  • The audit-proxy now sends audit logs to Cloud Audit Logging with gzip compressed to reduce egress bandwidth needed.

  • Removed accounts.google.com from the internet preflight check requirement.

  • The pre-defined dashboards are automatically present based on the presence of metrics.

  • Enabled auto repair on ReadonlyFilesystem node condition

  • Support the d character when using --log-since flag to take cluster snapshot. For example: gkectl diagnose snapshot --log-since=1d

  • A new CSI Workload preflight check was added to verify that workloads using vSphere PVs can work through CSI.

  • Preflight check failures for gkectl prepare now block install and upgrade operations.

  • The kubelet readonly port is now disabled by default for security enhancement. See Enable kubelet readonly port for instructions if you need to re-enable it for legacy reasons.

  • AIS Pods are now scheduled to run on control plane nodes instead of worker nodes.

The following issues are fixed in 1.16.0-gke.669:

  • Fixed the known issue that caused intermittent ssh errors on non-HA admin master after update or upgrade.
  • Fixed the known issue where upgrading enrolled admin cluster could fail due to membership update failure.
  • Fixed the issue where the CPv1 stackdriver operator had --is-kubeception-less=true specified by mistake.

  • Fixed the issue where clusters used the non-high-availability (HA) Connect Agent after an upgrade to 1.15.

  • Fixed the known issue of Cloud Audit Logging failure due to permission denied.

  • Fixed a known issue where the update operation cannot be fulfilled due to KSA signing key version unmatched.

  • Fixed a known issue where $ in the private registry username caused admin control plane machine startup failure.

  • Fixed a known issue where gkectl diagnose snapshot failed to limit the time window for journalctlcommands running on the cluster nodes when you take a cluster snapshot with the --log-since flag.

  • Fixed a known issue where node ID verification failed to handle hostnames with dots.

  • Fixed continuous increase of logging agent memory.

  • Fixed the issue that caused gcloud to fail to update the platform when the required-platform-version is already the current platform version.

  • Fixed an issue where cluster-api-controllers in a high-availability admin cluster had no Pod anti-affinity. This could allow the three clusterapi-controllers Pods not to be scheduled on different control-plane nodes.

  • Fixed the wrong admin cluster resource link annotation key that can cause the cluster to be enrolled again by mistake.

  • Fixed a known issue where node pool creation failed because of duplicated VM-Host affinity rules.

  • The preflight check for StorageClass parameter validations now throws a warning instead of a failure on ignored parameters after CSI Migration. StorageClass parameter diskformat=thin is now allowed and does not generate a warning.

  • Fixed a false error message for gkectl prepare when using a high-availability admin cluster.

  • Fixed an issue during the migration from the Seesaw load balancer to MetalLB that caused 'DeprecatedKubeception' always shows up in the diff.

  • Fixed a known issue where some cluster nodes couldn't access the HA control plane when the underlying network performs ARP suppression.

  • Removed unused Pod disruption budgets (such as kube-apiserver-pdb, kube-controller-manager-pdb, and kube-etcd-pdb) for Controlplane V2 user clusters

The following vulnerabilities are fixed in 1.16.0-gke.669:

August 17, 2023

Anthos clusters on VMware 1.14.7-gke.42 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.7-gke.42 runs on Kubernetes 1.25.10-gke.2100.

Upgraded VMware vSphere Container Storage Plug-in from 2.7.0 to 2.7.2.

The following issues are fixed in 1.14.7-gke.42:

  • Fixed a known issue that admin SSH public key has error after admin cluster upgrade or update.
  • Fixed a known issue that GARP reply sent by Seesaw doesn't set target IP.
  • Fixed an issue that /etc/vsphere/certificate/ca.crt was not updated after vsphere CA rotation on the Controlplane v2 user cluster control plane machines.
  • Fixed an issue that the CPv1 stackdriver operator had --is-kubeception-less=true specified by mistake.

The following vulnerabilities are fixed in 1.14.7-gke.42:

August 10, 2023

Anthos clusters on VMware 1.15.3-gke.47 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.15.3-gke.47 runs on Kubernetes 1.26.5-gke.2100.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

Upgrading an admin cluster with always-on secrets encryption enabled might fail.

An admin cluster upgrade from 1.14.x to 1.15.0 - 1.15.4 with always-on secrets encryption enabled might fail depending on whether the feature was enabled during cluster creation or during cluster update.

We recommend that you don't upgrade your admin cluster until a fix is available in 1.15.5. If you must upgrade to 1.15.0-1.15.4, do the steps in Preventing the upgrade failure before upgrading the cluster.

For information on working around an admin cluster failure because of this issue, see Upgrading an admin cluster with always-on secrets encryption enabled fails. Note that the workaround relies on you having the old encryption key backed up. If the old key is no longer available, you will have to recreate the admin cluster and all user clusters.

Anthos clusters on VMware 1.15.3 supports adding the gkeOnPremAPI section to your admin cluster configuration file and user cluster configuration file to enroll the clusters in the Anthos On-Prem API.

Upgraded VMware vSphere Container Storage Plug-in from 3.0 to 3.0.2. For more information, see the Plug-in release notes.

The following issues are fixed in 1.15.3-gke.47:

  • Fixed a known issue. that caused upgrading an admin cluster enrolled in the Anthos On-Prem API to fail.
  • Fixed an issue where audit logs are duplicated into an offline buffer even when they are successfully sent to Cloud Audit Logging.

The following vulnerabilities are fixed in 1.15.3-gke.47:

July 20, 2023

Anthos clusters on VMware 1.13.10-gke.42 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.10-gke.42 runs on Kubernetes 1.24.14-gke.2100.

  • Upgraded VMware vSphere Container Storage Plug-in from 2.6.2 to 2.7.2.
  • Added short names for Volume Snapshot CRDs.

The following issues are fixed in 1.13.10-gke.42:

  • Fixed an issue that CPv1 stackdriver operator has --is-kubeception-less=true specified by mistake.
  • Fixed an issue that /etc/vsphere/certificate/ca.crt is not updated after vsphere CA rotation on the Controlplane v2 user cluster control plane machines.
  • Fixed an issue where audit logs are duplicated into an offline buffer even when they are successfully sent to Cloud Audit Logs.
  • Fixed a known issue where $ in the private registry user name would cause admin control plane machine startup failure.
  • Fixed a known issue where the update operation cannot be fulfilled due to KSA signing key version unmatched.

The following vulnerabilities are fixed in 1.13.10-gke.42:

July 10, 2023

Anthos clusters on VMware 1.15.2-gke.44 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware. 1.15.2-gke.44 runs on Kubernetes 1.26.2-gke.1001.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

Upgrading an admin cluster with always-on secrets encryption enabled might fail.

An admin cluster upgrade from 1.14.x to 1.15.0 - 1.15.4 with always-on secrets encryption enabled might fail depending on whether the feature was enabled during cluster creation or during cluster update.

We recommend that you don't upgrade your admin cluster until a fix is available in 1.15.5. If you must upgrade to 1.15.0-1.15.4, do the steps in Preventing the upgrade failure before upgrading the cluster.

For information on working around an admin cluster failure because of this issue, see Upgrading an admin cluster with always-on secrets encryption enabled fails. Note that the workaround relies on you having the old encryption key backed up. If the old key is no longer available, you will have to recreate the admin cluster and all user clusters.

The following issues are fixed in 1.15.2-gke.44:

  • Fixed a bug where after an upgrade to 1.15, clusters used the non-high-availability (HA) Connect Agent.
  • Fixed a known issue where $ in the private registry username caused admin control plane machine startup failure.
  • Fixed a known issue where user cluster update failed after KSA signing key rotation.
  • Fixed a known issue where gkectl diagnose snapshot failed to limit the time window for journalctl commands running on the cluster nodes when you take a cluster snapshot with the --log-since flag.

The following vulnerabilities are fixed in 1.15.2-gke.44:

July 06, 2023

Anthos clusters on VMware 1.14.6-gke.23 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.6-gke.23 runs on Kubernetes 1.25.10-gke.1200.

The following issues are fixed in 1.14.6-gke.23:

  • Fixed a known issue where $ in the private registry username caused admin control plane machine startup failure.
  • Fixed a known issue where gkectl diagnose snapshot failed to limit the time window for journalctl commands running on the cluster nodes when you take a cluster snapshot with the --log-since flag.
  • Fixed a known issue where user cluster update failed after KSA signing key rotation.

The following vulnerabilities are fixed in 1.14.6-gke.23:

High-severity container vulnerabilities:

June 27, 2023

Security bulletin

A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.

For more information, see the GCP-2023-016 security bulletin.

Security bulletin

With CVE-2023-31436, an out-of-bounds memory access flaw was found in the Linux kernel's traffic control (QoS) subsystem in how a user triggers the qfq_change_class function with an incorrect MTU value of the network device used as lmax. This flaw allows a local user to crash or potentially escalate their privileges on the system.

For more information, see the GCP-2023-017 security bulletin.

Security bulletin

A new vulnerability (CVE-2023-2235) has been discovered in the Linux kernel that can lead to a privilege escalation on the node. For more information, see the GCP-2023-018 security bulletin.

June 20, 2023

Security bulletin

A new vulnerability, CVE-2023-0468, has been discovered in the Linux kernel that could allow an unprivileged user to escalate privileges to root when io_poll_get_ownership will keep increasing req->poll_refs on every io_poll_wake then overflow to 0 which will fput req->file twice and cause a struct file refcount issue. GKE clusters, including Autopilot clusters, with Container-Optimized OS using Linux Kernel version 5.15 are affected. GKE clusters using Ubuntu images or using GKE Sandbox are unaffected.

For more information, see the GCP-2023-015 security bulletin.

June 16, 2023

Security bulletin

Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).

For more information, see the GCP-2023-014 security bulletin

June 14, 2023

Anthos clusters on VMware 1.14.5-gke.41 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.5-gke.41 runs on Kubernetes 1.25.8-gke.1500.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.15, 1.14, and 1.13.

The component access service account key for an admin cluster using a private registry can be updated in 1.14.5 and later. See
Rotating service account keys for details.

The following issues are fixed in 1.14.5-gke.41:

  • Fixed a known issue where the kind cluster downloads container images from docker.io. These container images are now preloaded in the kind cluster container image.
  • Fixed a bug where disks may be out of order in the first boot, causing node bootstrap failure.
  • Fixed a known issue where node ID verification failed to handle hostnames with dots.
  • Fixed an issue where gcloud fails to update the platform when the required-platform-version is already the current platform version.
  • Fixed the Anthos Config Management gcloud issue that the policy controller state might be falsely reported as pending.
  • Fixed continuously increasing memory usage of the logging agent stackdriver-log-forwarder.
  • Fixed the wrong admin cluster resource link annotation key that can cause the cluster to be enrolled in the Anthos On-Prem API again by mistake.
  • Fixed a known issue where some cluster nodes couldn't access the HA control plane when the underlying network performs ARP suppression.
  • Fixed a known issue where vsphere-csi-secret is not updated during gkectl update credentials vsphere for admin cluster

The following vulnerabilities are fixed in 1.14.5-gke.41

Anthos clusters on VMware 1.13.9-gke.29 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.9-gke.29 runs on Kubernetes 1.24.11-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.15, 1.14, and 1.13.

The following issues are fixed in 1.13.9-gke.29:

  • Fixed a known issue where the kind cluster downloads container images from docker.io. These container images are now preloaded in the kind cluster container image.
  • Fixed the issue where gkectl failed to limit the time window for journalctl commands running on the cluster nodes when you take a cluster snapshot with the --log-since flag.
  • Fixed an issue where gcloud fails to update the platform when the required-platform-version is already the current platform version.
  • Fixed a known issue where nodes fail to register if the configured hostname contains a period.
  • Fixed the wrong admin cluster resource link annotation key that can cause the cluster to be enrolled again by mistake.

The following high-severity container vulnerabilities are fixed in 1.13.9-gke.29:

June 06, 2023

Security bulletin

A new vulnerability (CVE-2023-2878) has been discovered in the secrets-store-csi-driver where an actor with access to the driver logs could observe service account tokens. These tokens could then potentially be exchanged with external cloud providers to access secrets stored in cloud vault solutions. The severity of this Security Bulletin is None. For more information, see the GCP-2023-009 security bulletin.

June 05, 2023

Known issue

If you create a version 1.13.8 or version 1.14.4 admin cluster, or upgrade an admin cluster to version 1.13.8 or 1.14.4, the kind cluster pulls the following container images from docker.io:

  • docker.io/kindest/kindnetd
  • docker.io/kindest/local-path-provisioner
  • docker.io/kindest/local-path-helper

If docker.io isn't accessible from your admin workstation, the admin cluster creation or upgrade fails to bring up the kind cluster.

This issue affects the following versions of Anthos clusters on VMware:

  • 1.14.4
  • 1.13.8

For more information, including a workaround, see kind cluster pulls container images from docker.io on the Known issues page.

Security bulletin

A new vulnerability (CVE-2023-1872) has been discovered in the Linux kernel that can lead to a privilege escalation to root on the node. For more information, see the GCP-2023-008.

June 01, 2023

Anthos clusters on VMware 1.15.1-gke.40 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.15.1-gke.40 runs on Kubernetes 1.26.2-gke.1001.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.15, 1.14, and 1.13.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on VMware.

Upgrading an admin cluster with always-on secrets encryption enabled might fail.

An admin cluster upgrade from 1.14.x to 1.15.0 - 1.15.4 with always-on secrets encryption enabled might fail depending on whether the feature was enabled during cluster creation or during cluster update.

We recommend that you don't upgrade your admin cluster until a fix is available in 1.15.5. If you must upgrade to 1.15.0-1.15.4, do the steps in Preventing the upgrade failure before upgrading the cluster.

For information on working around an admin cluster failure because of this issue, see Upgrading an admin cluster with always-on secrets encryption enabled fails. Note that the workaround relies on you having the old encryption key backed up. If the old key is no longer available, you will have to recreate the admin cluster and all user clusters.

  • Fixed a known issue where node ID verification failed to handle hostnames with dots.

  • Fixed continuous increase of logging agent memory.

  • Fixed an issue where cluster-api-controllers in a high-availability admin cluster had no Pod anti-affinity. This could allow the three clusterapi-controllers Pods not to be scheduled on different control-plane nodes.

  • Fixed the wrong admin cluster resource link annotation key that can cause the cluster to be enrolled again by mistake.

  • Fixed a known issue where node pool creation failed because of duplicated VM-Host affinity rules.

  • The preflight check for StorageClass parameter validations now throws a warning instead of a failure on ignored parameters after CSI Migration. StorageClass parameter diskformat=thin is now allowed and does not generate a warning.

  • Fixed an issue where gkectl repair admin-master might fail with Failed to repair: failed to delete the admin master node object and reboot the admin master VM.

  • Fixed a race condition where some cluster nodes couldn't access the high-availability control plane when the underlying network performed ARP suppression.

  • Fixed a false error message for gkectl prepare when using a high-availability admin cluster.

  • Fixed an issue where during user cluster update, DeprecatedKubeception always shows up in the diff.

  • Fixed an issue where there were leftover Pods with failed status due to Predicate NodeAffinity failed during node re-creation.

Fixed the following vulnerabilities:

May 18, 2023

Security bulletin

Two new vulnerabilities (CVE-2023-1281, CVE-2023-1829) have been discovered in the Linux kernel that can lead to a privilege escalation to root on the node. For more information, see the GCP-2023-005 security bulletin.

May 15, 2023

Anthos clusters on VMware 1.13.8-gke.42 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.8-gke.42 runs on Kubernetes 1.24.11-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.15, 1.14, and 1.13.

  • Fixed a race condition where some cluster nodes couldn't access the HA control plane when the underlying network performed ARP suppression.

  • Fixed an issue where vsphere-csi-secret was not updated during gkectl update credentials vsphere for an admin cluster.

  • Disabled motd news on the ubuntu_containerd image to avoid unexpected connections to Canonical.

  • Fixed an issue where the Connect Agent continued using the older image after registry credential update.

  • Fixed an issue where cluster autoscaler ClusterRoleBindings in the admin cluster were accidentally deleted upon user cluster deletion. This fix removes dependency on ClusterRole, ClusterRoleBinding and ServiceAccount objects in the admin cluster.

  • Fixed an issue where Connect Agent in admin clusters might fail to be upgraded during cluster upgrade.

  • Fixed an issue where a cluster might not be registered when the initial membership creation attempt failed.

Fixed the following vulnerabilities:

May 02, 2023

Anthos clusters on VMware 1.15.0-gke.581 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.15.0-gke.581 runs on Kubernetes 1.26.2-gke.1001.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.15, 1.14, and 1.13.

Upgrading an admin cluster with always-on secrets encryption enabled might fail.

An admin cluster upgrade from 1.14.x to 1.15.0 - 1.15.4 with always-on secrets encryption enabled might fail depending on whether the feature was enabled during cluster creation or during cluster update.

We recommend that you don't upgrade your admin cluster until a fix is available in 1.15.5. If you must upgrade to 1.15.0-1.15.4, do the steps in Preventing the upgrade failure before upgrading the cluster.

For information on working around an admin cluster failure because of this issue, see Upgrading an admin cluster with always-on secrets encryption enabled fails. Note that the workaround relies on you having the old encryption key backed up. If the old key is no longer available, you will have to recreate the admin cluster and all user clusters.

  • CSI migration for the vSphere storage driver is enabled by default. A new storage preflight check and a new CSI workload preflight check verify that PersistentVolumes that used the old in-tree vSphere storage driver will continue to work with the vSphere CSI driver. There is a known issue during admin cluster upgrade. If you see a preflight check about a StorageClass diskformat parameter, you can use --skip-validation-cluster-health to skip the check. This issue will be fixed in a future release.

  • The minimum required version of vCenter and ESXi is 7.0 Update 2.

  • Preview: Support for vSphere 8.0

  • Preview: Support for VM-Host affinity for user cluster node pools

  • Preview: Support for High availability control plane for admin clusters

  • Preview: Support for system metrics collection using Google Cloud Managed Service for Prometheus

  • Preview: You can now filter application logs by namespace, Pod labels and content regex.

  • Preview: Support for storage policy in user clusters

  • Preview: You can now use gkectl diagnose snapshot --upload=true to upload a snapshot. And gkectl helps generate the Cloud Storage bucket with the format gs://anthos-snapshot[uuid]/vmware/$snapshot-name.

  • GA: Support for upgrade and rollback of node pool version

  • GA: gkectl get-config is a new command that locally generates cluster configuration files from an existing admin or user cluster.

  • GA: Support for multi-line parsing of Go and Java logs

  • GA: Support for manual load balancing in user clusters that enable ControlplaneV2

  • GA: Support for update of private registry credentials

  • GA: Metrics and logs in the bootstrap cluster are now uploaded to Google Cloud through Google Cloud's operations suite to provide better observability on admin cluster operations.

  • GA: vSphere CSI is now enabled for Windows node pools.

  • Fully managed Cloud Monitoring Integration dashboards. The new Integration Dashboard is automatically installed. You cannot make changes to the following dashboards, because they are fully managed by Google. However, you can make a copy of a dashboard and customize the copied version:

    • Anthos Cluster Control Plane Uptime
    • Anthos Cluster Node Status
    • Anthos Cluster Pod Status
    • Anthos Cluster Utilization Metering
    • Anthos Cluster on VMware VM Status
  • Admin cluster update operations are now managed by an admin cluster controller.

  • The Connect Agent now runs in high availability mode.

  • The metrics server now runs in high-availability mode.

  • Upgraded the VMware vSphere Container Storage Plug-in from 2.7 to 3.0. This includes support for Kubernetes version 1.26. For more information, see the plug-in release notes.

  • Upgraded Anthos Identity Service to hybrid_identity_charon_20230313_0730_RC00.

  • Switched the node selector from node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane and added toleration node-role.kubernetes.io/control-plane to system components.

  • Controlplane V2 is now the default for new user clusters.

  • Now when you delete a Controlplane V2 user cluster , the data disk is automatically deleted.

  • Cluster DNS now supports ordering policy for upstream servers.

  • Added admin cluster CA certificate validation to the admin cluster upgrade preflight check.

  • Upgraded Anthos Network Gateway to 1.4.4.

  • Updated anthos-multinet.

  • When you upload and share a snapshot using gkectl diagnose snapshot with a Google Support team service account service-[GOOGLE_CLOUD_PROJECT_NUMBER]@gcp-sa-anthossupport.iam.gserviceaccount.com, gkectl helps provision the service account automatically.

  • Upgraded node-exporter from 1.0.1 to 1.4.1.

  • Upgraded Managed Service for Prometheus for application metrics from 0.4 to 0.6.

  • We now allow storage DRS to be enabled in manual mode.

  • GKE connect is now required for admin clusters, and you cannot skip the corresponding validation. You can register existing admin clusters by using gkectl update admin.

  • We no longer silently skip saving empty files in diagnose snapshots, but instead collect the names of those files in a new empty_snapshots file in the snapshot tarball.

  • We now mount /opt/data using disk label data.

  • In the vSphere CSI driver, enabled improved-csi-idempotency and async-query-volume, and disabled trigger-csi-fullsync. This enhances the vSphere CSI driver to ensure volume operations are idempotent.

  • Changed the relative file path fields in the admin cluster configuration file to use absolute paths

  • Removed kubectl describe events in cluster snapshots for a better user experience. kubectl describe events fail when the target event expires. In contrast kubectl get events survive and provide enough debugging information.

Deprecations

  • Support for gkeadm on MAC and Windows is deprecated.

  • The enableWindowsDataplaneV2 field in the user cluster configuration file is deprecated.

  • The gkectl enroll cluster command is deprecated. Use gcloud to enroll a user cluster instead.

  • The following dashboards in the Cloud Monitoring Sample Library will be deprecated in a future release:

    • Anthos cluster control plane uptime
    • Anthos cluster node status
    • Anthos cluster pod status
    • Anthos utilization metering
    • GKE on-prem node status
    • GKE on-prem control plane uptime
    • GKE on-prem pod status
    • GKE on-prem vSphere vm health status
  • In a future release, the following customized dashboards will not be created when you create a new cluster:

    • GKE on-prem node status
    • GKE on-prem control plane uptime
    • GKE on-prem pod status
    • GKE on-prem vSphere vm health status
    • GKE on-prem Windows pod status
    • GKE on-prem Windows node status
  • Fixed the false error message generated by the cluster autoscaler about a missing ClusterRoleBinding. After a user cluster is deleted, that ClusterRoleBinding is no longer needed.

  • Fixed an issue where gkectl check-config failed (nil pointer error) during validation for Manual load balancing.

  • Fixed an issue where the cluster autoscaler did not work when Controlplane V2 was enabled.

  • Fixed an issue where using gkectl update to enable Cloud Audit Logs did not work.

  • Fixed an issue where a preflight check for Seesaw load balancer creation failed if the Seesaw group file already existed.

  • We now backfill the OnPremAdminCluster OSImageType field to prevent an unexpected diff during update.

  • Fixed an issue where disks might be out of order during the first boot.

  • Fixed an issue where the private registry credentials file for the user cluster could not be loaded.

  • Fixed an issue where the user-cluster node options and startup script used the cluster version instead of the node pool version.

  • Fixed an issue where gkectl diagnose cluster didn't check the health of control-plane Pods for kubeception user clusters.

  • Fixed an issue where KSASigningKeyRotation always showed as an unsupported change during user cluster update.

  • Fixed an issue where a cluster might not be registered when the initial membership creation attempt failed.

  • Fixed an issue where user cluster data disk validation used the cluster-level vCenter.datastore instead of masterNode.vsphere.datastore.

  • Fixed an issue where component-access-sa-key was missing in the admin-cluster-creds Secret after admin cluster upgrade.

  • Fixed an issue where during user cluster upgrade, the cluster state indicated that upgrade had completed before CA rotation had completed.

  • Fixed an issue where advanced networking components were evicted or not scheduled on nodes because of Pod priority.

  • Fixed a known issue where the calico-node Pod was unable to renew the auth token in the calico CNI kubeconfig file.

  • Fixed Anthos Identity Service metric exporting issues.

  • During preflight checks and cluster diagnosis, we now skip PersistentVolumes and PersistentVolumeClaims that use non-vSphere drivers.

  • Fixed a known issue where CIDR ranges could not be used in the IP block file.

  • Fixed an issue where auto resizing of CPU and memory for an admin cluster add-on node got reset by an admin cluster controller.

  • anet-operator can now be scheduled to a Windows node in a user cluster that has Controlplane V2 enabled.

May 01, 2023

Anthos clusters on VMware 1.14.4-gke.54 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.4-gke.54 runs on Kubernetes 1.25.8-gke.1500.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

Added admin cluster CA certificate validation to the admin cluster upgrade preflight check.

  • Fixed an issue where the Connect Agent continued using the older image after registry credential update.

  • Fixed an issue where the cluster autoscaler did not work when Controlplane V2 was enabled.

  • Fixed an issue where a cluster might not be registered when the initial membership creation attempt failed.

  • Fixed an issue where ClusterRoleBindings in the admin cluster were accidentally deleted upon user cluster deletion. This fix removes dependency on ClusterRole, ClusterRoleBinding and ServiceAccount objects in the admin cluster.

  • Fixed an issue where a preflight check for Seesaw load balancer creation failed if the Seesaw group file already existed.

  • Disabled motd news on the ubuntu_containerd image.

  • Fixed an issue where gkectl check-config failed at Manual LB slow validation with a nil pointer error.

  • Fix an issue where enabling Cloud Audit Logs with gkectl update did not work.

Fixed the following vulnerabilities:

April 13, 2023

Anthos clusters on VMware 1.12.7-gke.20 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.12.7-gke.20 runs on Kubernetes 1.23.17-gke.900.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

  • Added admin cluster CA certificate validation to the admin cluster upgrade preflight check.

  • We now allow storage DRS to be enabled in manual mode.

  • Fixed an issue where using gkectl update to enable Cloud Audit Logs did not work.

  • We now backfill the OnPremAdminCluster OSImageType field to prevent an unexpected diff during update.

  • Fixed an issue where a preflight check for Seesaw load balancer creation failed if the Seesaw group file already existed.

April 12, 2023

Kubernetes image registry redirect

As of March 21, 2023, traffic to k8s.gcr.io is redirected to registry.k8s.io, following the community announcement. This change is happening gradually to reduce disruption, and should be transparent for most Anthos clusters.

To check for edge cases and mitigate potential impact to your clusters, follow the step-by-step guidance in k8s.gcr.io Redirect to registry.k8s.io - What You Need to Know.

April 11, 2023

1.13.7 patch release

Anthos clusters on VMware 1.13.7-gke.29 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.7-gke.29 runs on Kubernetes 1.24.11-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

Fixed for 1.13.7

  • Fixed an issue where gkectl check-config fails at Manual LB slow validation with a nil pointer error.

  • Fixed a bug where enabling Cloud Audit Logs with gkectl update did not work.

  • Fixed an issue where a preflight check for Seesaw load balancer creation failed if the Seesaw group file already existed.

  • We now backfill the OnPremAdminCluster OSImageType field to prevent an unexpected diff during update.

Security bulletin

Two new vulnerabilities, CVE-2023-0240 and CVE-2023-23586, have been discovered in the Linux kernel that could allow an unprivileged user to escalate privileges. For more information, see the GCP-2023-003 security bulletin.

1.12.7-gke.19 bad release

Anthos clusters on VMware 1.12.7-gke.19 is a bad release and you should not use it. The artifacts have been removed from the Cloud Storage bucket.

April 03, 2023

Anthos clusters on VMware 1.14.3-gke.25 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.3-gke.25 runs on Kubernetes 1.25.5-gke.100.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

We now allow storage DRS to be enabled in manual mode.

  • We now backfill the OnPremAdminCluster OSImageType field to prevent an unexpected diff during cluster update.

  • Fixed an issue where gkectl diagnose cluster didn't check the health of control-plane Pods for kubeception user clusters.

  • Fixed an issue where the user-cluster node options and startup script used the cluster version instead of the node pool version.

Fixed the following vulnerabilities:

March 17, 2023

Anthos clusters on VMware 1.13.6-gke.32 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.6-gke.32 runs on Kubernetes 1.24.10-gke.2200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

  • Fixed an issue with Anthos Identity Service to better scale and handle concurrent authentication requests.

  • Fixed an issue where component-access-sa-key was missing in the admin-cluster-creds Secret after admin cluster upgrade.

Fixed the following vulnerabilities:

March 07, 2023

Anthos clusters on VMware 1.14.2-gke.37 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.2-gke.37 runs on Kubernetes 1.25.5-gke.100.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

We no longer silently skip saving empty files in diagnose snapshots, but instead collect the names of those files in a new empty_snapshots file in the snapshot tarball.

  • Fixed an issue where user cluster data disk validation used the cluster-level datastore vsphere.datastore instead of masterNode.vsphere.datastore.

  • Fixed an issue with Anthos Identity Service to better scale and handle concurrent authentication requests.

  • Fixed an issue where component-access-sa-key was missing in the admin-cluster-creds Secret after admin cluster upgrade.

  • Fixed an issue where user cluster upgrade triggered through the Google Cloud console might flap between ready and non-ready states until CA rotation fully completes.

  • Fixed an issue where gkectl diagnose cluster might generate false failure signals with non-vSphere CSI drivers.

  • Fixed an issue where admin cluster update doesn't wait for user control-plane machines to be re-created when using ControlPlaneV2.

Fixed the following vulnerabilities:

March 06, 2023

Cluster lifecycle improvements versions 1.13.1 and later

You can use the Google Cloud console or the gcloud CLI to upgrade user clusters managed by the Anthos On-Prem API. The upgrade steps differ depending on your admin cluster version. For more information, see the version of the documentation that corresponds to your admin cluster version:

1.12.6 patch release

Anthos clusters on VMware 1.12.6-gke.35 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.12.6-gke.35 runs on Kubernetes v1.23.16-gke.2400.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

  • Fixed a bug where KSASigningKeyRotation always shows as an unsupported change during user cluster update.
  • Fixed an issue with Anthos Identity Service to better scale and handle concurrent authentication requests.

  • Fixed an issue where component-access-sa-key was missing in the admin-cluster-creds Secret after admin cluster upgrade.

Fixed the following vulnerabilities:

March 01, 2023

A new vulnerability (CVE-2022-4696) has been discovered in the Linux kernel that can lead to a privilege escalation on the node. Anthos clusters on VMware running v1.12 and v1.13 are impacted. Anthos clusters on VMware running v1.14 or later are not affected.

For instructions and more details, see the Anthos clusters on VMware security bulletin.

February 13, 2023

Anthos clusters on VMware 1.13.5-gke.27 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.5-gke.27 runs on Kubernetes 1.24.9-gke.2500.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

  • Updated the Ubuntu image to ubuntu-gke-op-2004-1-13-v20230201 using node kernel version 5.4.0.1062.60.

  • Instead of ignoring snapshots files with empty content, we save their names in a new file named empty_snapshots.

During preflight checks and cluster diagnosis, we now skip PVs and PVCs that use non-vSphere drivers.

Fixed the following vulnerabilities:

January 31, 2023

Anthos clusters on VMware 1.14.1-gke.39 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.1-gke.39 runs on Kubernetes 1.25.5-gke.100.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

  • In the admin cluster configuration file, gkeadm now prepopulates caCertPath and the service account key paths with absolute paths instead of relative paths.

  • In the vSphere CSI driver, enabled improved-csi-idempotency, and async-query-volume, and disabled trigger-csi-fullsync. This enhances the vSphere CSI driver to ensure volume operations are idempotent.

  • Fixed a known issue where the calico-node Pod is unable to renew the auth token in the calico CNI kubeconfig file.

  • Fixed a known issue where CIDR ranges cannot be used in the IP block file.

Fixed the following vulnerabilities:

January 26, 2023

Anthos clusters on VMware 1.12.5-gke.34 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.12.5-gke.34 runs on Kubernetes 1.23.15-gke.2400.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

In the vSphere CSI driver, enabled improved-csi-idempotency, and async-query-volume, and disabled trigger-csi-fullsync. This enhances the vSphere CSI driver to ensure volume operations are idempotent.

  • If you specify a CIDR range (subnet) in the IP block file for your cluster nodes, the broadcast IP of the subnet, the network CIDR IP, and the network gateway IP will be excluded from the pool of addresses that get assigned to nodes.

  • Fixed a known issue where CIDR ranges cannot be used in the IP block file.

  • Fixed a bug where CA rotation appeared as an unsupported change during admin cluster update.

Fixed the following vulnerabilities:

January 25, 2023

Anthos clusters on VMware version 1.14.0 has a known issue where the calico-node Pod is unable to renew the auth token in the calico CNI kubeconfig file. For more information, see Pod create or delete errors due to Calico CNI service account auth token issue.

Because of this issue, you cannot use Anthos On-Prem API clients (Google Cloud console and gcloud CLI) to create and manage 1.14.0 clusters.

January 12, 2023

Anthos clusters on VMware 1.13.4-gke.19 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.4-gke.19 runs on Kubernetes 1.24.9-gke.100

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

  • In the vSphere CSI driver, enabled improved-csi-idempotency, and async-query-volume, and disabled trigger-csi-fullsync. This enhances the vSphere CSI driver to ensure volume operations are idempotent.

  • In the admin cluster configuration file, gkeadm now prepopulates caCertPath and the service account key paths with absolute paths instead of relative paths.

  • If you specify a CIDR range (subnet) in the IP block file for your cluster nodes, the broadcast IP of the subnet, the network CIDR IP, and the network gateway IP will be excluded from the pool of addresses that get assigned to nodes.
  • Fixed a bug where CIDR ranges cannot be used in an IP block file.

December 22, 2022

A new vulnerability (CVE-2022-2602) has been discovered in the io_uring subsystem in the Linux kernel that can allow an attacker to potentially execute arbitrary code.

For more information see the GCP-2022-025 security bulletin.

December 21, 2022

Anthos clusters on VMware 1.14.0-gke.430 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.14.0-gke.430 runs on Kubernetes 1.25.5-gke.100.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.14, 1.13, and 1.12.

  • Support for user cluster creation with Controlplane V2 enabled is now generally available. For more details on how to create a user cluster with this model, see Create a user cluster with Controlplane V2.
  • Preview: You can now roll back node pools to a previous working version if you detect an issue in the new version after a cluster upgrade. For more information, see Rolling back a node pool after an upgrade.
  • Preview: The following private registry updates are now available:
    • Support for private registry credentials using prepared Secrets is now available as a preview feature. A new privateRegistry field has been added in the Secrets configuration file.
    • Added a new privateRegistry section in the user cluster configuration file. You can use different private registry credentials for the user cluster and admin cluster. You can also use a different private registry address for user clusters with Controlplane V2 enabled.
    • You can also update private registry credentials for an admin cluster or user cluster with the gkectl update credentials command. For more information, see Update private registry credentials.
  • Cluster names are now included in kubeconfig files when creating a new admin cluster or user cluster. If you are upgrading your existing cluster to 1.14.0 or higher, the existing kubeconfig file is updated with the cluster name.
  • cluster-health-controller is now integrated with health-check-exporter to emit metrics based on the periodic health check results, making it easy to monitor and detect cluster health problems.
  • GA: The node pool update policy is generally available. With this feature, you can configure the value of maximumConcurrentNodePoolUpdate in the user cluster configuration file to 1. This will configure the maximum number of additional nodes spawned during cluster upgrade or update, which can potentially avoid two issues — resource quota limit issue and PDB deadlock issue. For more information, see Configure node pool update policy.
  • Support for vSphere cluster/host/network/datastore folders is generally available. You can use folders to group objects of the same type for easier management. For more information, see Specify vSphere folders in cluster configuration and the relevant sections in the admin cluster and user cluster configuration files.
  • Added a feature enabling cluster administrators to configure RBAC policies based on Azure Active Directory (AD) groups. Group information for users belonging to more than 200 groups can now be retrieved.
  • Upgraded Kubernetes from 1.24 to 1.25:
    • Migrated PDB API version from policy/v1beta1 to policy/v1. You must ensure that any workload PDB API version is updated to policy/v1 before upgrading your cluster to 1.14.0.
    • Migrated autoscaling/v2beta1 to autoscaling/v2.
    • Disabled CSI Migration for vSphere as this is enabled by default in Kubernetes 1.25.
  • Added storage validation that checks if in-use Kubernetes PersistentVolumes (PV) have disks present in the configured datastore, and if node.Status.VolumesAttached is consistent with the actual PV/disk attachment states during admin and user cluster upgrade preflight checks.
  • Updated gcloud version to 410.0.0 on the admin workstation.
  • Upgraded VMware vSphere Container Storage Plug-in from 2.5 to 2.7. This version bump includes support for Kubernetes version 1.25. For more information, see VMware vSphere Container Storage Plug-in 2.7 Release Notes.
  • In the generated user cluster configuration template, the prepopulated value for enableDataplaneV2 is now true.
  • Removed unnecessary RBAC policies for managing the lifecycle of user clusters in the Google Cloud console.
  • Updated the parser of container logs to extract severity level.
  • Simplified the cluster snapshot uploading process by automatically retrieving GKE connect-register service account key, and making the flag --service-account-key-file optional. When the cluster is not registered correctly, and no additional service account key file is passed in through the flag, the gkectl diagnose snapshot command will use the GOOGLE_APPLICATION_CREDENTIALS environment variable to authenticate the request.
  • Upgraded Container-Optimized OS to m101.
  • In the admin cluster and user cluster configuration file templates, loadbalancer.kind field is now prepopulated with MetalLB.

A known issue has been discovered. See the January 25, 2023 release note.

December 20, 2022

Anthos clusters on VMware 1.12.4-gke.42 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.12.4-gke.42 runs on Kubernetes 1.23.13-gke.1700.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.13, 1.12, and 1.11.

  • Changed the relative file path fields in the admin cluster configuration file to use absolute paths.
  • Added yq tool in the admin workstation.

December 15, 2022

Anthos clusters on VMware 1.13.3-gke.26 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.3-gke.26 runs on Kubernetes 1.24.7-gke.1700.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.13, 1.12, and 1.11.

  • Added yq tool in the admin workstation to simplify troubleshooting.
  • Upgraded VMware vSphere Container Storage Plug-in from 2.5 to 2.6.2. This version bump includes support for Kubernetes version 1.24. For more information, see VMware vSphere Container Storage Plug-in 2.6 Release Notes.
  • Added storage validation that checks Kubernetes PersistentVolumes and vSphere virtual disks as part of admin and user cluster upgrade preflight checks.
  • Fixed an issue where anet-operator could be scheduled to a Windows node with enableControlplaneV2: true.
  • Fixed OOM events associated with monitoring-operator- Pods by increasing memory limit to 1GB.
  • Fixed the issue where deleting a user cluster also deleted cluster-health-controller and vsphere-metrics-exporter ClusterRole objects.
  • Fixed the following vulnerabilities:

December 08, 2022

Anthos clusters on VMware 1.11.6-gke.18 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.11.6-gke.18 runs on Kubernetes 1.22.15-gke.3300.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.13, 1.12, and 1.11.

November 17, 2022

Anthos clusters on VMware 1.13.2-gke.26 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.2-gke.26 runs on Kubernetes 1.24.7-gke.1400.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.13, 1.12, and 1.11.

  • Fixed a validation error where the GKE Hub membership is not found when using a gcloud version that is not bundled with the admin workstation.
  • Fixed the issue where the admin cluster might fail to register due to naming conflicts.
  • Fixed the issue where the Connect Agent in the admin cluster does not upgrade after a failure to upgrade nodes in the user cluster control plane.
  • Fixed a bug where running gkectl diagnose snapshot using system scenario did not capture Cluster API resources in the default namespace.
  • Fixed the issue during admin cluster creation where gkectl check-config fails due to missing OS images, if gkectl prepare is not run first.
  • Fixed the unspecified Internal Server error in ClientConfig when using the Anthos Identity Service (AIS) hub feature to manage the OpenID Connect (OIDC) configuration.
  • Fixed the issue of /var/log/audit/ filling up disk space on the admin workstation.
  • Fixed an issue where cluster deletion may be stuck at node draining when the user cluster control plane and node pools are on different datastores.
  • Fixed the issue where nodes fail to register if the configured hostname in the IP block file contains one or more periods.
  • Fixed the following vulnerabilities:

November 10, 2022

Anthos clusters on VMware 1.11.5-gke.14 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.11.5-gke.14 runs on Kubernetes 1.22.15-gke.2200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.13, 1.12, and 1.11.

November 09, 2022

Two new vulnerabilities, CVE-2022-2585 and CVE-2022-2588, have been discovered in the Linux kernel that can lead to a full container break out to root on the node.

For more information, see the GCP-2022-024 security bulletin.

November 07, 2022

A security vulnerability, CVE-2022-39278, has been discovered in Istio, which is used in Anthos Service Mesh, that allows a malicious attacker to crash the control plane.

For instructions and more details, see the Anthos clusters on VMware security bulletin.

November 01, 2022

Anthos clusters on VMware 1.13.1-gke.35 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.1-gke.35 runs on Kubernetes 1.24.2-gke.1900.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.13, 1.12, and 1.11.

  • Increased logging granularity for the cluster backup operation including indicating status for each step of the process.

October 28, 2022

A new vulnerability, CVE-2022-20409, has been discovered in the Linux kernel that could allow an unprivileged user to escalate to system execution privilege.

For instructions and more details, see the Anthos clusters on VMware security bulletin.

October 27, 2022

A new vulnerability, CVE-2022-3176, has been discovered in the Linux kernel that can lead to local privilege escalation. This vulnerability allows an unprivileged user to achieve full container breakout to root on the node.

For instructions and more details, see the Anthos clusters on VMware security bulletin.

October 25, 2022

Anthos clusters on VMware 1.12.3-gke.23 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.12.3-gke.23 runs on Kubernetes 1.23.8-gke.1900.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.13, 1.12, and 1.11.

  • Fixed the issue of a race condition that blocks the deletion of an old machine object during cluster upgrade or update.
  • Fixed an issue for clusters enabled with Anthos Network Gateway where the NetworkGatewayGroup object may erroneously report nodes as having NotHealthy status.
  • Fixed an issue where creating or updating NetworkGatewayGroup objects fails because of a webhook IP conflict error.
  • Fixed the following vulnerabilities:

October 13, 2022

Anthos clusters on VMware 1.11.4-gke.32 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.11.4-gke.32 runs on Kubernetes 1.22.8-gke.204.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.13, 1.12, and 1.11.

October 12, 2022

The Connect Agent version used in Anthos clusters on VMware versions 1.8 and earlier is no longer supported. If you upgrade your user cluster to these versions, the gkectl updgrade cluster command may fail. If you encounter this issue and need further assistance, you should contact Google Support.

October 11, 2022

If you use gcloud anthos version 1.4.2, and authenticate an Anthos cluster on VMware with gcloud anthos auth, the command fails with the following error:

Decryption failed, no keys in the current key set could decrypt the payload.

To resolve this, you must upgrade gcloud anthos to 1.4.3 or above (gcloud SDK 397.0.0 or above) to authenticate clusters with gcloud anthos auth.

September 29, 2022

Anthos clusters on VMware 1.13.0-gke.525 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.13.0-gke.525 runs on Kubernetes 1.24.2-gke.1900.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.13, 1.12, and 1.11.

vSphere versions below 7.0 Update 1 are no longer supported in Anthos clusters on VMware. You must upgrade vSphere (both ESXi and vCenter) to version 7.0 Update 1 or above before you can upgrade to Anthos clusters on VMware 1.13.0. If you want to use the vSphere Container Storage Interface driver or NFSv3, then you must upgrade to vSphere 7.0 Update 2 or a later update of version 7.0.

Cluster life-cycle Improvements:

  • GA: A new asynchronous variation of the user cluster upgrade is now supported. With this variation, the gkectl upgrade cluster command starts the upgrade and completes. You don't need to watch the output of the command for the entire duration of the upgrade. For more details, see Upgrade a user cluster.
  • Preview: You can now update node pools either sequentially or maintain the default parallel behavior by specifying the value of maximumConcurrentNodePoolUpdate in your user cluster configuration file. Setting the value to 1 will configure the node pool update to be sequential, which can potentially avoid two issues — resource quota limit issue and PDB deadlock issue.
  • Introduced an admin cluster controller for managing the admin cluster lifecycle.
  • Added new preflight checks:
    • Check that node IPs are in the subnet for IPAM.
    • A new preflight check was added to validate the clusterLocation field under stackdriver and cloudAuditLogging. This preflight check requires the component access service account to have the compute.viewer role, and the compute.googleapis.com to be allowlisted in the HTTP proxy and firewall settings. If you use an invalid value in the clusterLocation, the preflight check will fail. You can correct the invalid clusterLocation by removing the stackdriver and/or cloudAuditLogging configurations from the admin or user cluster configuration files, applying the changes with gkectl update, and then add the corrected configurations back. Or, you can use --skip-validation-gcp to skip the check. Note that having an invalid clusterLocation will cause a failure to export logs and metrics.
    • For a cluster in static IP mode, you need to have one IP address for each node and an additional IP address. This additional IP address will be used for a temporary node during cluster update, upgrade and auto-repair.
    • Validate that IP addresses are not in docker IP range in IPAM mode.
    • Check to make sure there is no node port collision among different user clusters in manual load balancing mode.
    • Check datastore size to ensure it has enough capacity for surge machine.
    • Check for an available IP address for creating Windows VM template in IPAM mode.
    • PDB preflight check to prevent multiple PDBs from matching with the same pod.

Platform enhancements:

  • GA: Support for cos OS image type in admin cluster nodes is now generally available. You can update the admin node image type with the gkectl update admin command.
  • Preview: A new user cluster deployment model with support for multi-vCenter deployments is available as a preview feature. For more details on how to create a user cluster with this new model, see Create a user cluster with a new installation model.
  • Preview: vSphere CSI volume snapshot is now available as a preview feature. This feature provides the ability to create volume snapshots and restore volumes from snapshots using VMware Cloud Native Storage. To use this feature, you must update both vCenter Server and ESXi to version 7.0 Update 3 or later.

Security enhancements:

  • GA: Support for storing credentials for user clusters as Kubernetes Secrets is generally available.

    • With this feature, users can prepare credentials for the user cluster, and store them as Kubernetes Secrets in the admin cluster before a user cluster is created. After credential preparation, users can delete the Secrets configuration file which contains the user cluster credentials from the admin workstation. When creating a user cluster, the prepared credentials will be used. For more details, see Configure prepared credentials for user clusters.
  • Kubernetes service account (KSA) Signing Key rotation is supported on user clusters. For more details, see Rotate KSA signing keys.

  • GA: Component access SA key rotation for both admin and user clusters is generally available.

  • GA: You can set up Connect gateway to use Google Group membership for authorization. For more information, see Set up the Connect gateway with Google Groups.

  • Changed kube-scheduler, kube-etcd, kube-apiserver and Key Management Service (KMS) components to run in rootless mode in the user cluster.

Simplify day-2 operations:

  • Preview: Added support of multi-line parsing for Go and Java logs.
  • GA: Launched the enablement of Google Cloud Managed Service for Prometheus to track metrics in Anthos on vSphere clusters, and introduced two separate flags to enable logging and monitoring for user applications separately: EnableCloudLoggingForApplications and EnableGMPForApplications. You can monitor and alert on the applications using Prometheus with Google-managed Prometheus without managing and operating Prometheus. You can set enableGMPForApplications in the Stackdriver spec to enable Google Managed Prometheus for application metrics without any other manual steps, and the Google Managed Prometheus components are then set up automatically. See Enable Managed Service for Prometheus for user applications for details.

  • Added a new Anthos Utilization Metering dashboard in Cloud Monitoring to monitor cluster health. The dashboard shows CPU and memory utilization in the clusters by namespace and Pod labels.

  • Upgraded to Ubuntu 20.04 and containerd 1.6.
  • connectgateway.googleapis.com API is now required to create new clusters in 1.13.0.
  • Updated the gcloud version in the admin workstation to 401.0.0.
  • Increased the default boot disk size for the admin workstation to 100GB.
  • SImplified the gkectl diagnose snapshot scenario usage. The --scenario flag is no longer needed for the admin cluster snapshot. Use system (default) or all values to specify scenarios for the user cluster snapshot. For more details, see Diagnosing cluster issues.
  • Improved gkectl diagnose cluster to detect and diagnose two general issues:
    • Node draining issues can block cluster upgrade
    • Kubernetes Cluster API resource managed by an Anthos clusters on VMware bundle might be accidentally modified which can cause failure of system components, or cluster upgrade or update failure.
  • Enforced admin cluster registration with preflight checks.

    • This also applies to admin clusters to be upgraded to 1.13. You can run gkectl update admin to register existing 1.12 admin clusters.
    • You can skip this check with the --skip-validation-config flag if you cannot register admin clusters for certain reasons.
  • Configuration for Logging and Monitoring is now enforced in admin and user cluster configuration files during creation preflight checking. You can run gkectl update cluster and gkectl update admin to enable Logging and Monitoring in existing 1.12 user or admin clusters before upgrading to 1.13. Otherwise, upgrade preflight checks will emit a warning. You can skip these checks with the --skip-validation-stackdriver flag if you cannot enable Logging and Monitoring for certain reasons. However, enabling Logging and Monitoring is strongly recommended to get better Google support, and there is no charge for this service on Anthos.

  • When Logging and Monitoring is enabled, the values of the gkeConnect.projectID field, stackdriver.projectID field, and cloudAuditLogging.projectID field must all be the same in the cluster configuration files. Otherwise, cluster creation preflight checks would fail with an error, and upgrade preflight checks would emit a warning. You can also skip these checks with the --skip-validation-stackdriver flag, but this is not recommended as using different project IDs for stackdriver and gkeconnect may cause friction during support and fleet management. Note you can still send logs and metrics to a different project through Cloud Logging sinks and metric viewer scoping.

  • Migrated metrics-server and addon-resizer to a new namespace: gke-managed-metrics-server.

  • Refined kube-state-metrics so that only core metrics are collected by default. Fewer resources are needed to collect this optimized set of metrics, which improves overall performance and scalability.

  • Fixed the issue of cloud-init log not showing in the serial console for Ubuntu.
  • Fixed the issue where user cluster check-config fails when the admin cluster uses cos as the osImageType.
  • Updated virtual hardware version to version 15 for creating VMs in Anthos cluster on VMware 1.13.0.
  • Fixed the issue of two missing metrics, scheduler and controller-manager, in the admin and user cluster.
  • Fixed the issue of an empty CPU readiness chart in OOTB dashboards that was caused by deprecated metrics.
  • Fixed the issue where you may not be able to add a new user cluster if a user cluster is stuck in the deletion process, and your admin cluster is set up with a MetalLB load balancer configuration.
  • Fixed the following vulnerabilities:
  • In the configuration file template generated by gkectl create-config cluster, the pre-populated value for the commented field kubeception is shown as false, while the default value is true.
  • In the configuration file template generated by gkectl create-config admin, gkeConnect is shown as an optional section, however it is actually a required section.

September 28, 2022

Anthos clusters on VMware 1.12.2-gke.21 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.12.2-gke.21 runs on Kubernetes 1.23.8-gke.1900.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.12, 1.11, and 1.10.

  • Fixed the issue where you may not be able to add a new user cluster if a user cluster is stuck in the deletion process, and your admin cluster is set up with a MetalLB load balancer configuration.
  • Fixed an issue where istiod starts up very slowly when connectivity to the Google Cloud metadata service is partially broken.
  • Fixed the issue where the admin control plane VM template is deleted after a resumed admin cluster upgrade attempt.
  • Fixed the issue where user cluster check-config fails when the admin cluster uses cos as the osImageType.
  • Fixed the following vulnerabilities:

September 08, 2022

Anthos clusters on VMware 1.10.7-gke.15 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.10.7-gke.15 runs on Kubernetes 1.21.14-gke.2100.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.12, 1.11, and 1.10.

Fixed for v1.10.7

Anthos clusters on VMware 1.11.3-gke.45 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.11.3-gke.45 runs on Kubernetes 1.22.8-gke.204.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.12, 1.11, and 1.10.

The gkectl diagnose cluster command automatically runs when gkectl diagnose snapshot is run, and the output is saved in a new folder in the snapshot called /diagnose-report.

Fixed for v1.11.3

August 25, 2022

Anthos clusters on VMware 1.12.1-gke.57 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.12.1-gke.57 runs on Kubernetes 1.23.5-gke.1505.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.12, 1.11, and 1.10.

  • GA: You can now have your GKE clusters in separate vSphere clusters. With this feature, you can deploy the admin cluster in one vSphere cluster, and a user cluster in a different vSphere cluster.
  • Fixed the issue where mounting emptyDir volume with exec option on Container-Optimized OS (COS) nodes fails with permission error.
  • Fixed the issue where enabling and disabling cluster autoscaler sometimes prevents nodepool replicas from being updated.
  • Fixed the manual node repair issue where manually adding the onprem.cluster.gke.io/repair-machine Machine annotation can trigger VM recreation without deleting the Machine object.
  • Switched back to cgroup v1 (hybrid) for Container Optimized OS (COS) nodes because cgroup v2 (unified) could potentially cause instability for your workloads in a COS cluster.
  • Fixed the issue where running gkectl repair admin-master after a failed admin cluster upgrade attempt caused subsequent admin upgrade attempts to fail. A preflight check has been added for gkectl repair admin-master to prevent the process from using a template that doesn't match the admin cluster checkpoint.
  • Fixed the issue where kubectl describe might error or timeout if resource number is too high during a cluster snapshot.
  • Fixed the following vulnerabilities:

August 12, 2022

Anthos clusters on VMware 1.10.6-gke.36 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.10.6-gke.36 runs on Kubernetes 1.21.14-gke.2100.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.12, 1.11, and 1.10.

  • Fixed the issue where mounting emptyDir volume with exec option on Container-Optimized OS (COS) nodes fails with permission error.
  • Fixed the issue where enabling and disabling cluster autoscaler sometimes prevents nodepool replicas from being updated.
  • Fixed the following vulnerabilities:

August 02, 2022

A new vulnerability CVE-2022-2327 has been discovered in the Linux kernel that can lead to local privilege escalation. This vulnerability allows an unprivileged user to achieve a full container breakout to root on the node.

For more information, see the GCP-2022-018 security bulletin.

July 27, 2022

Anthos clusters on VMware 1.11.2-gke.53 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.11.2-gke.53 runs on Kubernetes 1.22.8-gke.204.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.12, 1.11, and 1.10.

  • Fixed a known issue in which the cluster backup feature affected the inclusion of always-on secrets encryption keys in the backup.
  • Fixed a known issue of high-resource usage when AIDE runs as a cron job, by disabling AIDE by default. This fix affects compliance with CIS L1 Server benchmark 1.4.2: Ensure filesystem integrity is regularly checked. Customers can opt in to re-enable the AIDE if needed. To re-enable the AIDE cron job, see Configure AIDE cron job.
  • Fixed a known issue where gke-metrics-agent DaemonSet has frequent CrashLoopBackOff errors by upgrading to gke-metrics-agent v1.1.0-anthos.14.
  • Fixed the following vulnerabilities:

July 19, 2022

Anthos clusters on VMware 1.9.7-gke.8 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.9.7-gke.8 runs on Kubernetes 1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.12, 1.11, and 1.10.

  • Fixed a known issue in which the cluster backup feature affected the inclusion of always-on secrets encryption keys in the backup.
  • Fixed a known issue of high-resource usage when AIDE runs as a cron job, by disabling AIDE by default. This fix affects compliance with CIS L1 Server benchmark 1.4.2: Ensure filesystem integrity is regularly checked. Customers can opt in to re-enable the AIDE if needed. To re-enable the AIDE cron job, see Configure AIDE cron job.
  • Fixed the following vulnerabilities:

July 07, 2022

Anthos clusters on VMware v1.12.0-gke.446 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware v1.12.0-gke.446 runs on Kubernetes v1.23.5-gke.1504.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.12, 1.11, and 1.10.

Announcements

  • vSphere releases for versions lower than version 7.0 Update 2 are deprecated in Kubernetes 1.24. VMware's General Support for vSphere 6.7 will end on October 15, 2022. Customers are recommended to upgrade vSphere (both ESXi and vCenter) to version 7.0 Update 2 or above. vSphere versions less than version 7.0 Update 2 will no longer be supported in Anthos clusters on VMware in an upcoming version. You must upgrade vSphere to 7.0 Update 2 or above before you can upgrade to Anthos clusters on VMware 1.13.0.

  • Beta versions of VolumeSnapshot CRDs are deprecated in Kubernetes v1.20 and are unsupported in the Kubernetes v1.24 release.
    The upcoming Anthos clusters on VMware version 1.13 release will no longer serve v1beta1 VolumeSnapshot CRDs. Make sure that you migrate manifests and API clients to use snapshot.storage.k8s.io/v1 API version, available since Kubernetes v1.20. All existing persisted objects remain accessible via the new snapshot.storage.k8s.io/v1 APIs.

  • The dockershim component in Kubernetes enables cluster nodes to use the Docker Engine container runtime. However, Kubernetes 1.24 removed the dockershim component. Starting from Anthos clusters on VMware version 1.12.0, you cannot create new clusters that use the Docker Engine container runtime. All new clusters must use the default container runtime Containerd. A cluster update will also be blocked if you want to switch from containerd node pool to docker node pool, or if you add new docker node pools. For existing version 1.11.x clusters with docker node pools, you can continue upgrading it to version 1.12.0, but you must update the node pools to use containerd before you can upgrade to version 1.13.0 in the future.

Breaking changes:

In Kubernetes 1.23, the rbac.authorization.k8s.io/v1alpha1 API version is removed. Instead, use the rbac.authorization.k8s.io/v1 API. See the Kubernetes 1.23.5 release notes.

Platform enhancements:

  • General Availability (GA): Separate vSphere data centers for the admin cluster and the user clusters are supported.
  • GA: Anthos Identity service LDAP authentication is supported.
  • GA: User cluster control-plane node and admin cluster add-on node auto sizing is supported.

Security enhancements:

  • Preview: Preparing credentials for user clusters as Kubernetes secrets before cluster creation.

    • The credential preparation feature prepares the credentials before a user cluster is created. After credential preparation, user cluster credentials are saved as versioned Kubernetes secrets in the admin cluster, and the template which is used for credential preparation can be deleted from the admin workstation. When creating a user cluster, it only needs to configure the namespace and the versions of the prepared secrets in the user cluster config file. Using this feature can help protect user cluster credentials.
  • Preview: The gkectl update credentials command supports rotating the component access SA key for both the admin and the user clusters.

  • The COS node image shipped in version 1.12.0 is qualified with the Center for Internet Security (CIS) L1 Server Benchmark.

  • The gkectl update credentials command supports register service account key rotation.

Cluster lifecycle Improvements:

  • Preview: You can configure the time duration of Pod Disruption Budget (PDB) violation timeout during a node drain. The default behavior is to always block on a PDB violation and to not force-delete pods during node drain, to avoid unexpected data corruption, and this default is unchanged. In certain cases, when users want to unblock the PDB violation deadlock with the bound timeout during cluster upgrade, they can apply the special annotation onprem.cluster.gke.io/pdb-violation-timeout: TIMEOUT on the machine objects.

Simplify day-2 operations

  • Preview: Launched the enablement of Google Cloud Managed Service for Prometheus to track metrics in Anthos on vSphere clusters, and introduced two separate flags to enable logging and monitoring for user applications separately: EnableCloudLoggingForApplications and EnableGMPForApplications. The legacy flag EnableStackdriverForApplications is deprecated, and will be removed in a future release. Customers can monitor and alert on the applications using Prometheus with Google-managed Prometheus without managing and operating Prometheus. Customers can set enableGMPForApplications in the Stackdriver spec to enable Google Managed Prometheus for application metrics without any other manual steps, and the Google Managed Prometheus components are then set up automatically. See Enable Managed Service for Prometheus for user applications for details.

  • All sample dashboards to monitor cluster health are available in Cloud Monitoring sample dashboards. Customers can install the dashboards with one click. See Install sample dashboards.

  • Improvements to cluster diagnosis: The gkectl diagnose cluster command automatically runs when gkectl diagnose snapshot is run, and the output is saved in a new folder in the snapshot called /diagnose-report.

  • The gkectl diagnose cluster command surfaces more detailed information for issues arising from virtual machine creation.

  • A validation check for the existence of an OS image has been added to the gkectl update admin and gkectl diagnose cluster commands.

  • A blocking preflight check has been added. This check validates that the vCenter.datastore specified in the cluster configuration file doesn't belong to a DRS-enabled datastore cluster.

Functionality changes:

  • Upgraded COS from m93 to m97, and containerd to 1.6 on COS.

  • Metrics agent: Upgraded gke-metrics-agent from 1.1.0 to 1.8.3, which fixes some application metrics issues. The offline buffer in the metrics agent can now discard old data based on the age of metrics data, in addition to the total size of buffer. Metrics data is stored in an offline buffer for at most 22 hours in case of a network outage.

  • New metrics: Added 7 resource utilization metrics.

    • k8s_container:
      • container/cpu/request_utilization
      • container/cpu/limit_utilization
      • container/memory/request_utilization
      • container/memory/limit_utilization
    • k8s_node:
      • node/cpu/allocatable_utilization
      • node/memory/allocatable_utilization
    • k8s_pod:
      • pod/volume/utilization

Fixes

Known issues:

  • On the out-of-the-box monitoring dashboards, the GKE on-prem Windows pod status and GKE on-prem Windows node status also show data from Linux clusters.

  • The scheduler metrics, such as scheduler_pod_scheduling_attempts, are not collected in version 1.12.0 due to a configuration issue in the metric collector.

In version 1.12.0, cgroup v2 (unified) is enabled by default for Container Optimized OS (COS) nodes. This could potentially cause instability for your workloads in a COS cluster. We will switch back to cgroup v1 (hybrid) in version 1.12.1. If you are considering using version 1.12 with COS nodes, we suggest that you wait until the 1.12.1 release.

June 24, 2022

Three new memory corruption vulnerabilities (CVE-2022-29581, CVE-2022-29582, CVE-2022-1116) have been discovered in the Linux kernel. These vulnerabilities allow an unprivileged user with local access to the cluster to achieve a full container breakout to root on the node. For more information, refer to the GCP-2022-016 security bulletin.

June 16, 2022

Anthos clusters on VMware 1.10.5-gke.26 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.10.5-gke.26 runs on Kubernetes 1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.11, 1.10, and 1.9.

Fixed for version 1.10.5

  • Fixed the issue where admin cluster backup did not back up always-on secrets encryption keys. This caused repairing an admin cluster using gkectl repair master --restore-from-backup to fail when always-on secrets encryption was enabled.

  • Fixed the issue of high resource usage when AIDE runs as a cron job by disabling AIDE by default. This fix will affect compliance with CIS L1 Server benchmark 1.4.2: Ensure filesystem integrity is regularly checked.

    To re-enable the AIDE cron job, see Configure AIDE cron job.

Fixed the following vulnerabilities

June 03, 2022

Cluster lifecycle improvements

GA: You can use the Cloud console to create, update, and delete Anthos on VMware user clusters. For more information, see Create a user cluster in the Cloud console.

May 26, 2022

Anthos clusters on VMware 1.11.1-gke.53 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.11.1-gke.53 runs on Kubernetes 1.22.8-gke.200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.11, 1.10, and 1.9.

Fixed for v1.11.1

  • Fixed the known issue where v1.11.0 user clusters cannot be created with a v1.10.x admin cluster.

  • Fixed the issue where the gkectl logs might be truncated when admin cluster creation has failed.

  • Fixed the issue that Anthos Identity Service with LDAP failed to authenticate against some older Active Directory servers when the user id contains a comma.

Fixed the following vulnerabilities

High-severity CVEs

Medium-severity CVEs

Anthos clusters on VMware 1.10.4-gke.32 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.10.4-gke.32 runs on Kubernetes 1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.11, 1.10, and 1.9.

Fixed for v1.10.4

Fixed the following vulnerabilities

High-severity CVEs

RBAC fixes

  • anetd

    • Changed to use kubelet kubeconfig to only allow the anetd to update its own node resource, and the pod resources that are running on the node.
  • antrea-controller / anetd-win

    • Instead of reusing the RBAC config for anetd, created a dedicated RBAC config for antrea and reduced the unnecessary permissions.
  • clusterdns-controller

    • Scoped down clusterdns permissions to default resource name.
    • Scoped down configmap permissions to coredns resource name.
    • Removed create/delete permissions for configmaps. The coredns configmap is now created by the bundle, with create-only annotation to ensure we don't overwrite existing config on upgrade.
  • dns-autoscaler

    • Removed unneeded permissions, and scoped down needed permissions to a particular resource using resourceNames.
    • Restricted get configmap for dns autoscaler.
  • gke-usage-metering

    • Restricted the permission to the kube-system namespace where possible
  • seesaw-load-balancer

    • Restricted the permission by setting resource names.

May 19, 2022

Anthos clusters on VMware 1.9.6-gke.1 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.9.6-gke.1 runs on Kubernetes 1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.11, 1.10, and 1.9.

Secret encryption key rotation does not fail when the cluster has more than 1000 secrets.

Fixed the following vulnerabilities

Changed scope of certain RBAC permissions

We have scoped down the over-privileged RBAC permissions for the following components in this release:

  • clusterdns-controller:

    • Scope down clusterdns permissions to 'default' resource name.
    • Scope down configmap permissions to 'coredns' resource name.
    • Remove create/delete permissions for configmaps.
  • seesaw-load-balancer:

    • Restrict the permission to access secrets by specifying certain secret names instead of allowing the access for all secrets.
  • coredns-autoscaler:

    • Reduce the get configmap permission to a specific configmap resource name.
  • anetd / anet-operator:

    • Changed to use kubelet kubeconfig to restrict the anetd to only update its own node resource, and the pod resources that are running on the node.
  • gke-usage-metering:

    • Restrict the permission to only kube-system namespace.
  • ANG (Anthos Network Gateway)

    • Remove/modify RBAC roles and lower the use of kube-rbac proxy in ANG.

May 02, 2022

Creating a 1.11.0 user cluster with a 1.10 admin cluster fails. If you need a 1.11.0 user cluster, use the following workaround:

  1. Create a 1.10 user cluster.

  2. Upgrade the user cluster to 1.11.0.

  3. Optionally, upgrade the admin cluster to 1.11.0. After the admin cluster is upgraded, you can create 1.11.0 user clusters.

For details on how to upgrade, see Upgrading Anthos clusters on VMware.

April 28, 2022

Two security vulnerabilities, CVE-2022-1055 and CVE-2022-27666, have been discovered in the Linux kernel. Each can lead to a local attacker being able to perform a container breakout, privilege escalation on the host, or both. These vulnerabilities affect all Linux node operating systems (Container-Optimized OS and Ubuntu). For instructions and more details, see the GCP-2022-014 security bulletin.

April 27, 2022

Anthos clusters on VMware 1.11.0-gke.543 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.11.0-gke.543 runs on Kubernetes v1.22.8-gke.200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.11, 1.10, and 1.9.

  • The structure of the Anthos clusters on VMware documentation is substantially different from previous versions. For details, see New documentation structure.

  • Dockershim, the Docker Engine integration code in Kubernetes, was deprecated in Kubernetes 1.20, and will be removed in Kubernetes 1.24. Thus, the ubuntu OS node image type will not be supported at that time. You should plan to convert your node pools to use either the ubuntu_containerd or the cos OS image type as soon as possible. For more details, see Using containerd for the container runtime.

  • The connect project is now called fleet host project. For more information, see Fleet host project.

  • Kubernetes 1.22 has deprecated certain APIs, a list of which can be found in Kubernetes 1.22 deprecated APIs. In your manifests and API clients, you need to replace references to the deprecated APIs with references to the newer API calls. For more information, see the What to do section in the Deprecated API Migration Guide.

  • Several Anthos metrics have been deprecated for which data is no longer collected. For a list of deprecated metrics, including instructions to migrate to replacement metrics, see Replace deprecated metrics in dashboard.

Cluster lifecycle Improvements:

  • Admin cluster creation is now resumable. If admin cluster creation fails at any step, you can now rerun gkectl create admin to resume the admin cluster creation.

Platform enhancements:

  • Windows Node Pool:

    • GA: Support for Windows Dataplane V2 is generally available. Windows Dataplane V2 is now enabled by default for Windows node pools. This means that containerd is also enabled by default for Windows node pools.
    • Added deprecation notice for Windows nodes that Docker and Flannel will be removed in a subsequent version. If you are using Docker container runtime, you should update your user cluster configuration with gkectl update cluster to use containerd and Windows Dataplane V2 instead.
    • Added support for idempotent Windows startup script execution after node reboot.
    • New Windows Server 2019 OS build version 10.0.17763.2565 has been qualified for Anthos 1.11.0.
  • Egress NAT Gateway:

    • GA: Egress NAT Gateway is now generally available. With this feature, you can configure source network address translation (SNAT) so that certain egress traffic from user clusters is given a predictable source IP address. This enables return traffic from workloads outside the originating cluster to reach the cluster. For more information, see Configuring an egress NAT gateway.
  • MetalLB:

    • GA: The new load balancer option, MetalLB, is now generally available as another bundled software load balancer in addition to Seesaw.
  • Multinic logs:

    • The Fluent Bit Logging agent can now collect logs for Pods with multiple network interfaces, and send them to Cloud Logging. Logs will be collected as system logs and no extra charges will apply.

Security enhancements: - Admin cluster CA Certificate Rotation:

  • GA: You can now use gkectl to rotate system root CA certificates for admin clusters.

Simplify day-2 operations:

  • GA: gkectl update admin supports registering an existing admin cluster.
  • Cluster diagnosis improvements:
    • gkectl diagnose cluster automatically runs during admin or user cluster upgrade failure.
    • gkectl diagnose cluster searches and surfaces related events for any validation failure.
  • GA: gkectl update supports enabling and disabling of Cloud Logging and Cloud Monitoring in an existing cluster. You can also enable or disable logging to Cloud Audit Logs with gkectl update on both admin and user clusters.
  • Changes made to the metrics-server-config ConfigMap are now preserved across cluster upgrades.

Terminology changes:

The connect project is now called fleet host project. For more information, see Fleet host project.

We have removed the over-privileged RBAC permissions for the following components.

RBAC policies applied to service account on the admin cluster

When you register a 1.11.0+ admin cluster to a fleet, a service account is created with the needed role-based access control (RBAC) policies that lets the Connect agent send requests to the admin cluster's Kubernetes API server on behalf of the service account. The service account and RBAC policies are needed so that you can manage the lifecycle of your user clusters in the Google Cloud console. For more information, see Admin cluster RBAC policies.

April 18, 2022

Anthos clusters on VMware 1.10.3-gke.49 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.10.3-gke.49 runs on Kubernetes 1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.10, 1.9, and 1.8.

  • Fixed issue where scale down sometimes took longer than expected when cluster autoscaling is enabled in a Dataplane-v2 cluster.
  • Fixed issue where the state of an admin cluster that uses a COS image is lost during an admin cluster upgrade or admin cluster control plane repair.
  • Added keep-alive configuration to avoid timeout issues for long running vSphere operations in gkeadm.
  • RBAC fixes:

    • coredns-autoscaler:
    • Removed configmaps create permission.
    • Removed replicasets/scale permissions.
    • Removed replicationcontrollers/scale permissions.
    • Scoped down deployments/scale permissions to coredns resource name.

    • clusterdns-controller:

      • Scoped down clusterdns permissions to default resource name.
      • Scoped down configmap permissions to coredns resource name.
      • Removed create/delete permissions for configmaps. The coredns configmap is now created by the bundle, with create-only annotation to ensure we don't overwrite existing config on upgrade.
    • auto-resize controller:

    • Scoped down leases permissions to onprem-auto-resize-leader-election resource name.

    • Scoped down configmaps permissions to onprem-auto-resize-leader-election resource name.

    • load-balancer-f5:

    • Removed get list watch create patch delete permissions for configmaps.

    • Removed update create patch for events nodes.

    • Removed create permissions for services/status and services.

    • Removed view permission for secret bigip-login-9t8mzp.

  • Fixed high-severity CVEs:

April 12, 2022

A security vulnerability, CVE-2022-23648, has been discovered in containerd's handling of path traversal in the OCI image volume specification. Containers launched through containerd's CRI implementation with a specially-crafted image configuration could gain full read access to arbitrary files and directories on the host.

For more information, see the GCP-2022-013 security bulletin.

April 11, 2022

A security vulnerability, CVE-2022-0847, has been discovered in the Linux kernel version 5.8 and later that can potentially escalate container privileges to root.

For more information, see the GCP-2022-012 security bulletin.

March 24, 2022

Anthos clusters on VMware 1.9.5-gke.2 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.9.5-gke.2 runs on Kubernetes v1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.10, 1.9, and 1.8.

March 15, 2022

Anthos clusters on VMware 1.8.8-gke.1 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.8.8-gke.1 runs on Kubernetes v1.20.12-gke.1500.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.10, 1.9, and 1.8.

  • Clusters with enableDataplaneV2 set to true can experience connectivity issues between Pods due to anetd daemons (running as a Daemonset) entering a software deadlock. While in this state, anetd daemons will see stale nodes (previously deleted nodes) as peers and miss newly added nodes as new peers. If you have experienced this issue, follow these instructions to restart the anetd daemons and restore connectivity.

March 03, 2022

Anthos clusters on VMware 1.10.2-gke.34 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.10.2-gke.34 runs on Kubernetes 1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.10, 1.9, and 1.8.

Changes

  • gkectl diagnose now reports a broken cluster caused by an admin cluster registration error during creation.

Fixes

  • Fixed issue: Failure to register admin cluster during creation

    • You can upgrade an admin cluster to version 1.10.2 without applying the documented mitigation, even if the cluster failed to register with the provided gkeConnect configuration during its creation. You can fix the registration issue by running gkectl update admin with the correct gkeConnect configuration after upgrade.
    • If the cluster registration failed when creating a version 1.10.2 admin cluster, no mitigation is needed to upgrade to later versions after version 1.10.2.
  • Fixed ".local" DNS lookup issue caused by Ubuntu 20.04 systemd-resolved configuration changes.

  • Fixed issue where Docker bridge IP incorrectly used 172.17.0.1/16 instead of 169.254.123.1/24.

  • Fixed unexpectedly high network traffic to monitoring.googleapis.com in a newly created cluster.

  • Fixed an issue that admin cluster creation or upgrade might be interrupted by temporary vCenter connection issue.

  • Fixed critical CVEs:

  • Fixed this high-severity CVE:

When cluster autoscaling is enabled in a Dataplane-v2 cluster, scale down may sometimes take longer than expected. For example, it may take approximately 20 minutes instead of 10 minutes as in a normal case.

February 24, 2022

The Envoy project recently discovered a set of vulnerabilities. All issues listed in the security bulletin are fixed in Envoy release 1.21.1. For more information, see the GCP-2022-008 security bulletin.

February 23, 2022

Anthos clusters on VMware 1.9.4-gke.3 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.9.4-gke.3 runs on Kubernetes v1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.10, 1.9, and 1.8.

Fixes

  • Upgraded Cilium to version 1.10.5.

    • This upgrade also fixed the issue where unreachable node endpoints caused application 503 errors. Previously, when cilium-health status was run in anetd daemons, the output showed stale remote nodes.
  • Fixed unexpectedly high network traffic to monitoring.googleapis.com in a newly created cluster.

  • Fixed these high-severity CVEs:

When cluster autoscaling is enabled in a Dataplane-v2 cluster, scale down may sometimes take longer. For example, it may take approximately 20 minutes instead of 10 minutes as in a normal case.

February 17, 2022

Anthos clusters on VMware 1.8.7-gke.0 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.8.7-gke.0 runs on Kubernetes v1.20.12-gke.1500.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.10, 1.9, and 1.8.

Fixes:

February 14, 2022

A security vulnerability, CVE-2022-0492, has been discovered in the Linux kernel's cgroup_release_agent_write function. The attack uses unprivileged user namespaces, and under certain circumstances, this vulnerability can be exploitable for container breakout. For more information, see the GCP-2022-006 security bulletin.

February 11, 2022

A security vulnerability, CVE-2021-43527, has been discovered in any binary that links to the vulnerable versions of libnss3 found in NSS (Network Security Services) versions prior to 3.73 or 3.68.1. Applications using NSS for certificate validation or other TLS, X.509, OCSP or CRL functionality may be impacted, depending on how they configure NSS.

For more information, see the GCP-2022-005 security bulletin.

February 10, 2022

Anthos clusters on VMware 1.10.1-gke.19 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.10.1-gke.19 runs on Kubernetes v1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.10, 1.9, and 1.8.

  • Removed unintentional infrastructure log lines from the cluster snapshot.
  • Upgraded the Connect Agent version to 20211210-01-00.

    • This upgrade also fixed the issue where the Connect Agent restarts unexpectedly on either a newly-created cluster or an existing cluster that uses Anthos Identity Service to manage the Anthos Identity Service ClientConfig.
  • Fixed two high severity CVEs:

  • Fixed the short metric probing interval issue that sends a high volume of traffic to the monitoring.googleapis.com endpoint in a cluster.

  • If your admin cluster failed to register with the provided gkeConnect spec during creation, upgrading to a later 1.9 or 1.10 release will fail with the following error:

    failed to migrate to first admin trust chain: failed to parse current version "": invalid version: "" failed to migrate to first admin trust chain: failed to parse current version "": invalid version: ""

    If you have experienced this issue, follow these instructions to fix the gkeConnect registration issue before you upgrade your admin cluster.

February 07, 2022

A security vulnerability, CVE-2021-4034, has been discovered in pkexec, a part of the Linux policy kit package (polkit), that allows an authenticated user to perform a privilege escalation attack. PolicyKit is generally used only on Linux desktop systems to allow non-root users to perform actions such as rebooting the system, installing packages, restarting services, and so forth, as governed by a policy.

For instructions and more details, see the GCP-2022-004 security bulletin.

February 01, 2022

Three security vulnerabilities, CVE-2021-4154, CVE-2021-22600, and CVE-2022-0185, have been discovered in the Linux kernel, each of which can lead to either a container breakout, privilege escalation on the host, or both. These vulnerabilities affect all Linux node operating systems (COS and Ubuntu) on Anthos clusters on VMware.

For instructions and more details, see the GCP-2022-02 security bulletin.

January 24, 2022

Anthos clusters on VMware 1.9.3-gke.4 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.9.3-gke.4 runs on Kubernetes v1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.10, 1.9, and 1.8.

Fixes for version 1.9.3:

  • Fixed issue where special characters in the vSphere username are not properly escaped.

Changes in version 1.9.3:

  • Upgraded the Connect Agent version to 20211210-01-00.

    • This upgrade also fixed the issue where the Connect Agent restarts unexpectedly on a newly-created cluster that uses Anthos Identity Service to manage the Anthos Identity Service ClientConfig.

Known issue in version 1.9.3:

  • The Connect Agent restarts unexpectedly on an existing cluster that uses Anthos Identity Service to manage the Anthos Identity Service ClientConfig. If you have experienced this issue, follow these instructions to upgrade the Connect Agent version.

Anthos clusters on VMware 1.8.6-gke.4 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.8.6-gke.4 runs on Kubernetes 1.20.12-gke.1500.

Fixes for version 1.8.6:

  • Fixed issue where special characters in the vSphere username are not properly escaped.

December 23, 2021

  • When deploying Anthos clusters on VMware releases with a version number of 1.9.0 or higher, that have the Seesaw bundled load balancer in an environment that uses NSX-T stateful distributed firewall rules, stackdriver-operator might fail to create gke-metrics-agent-conf ConfigMap and cause gke-connect-agent Pods to be in a crash loop. The underlying issue is that stateful NSX-T distributed firewall rules terminate the connection from a client to the user cluster API server through the Seesaw load balancer because Seesaw uses asymmetric connection flows. The integration issue with NSX-T distributed firewall rules affect all Anthos clusters on VMWare releases that use Seesaw. You might see similar connection problems on your own applications when they create large Kubernetes objects whose sizes are bigger than 32K. Follow these instructions to disable NSX-T distributed firewall rules, or to use stateless distributed firewall rules for Seesaw VMs.

  • If your clusters use a manual load balancer, follow these instructions to configure your load balancer to reset client connections when it detects a backend node failure. Without this configuration, clients of the Kubernetes API server might stop responding for several minutes when a server instance goes down.

December 22, 2021

Anthos clusters on VMware 1.10.0-gke.194 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.10.0-gke.194 runs on Kubernetes v1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.10, 1.9, and 1.8.

  • vCenter/ESXi host 6.7u2 and below is no longer supported. Upgrade your vCenter environment to a supported version (6.7U3 and above) before upgrading your clusters.

  • The diskformat parameter is removed from the standard vSphere driver StorageClass as the parameter has been deprecated in Kubernetes 1.21.

  • Preview: Egress NAT gateway:

    • To enable an egress NAT gateway, the advancedNetworking section in the user cluster configuration file replaces the now-deprecated enableAnthosNetworkGateway section.

    • You must create a NetworkGatewayGroup object (previously AnthosNetworkGateway) to configure the egress NAT gateway.

    • Any admin or user clusters that are version 1.9 or earlier, and that are enabled with Anthos Network Gateway, cannot be upgraded. You must delete and recreate those clusters following these instructions.

Cluster lifecycle Improvements:

  • An admin cluster upgrade is resumable after a previous failed admin cluster upgrade attempt.

  • GA: Admin cluster registration during new cluster creation is generally available.

  • Preview: Admin cluster registration when updating existing clusters is available as a preview feature.

Platform enhancements:

  • Preview: A new load balancer option, MetalLB, is available as another bundled software load balancer in addition to Seesaw.This will be the default load balancer choice instead of Seesaw when GA.

  • GA: Support for user cluster node pool autoscaling is generally available.

  • Preview: You can create admin cluster nodes and user cluster control-plane nodes with Container-Optimized OS by specifying the osImageType as cos in the admin cluster configuration file.

  • Windows Node Pool:

    • Preview: The containerd runtime is now available for Windows node pools when Dataplane V2 for Windows is enabled.
    • Node Problem Detector checks containerd service health on the nodes and surfaces problems to the API Server. For version 1.10.0, NPD does not attempt to repair the containerd service.
    • Containerd logs are exported to the Cloud Console.

    • CSI proxy is deployed automatically onto Windows nodes. You can install and use a Windows CSI driver of your choice, such as the SMB CSI driver.

  • GA: The multi-NIC capability to provide additional network interfaces to your Pods is generally available.

  • GA: You can upgrade to Ubuntu 20.04 and containerd 1.5.

Security enhancements:

  • User cluster control plane certificates are automatically rotated at each cluster upgrade. 

Simplify day-2 operations:

  • Preview: gkectl update admin supports the enabling and disabling of Cloud Monitoring and Cloud Logging in the admin cluster. 

  • Changed the collection of application metrics to use a more scalable monitoring pipeline based on OpenTelemetry. This change significantly reduces the amount of resources required to collect metrics.

  • Updated the parser of containerd and kubelet node logs to extract severity level.

  • Introduced the --share-with optional flag in the gkectl diagnose snapshot command to share the read permission after uploading the snapshot to a Google Cloud Storage bucket.

Functionality changes:

  • Replaced the SSH tunnel with Konnectivity service for communication between the user cluster control plane and the user cluster nodes. The Kubernetes SSH tunnel has been deprecated. 

    • You must create two additional firewall rules so that user worker nodes can access ports 8132 on the user control-plane VIP address and get return packets. This is required for the Konnectivity service.

    • Introduced a new konnectivityServerNodePort field in the user cluster manual load balancer configuration. This field is required when creating or upgrading a user cluster, with manual load balancer mode, to version 1.10. 

  • The Ubuntu OS image is upgraded from 18.04 to 20.04 LTS.

    • The python command is no longer available. Any python command should be updated to python3 instead, and the syntax should be updated to Python 3.

    • /etc/resolv.conf now points to /run/systemd/resolve/stub-resolv.conf, instead of /run/systemd/resolve/resolv.conf.

    • The Ubuntu CIS benchmark version changed from v2.0.1 for Ubuntu 18.04 LTS to v1.0.0 for Ubuntu 20.04 LTS.

  • Upgraded COS from m89 to m93.

  • Upgraded containerd from 1.4 to 1.5 on Ubuntu and COS.

  • Changed gkectl diagnose snapshot to use the --all-with-logs scenario by default.

  • The gkeadm command copies the admin workstation configuration file to the admin workstation during creation so it can be used as a backup to re-create the admin workstation later.

  • Increased the Pod priority of kube-state-metrics to improve its reliability when the cluster is under resource contention.

  • Fixed an issue that the Windows nodes were assigned with duplicated IP addresses.

  • Fixed CVE-2021-32760. Because of Ubuntu PPA version pinning, this vulnerability might still be reported by certain vulnerability scanning tools, and thus appear as a false positive even though the underlying vulnerability has been patched.

  • Because of the change to use an OpenTelemetry-based scalable monitoring pipeline for application metrics, Horizontal Pod Autoscaling with user-defined metrics does not work in 1.10.0 unless you explicitly set scalableMonitoring to false, while also ensuring that both enableStackdriverForApplications and enableCustomMetricsAdapter are set to true, in the Stackdriver object.

    As a workaround, you can install a custom Prometheus adapter if you want to use Horizontal Pod Autoscaling with user-defined metrics while still keeping the scalable monitoring default setting for application metrics.

  • Because of a COS 93 configuration issue, IPv6 dualstack does not work correctly for COS node pool nodes in version 1.10.0. If you are using IPv6 dualstack with a COS node pool, wait for an upcoming patch release that addresses this issue.

  • If an admin cluster is created with osImagetype of cos, and you have rotated the audit logging service account key with gkectl update admin, the changes are overridden after the admin cluster control-plane node reboot. In that case, re-run the update command after the admin cluster control-plane node reboot to apply those changes.

  • On COS nodes, the NTP server is configured to time.google.com by default. In DHCP mode, this setting cannot be overridden to use the NTP server provided by your DHCP server. The issue will be fixed in an upcoming patch release. Before then, you can deploy a DaemonSet to override the NTP setting if you want to use a different NTP server in your COS node pool.

November 30, 2021

Anthos clusters on VMware 1.7.6-gke.6 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.7.6-gke.6 runs on Kubernetes v1.19.15-gke.1900.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.9, 1.8, and 1.7.

  • Fixed issue where special characters in the vSphere username are not properly escaped.
  • Alleviated the high CPU and memory usage by /etc/cron.daily/aide discussed in this issue.
  • Fixed issue where user cluster node is not synching time.
  • Fixed CVE-2021-41103. Because of Ubuntu PPA version pinning, this vulnerability might still be reported by certain vulnerability scanning tools, and appear as a false positive even though the underlying vulnerability has been patched.

November 29, 2021

Anthos clusters on VMware 1.8.5-gke.3 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.8.5-gke.3 runs on Kubernetes v1.20.9-gke.701.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.9, 1.8, and 1.7.

  • Fixed issue where special characters in the vSphere username are not properly escaped.
  • Alleviated the high CPU and memory usage by /etc/cron.daily/aide discussed in this issue.
  • Fixed issue where user cluster node is not synching time.
  • Fixed CVE-2021-41103. Because of Ubuntu PPA version pinning, this vulnerability might still be reported by certain vulnerability scanning tools, and appear as a false positive even though the underlying vulnerability has been patched.

November 18, 2021

Anthos clusters on VMware 1.9.2-gke.4 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.9.2-gke.4 runs on Kubernetes v1.21.5-gke.1200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.9, 1.8, and 1.7.

With version 1.9.2, cert-manager is installed in the cert-manager namespace. Previously, for versions 1.8.2 to 1.9.1, cert-manager was installed in the kube-system namespace.

The cert-manager version is upgraded from 1.0.3 to 1.5.4.

If you already use any ClusterIssuer with a different cluster resource namespace from the default cert-manager namespace, follow these steps if you upgrade to version 1.9.2.

   * Manually copy the related certificates, secrets, or issuers to the cert-manager namespace to use the installed cert-manager after upgrading to 1.9.2.    

   * If you need to use a different version of cert-manager, or if you need to install it in a different namespace, follow these instructions each time that you upgrade your cluster. 

Fixes:

  • Fixed issue with cilium-operator not reconciling CiliumNode for Windows nodes when updating the cluster to add Windows node pools.
  • Fixed issue which could temporarily result in no healthy CoreDNS pods present during cluster operations.
  • Fixed issue where you cannot run gkectl upgrade loadbalancer on a user cluster seesaw load balancer.
  • Fixed issue where node_filesystem metrics report gives wrong size for /run.
  • Fixed CVE-2021-37159. Because of Ubuntu PPA version pinning, this vulnerability might still be reported as a false positive by certain vulnerability scanning tools, although the underlying vulnerability has been patched in the 1.9.2 release.
  • Fixed issue where user cluster node is not synching time.
  • Alleviated the high CPU and memory usage by /etc/cron.daily/aide discussed in this issue.

October 29, 2021

The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem.

For more information, see the GCP-2021-011 security bulletin.

October 27, 2021

Anthos clusters on VMware 1.8.4-gke.1 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.8.4-gke.1 runs on Kubernetes v1.20.9-gke.701.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.9, 1.8, and 1.7.

Fixes for version 1.8.4:

  • Fixed high-severity CVE-2021-3711.
  • Fixed gkectl check-config failure when Anthos clusters are configured with a proxy whose url contains special characters.
  • Fixed "cert-manager" cainjector leader-election failure.

Known issue in version 1.8.4:

If you have already installed your own cert-manager in your cluster, read the suggested mitigation before upgrading to a version >=1.8.2 in order to avoid an installation conflict with the cert-manager deployed by Anthos clusters on VMware.

  • Installing your cert-manager with Apigee may also result in a conflict with the cert-manager deployed by Anthos clusters on VMware. To avoid this, read the suggested mitigation before upgrading to this version.

Anthos clusters on VMware 1.7.5-gke.0 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.7.5-gke.0 runs on Kubernetes v1.19.12-gke.2101.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.9, 1.8, and 1.7.

Fixes for version 1.7.5:

Fixed gkectl check-config failure when Anthos clusters are configured with a proxy whose url contains special characters.

October 21, 2021

A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742. Ingress-nginx custom snippets allow retrieval of ingress-nginx service account tokens and secrets across all namespaces. For more information, see the GCP-2021-024 security bulletin.

October 20, 2021

Anthos clusters on VMware 1.9.1-gke.6 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.9.1-gke.6 runs on Kubernetes v1.21.5-gke.400.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.9, 1.8, and 1.7.

  • In version 1.9.0, there was a known issue with restoring an admin cluster using a backup when using a private registry. That has been fixed in version 1.9.1.
  • Fixed gkectl check-config failure that occurs when Anthos clusters are configured with a proxy whose url contains special characters.
  • Fixed "cert-manager" cainjector leader-election failure.

If you have already installed your own cert-manager in your cluster, read the suggested mitigation before upgrading to a version >=1.8.2 in order to avoid an installation conflict with the cert-manager deployed by Anthos clusters on VMware.

  • Installing your cert-manager with Apigee may also result in a conflict with the cert-manager deployed by Anthos clusters on VMware. To avoid this, read the suggested mitigation before upgrading to this version.

October 04, 2021

A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server. For more information, see the GCP-2021-021 security bulletin.

September 29, 2021

Anthos clusters on VMware 1.9.0-gke.8 is now available. To upgrade, see Upgrading Anthos clusters on VMware. Anthos clusters on VMware 1.9.0-gke.8 runs on Kubernetes v1.21.4-gke.200.

The supported versions offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware are 1.9, 1.8, and 1.7.

Features:

Cluster lifecycle Improvements:

  • GA: You can register an admin cluster during its creation by filling in the gkeConnect section in the admin cluster configuration file, similar to user cluster registration.

Platform enhancements:

  • Preview: User clusters can now be in a different vSphere datacenter from the admin cluster, resulting in datacenter isolation between the admin cluster and user clusters. This provides greater resiliency in the case of vSphere environment failures.

  • GA: Support for Windows node pools is generally available.This release adds:

    • Preview: Windows DataplaneV2 support, which allows for using Windows Network Policy
    • Node Problem Detector (NPD) support on Windows
    • Streamlined process for preparing Windows images in a private registry
    • Enhanced Flannel CNI support on Windows

    The upstream fixes for the "Windows Pod stuck at terminating status" error are also applied to this release, which improves the stability of running Windows workloads.

  • GA: Support for Container-Optimized OS (COS) node pools is generally available.

  • GA: CoreDNS is now the cluster DNS provider.

    • Clusters that are upgraded to 1.9 will have their KubeDNS provider replaced with CoreDNS. During the upgrade, CoreDNS is first deployed and then KubeDNS is removed, so applications should not observe DNS unavailability. However before upgrading, ensure that your cluster has enough additional resources to deploy CoreDNS. CoreDNS requires 100 millicpu and 170 MiB of memory per instance, all clusters require a minimum of 2 instances, and there is an additional instance deployed for every 16 nodes in the cluster.
    • You can configure cluster DNS options such as upstream name servers by using the new ClusterDNS custom resource.

Security enhancements:

  • GA: Always-on secrets encryption: You can enable secrets encryption with internally generated keys instead of a hardware security module (HSM). Use the gkectl update command to rotate these keys or to enable or disable secrets encryption after cluster creation.
  • Preview: Windows network policy support. This release introduces a new network plugin, Antrea, for Windows nodes. In addition to network connectivity and services support, it provides network policy support. When creating a user cluster, you can set enableWindowsDataplaneV2 to true to enable this feature. Enabling this feature replaces Flannel with Antrea on Windows nodes.
  • Preview: Azure AD group support for Authentication: This feature allows cluster admins to configure RBAC policies based on Azure AD groups for authorization in clusters. This supports retrieval of groups information for users belonging to more than 200 groups, thus overcoming a limitation of regular OIDC configured with Azure AD as the identity provider.

Simplify day-2 operations:

  • Preview: When creating a user cluster, you can set enableVMTracking in the configuration file to true to enable vSphere tag creation and attachment to the VMs in the user cluster. This allows easy mapping of VMs to clusters and node pools. See Enable VM tracking.
  • GA: New metrics agents based on open telemetry are introduced to improve reliability, scalability and resource usage.
  • Preview: You can enable or disable Stackdriver with gkectl update on existing user clusters. You can enable or disable cloud audit logging and monitoring with gkectl update on both admin and user clusters.

Breaking changes:

  • User cluster registration is now required and enforced. You must fill in the gkeConnect section of the user cluster configuration file before creating a new user cluster. You cannot upgrade a user cluster unless that cluster is registered. To unblock the cluster upgrade, add the gkeConnect section to the configuration file and run gkectl update cluster to register an existing 1.8 user cluster.

  • User clusters must be upgraded before the admin cluster. The flag --force-upgrade-admin to allow the old upgrade flow (admin cluster upgrade first) is no longer supported.

  • The following requirements are now enforced when you create a cluster that has logging and monitoring enabled.

    • The Config Monitoring for Ops API is enabled in your logging-monitoring project.
    • The Ops Config Monitoring Resource Metadata Writer role is granted to your logging-monitoring service account.
    • The URL opsconfigmonitoring.googleapis.com is added to your proxy allowlist (if applicable).

Changes:

  • There is now a checkpoint file for the admin cluster, located in the same datastore folder as the admin cluster data disk, with the name DATA_DISK_NAME-checkpoint.yaml, or DATA_DISK_NAME.yaml if the length of DATA_DISK_NAME is greater than the filename length limit. This file is required for future upgrades and should be considered as important as the admin cluster data disk.

    Note: If you have enabled VM encryption in vCenter, you must grant Cryptographer.Access permission to the vCenter credentials specified in your admin cluster configuration file, before trying to create or upgrade your admin cluster.

  • The admin cluster backup with gkectl preview feature introduced in 1.8 now allows updates to clusterBackup.datastore. This datastore may be different from vCenter.datastore so long as it is in the same datacenter as the cluster.

  • The k8s 1.21 release includes the following metrics changes:

    • Add new field status for storage_operation_duration_seconds, so that you can know about all status storage operation latency.
    • The storage metrics storage_operation_errors_total and storage_operation_status_count are marked deprecated. In both cases, the storage_operation_duration_seconds metric can be used to recover equivalent counts (using status=fail-unknown in the case of storage_operations_errors_total).

    • Rename the metric etcd_object_counts to apiserver_storage_object_counts and mark it as stable. The original etcd_object_counts metrics name is marked as "Deprecated" and will be removed in the future.

  • A new GKE on-prem control plane uptime dashboard is introduced with a new metric, kubernetes.io/anthos/container/uptime, for component availability. The old GKE on-prem control plane status dashboard and old kubernetes.io