This documentation is for the current version of Anthos clusters on AWS, released in November 2021. See the Release notes for more information. For documentation on the previous generation of Anthos clusters on AWS, see Previous generation.

Quotas and limits

This page lists the quotas and limits that apply to Anthos clusters on AWS.

Cluster versions

Kubernetes 1.26

1.26.2-gke.1001

Kubernetes OSS release notes

  • Feature: Updated OS image to Ubuntu 22.04. cgroupv2 is now used as the default control group configuration.

    • Ubuntu 22.04 uses cgroupv2 by default. We recommend that you check if any of your applications access the cgroup filesystem. If they do, they must be updated to use cgroupv2. Some example applications that might require updates to ensure compatibility with cgroupv2 are:
      • Third-party monitoring and security agents that depend on the cgroup filesystem.
      • If cAdvisor is being used as a stand-alone DaemonSet for monitoring Pods and containers, it should be updated to version v0.43.0 or later.
      • If you are using JDK, we recommend that you use version 11.0.16 and later, or version 15 and later. These versions fully support cgroupv2.
      • If you are using the uber-go/automaxprocs package, make sure to use version v1.5.1 or higher.
      • Ubuntu 22.04 removes the timesyncd package. Instead, chrony is now used for the Amazon Time Sync Service.
    • For more information, see the Ubuntu release notes

  • Feature: Sends metrics for control plane components to Cloud Monitoring. This includes a subset of the Prometheus metrics from kube-apiserver, etcd, kube-scheduler, kube-controller-manager. Metrics names use the prefix kubernetes.io/anthos/.

  • Feature: Enabled sending Kubernetes resource metadata to Google Cloud Platform, improving both the user interface and cluster metrics. For the metadata to be ingested properly, customers need to enable the Config Monitoring for Ops API. This API can be enabled either in the Google Cloud Console , or by manually enabling the opsconfigmonitoring.googleapis.com API in the gcloud CLI. Additionally, customers must follow the steps outlined in the Authorize Cloud Logging/Monitoring documentation to add the necessary IAM bindings. If applicable, add opsconfigmonitoring.googleapis.com to your Proxy Allowlist.

  • Feature: Added preview feature for creating Spot AWS node pool.

  • Feature: Creating node pools using ARM-based (Graviton) instance types is now GA.

  • Feature: Enabled kubelet graceful node shutdown. Non-system Pods are given 15 seconds to terminate, after which system Pods (with the system-cluster-critical or system-node-critical priority classes) have 15 seconds to gracefully terminate.

  • Feature: Enabled Node auto repair feature in preview mode. Please contact your account team to opt into the preview.

  • Feature: Added tags to dynamically created EFS Access Point resource.

  • Feature: Clusters now have per-node-pool subnet security group rules instead of VPC-wide rules

    • Previously, the control plane allowed inbound traffic from the entire primary IP range of the VPC on ports TCP/443 and TCP/8123, which are used by node pools.
    • Now, the control plane narrows the allowed inbound traffic to each IP range of the node pool subnets on ports TCP/443 and TCP/8123; multiple node pools can share one subnet.
    • This change supports node pools running outside of the VPC's primary IP range and improves the security of the control plane.
    • If you relied on the VPC-wide security group rule for allowing traffic from outside of the cluster (e.g. from a bastion host for kubectl), then as part of the upgrade you should create a security group, add a VPC-wide rule to it, and attach the security group to the control plane (via the AwsCluster.controlPlane.securityGroupIds field).

  • Known Issue: Kubernetes 1.26.2 will incorrectly apply the default StorageClass to PersistentVolumeClaims which have the deprecated annotation volume.beta.kubernetes.io/storage-class.

  • Security Fix: Set the hop limit of the IMDS emulator response to 1. This secures the communication of IMDS data between the emulator and a workload.

  • Bug Fixes: Newly-created clusters now use etcd v3.4.21 for improved stability. Existing clusters of previous versions were already using etcd v3.5.x and will not be downgraded to v3.4.21 during cluster upgrade; these clusters will instead use v3.5.6.

Kubernetes 1.25

1.25.7-gke.1000

Kubernetes OSS release notes.

  • Feature: Added tags to dynamically created EFS Access Point resource.

  • Bug Fixes: Newly-created clusters now use etcd v3.4.21 for improved stability. Existing clusters of previous versions were already using etcd v3.5.x and will not be downgraded to v3.4.21 during cluster upgrade; these clusters will instead use v3.5.6.

1.25.6-gke.1600

Kubernetes OSS release notes.

1.25.5-gke.2000

Kubernetes OSS release notes.

  • Fixed an issue where certain errors were not propagated and reported during cluster create/update operations.
  • Fixed an issue with AWS EFS CSI driver where EFS hostnames can't be resolved when AWS VPC is configured to use a custom DNS server.
  • Updated Anthos Identity Service to better handle concurrent authentication webhook requests.

  • Updated fluent-bit to v1.9.9 to fix CVE-2022-42898.

  • Security Fixes

  • Fixed an issue where authentication through the Anthos Service Mesh dashboard failed due to inability to impersonate end user.

1.25.5-gke.1500

Kubernetes OSS release notes.

  • Known issue: Some UI surfaces in Google Cloud console can't authorize to the cluster and might display the cluster as unreachable. A workaround is to manually apply RBAC permitting user impersonation. For details, see Troubleshooting.

  • Security Fixes

1.25.4-gke.1300

Kubernetes OSS release notes.

  • Known issue: Some UI surfaces in Google Cloud console can't authorize to the cluster and might display the cluster as unreachable. A workaround is to manually apply RBAC permitting user impersonation. For details, see Troubleshooting.

  • Removed deprecated in-tree volume plugins flocker, quobyte and storageos.

  • Security Fixes

  • Security enhancement: Restricted static pods running on the cluster's control plane VMs to run as non-root Linux users.

  • Feature: Added support for dynamically updating AWS node pool security groups. To update security groups, you must have the following permissions in your API role -

    • ec2:ModifyInstanceAttribute
    • ec2:DescribeInstances

  • Feature: Added support for dynamically updating AWS node pool tags. To update node pool tags, you must have the following permissions in your API role -

    • autoscaling:CreateOrUpdateTags
    • autoscaling:DeleteTags
    • ec2:CreateTags
    • ec2:DeleteTags
    • ec2:DescribeLaunchTemplates

  • Feature: EFS dynamic provisioning is now available in GA for clusters at version 1.25 or later. To use this feature, you must add the following permissions to the control plane role:

    • ec2:DescribeAvailabilityZones
    • elasticfilesystem:DescribeAccessPoints
    • elasticfilesystem:DescribeFileSystems
    • elasticfilesystem:DescribeMountTargets
    • elasticfilesystem:CreateAccessPoint
    • elasticfilesystem:DeleteAccessPoint

  • Uploading of workload metrics using Google Managed Service for Prometheus with managed collection to Cloud Monarch is now available in GA.

  • Feature: Added support to enable and update CloudWatch metrics collection on AWS node pool's auto scaling group. To enable or update metrics collection via create or update API, you must add the following permissions to your API role:

    • autoscaling:EnableMetricsCollection
    • autoscaling:DisableMetricsCollection

  • Feature: Azure AD GA. This feature allows cluster admins to configure RBAC policies based on Azure AD groups for authorization in clusters. This supports retrieval of groups information for users belonging to more than 200 groups, thus overcoming a limitation of regular OIDC configured with Azure AD as the identity provider.

  • Architecture: Added a new token manager (gke-token-manager) to generate tokens for control plane components, using the service account signing key. Benefits:

    1. Eliminate the dependency on kube-apiserver for control plane components to authenticate to Google services. Previously, control plane components would use the TokenRequest API and were reliant on a healthy kube-apiserver. Whereas now the gke-token-manager component mints the tokens directly using the service account signing key.
    2. Eliminate the RBAC for generating token for controlplane components.
    3. Uncouple the logging and kube-apiserver. So that the logging can be ingested before the kube-apiserver is up.
    4. Make the controlplane more resilience. When the kube-apiserver is out of service the controlplane components can still get the tokens and keep working.

  • As a preview feature, ingest a variety of metrics from the control plane components to Cloud Monitoring, including kube-apiserver, etcd, kube-scheduler and kube-controller-manager.

  • Feature: Users in a Google Group can access AWS clusters using Connect Gateway by granting necessary RBAC permission to the Group. More details at Set up the Connect gateway with Google Groups.

  • Fixed an issue which could result in outdated versions of gke-connect-agent not being removed after cluster upgrades.

Kubernetes 1.24

1.24.11-gke.1000

Kubernetes OSS release notes.

  • Bug Fixes: Newly-created clusters now use etcd v3.4.21 for improved stability. Existing clusters of previous versions were already using etcd v3.5.x and will not be downgraded to v3.4.21 during cluster upgrade; these clusters will instead use v3.5.6.

1.24.10-gke.1200

Kubernetes OSS release notes.

1.24.9-gke.2000

Kubernetes OSS release notes.

  • Fixed an issue where certain errors were not propagated and reported during cluster create/update operations.
  • Updated Anthos Identity Service to better handle concurrent authentication webhook requests.

  • Updated fluent-bit to v1.9.9 to fix CVE-2022-42898.

  • Security Fixes

1.24.9-gke.1500

Kubernetes OSS release notes.

1.24.8-gke.1300

Kubernetes OSS release notes.

1.24.5-gke.200

Kubernetes OSS release notes.

1.24.3-gke.2200

Kubernetes OSS release notes.

  • Fix a bug where creating a Kubernetes Service resource with type LoadBalancer and annotation service.beta.kubernetes.io/aws-load-balancer-type: nlb, would remain with an empty target group. See https://github.com/kubernetes/cloud-provider-aws/issues/301

1.24.3-gke.2100

Kubernetes OSS release notes.

  • Upload Kubernetes resource metrics to Google Cloud Monitoring for Windows node pools.
  • Provided a webhook for easy IMDS emulator injection.
  • go1.18 stops accepting certificates signed with the SHA-1 hash algorithm by default. Admission/conversion webhooks or aggregated server endpoints using these insecure certificates will break by default in 1.24. The environment variable GODEBUG=x509sha1=1 is set in Anthos on-AWS clusters as a temporary workaround to let these insecure certificates continue to work. However, the go team is anticipated to remove support on this workaround in the near coming releases. Customers should check and ensure there aren't any admission/conversion webhooks or aggregated server endpoints that are using such insecure certificates before upgrading to the upcoming breaking version.
  • Anthos clusters on AWS now supports EFS dynamic provisioning in preview mode, for Kubernetes clusters at version 1.24 or later. To use this feature, you must add the following permissions to the control plane role: ec2:DescribeAvailabilityZones elasticfilesystem:DescribeAccessPoints elasticfilesystem:DescribeFileSystems elasticfilesystem:DescribeMountTargets elasticfilesystem:CreateAccessPoint elasticfilesystem:DeleteAccessPoint

  • Improve network connectivity checks during cluster and node pool creation to help troubleshooting.

  • Security Fixes

  • Support updates to AWS control plane tags. To update tags, you need to add the following permissions to the API role - autoscaling:CreateOrUpdateTags autoscaling:DeleteTags ec2:CreateTags ec2:DescribeLaunchTemplates ec2:DescribeSecurityGroupRules ec2:DeleteTags elasticloadbalancing:AddTags elasticloadbalancing:RemoveTags

  • Upload workload metrics using Google Managed Service for Prometheus to Cloud Monarch is available as invite only private preview.

Supported regions

Google Cloud Region AWS region
asia-east2 ap-east-1
asia-northeast2 ap-northeast-1
ap-northeast-3
asia-south1 ap-south-1
asia-southeast1 ap-east-1
ap-northeast-1
ap-southeast-1
asia-southeast2 ap-southeast-3
australia-southeast1 ap-southeast-2
europe-north1 eu-north-1
europe-west1 eu-central-1
eu-north-1
eu-south-1
eu-west-1
eu-west-2
eu-west-3
europe-west2 eu-west-2
europe-west3 eu-central-1
europe-west9 eu-west-3
northamerica-northeast1 ca-central-1
southamerica-east1 sa-east-1
us-east4 ca-central-1
us-east-1
us-east-2
us-west1 us-west-1
us-west-2

Supported VM types

The following AWS VM sizes are supported:

Type Size
C5 Instances c5.large
C5 Instances c5.xlarge
C5 Instances c5.2xlarge
C5 Instances c5.4xlarge
C5 Instances c5.9xlarge
C5 Instances c5.12xlarge
C5 Instances c5.18xlarge
C5 Instances c5.24xlarge
C5 Instances c5a.large
C5 Instances c5a.xlarge
C5 Instances c5a.2xlarge
C5 Instances c5a.4xlarge
C5 Instances c5a.9xlarge
C5 Instances c5a.12xlarge
C5 Instances c5a.18xlarge
C5 Instances c5a.24xlarge
C5 Instances c5ad.large
C5 Instances c5ad.xlarge
C5 Instances c5ad.2xlarge
C5 Instances c5ad.4xlarge
C5 Instances c5ad.9xlarge
C5 Instances c5ad.12xlarge
C5 Instances c5ad.18xlarge
C5 Instances c5ad.24xlarge
C5 Instances c5d.large
C5 Instances c5d.xlarge
C5 Instances c5d.2xlarge
C5 Instances c5d.4xlarge
C5 Instances c5d.9xlarge
C5 Instances c5d.12xlarge
C5 Instances c5d.18xlarge
C5 Instances c5d.24xlarge
I3en Instances i3en.large
I3en Instances i3en.xlarge
I3en Instances i3en.2xlarge
I3en Instances i3en.3xlarge
I3en Instances i3en.6xlarge
I3en Instances i3en.12xlarge
I3en Instances i3en.24xlarge
M5 Instances m5.large
M5 Instances m5.xlarge
M5 Instances m5.2xlarge
M5 Instances m5.4xlarge
M5 Instances m5.8xlarge
M5 Instances m5.12xlarge
M5 Instances m5.16xlarge
M5 Instances m5.24xlarge
M5 Instances m5a.large
M5 Instances m5a.xlarge
M5 Instances m5a.2xlarge
M5 Instances m5a.4xlarge
M5 Instances m5a.8xlarge
M5 Instances m5a.12xlarge
M5 Instances m5a.16xlarge
M5 Instances m5a.24xlarge
M5 Instances m5ad.large
M5 Instances m5ad.xlarge
M5 Instances m5ad.2xlarge
M5 Instances m5ad.4xlarge
M5 Instances m5ad.8xlarge
M5 Instances m5ad.12xlarge
M5 Instances m5ad.16xlarge
M5 Instances m5ad.24xlarge
M5 Instances m5d.large
M5 Instances m5d.xlarge
M5 Instances m5d.2xlarge
M5 Instances m5d.4xlarge
M5 Instances m5d.8xlarge
M5 Instances m5d.12xlarge
M5 Instances m5d.16xlarge
M5 Instances m5d.24xlarge
R5 Instances r5.large
R5 Instances r5.xlarge
R5 Instances r5.2xlarge
R5 Instances r5.4xlarge
R5 Instances r5.8xlarge
R5 Instances r5.12xlarge
R5 Instances r5.16xlarge
R5 Instances r5.24xlarge
R5 Instances r5a.large
R5 Instances r5a.xlarge
R5 Instances r5a.2xlarge
R5 Instances r5a.4xlarge
R5 Instances r5a.8xlarge
R5 Instances r5a.12xlarge
R5 Instances r5a.16xlarge
R5 Instances r5a.24xlarge
R5 Instances r5ad.large
R5 Instances r5ad.xlarge
R5 Instances r5ad.2xlarge
R5 Instances r5ad.4xlarge
R5 Instances r5ad.8xlarge
R5 Instances r5ad.12xlarge
R5 Instances r5ad.16xlarge
R5 Instances r5ad.24xlarge
R5 Instances r5d.large
R5 Instances r5d.xlarge
R5 Instances r5d.2xlarge
R5 Instances r5d.4xlarge
R5 Instances r5d.8xlarge
R5 Instances r5d.12xlarge
R5 Instances r5d.16xlarge
R5 Instances r5d.24xlarge
T3 Instances t3.medium
T3 Instances t3.large
T3 Instances t3.xlarge
T3 Instances t3.2xlarge
T3 Instances t3a.medium
T3 Instances t3a.large
T3 Instances t3a.xlarge
T3 Instances t3a.2xlarge

Node image types

Anthos clusters on AWS cluster nodes run Ubuntu version 20.04. The image is similar to GKE's Ubuntu node image.

ContainerOS nodes are currently not supported.

Node pool sizes

Anthos clusters on AWS supports node pools containing up to 50 nodes.

Cluster and node pool quotas

Anthos clusters on AWS imposes several default quotas. To increase them, contact Google Cloud support.

Starting from Kubernetes 1.26, the scalability of Anthos clusters on AWS has been enhanced, and allows up to 1000 nodes per cluster. To increase your quotas , contact Google Cloud support.

Quota Name Default value Upper limit
Number of clusters per Google Cloud project 20 NA
Number of node pools per cluster 10 NA
Number of nodes per cluster 500 1000
Number of nodes per node pool 50 500
Number of pods per node 110 110

In addition to these quotas, your Anthos clusters on AWS installation is subject to any AWS service quotas on your AWS account, including the following:

For more information, see the AWS Service Quotas console.