Creating a VPC-native cluster

This page explains how to configure VPC-native clusters in Google Kubernetes Engine.

Overview

In GKE, clusters can be distinguished according to the way they route traffic from one Pod to another Pod. A cluster that uses alias IP address ranges is called a VPC-native cluster. A cluster that uses Google Cloud Routes is called a routes-based cluster.

Benefits of VPC-native clusters

VPC-native clusters have several benefits:

  • Pod IP addresses are natively routable within the cluster's VPC network and other VPC networks connected to it by VPC Network Peering.
  • Pod IP addresses are reserved in the VPC network before the Pods are created in your cluster. This prevents conflict with other resources in the VPC network and allows you to better plan IP address allocations.
  • Pod IP address ranges do not depend on custom static routes. They do not consume the system-generated and custom static routes quota. Instead, automatically-generated subnet routes handle routing for VPC-native clusters.
  • You can create firewall rules that apply to just Pod IP address ranges instead of any IP address on the cluster's nodes.
  • Pod IP address ranges, and subnet secondary IP address ranges in general, are accessible from on-premises networks connected with Cloud VPN or Cloud Interconnect using Cloud Routers.

Restrictions

  • You cannot convert a VPC-native cluster into a routes-based cluster, and you cannot convert a routes-based cluster into a VPC-native cluster.

  • VPC-native clusters require VPC networks. Legacy networks are not supported.

  • As with any GKE cluster, Service (ClusterIP) addresses are only available from within the cluster. If you need to access a Kubernetes Service from VM instances outside of the cluster, but within the cluster's VPC network and region, create an internal TCP/UDP load balancer.

IP address ranges for VPC-native clusters

When you create a VPC-native cluster, you specify a subnet in a VPC network. The cluster uses three unique subnet IP address ranges:

  • It uses the subnet's primary IP address range for all node IP addresses.
  • It uses one secondary IP address range for all Pod IP addresses.
  • It uses another secondary IP address range for all Service (cluster IP) addresses.

The following table provides a summary of IP address ranges for nodes, Pods and Services:

Range Explanation Example
Nodes

Node IP addresses are assigned from the primary IP address range of the subnet associated with your cluster.

Both node IP addresses and the size of the subnet's secondary IP address range for Pods limit the number of nodes that a cluster can support. Refer to node limiting ranges for more information.

If you plan to create a 900-node cluster, the primary IP address range of the cluster's subnet must be at least a /22 (2(32-22) = 210 = 1,024 addresses). Of those 1,024 addresses, 1,020 are usable because four IP addresses are reserved in every primary IP address range.

Refer to Subnet primary IP address range and Subnet secondary IP address range for Pods for more information.

Pods

Pod IP addresses are taken from the cluster subnet's secondary IP address range for Pods. Unless you set a different maximum number of Pods per node, GKE allocates a /24 alias IP range (256 addresses) to each node for the Pods running on it. On each node, those 256 alias IP addresses are used to support up to 110 Pods.

For a 900-node cluster supporting up to 110 Pods per node, you need 900 × 256 = 230,400 IP addresses for Pods. (Each node is allocated an alias IP range whose netmask's size is /24.) This cluster requires a subnet whose secondary IP range for Pods has a subnet mask no larger than /14. This secondary IP range range provides 2(32-14) = 218 = 262,144 IP addresses for Pods.

Refer to Subnet secondary IP address range for Pods for more information.

Services

Service (cluster IP) addresses are taken from the cluster's subnet's secondary IP address range for Services. You must ensure this range is large enough to provide addresses for all the Kubernetes Services you host in your cluster.

For a cluster that runs up to 3000 Services, you need 3000 cluster IP addresses. You need a secondary range of size /20 or larger. A /20 range of IP addresses results in 2(32-20) = 212 = 4,096 IP addresses.

Refer to Subnet secondary IP address range for Services for more information.

Internal IP addresses

The IP addresses you use for your VPC-native cluster's subnets must come from a valid subnet range. The valid ranges include private IP addresses (RFC 1918 and others) and privately reused public IP addresses. See Valid ranges and Restricted ranges in the Virtual Private Cloud documentation for more information about valid subnet ranges.

See the Using non-RFC 1918 private IP address ranges section for instructions on enabling the use of these ranges.

See the Enable privately reused public IP address ranges section for instructions on the use of these ranges in private clusters.

Secondary range assignment methods

You can assign Pod IP address ranges and Service (ClusterIP) address ranges to a VPC-native cluster using one of two methods:

Managed by GKE (default)

GKE can create and manage the subnet's secondary ranges for you. When you create the cluster, you specify either a complete CIDR range or the size of a netmask for both the Pods and Services. For example, you can specify 10.1.0.0/16 for Pods and 10.2.0.0/20 for Services, or you can specify /16 for Pods and /20 for Services.

If you create the cluster and subnet simultaneously, Pod and Service IP address ranges are managed by GKE.

User-managed

You can create the subnet's secondary IP address ranges, then create a cluster that uses those ranges. If you manually create the secondary ranges, you must manage them yourself.

Differences with routes-based clusters

The allocation scheme for Pod and Service (ClusterIP) addresses is different than the scheme used by a routes-based cluster. Instead of specifying a single CIDR for Pods and Services together, you must choose or create two secondary IP address ranges in cluster's subnet: one for Pods and another for Services.

Shared VPC considerations

When creating a VPC-native cluster in a Shared VPC environment, a project owner, editor, or IAM member with the Network Admin role in the Shared VPC host project must create the cluster's subnet and secondary IP address ranges manually. A service project admin who creates a cluster must at least have subnet-level permissions to the desired subnet in the Shared VPC network's host project.

In a Shared VPC environment, secondary IP address ranges cannot be managed by GKE. A Network Admin in the Shared VPC host project must create the subnet and secondary IP address ranges before you can create the cluster. For an example showing how to set up a VPC-native cluster in a Shared VPC network, refer to Setting up clusters with Shared VPC.

IP address range planning

Use the information in the following sections to help you calculate sizes for primary and secondary IP address ranges of the subnet used by your cluster.

Subnet primary IP address range

Every subnet must have a primary IP address range. You can expand the primary IP address range at any time, even when Google Cloud resources use the subnet; however, you cannot shrink or change a subnet's primary IP address scheme after the subnet has been created. The first two and last two IP addresses of a primary IP address range are reserved by Google Cloud.

The following table shows the maximum number of nodes you can create in all clusters that use the subnet, given the size of the subnet's primary IP address range.

Subnet primary IP range Maximum nodes
/29
Minimum size for a subnet's primary IP range
4 nodes
/28 12 nodes
/27 28 nodes
/26 60 nodes
/25 124 nodes
/24 252 nodes
/23 508 nodes
/22 1,020 nodes
/21 2,044 nodes
/20
Default size of a subnet's primary IP range in auto mode networks
4,092 nodes
/19 8,188 nodes
/8
Maximum size for a subnet's primary IP range
16,777,212 nodes

Useful formulas

You can use the following formulas to:

  • Calculate the maximum number of nodes, N, that a given netmask can support. Use S for the size of the netmask, whose valid range is between 8 and 29, inclusive.

    N = 2(32 -S) - 4

  • Calculate the size of the netmask, S, required to support a maximum of N nodes:

    S = 32 - ⌈log2(N + 4)⌉

    ⌈⌉ is the ceiling (least integer) function, meaning round up to the next integer. The valid range for the size of the netmask, S, is between 8 and 29, inclusive.

Subnet secondary IP address range for Pods

Carefully plan your secondary IP address range for Pods. Though it is possible to replace a subnet's secondary IP address range, doing so is not supported because it has the potential to put the cluster in an unstable state.

The following table shows the maximum number of nodes and Pods you can create in all clusters that use the subnet, given the size of the subnet's secondary IP address range used by Pods. This table assumes the maximum number of Pods per node is 110 (the default and largest possible Pod density).

Subnet secondary IP range for Pods Maximum Pod IP addresses Maximum nodes Maximum Pods
/24
Smallest possible Pod IP range when the secondary range assignment method is user-managed
256 addresses 1 node 110 Pods
/23
Only possible when the secondary range assignment method is user-managed
512 addresses 2 nodes 220 Pods
/22
Only possible when the secondary range assignment method is user-managed
1,024 addresses 4 nodes 440 Pods
/21
Smallest possible Pod IP range when the secondary range assignment method is managed by GKE
2,048 addresses 8 nodes 880 Pods
/20 4,096 addresses 16 nodes 1,760 Pods
/19 8,192 addresses 32 nodes 3,520 Pods
/18 16,384 addresses 64 nodes 7,040 Pods
/17 32,768 addresses 128 nodes 14,080 Pods
/16 65,536 addresses 256 nodes 28,160 Pods
/15 131,072 addresses 512 nodes 56,320 Pods
/14
Default size for the subnet's secondary IP range for Pods when the secondary range assignment method is managed by GKE
262,144 addresses 1,024 nodes 112,640 Pods
/13 524,288 addresses 2,048 nodes 225,280 Pods
/12 1,048,576 addresses 4,096 nodes 450,560 Pods
/11
Largest possible Pod address range
2,097,152 addresses 8,192 nodes 901,120 Pods

If you have changed the maximum number of Pods per node, you can use the following formulas to calculate the maximum number of nodes and Pods that a subnet's secondary IP address range for Pods can support:

  1. Calculate the size of the netmask of each node's Pod range, M.
    M = 31 - ⌈log2(Q)⌉ where:

    • Q is the number of Pods per node
    • ⌈⌉ is the ceiling (least integer) function, meaning round up to the next integer
  2. Calculate the maximum number of nodes, N, that the subnet's secondary IP address range for Pods can support:
    N = 2(M - S) where:

    • M is the size of the netmask of each node's alias IP address range for Pods, calculated in the first step
    • S is the size of the subnet mask of the subnet's secondary IP address range
  3. Calculate the maximum number of Pods, P, that the subnet's secondary IP address range for Pods can support:
    P = N × Q where:

    • N is the maximum number of nodes, calculated in the previous step
    • Q is the number of Pods per node

Subnet secondary IP address range for Services

Carefully plan your secondary IP address range for Services. Because this is also a subnet secondary IP address range, you can only replace it when no Google Cloud resources use it. This range cannot be changed as long as a cluster uses it for Services (cluster IP addresses).

Unlike node and Pod IP address ranges, each cluster must have a unique subnet secondary IP address range for Services and cannot be sourced from a shared primary or secondary IP range.

The following table shows the maximum number of Services you can create in a single cluster using the subnet, given the size of the subnet's secondary IP address range for Services.

Secondary IP range for Services Maximum number of Services
/27
Smallest possible Service address range
32 Services
/26 64 Services
/25 128 Services
/24 256 Services
/23 512 Services
/22 1,024 Services
/21 2,048 Services
/20
Default size for the subnet's secondary IP range for Services when the secondary range assignment method is managed by GKE
4,096 Services
/19 8,192 Services
/18 16,384 Services
/17 32,768 Services
/16
Largest possible Service address range
65,536 Services

Node limiting ranges

The maximum number of Pods and Services for a given GKE cluster is limited by the size of the cluster's secondary ranges. The maximum number of nodes in the cluster is limited by the size of the cluster's subnet's primary IP address range and the cluster's Pod address range.

The Cloud Console shows error messages like the following to indicate that either the subnet's primary IP address range or the cluster's Pod IP address range (the subnet's secondary IP address range for Pods) has been exhausted:

Instance [node name] creation failed: IP space of [cluster subnet] is
exhausted

Before you begin

Before you start, make sure you have performed the following tasks:

Set up default gcloud settings using one of the following methods:

  • Using gcloud init, if you want to be walked through setting defaults.
  • Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

  1. Run gcloud init and follow the directions:

    gcloud init

    If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

    gcloud init --console-only
  2. Follow the instructions to authorize gcloud to use your Google Cloud account.
  3. Create a new configuration or select an existing one.
  4. Choose a Google Cloud project.
  5. Choose a default Compute Engine zone.

Using gcloud config

  • Set your default project ID:
    gcloud config set project project-id
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone compute-zone
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region compute-region
  • Update gcloud to the latest version:
    gcloud components update

Procedures

Use the following procedures to create a VPC-native cluster and verify its configured Pod and Service IP address ranges.

Creating a cluster in an existing subnet

The following instructions demonstrate how to create a VPC-native GKE cluster in an existing subnet with your choice of secondary range assignment method.

gcloud

  • To use a secondary range assignment method of managed by GKE:

    gcloud container clusters create cluster-name \
      --region=region \
      --enable-ip-alias \
      --subnetwork=subnet-name \
      --cluster-ipv4-cidr=pod-ip-range \
      --services-ipv4-cidr=services-ip-range
    
  • To use a secondary range assignment method of user- managed:

    gcloud container clusters create cluster-name \
      --region=region \
      --enable-ip-alias \
      --subnetwork=subnet-name \
      --cluster-secondary-range-name=secondary-range-pods \
      --services-secondary-range-name=secondary-range-services
    

Replace the placeholders with valid values:

  • cluster-name is the name of the GKE cluster.
  • region is the region in which the cluster is created. To create a zonal cluster, replace this flag with --zone=zone, where zone is a Google Cloud zone.
  • subnet-name is the name of an existing subnet. The subnet's primary IP address range is used for nodes. The subnet must exist in the same region as the one used by the cluster. If omitted, GKE attempts to use a subnet in the default VPC network in the cluster's region.
  • If the secondary range assignment method is managed by GKE:
    • pod-ip-range is an IP address range in CIDR notation (for example, 10.0.0.0/14) or the size of a CIDR block's subnet mask (for example, /14). This is used to create the subnet's secondary IP address range for Pods.
    • services-ip-range is an IP address range in CIDR notation (for example, 10.4.0.0/19) or the size of a CIDR block's subnet mask (for example, /19). This is used to create the subnet's secondary IP address range for Services.
  • If the secondary range assignment method is user-managed:
    • secondary-range-pods is the name of an existing secondary IP address range in the specified subnet-name. GKE uses the entire subnet secondary IP address range for the cluster's Pods.
    • secondary-range-services is the name of an existing secondary IP address range in the specified subnet-name. GKE uses the entire subnet secondary IP address range for the cluster's Services.

Console

  1. Visit the Google Kubernetes Engine menu in Cloud Console.

    Visit the Google Kubernetes Engine menu

  2. Click the Create cluster button.

  3. From the navigation pane, under Cluster, click Networking.

  4. In the Network drop-down list, select a VPC.

  5. In the Node subnet drop-down list, select a subnet for the cluster.

  6. Ensure the Enable VPC-native traffic routing (uses alias IP) checkbox is selected.

  7. Select the Automatically create secondary ranges checkbox if you want the secondary range assignment method to be managed by GKE. Clear this checkbox if you have already created secondary ranges for the chosen subnet and would like the secondary range assignment method to be user-managed.

  8. In the Pod address range field, enter a pod range (example: 10.0.0.0/14).

  9. In the Service address range field, enter a service range (example: 10.4.0.0/19).

  10. Configure your cluster as desired.

  11. Click Create.

API

When you create a VPC-native cluster, you define an IPAllocationPolicy object. You can reference existing subnet secondary IP address ranges or you can specify CIDR blocks. Reference existing subnet secondary IP address ranges to create a cluster whose secondary range assignment method is user-managed. Provide CIDR blocks if you want the range assignment method to be managed by GKE.

{
  "name": cluster-name,
  "description": description,
  ...
  "ipAllocationPolicy": {
    "useIpAliases": true,
    "clusterIpv4CidrBlock"      : string,
    "servicesIpv4CidrBlock"     : string,
    "clusterSecondaryRangeName" : string,
    "servicesSecondaryRangeName": string,

  },
  ...
}

where:

  • "clusterIpv4CidrBlock" is the size/location of the CIDR range for Pods. Determines the size of the secondary range for Pods, and can be IP/size in CIDR notation (such as 10.0.0.0/14) or /size (like /14). An empty space with the given size is chosen from the available space in your VPC. If left blank, a valid range is found and created with a default size.
  • "servicesIpv4CidrBlock" is the size/location of the CIDR range for Services. See description of "clusterIpv4CidrBlock".
  • "clusterSecondaryRangeName" is the name of the secondary range for Pods. The secondary range must already exist and belong to the subnetwork associated with the cluster (such as the subnetwork specified with the --subnetwork flag).
  • "serviceSecondaryRangeName" is the name of the secondary range for Services. The secondary range must already exist and belong to the subnetwork associated with the cluster (such as the subnetwork specified with by --subnetwork flag).

Terraform

You can easily create a VPC-native cluster via Terraform using a Terraform module.

For example, you can add this block to your Terraform configuration:

module "gke" {
  source  = "terraform-google-modules/kubernetes-engine/google"
  version = "~> 9.1"

  project_id        = "project-id"
  name              = "cluster-name"
  region            = "region"
  network           = "network-name"
  subnetwork        = "subnet-name"
  ip_range_pods     = "secondary-range-pods"
  ip_range_services = "secondary-range-services"
}

Replace the following:

  • project-id is the project ID in which the cluster is created.
  • cluster-name is the name of the GKE cluster.
  • region is the region in which the cluster is created.
  • network-name is the name of an existing network.
  • subnet-name is the name of an existing subnet. The subnet's primary IP address range is used for nodes. The subnet must exist in the same region as the one used by the cluster.
  • secondary-range-pods is the name of an existing secondary IP address range in the specified subnet-name. GKE uses the entire subnet secondary IP address range for the cluster's Pods.
  • secondary-range-services is the name of an existing secondary IP address range in the specified subnet-name. GKE uses the entire subnet secondary IP address range for the cluster's Services.

Creating a cluster and subnet simultaneously

The following directions demonstrate how to create a VPC-native GKE cluster and subnet at the same time. The secondary range assignment method is managed by GKE when you perform these two steps with one command.

gcloud

To create a VPC-native cluster and subnet simultaneously:

gcloud container clusters create cluster-name \
    --region=region \
    --enable-ip-alias \
    --create-subnetwork name=subnet-name,range=node-ip-range \
    --cluster-ipv4-cidr=pod-ip-range \
    --services-ipv4-cidr=services-ip-range

where:

  • cluster-name is the name of the GKE cluster.
  • region is the region in which the cluster is created. To create a zonal cluster, replace this flag with --zone=zone, where zone is a GKE zone.
  • subnet-name is the name of the subnet to create. The subnet's region is the same region as the cluster (or the region containing the zonal cluster). Use an empty string (name="") if you want GKE to generate a name for you.
  • node-ip-range is an IP address range in CIDR notation (for example, 10.5.0.0/20) or the size of a CIDR block's subnet mask (for example, /20). This is used to create the subnet's primary IP address range for nodes. If omitted, GKE chooses an available IP range in the VPC with a size of /20.
  • pod-ip-range is an IP address range in CIDR notation (for example, 10.0.0.0/14) or the size of a CIDR block's subnet mask (for example, /14). This is used to create the subnet's secondary IP address range for Pods. If omitted, GKE uses /14, the default Pod IP address range size.
  • services-ip-range is an IP address range in CIDR notation (for example, 10.4.0.0/19) or the size of a CIDR block's subnet mask (for example, /19). This is used to create the subnet's secondary IP address range for Services. If omitted, GKE uses /20, the default Services IP address range size.

Console

You cannot create a cluster and subnet simultaneously using the Cloud Console. Instead, first create a subnet then create the cluster in an existing subnet.

API

To create a VPC-native cluster, define an IPAllocationPolicy object in your cluster resource:

{
  "name": cluster-name,
  "description": description,
  ...
  "ipAllocationPolicy": {
    "useIpAliases": true,
    "createSubnetwork": true,
    "subnetworkName": subnet-name
  },
  ...
}

The createSubnetwork field automatically creates and provisions a subnetwork for the cluster. The subnetworkName field is optional; if left empty, a name is automatically chosen for the subnetwork.

Using non-RFC 1918 private IP address ranges

GKE clusters using 1.14.2-gke.1 and later can use private IP address ranges outside of the RFC 1918 ranges for nodes, Pods, and Services. See valid ranges in the VPC network documentation for a list of non-RFC 1918 private ranges that can be used as internal IP addresses for subnet ranges.

Non-RFC 1918 private IP address ranges are compatible with both private clusters and non-private clusters.

Non-RFC 1918 private ranges are subnet ranges — you can use them exclusively or in conjunction with RFC 1918 subnet ranges. Nodes, Pods, and Services continue to use subnet ranges as described in IP ranges for VPC-native clusters. If you use non-RFC 1918 ranges, keep the following in mind:

  • Subnet ranges, even those using non-RFC 1918 ranges, must be assigned manually or by GKE before the cluster's nodes are created. You cannot switch to or cease using non-RFC 1918 subnet ranges unless you replace the cluster.

  • Internal TCP/UDP load balancers only use IP addresses from the subnet's primary IP address range. To create an internal TCP/UDP load balancer with a non-RFC 1918 address, your subnet's primary IP address range must be non-RFC 1918.

Destinations outside your cluster might have difficulties receiving traffic from private, non-RFC 1918 ranges. For example, RFC 1112 (class E) private ranges are typically used as multicast addresses. If a destination outside of your cluster cannot process packets whose sources are private IP addresses outside of the RFC 1918 range, you can do both of the following:

  • Use an RFC 1918 range for the subnet's primary IP address range. This way, nodes in the cluster use RFC 1918 addresses.

  • Ensure that your cluster is running the IP masquerade agent and that the destinations are not in the nonMasqueradeCIDRs list. This way, packets sent from Pods have their sources changed (SNAT) to node addresses, which are RFC 1918.

Enable privately reused public IP address ranges

GKE clusters using 1.14.2-gke.1 and later can privately reuse certain public IP address ranges as internal, subnet IP address ranges. You can privately reuse any public IP address except for certain restricted ranges as described the VPC network documentation.

Your cluster must be a private cluster in order to use privately reused public IP address ranges. Non-private VPC-native clusters and routes-based clusters are not supported.

Privately reused public ranges are subnet ranges – you can use them exclusively or in conjunction with other subnet ranges that use private addresses. Nodes, Pods, and Services continue to use subnet ranges as described in IP ranges for VPC-native clusters. Keep the following in mind when re-using public IP addresses privately:

  • When you reuse a public IP address range as a subnet range, your cluster can no longer communicate with systems on the Internet that use that public range – the range becomes an internal IP address range in the cluster's VPC network.

  • Subnet ranges, even those that privately reuse public IP address ranges, must be assigned manually or by GKE before the cluster's nodes are created. You cannot switch to or cease using privately reused public IP addresses unless you replace the cluster.

  • Packets sent from Pods that use a privately reused public IP address range cannot have their sources changed (SNAT) to node addresses. Consequently, you must create your cluster with the --disable-default-snat flag. For more details about this flag, refer to IP masquerading in GKE.

  • Because a cluster whose Pods privately reuse public IP address ranges must be a private cluster, your cluster must use a NAT solution such as Cloud NAT if Pods need to send traffic to destinations on the internet. When using Cloud NAT, you must at least configure the NAT gateway to apply to the cluster's subnet's secondary IP address range for Pods. In this way, Cloud NAT performs SNAT from packets sent from Pod IP addresses because the cluster's IP masquerade configuration must have the default SNAT behavior turned off.

Example cluster with privately reused public ranges

The following example uses gcloud to create a cluster that uses privately re-used public IP address ranges. You must use the following flags:

  • --enable-ip-alias: This creates a VPC-native cluster, which is required when you privately reuse public IP address ranges.
  • --enable-private-nodes: This creates a private cluster, which is required when you privately reuse public IP address ranges.
  • --disable-default-snat: This option is required if you privately reuse public IP addresses for your Pods or nodes. Disabling SNAT is required so that responses can be routed to the Pod that originated the traffic.

This command creates a VPC-native, private cluster whose:

  • nodes use the 10.0.0.0/24 primary IP address range of the subnet.
  • Pods privately reuse the 5.0.0.0/16 public IP address range as the subnet's secondary IP address range for Pods.
  • Services privately reuse the 5.1.0.0/16 public IP address range as the subnet's secondary IP address range for Services.
gcloud container clusters create cluster-name \
  --enable-ip-alias \
  --enable-private-nodes \
  --disable-default-snat \
  --zone=zone \
  --create-subnetwork name=cluster-subnet,range=10.0.0.0/24 \
  --cluster-ipv4-cidr=5.0.0.0/16 \
  --services-ipv4-cidr=5.1.0.0/16 \
  --master-ipv4-cidr=master-CIDR

Verifying Pod and Service ranges

After you create a VPC-native cluster, you can verify its Pod and Service ranges.

gcloud

To verify the cluster, run the following command:

gcloud container clusters describe cluster-name

In the command output, look under the ipAllocationPolicy field:

  • clusterIpv4Cidr is the secondary range for Pods
  • servicesIpv4Cidr is the secondary range for Services

Console

To verify the cluster, perform the following steps:

  1. Visit the Google Kubernetes Engine menu in Cloud Console.

    Visit the Google Kubernetes Engine menu

  2. Select the desired cluster.

The secondary ranges are displayed in the Cluster section under the Details tab:

  • Container address range is the secondary range for Pods
  • Service address range is the secondary range for Services

Troubleshooting

This section provides guidance for resolving issues related to VPC-native clusters.

The resource 'projects/[PROJECT_NAME]/regions/XXX/subnetworks/default' is not ready

Potential causes
There are parallel operations on the same subnet. For example, another VPC-native cluster is being created, or a secondary range is being added or deleted on the subnet.
Resolution
Retry the command.

Invalid value for field 'resource.secondaryIpRanges[1].ipCidrRange': 'XXX'. Invalid IPCidrRange: XXX conflicts with existing subnetwork 'default' in region 'XXX'."

Potential causes

Another VPC-native cluster is being created at the same time, and is attempting to allocate the same ranges in the same VPC network.

The same secondary range is being added to the subnetwork in the same VPC network.

Resolution

If this error is returned on cluster creation when no secondary ranges were specified, retry the cluster creation command.

Not enough free IP space for Pods

Symptoms

Cluster is stuck in a provisioning state for an extended period of time

Cluster creation returns a Managed Instance Group (MIG) error

New nodes cannot be added to an existing cluster

Potential causes

Unallocated space in the Pod IP address range is not large enough for the nodes requested in the cluster. For example, if a cluster's Pod IP address range has a netmask whose size is /23 (512 address), and the maximum Pods per node is 110, you cannot create any more than two nodes. (Each node is assigned an alias IP address range with a netmask whose size is /24.)

Solutions

Create a replacement cluster after reviewing and planning appropriately sized primary and secondary IP address ranges. Refer to IP ranges for VPC-native clusters and IP range planning.

If you cannot replace the cluster, you might be able to work around the problem if you can create a new node pool with a smaller maximum number of Pods per node. If possible, migrate workloads to that node pool then delete the previous node pool. Reducing the maximum number of Pods per node allows you to support more nodes on a fixed secondary IP address range for Pods. Refer to Subnet secondary IP address range for Pods and Node limiting ranges for details about the calculations involved.

Confirm whether default sNAT is disabled

Use the following command to check the status of default sNAT:

gcloud container clusters describe cluster-name

Replace cluster-name with the name of your cluster.

The output will include a disableDefaultSnat field like this:

networkConfig:
  disableDefaultSnat: true
  network: ...

Cannot use --disable-default-snat without --enable-ip-alias

This error message, and must disable default sNAT (--disable-default-snat) before using public IP address privately in the cluster, mean that you should explicitly set the --disable-default-snat flag when creating the cluster since you are using public IP addresses in your private cluster.

If you see error messages like cannot disable default sNAT ... , this means the default SNAT can't be disabled in your cluster. Please review your cluster configuration.

Debugging Cloud NAT with default sNAT disabled

If you have a private cluster created with the --disable-default-snat flag and have set up Cloud NAT for internet access and you aren't seeing internet-bound traffic from your Pods, make sure that the Pod range is included in the Cloud NAT configuration.

If there is a problem with Pod to Pod communication, examine the iptables rules on the nodes to verify that the Pod ranges are not masqueraded by iptables rules. See the IP masquerade example iptables output for how to list the iptables rules and what the default rules are. If you have not configured and IP masquerade agent for the cluster, GKE automatically ensures that Pod to Pod communication is not masqueraded. However, if an IP masquerade agent is configured, it will override the default IP masquerade rules. Verify that additional rules are configured in the IP masquerade agent to ignore masquerading the Pod ranges.

What's next