Creating a VPC-native cluster

This page explains how to configure VPC-native clusters in Google Kubernetes Engine.

Overview

In GKE, clusters can be distinguished according to the way they route traffic from one Pod to another Pod. A cluster that uses alias IP ranges is called a VPC-native cluster. A cluster that uses Google Cloud Routes is called a routes-based cluster.

Benefits VPC-native clusters

VPC-native clusters have several benefits:

  • Pod IP addresses are natively routable within the cluster's VPC network and other VPC networks connected to it by VPC Network Peering.
  • Pod IPs are reserved in the VPC network before the Pods are created in your cluster. This prevents conflict with other resources in the VPC network and allows you to better plan IP address allocations.
  • Pod IP ranges do not depend on custom static routes. They do not consume the system-generated and custom static routes quota. Instead, automatically-generated subnet routes handle routing for VPC-native clusters.
  • You can create firewall rules that apply to just Pod IP ranges instead of any IP address on the cluster's nodes.
  • Nodes in a VPC-native cluster have anti-spoofing enabled. VPC networking performs anti-spoofing checks which prevent nodes from being able to send packets with arbitrary source IP addresses. (Anti-spoofing is disabled for routes-based clusters.)
  • Pod IP ranges, and subnet secondary IP ranges in general, can be shared with on-premises networks connected with Cloud VPN or Cloud Interconnect using Cloud Routers.
  • Alias IP ranges allow Pods to directly access hosted services without using a NAT gateway.

Restrictions

  • You cannot convert a VPC-native cluster into a routes-based cluster, and you cannot convert a routes-based cluster into a VPC-native cluster.

  • VPC-native clusters require VPC networks. Legacy networks are not supported.

  • As with any GKE cluster, Service (ClusterIP) addresses are only available from within the cluster. If you need to access a Kubernetes Service from VM instances outside of the cluster, but within the cluster's VPC network and region, create an internal TCP/UDP load balancer.

IP ranges for VPC-native clusters

When you create a VPC-native cluster, you specify a subnet in a VPC network. The cluster uses three unique subnet IP ranges:

  • It uses the subnet's primary IP range for all node IP addresses.
  • It uses one secondary IP range for all Pod IP addresses.
  • It uses another secondary IP range for all Service (cluster IP) addresses.

The following table provides a summary of IP address ranges for nodes, Pods and Services:

Range Explanation Example
Nodes

Node IP addresses are assigned from the primary IP address range of the subnet associated with your cluster.

Both node IP addresses and the size of the subnet's secondary IP range for Pods limit the number of nodes that a cluster can support. Refer to node limiting ranges for more information.

If you plan to create a 900-node cluster, the primary IP address range of the cluster's subnet must be at least a /22 (2(32-22) = 210 = 1,024 addresses). Of those 1,024 addresses, 1,020 are usable because four IP addresses are reserved in every primary IP address range.

Refer to Subnet primary IP address range and Subnet secondary IP address range for Pods for more information.

Pods

Pod IP addresses are taken from the cluster subnet's secondary IP address range for Pods. Unless you set a different maximum number of Pods per node, GKE allocates a /24 alias IP range (256 addresses) to each node for the Pods running on it. On each node, those 256 alias IP addresses are used to support up to 110 Pods.

For a 900-node cluster supporting up to 110 Pods per node, you need 900 × 256 = 230,400 IP addresses for Pods. (Each node is allocated an alias IP range whose netmask's size is /24.) This cluster requires a subnet whose secondary IP range for Pods has a subnet mask no larger than /14. This secondary IP range range provides 2(32-14) = 218 = 262,144 IP addresses for Pods.

Refer to Subnet secondary IP address range for Pods for more information.

Services

Service (cluster IP) addresses are taken from the cluster's subnet's secondary IP address range for Services. You must ensure this range is large enough to provide addresses for all the Kubernetes Services you host in your cluster.

For a cluster that runs up to 3000 Services, you need 3000 cluster IP addresses. You need a secondary range of size /20 or larger. A /20 range of IP addresses results in 2(32-20) = 212 = 4,096 IP addresses.

Refer to Subnet secondary IP address range for Services for more information.

Secondary range assignment methods

You can assign Pod IP ranges and Service (ClusterIP) address ranges to a VPC-native cluster using one of two methods:

Managed by GKE (default)

GKE can create and manage the subnet's secondary ranges for you. When you create the cluster, you specify either a complete CIDR range or the size of a netmask for both the Pods and Services. For example, you can specify 10.1.0.0/16 for Pods and 10.2.0.0/20 for Services, or you can specify /16 for Pods and /20 for Services.

If you create the cluster and subnet simultaneously, Pod and Service IP ranges are managed by GKE.

User-managed

You can create the subnet's secondary IP ranges, then create a cluster that uses those ranges. If you manually create the secondary ranges, you must manage them yourself.

Differences with routes-based clusters

The allocation scheme for Pod and Service (ClusterIP) addresses is different than the scheme used by a routes-based cluster. Instead of specifying a single CIDR for Pods and Services together, you must choose or create two secondary IP ranges in cluster's subnet: one for Pods and another for Services.

Shared VPC considerations

When creating a VPC-native cluster in a Shared VPC environment, a project owner, editor, or IAM member with the Network Admin role in the Shared VPC host project must create the cluster's subnet and secondary IP ranges manually. A service project admin who creates a cluster must at least have subnet-level permissions to the desired subnet in the Shared VPC network's host project.

In a Shared VPC environment, secondary IP ranges cannot be managed by GKE. A Network Admin in the Shared VPC host project must create the subnet and secondary IP ranges before you can create the cluster. For an example showing how to set up a VPC-native cluster in a Shared VPC network, refer to Setting up clusters with Shared VPC.

IP range planning

Use the information in the following sections to help you calculate sizes for primary and secondary IP ranges of the subnet used by your cluster.

Subnet primary IP address range

Every subnet must have a primary IP address range. You can expand the primary IP address range at any time, even when Google Cloud resources use the subnet; however, you cannot shrink or change a subnet's primary IP address scheme after the subnet has been created. The first two and last two IP addresses of a primary IP range are reserved by Google Cloud.

The following table shows the maximum number of nodes you can create in all clusters that use the subnet, given the size of the subnet's primary IP address range.

Subnet primary IP range Maximum nodes
/29
Minimum size for a subnet's primary IP range
4 nodes
/28 12 nodes
/27 28 nodes
/26 60 nodes
/25 124 nodes
/24 252 nodes
/23 508 nodes
/22 1,020 nodes
/21 2,044 nodes
/20
Default size of a subnet's primary IP range in auto mode networks
4,092 nodes
/19 8,188 nodes
/8
Maximum size for a subnet's primary IP range
16,777,212 nodes

Useful formulas

You can use the following formulas to:

  • Calculate the maximum number of nodes, N, that a given netmask can support. Use S for the size of the netmask, whose valid range is between 8 and 29, inclusive.

    N = 2(32 -S) - 4

  • Calculate the size of the netmask, S, required to support a maximum of N nodes:

    S = 32 - ⌈log2(N + 4)⌉

    ⌈⌉ is the ceiling (least integer) function, meaning round up to the next integer. The valid range for the size of the netmask, S, is between 8 and 29, inclusive.

Subnet secondary IP address range for Pods

Carefully plan your secondary IP range for Pods. Though it is possible to replace a subnet's secondary IP address range, doing so is not supported because it has the potential to put the cluster in an unstable state.

The following table shows the maximum number of nodes and Pods you can create in all clusters that use the subnet, given the size of the subnet's secondary IP address range used by Pods. This table assumes the maximum number of Pods per node is 110 (the default and largest possible Pod density).

Subnet secondary IP range for Pods Maximum Pod IP addresses Maximum nodes Maximum Pods
/24
Smallest possible Pod IP range when the secondary range assignment method is user-managed
256 addresses 1 node 110 Pods
/23
Only possible when the secondary range assignment method is user-managed
512 addresses 2 nodes 220 Pods
/22
Only possible when the secondary range assignment method is user-managed
1,024 addresses 4 nodes 440 Pods
/21
Smallest possible Pod IP range when the secondary range assignment method is managed by GKE
2,048 addresses 8 nodes 880 Pods
/20 4,096 addresses 16 nodes 1,760 Pods
/19 8,192 addresses 32 nodes 3,520 Pods
/18 16,384 addresses 64 nodes 7,040 Pods
/17 32,768 addresses 128 nodes 14,080 Pods
/16 65,536 addresses 256 nodes 28,160 Pods
/15 131,072 addresses 512 nodes 56,320 Pods
/14
Default size for the subnet's secondary IP range for Pods when the secondary range assignment method is managed by GKE
262,144 addresses 1,024 nodes 112,640 Pods
/13 524,288 addresses 2,048 nodes 225,280 Pods
/12 1,048,576 addresses 4,096 nodes 450,560 Pods
/11
Largest possible Pod address range
2,097,152 addresses 8,192 nodes 901,120 Pods

If you have changed the maximum number of Pods per node, you can use the following formulas to calculate the maximum number of nodes and Pods that a subnet's secondary IP range for Pods can support:

  1. Calculate the size of the netmask of each node's Pod range, M.
    M = 31 - ⌈log2(Q)⌉ where:

    • Q is the number of Pods per node
    • ⌈⌉ is the ceiling (least integer) function, meaning round up to the next integer
  2. Calculate the maximum number of nodes, N, that the subnet's secondary IP range for Pods can support:
    N = 2(M - S) where:

    • M is the size of the netmask of each node's alias IP range for Pods, calculated in the first step
    • S is the size of the subnet mask of the subnet's secondary IP range
  3. Calculate the maximum number of Pods, P, that the subnet's secondary IP range for Pods can support:
    P = N × Q where:

    • N is the maximum number of nodes, calculated in the previous step
    • Q is the number of Pods per node

Subnet secondary IP address range for Services

Carefully plan your secondary IP range for Services. Because this is also a subnet secondary IP range, you can only replace it when no Google Cloud resources use it. This range cannot be changed as long as a cluster uses it for Services (cluster IP addresses).

Unlike node and Pod IP ranges, each cluster must have a unique subnet secondary IP range for Services.

The following table shows the maximum number of Services you can create in a single cluster using the subnet, given the size of the subnet's secondary IP address range for Services.

Secondary IP range for Services Maximum number of Services
/27
Smallest possible Service address range
32 Services
/26 64 Services
/25 128 Services
/24 256 Services
/23 512 Services
/22 1,024 Services
/21 2,048 Services
/20
Default size for the subnet's secondary IP range for Services when the secondary range assignment method is managed by GKE
4,096 Services
/19 8,192 Services
/18 16,384 Services
/17 32,768 Services
/16
Largest possible Service address range
65,536 Services

Node limiting ranges

The maximum number of Pods and Services for a given GKE cluster is limited by the size of the cluster's secondary ranges. The maximum number of nodes in the cluster is limited by the size of the cluster's subnet's primary IP address range and the cluster's Pod address range.

The Cloud Console shows error messages like the following to indicate that either the subnet's primary IP address range or the cluster's Pod IP address range (the subnet's secondary IP address range for Pods) has been exhausted:

Instance [node name] creation failed: IP space of [cluster subnet] is
exhausted

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region [COMPUTE_REGION]
  • Update gcloud to the latest version:
    gcloud components update

Procedures

Use the following procedures to create a VPC-native cluster and verify its configured Pod and Service IP ranges.

Creating a cluster in an existing subnet

The following directions demonstrate how to create a VPC-native GKE cluster in an existing subnet with your choice of secondary range assignment method.

Console

The following steps create a cluster using the Cloud Console.

  1. Go to the Google Kubernetes Engine menu in Cloud Console.

    Visit the Google Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure your cluster as desired. Then, click Availability, networking, security, and additional features.

  4. In the VPC-native section, ensure that Enable VPC-native (Using alias IP) is selected.

  5. Choose a VPC in the Network pop-up button.

  6. Choose a subnet for the cluster from the Node subnet pop-up button.

  7. Check Automatically create secondary ranges if you want the secondary range assignment method to be managed by GKE. Un-check this box if you have already created secondary ranges for the chosen subnet and would like the secondary range assignment method to be user-managed.

  8. Fill in the Pod address range field with the pod range (example: 10.0.0.0/14).

  9. Fill in the Service address range field with the service range (example: 10.4.0.0/19).

  10. Make other configuration selections for your cluster, then click Create.

gcloud

  • To use a secondary range assignment method of managed by GKE:

    gcloud container clusters create CLUSTER_NAME \
      --region=REGION \
      --enable-ip-alias \
      --subnetwork=SUBNET_NAME \
      --cluster-ipv4-cidr=POD_IP_RANGE \
      --services-ipv4-cidr=SERVICES_IP_RANGE
    
  • To use a secondary range assignment method of user- managed:

    gcloud container clusters create CLUSTER_NAME \
      --region=REGION \
      --enable-ip-alias \
      --subnetwork=SUBNET_NAME \
      --cluster-secondary-range-name=SECONDARY_RANGE_PODS \
      --services-secondary-range-name=SECONDARY_RANGE_SERVICES
    

Replace the placeholders with valid values:

  • CLUSTER_NAME is the name of the GKE cluster
  • REGION is the region in which the cluster is created. To create a zonal cluster, replace this flag with --zone=ZONE, where ZONE is a Google Cloud zone.
  • SUBNET_NAME is the name of an existing subnet. The subnet's primary IP range is used for nodes. The subnet must exist in the same region as the one used by the cluster. If omitted, GKE attempts to use a subnet in the default VPC network in the cluster's region.
  • If the secondary range assignment method is managed by GKE:
    • POD_IP_RANGE is an IP address range in CIDR notation (for example, 10.0.0.0/14) or the size of a CIDR block's subnet mask (for example, /14). This is used to create the subnet's secondary IP address range for Pods.
    • SERVICES_IP_RANGE is an IP address range in CIDR notation (for example, 10.4.0.0/19) or the size of a CIDR block's subnet mask (for example, /19). This is used to create the subnet's secondary IP address range for Services.
  • If the secondary range assignment method is user-managed:
    • SECONDARY_RANGE_PODS is the name of an existing secondary IP address range in the specified SUBNET_NAME. GKE uses the entire subnet secondary IP range for the cluster's Pods.
    • SECONDARY_RANGE_SERVICES is the name of an existing secondary IP address range in the specified SUBNET_NAME. GKE uses the entire subnet secondary IP range for the cluster's Services.

API

When you create a VPC-native cluster, you define an IPAllocationPolicy object. You can reference existing subnet secondary IP ranges or you can specify CIDR blocks. Reference existing subnet secondary IP ranges to create a cluster whose secondary range assignment method is user-managed. Provide CIDR blocks if you want the range assignment method to be managed by GKE.

{
  "name": [CLUSTER_NAME],
  "description": [DESCRIPTION],
  ...
  "ipAllocationPolicy": {
    "useIpAliases": true,
    "clusterIpv4CidrBlock"      : string,
    "servicesIpv4CidrBlock"     : string,
    "clusterSecondaryRangeName" : string,
    "servicesSecondaryRangeName": string,

  },
  ...
}

where:

  • "clusterIpv4CidrBlock" is the size/location of the CIDR range for Pods. Determines the size of the secondary range for Pods, and can be IP/size in CIDR notation (such as 10.0.0.0/14) or /size (like /14). An empty space with the given size is chosen from the available space in your VPC. If left blank, a valid range is found and created with a default size.
  • "servicesIpv4CidrBlock" is the size/location of the CIDR range for Services. See description of "clusterIpv4CidrBlock".
  • "clusterSecondaryRangeName" is the name of the secondary range for Pods. The secondary range must already exist and belong to the subnetwork associated with the cluster (such as the subnetwork specified with the --subnetwork flag).
  • "serviceSecondaryRangeName" is the name of the secondary range for Services. The secondary range must already exist and belong to the subnetwork associated with the cluster (such as the subnetwork specified with by --subnetwork flag).

Creating a cluster and subnet simultaneously

The following directions demonstrate how to create a VPC-native GKE cluster and subnet at the same time. The secondary range assignment method is managed by GKE when you perform these two steps with one command.

Console

You cannot create a cluster and subnet simultaneously using the Cloud Console. Instead, first create a subnet then create the cluster in an existing subnet.

gcloud

To create a VPC-native cluster and subnet simultaneously:

gcloud container clusters create CLUSTER_NAME \
    --region=REGION \
    --enable-ip-alias \
    --create-subnetwork name=SUBNET_NAME,range=NODE_IP_RANGE \
    --cluster-ipv4-cidr=POD_IP_RANGE \
    --services-ipv4-cidr=SERVICES_IP_RANGE

--create-subnetwork name=my-subnet,range=/21

  • CLUSTER_NAME is the name of the GKE cluster
  • REGION is the region in which the cluster is created. To create a zonal cluster, replace this flag with --zone=ZONE, where ZONE is a GKE zone.
  • SUBNET_NAME is the name of the subnet to create. The subnet's region is the same region as the cluster (or the region containing the zonal cluster). Use an empty string (name="") if you want GKE to generate a name for you.
  • NODE_IP_RANGE is an IP address range in CIDR notation (for example, 10.5.0.0/20) or the size of a CIDR block's subnet mask (for example, /20). This is used to create the subnet's primary IP address range for nodes. If omitted, GKE chooses an available IP range in the VPC with a size of /20.
  • POD_IP_RANGE is an IP address range in CIDR notation (for example, 10.0.0.0/14) or the size of a CIDR block's subnet mask (for example, /14). This is used to create the subnet's secondary IP address range for Pods. If omitted, GKE uses /14, the default Pod IP range size.
  • SERVICES_IP_RANGE is an IP address range in CIDR notation (for example, 10.4.0.0/19) or the size of a CIDR block's subnet mask (for example, /19). This is used to create the subnet's secondary IP address range for Services. If omitted, GKE uses /20, the default Services IP range size.

API

To create a VPC-native cluster, define an IPAllocationPolicy object in your cluster resource:

{
  "name": [CLUSTER_NAME],
  "description": [DESCRIPTION],
  ...
  "ipAllocationPolicy": {
    "useIpAliases": true,
    "createSubnetwork": true,
    "subnetworkName": [SUBNET_NAME]
  },
  ...
}

createSubnetwork automatically creates and provisions a subnetwork for the cluster. subnetworkName is optional; if left empty, a name is automatically chosen for the subnetwork.

Verifying Pod and Service ranges

After you create a VPC-native cluster, you can verify its Pod and Service ranges.

gcloud

To verify the cluster, run the following command:

gcloud container clusters describe [CLUSTER_NAME]

In the command output, look under the ipAllocationPolicy field:

  • clusterIpv4Cidr is the secondary range for Pods
  • servicesIpv4Cidr is the secondary range for Services

Console

To verify the cluster, perform the following steps:

  1. Visit the Google Kubernetes Engine menu in Cloud Console.

    Visit the Google Kubernetes Engine menu

  2. Select the desired cluster.

The secondary ranges are displayed in the Cluster section under the Details tab:

  • Container address range is the secondary range for Pods
  • Service address range is the secondary range for Services

Troubleshooting

This section provides guidance for resolving issues related to VPC-native clusters.

The resource 'projects/[PROJECT_NAME]/regions/XXX/subnetworks/default' is not ready

Potential causes
There are parallel operations on the same subnet. For example, another VPC-native cluster is being created, or a secondary range is being added or deleted on the subnet.
Resolution
Retry the command.

Invalid value for field 'resource.secondaryIpRanges[1].ipCidrRange': 'XXX'. Invalid IPCidrRange: XXX conflicts with existing subnetwork 'default' in region 'XXX'."

Potential causes

Another VPC-native cluster is being created at the same time, and is attempting to allocate the same ranges.

The same secondary range is being added to the subnetwork.

Resolution

If this error is returned on cluster creation when no secondary ranges were specified, retry the cluster creation command.

Not enough free IP space for Pods

Symptoms

Cluster is stuck in a provisioning state for an extended period of time

Cluster creation returns a Managed Instance Group (MIG) error

New nodes cannot be added to an existing cluster

Potential causes

Unallocated space in the Pod IP range is not large enough for the nodes requested in the cluster. For example, if a cluster's Pod IP range has a netmask whose size is /23 (512 address), and the maximum Pods per node is 110, you cannot create any more than two nodes. (Each node is assigned an alias IP range with a netmask whose size is /24.)

Solutions

Create a replacement cluster after reviewing and planning appropriately sized primary and secondary IP ranges. Refer to IP ranges for VPC-native clusters and IP range planning.

If you cannot replace the cluster, you might be able to work around the problem if you can create a new node pool with a smaller maximum number of Pods per node. If possible, migrate workloads to that node pool then delete the previous node pool. Reducing the maximum number of Pods per node allows you to support more nodes on a fixed secondary IP range for Pods. Refer to Subnet secondary IP address range for Pods and Node limiting ranges for details about the calculations involved.

What's next

Var denne siden nyttig? Si fra hva du synes:

Send tilbakemelding om ...

Kubernetes Engine Documentation