Creating VPC-native clusters using Alias IPs

This page explains how to create VPC-native clusters in Kubernetes Engine using Alias IPs.

Overview

With Alias IPs, Kubernetes Engine clusters can allocate IP addresses from a CIDR block known to Google Cloud Platform (GCP). This makes your cluster more scalable, and allows your cluster to better interact with other GCP products and entities.

Benefits

Using Alias IPs has several benefits:

  • Pod IPs are natively routable within the GCP network (including via VPC Network Peering), and no longer use up route quota.
  • Pod IPs are reserved within the network ahead of time, which prevents conflict with other compute resources.
  • The networking layer can perform anti-spoofing checks to ensure that egress traffic is not sent with arbitrary source IPs.
  • Aliased IPs can be announced through BGP by Cloud Router.
  • Firewall controls for Pods can be applied separately from their nodes.
  • Alias IPs allow Pods to directly access hosted services without using a NAT gateway.

Restrictions

  • You cannot currently migrate an existing cluster that uses routes for Pod routing to a cluster that uses Alias IPs.
  • Cluster IPs for internal Services remain only available from within the cluster. If you want to access a Kubernetes Service from within the VPC, but from outside of the cluster (for example, from a Compute Engine instance), use an internal load balancer.

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • Set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • Update all gcloud commands to the latest version:
    gcloud components update

Secondary ranges

Clusters created with Alias IPs use two secondary ranges to allocate IP addresses in your VPC:

  • A range that allocates Alias IPs for Pod IPs
  • A range that reserves space for ClusterIP Services to prevent reuse of Service IPs within your VPC

Alias IPs' allocation scheme is different than the scheme used with advanced routes. With advanced routes, one CIDR range is allocated for the whole cluster, and range is divided between Pods and services (the last /20 of the range). With Alias IPs, the Service range is a distinct CIDR range outside of the cluster range.

Secondary range management

There are two ways to associate secondary ranges with your cluster:

Managed by Kubernetes Engine
You can have the control plane manage the secondary ranges. This is the default mode used when you create a cluster with Alias IPs and when you use the CIDR ranges/netmask flags (--cluster-ipv4-cidr and --services-ipv4-cidr).
You can customize secondary range size and placement by explicitly naming a CIDR range (for example, 10.0.0.0/16) or by defining a netmask size (such as /16). When you create or delete a cluster in this way, Kubernetes Engine automatically creates or deletes the secondary ranges in your VPC.
User-managed
You can manually create the secondary ranges, then create a cluster with those ranges. If you manually create secondary ranges, you must manage them yourself.
You must manually create secondary ranges for clusters on shared VPC networks. With shared VPC, users who create Kubernetes Engine clusters cannot create secondary ranges with their tenant credentials. The owner of the host project creates the secondary ranges and passes them to the tenant for use with their cluster.

Considerations for cluster sizing

The maximum number of nodes and Services for a given Kubernetes Engine cluster is determined by the size of the cluster's secondary ranges. You cannot change secondary range sizes after you create a cluster. When you create a cluster, ensure that you choose secondary ranges large enough to accommodate the cluster's anticipated growth.

The following table outlines the three ranges you must consider: nodes, Pods and Services:

Range Guidance
Nodes

Node IPs are taken from the subnetwork associated with the cluster. Your cluster subnetwork must be large enough to fit the total number of nodes in your cluster.

For example, if you plan to create a 900-node cluster, the subnet used with the cluster must be at least a /22 in size. A subnet size /22 contains 2(32-22) = 210 = 1024 IP addresses, which is sufficient for the 900 node IPs needed for the cluster.

Pods

Each node in the cluster is currently allocated a /24 (2(32-24) = 28 = 256) block of Pod IPs. The Pod IPs are taken from the associated secondary range for Pods. The Pod range as determined by the --cluster-ipv4-cidr or --cluster-secondary-range-name flags must be at least large enough to fit (total number of nodes × 256) IP addresses.

For example, for a 900-node cluster, you need 900 × 256 = 230,400 IP addresses. The IPs must come in /24-sized blocks, as that is the granularity assigned to a node. You need a secondary range of size /14 or larger. A /14 range of IPs results in 2(32-14) = 218 ≈ 262k IP addresses.

Services

Every cluster needs to reserve a range of IPs for Kubernetes Service cluster IPs. The Service IPs are assigned from the associated secondary range for Services. You must ensure that the block of IPs is sufficient for the total number of Services that you anticipate to run in the cluster. You define the ranges defined using the --services-ipv4-cidr or --services-secondary-range-name flags.

For example, for a cluster that runs at most 3000 Services, you need 3000 IP addresses to be used for cluster IPs. You need a secondary range of size /20 or larger. A /20 range of IPs results in 2(32-20) = 212 ≈ 4k IP addresses.

Defaults and limits for range sizes

Range Default size Minimum size Maximum size
Pods

/14 = 218 = 262,000 IP addresses allocated for Pods

/19 = 213 = 8000 IP addresses

/11 = 221 = approximately two million IP addresses

Services

/20 = 220 = 4096 IP addresses allocated for Services

/24 = 28 = 256 IP addresses allocated

/16 = 216 = 64k IP addresses allocated for Services

Creating a cluster with Alias IPs

You can create a VPC-native cluster with Alias IPs using Google Cloud Platform Console, the gcloud command-line tool, or the Kubernetes Engine API.

Console

To create a cluster with Alias IPs, perform the following steps:

  1. Visit the Kubernetes Engine menu in GCP Console.

    Visit the Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure your cluster as desired. Then, click More.
  4. From the VPC-native (using alias IP) drop-down menu, select Enabled.
  5. Click Create.

gcloud

To create the cluster with Alias IPs, run the following command, where [CLUSTER_NAME] is the name you choose for the cluster:

gcloud container clusters create [CLUSTER_NAME] --enable-ip-alias

API

To create a cluster with Alias IPs, define a IPAllocationPolicy object in your cluster resource:

{
  "name": [CLUSTER_NAME],
  "description": [DESCRIPTION],
  ...
  “ipAllocationPolicy”: {
    "useIpAliases": true,
    “clusterIpv4CidrBlock”      : string,
    “servicesIpv4CidrBlock”     : string,
    “clusterSecondaryRangeName” : string,
    “servicesSecondaryRangeName”: string,

  },
  ...
}

where:

  • "useIpAliases": true creates the cluster with Alias IPs
  • "clusterIpv4CidrBlock" is the size/location of the CIDR range for Pods. Determines the size of the secondary range for Pods, and can be IP/size in CIDR notation (such as 10.0.0.0/14) or /size (like /14). An empty space with the given size is chosen from the available space in your VPC. If left blank, a valid range is found and created with a default size.
  • "servicesIpv4CidrBlock" is the size/location of the CIDR range for Services. See description of "clusterIpv4CidrBlock".
  • "clusterSecondaryRangeName" is the name of the secondary range for Pods. The secondary range must already exist and belong to the subnetwork associated with the cluster (such as the subnetwork specified with the --subnetwork flag).
  • "clusterSecondaryRangeName" is the name of the secondary range for Services. The secondary range must already exist and belong to the subnetwork associated with the cluster (such as the subnetwork specified with by --subnetwork flag).

For more cluster examples, see Examples.

Verifying the cluster's secondary ranges

After you create a cluster with Alias IPs, you should verify the cluster's ranges.

Console

To verify the cluster, perform the following steps:

  1. Visit the Kubernetes Engine menu in GCP Console.

    Visit the Kubernetes Engine menu

  2. Select the desired cluster.

The secondary ranges are displayed in the Cluster menu under the Details tab:

  • Container address range is the secondary range for Pods
  • Service address range is the secondary range for Services

gcloud

To verify the cluster, run the following command:

gcloud container clusters describe [CLUSTER_NAME]

In the command output, look under the ipAllocationPolicy field:

  • clusterIpv4Cidr is the secondary range for Pods
  • servicesIpv4Cidr is the secondary range for Services

Examples

The following sections provide examples for using Alias IPs.

Creating a cluster with specific IP ranges

Console

This creates my-cluster with the given Pod and Service ranges. Secondary ranges are automatically created, attached to the default subnetwork, and managed by Kubernetes Engine:

  1. Visit the Kubernetes Engine menu in GCP Console.

    Visit the Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure your cluster as desired. Then, click More.
  4. From the VPC-native (using alias IP) drop-down menu, select Enabled.
  5. Fill in the Container address range field with the pod range (example: 10.0.0.0/14)
  6. Fill in the Service address range field with the service range (example: 10.4.0.0/19)

gcloud

This command creates my-cluster with the given Pod and Service ranges. Secondary ranges are automatically created, attached to the default subnetwork, and managed by Kubernetes Engine:

gcloud container clusters create my-cluster \
  --enable-ip-alias --cluster-ipv4-cidr=10.0.0.0/14 \
  --services-ipv4-cidr=10.4.0.0/19

Creating a cluster with specific sizes on a different subnetwork

This command creates my-cluster with the given Pod and Service range sizes. CIDR range location is determined from the free space in the VPC. Secondary ranges is created attached to the subnetwork my-subnet. The secondary ranges are managed by Kubernetes Engine:

gcloud container clusters create my-cluster \
  --subnetwork my-subnet
  --enable-ip-alias --cluster-ipv4-cidr=/16 \
  --services-ipv4-cidr=/22

Creating a cluster with existing secondary ranges

For details on creating secondary ranges for Alias IPs, refer to Alias IP Ranges Overview.

You need to create two secondary ranges on the subnet, one called my-pods for Pods and another called my-services for Services:

gcloud compute networks subnets my-subnet \
    --add-secondary-ranges=my-pods:10.0.0.0/16,my-services:10.1.0.0/16
    --region us-central1

Then, to create your cluster:

Console

Perform the following steps:

  1. Visit the Kubernetes Engine menu in GCP Console.

    Visit the Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure your cluster as desired. Then, click More.
  4. From the VPC-native (using alias IP) drop-down menu, select Enabled.
  5. Set Automatically create secondary ranges to false.
  6. Select the your Network and Node subnet you want to use with your cluster.
  7. Select the Container secondary range and Services secondary range to use with your cluster.

gcloud

Run the following command:

gcloud container clusters create my-cluster \
--enable-ip-alias \
--cluster-secondary-range-name=my-pods \
--services-secondary-range-name=my-services

Creating a subnetwork automatically

The following procedures create a new alias IP cluster with an automatically-generated subnetwork.

gcloud

To create a new cluster that uses Alias IPs, run the following command:

gcloud container clusters create --enable-ip-alias --create-subnetwork name=my-cluster-subnet

In this command, the new cluster is automatically configured with IP ranges and a subnetwork. You can either provide a name for the subnetwork (in this example, name=my-cluster-subnet) or provide an empty string ("") to have a name automatically generated.

To configure the cluster yourself, run the following command:

gcloud container clusters create [CLUSTER_NAME] --enable-ip-alias \
--create-subnetwork="" [--cluster-ipv4-cidr [RANGE] --services-ipv4-cidr [RANGE] ]

In this command:

  • [CLUSTER_NAME] is the name you choose for the cluster.
  • --enable-ip-alias flag indicates that the cluster uses Alias IPs.
  • --create-subnetwork flag causes a subnetwork for the cluster to be automatically created.
  • The optional --cluster-ipv4-cidr flag indicates the size and location of the cluster's CIDR range. [RANGE] can be in the form of [IP]/[SIZE], such as 10.0.0.0/18, or simply /[SIZE], which causes the IP address to be automatically assigned. If this flag is omitted, a CIDR range is automatically assigned with default sizes.
  • The optional --services-ipv4-cidr flag indicates the size and location of the Service's CIDR range. The [RANGE] specifications are identical to --cluster-ipv4-cidr. The range cannot overlap with --cluster-ip4-cidr and vice-versa. If this flag is omitted, a CIDR range is automatically assigned with default sizes.

API

To create a cluster with Alias IPs, define a IPAllocationPolicy object in your cluster resource:

{
  "name": [CLUSTER_NAME],
  "description": [DESCRIPTION],
  ...
  “ipAllocationPolicy”: {
    "useIpAliases": true,
    "createSubnetwork": true,
    "subnetworkName": [SUBNET_NAME]
  },
  ...
}

"useIpAliases": true creates the cluster with Alias IPs. createSubnetwork automatically creates and provisions a subnetwork for the cluster. subnetworkName is optional; if left empty, a name is automatically chosen for the subnetwork.

Troubleshooting

This section provides guidance for resolving issues related to Alias IPs.

Not enough free IP space for Pod

Symptoms
  • Cluster is stuck in a provisioning state for an extended period of time
  • Cluster creation returns a Managed Instance Group (MIG) error
  • New nodes cannot be added to an existing cluster
Potential causes

Unallocated space in the Pods secondary range is not large enough for the nodes requested in the cluster. Refer to Considerations for cluster sizing for guidance in properly sizing IP ranges.

If this issue occurs during cluster creation, delete the cluster stuck in the provisioning state and create another with a secondary range that has sufficient space for the cluster. For example, if a user specifies a secondary CIDR Range smaller than /24 and assigns it to a cluster, then the cluster can become in stuck in the provisioning state.

If this issue occurs during node pool resize, you must make room for the new nodes by removing existing nodes.

What's next

Send feedback about...

Kubernetes Engine