Creating VPC-native clusters using Alias IPs

This page explains how to create VPC-native clusters in Google Kubernetes Engine using Alias IPs.

Overview

With Alias IPs, GKE clusters can allocate IP addresses from a CIDR block known to Google Cloud Platform (GCP). This makes your cluster more scalable, and allows your cluster to better interact with other GCP products and entities.

Benefits

Using Alias IPs has several benefits:

  • Pod IPs are natively routable within the GCP network (including via VPC Network Peering), and no longer use up route quota.
  • Pod IPs are reserved within the network ahead of time, which prevents conflict with other compute resources.
  • The networking layer can perform anti-spoofing checks to ensure that egress traffic is not sent with arbitrary source IPs.
  • Aliased IPs can be announced through BGP by Cloud Router.
  • Firewall controls for Pods can be applied separately from their nodes.
  • Alias IPs allow Pods to directly access hosted services without using a NAT gateway.

Restrictions

  • You cannot currently migrate a VPC-native cluster to one that uses advanced routes.
  • You cannot currently migrate an existing cluster that uses routes for Pod routing to a cluster that uses Alias IPs.
  • Cluster IPs for internal Services remain only available from within the cluster. If you want to access a Kubernetes Service from within the VPC, but from outside of the cluster (for example, from a Compute Engine instance), use an internal load balancer.

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • If you are working with regional clusters, set your default compute region:
    gcloud config set compute/region [COMPUTE_REGION]
  • Update gcloud to the latest version:
    gcloud components update

Secondary ranges

Clusters created with Alias IPs use two secondary ranges to allocate IP addresses in your VPC:

  • A range that allocates Alias IPs for Pod IPs
  • A range that reserves space for ClusterIP Services to prevent reuse of Service IPs within your VPC

Alias IPs' allocation scheme is different than the scheme used with advanced routes. With advanced routes, one CIDR range is allocated for the whole cluster, and range is divided between Pods and services (the last /20 of the range). With Alias IPs, the Service range is a distinct CIDR range outside of the cluster range.

Secondary range management

There are two ways to associate secondary ranges with your cluster:

Managed by GKE
You can have the control plane manage the secondary ranges. This is the default mode used when you create a cluster with Alias IPs and when you use the CIDR ranges/netmask flags (--cluster-ipv4-cidr and --services-ipv4-cidr).
You can customize secondary range size and placement by explicitly naming a CIDR range (for example, 10.0.0.0/16) or by defining a netmask size (such as /16). When you create or delete a cluster in this way, GKE automatically creates or deletes the secondary ranges in your VPC.
User-managed
You can manually create the secondary ranges, then create a cluster with those ranges. If you manually create secondary ranges, you must manage them yourself.
You must manually create secondary ranges for clusters on shared VPC networks. With shared VPC, users who create GKE clusters cannot create secondary ranges with their tenant credentials. The owner of the host project creates the secondary ranges and passes them to the tenant for use with their cluster.

Considerations for cluster sizing

The maximum number of nodes and Services for a given GKE cluster is determined by the size of the cluster's secondary ranges. You cannot change secondary range sizes after you create a cluster. When you create a cluster, ensure that you choose secondary ranges large enough to accommodate the cluster's anticipated growth.

The following table outlines the three ranges you must consider: nodes, Pods and Services:

Range Guidance
Nodes

Node IPs are taken from the subnetwork associated with the cluster. Your cluster subnetwork must be large enough to fit the total number of nodes in your cluster.

For example, if you plan to create a 900-node cluster, the subnet used with the cluster must be at least a /22 in size. A subnet size /22 contains 2(32-22) = 210 = 1024 IP addresses, which is sufficient for the 900 node IPs needed for the cluster.

Pods

Each node in the cluster is currently allocated a /24 (2(32-24) = 28 = 256) block of Pod IPs. The Pod IPs are taken from the associated secondary range for Pods. The Pod range as determined by the --cluster-ipv4-cidr or --cluster-secondary-range-name flags must be at least large enough to fit (total number of nodes × 256) IP addresses.

For example, for a 900-node cluster, you need 900 × 256 = 230,400 IP addresses. The IPs must come in /24-sized blocks, as that is the granularity assigned to a node. You need a secondary range of size /14 or larger. A /14 range of IPs results in 2(32-14) = 218 ≈ 262k IP addresses.

Services

Every cluster needs to reserve a range of IPs for Kubernetes Service cluster IPs. The Service IPs are assigned from the associated secondary range for Services. You must ensure that the block of IPs is sufficient for the total number of Services that you anticipate to run in the cluster. You define the ranges defined using the --services-ipv4-cidr or --services-secondary-range-name flags.

For example, for a cluster that runs at most 3000 Services, you need 3000 IP addresses to be used for cluster IPs. You need a secondary range of size /20 or larger. A /20 range of IPs results in 2(32-20) = 212 ≈ 4k IP addresses.

Defaults and limits for range sizes

Range Default size Minimum size Maximum size
Pods

/14 = 218 = 262,000 IP addresses allocated for Pods

/19 = 213 = 8000 IP addresses

/11 = 221 = approximately two million IP addresses

Services

/20 = 212 = 4096 IP addresses allocated for Services

/24 = 28 = 256 IP addresses allocated

/16 = 216 = 64k IP addresses allocated for Services

Creating a cluster with Alias IPs

You can create a VPC-native cluster with Alias IPs using Google Cloud Platform Console, the gcloud command-line tool, or the GKE API.

gcloud

To create the cluster with Alias IPs, run the following command, where [CLUSTER_NAME] is the name you choose for the cluster:

gcloud container clusters create [CLUSTER_NAME] --enable-ip-alias

Console

To create a cluster with Alias IPs, perform the following steps:

  1. Visit the Google Kubernetes Engine menu in GCP Console.

    Visit the Google Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure your cluster as desired. Then, click Advanced options.
  4. In the VPC-native section, select Enable VPC-native (using alias IP).
  5. Click Create.

API

To create a cluster with Alias IPs, define a IPAllocationPolicy object in your cluster resource:

{
  "name": [CLUSTER_NAME],
  "description": [DESCRIPTION],
  ...
  "ipAllocationPolicy": {
    "useIpAliases": true,
    "clusterIpv4CidrBlock"      : string,
    "servicesIpv4CidrBlock"     : string,
    "clusterSecondaryRangeName" : string,
    "servicesSecondaryRangeName": string,

  },
  ...
}

where:

  • "useIpAliases": true creates the cluster with Alias IPs
  • "clusterIpv4CidrBlock" is the size/location of the CIDR range for Pods. Determines the size of the secondary range for Pods, and can be IP/size in CIDR notation (such as 10.0.0.0/14) or /size (like /14). An empty space with the given size is chosen from the available space in your VPC. If left blank, a valid range is found and created with a default size.
  • "servicesIpv4CidrBlock" is the size/location of the CIDR range for Services. See description of "clusterIpv4CidrBlock".
  • "clusterSecondaryRangeName" is the name of the secondary range for Pods. The secondary range must already exist and belong to the subnetwork associated with the cluster (such as the subnetwork specified with the --subnetwork flag).
  • "clusterSecondaryRangeName" is the name of the secondary range for Services. The secondary range must already exist and belong to the subnetwork associated with the cluster (such as the subnetwork specified with by --subnetwork flag).

For more cluster examples, see Examples.

Verifying the cluster's secondary ranges

After you create a cluster with Alias IPs, you should verify the cluster's ranges.

gcloud

To verify the cluster, run the following command:

gcloud container clusters describe [CLUSTER_NAME]

In the command output, look under the ipAllocationPolicy field:

  • clusterIpv4Cidr is the secondary range for Pods
  • servicesIpv4Cidr is the secondary range for Services

Console

To verify the cluster, perform the following steps:

  1. Visit the Google Kubernetes Engine menu in GCP Console.

    Visit the Google Kubernetes Engine menu

  2. Select the desired cluster.

The secondary ranges are displayed in the Cluster section under the Details tab:

  • Container address range is the secondary range for Pods
  • Service address range is the secondary range for Services

Examples

The following sections provide examples for using Alias IPs.

Creating a cluster with specific IP ranges

gcloud

This command creates my-cluster with the given Pod and Service ranges. Secondary ranges are automatically created, attached to the default subnetwork, and managed by GKE:

gcloud container clusters create my-cluster \
  --enable-ip-alias --cluster-ipv4-cidr=10.0.0.0/14 \
  --services-ipv4-cidr=10.4.0.0/19

Console

This creates my-cluster with the given Pod and Service ranges. Secondary ranges are automatically created, attached to the default subnetwork, and managed by GKE:

  1. Visit the Google Kubernetes Engine menu in GCP Console.

    Visit the Google Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure your cluster as desired. Then, click Advanced options.
  4. In the VPC-native section, select Enable VPC-native (Using alias IP).
  5. Fill in the Pod address range field with the pod range (example: 10.0.0.0/14)
  6. Fill in the Service address range field with the service range (example: 10.4.0.0/19)

Creating a cluster with specific sizes on a different subnetwork

This command creates my-cluster with the given Pod and Service range sizes. CIDR range location is determined from the free space in the VPC. Secondary ranges is created attached to the subnetwork my-subnet. The secondary ranges are managed by GKE:

gcloud container clusters create my-cluster \
  --subnetwork my-subnet
  --enable-ip-alias --cluster-ipv4-cidr=/16 \
  --services-ipv4-cidr=/22

Creating a cluster with existing secondary ranges

For details on creating secondary ranges for Alias IPs, refer to Alias IP Ranges Overview.

You need to create two secondary ranges on the subnet, one called my-pods for Pods and another called my-services for Services:

gcloud compute networks subnets update my-subnet \
    --add-secondary-ranges my-pods=10.0.0.0/16,my-services=10.1.0.0/16
    --region us-central1

Then, to create your cluster:

gcloud

Run the following command:

gcloud container clusters create my-cluster \
--enable-ip-alias \
--cluster-secondary-range-name=my-pods \
--services-secondary-range-name=my-services

Console

Perform the following steps:

  1. Visit the Google Kubernetes Engine menu in GCP Console.

    Visit the Google Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure your cluster as desired. Then, click Advanced options.
  4. From the VPC-native section, select Enable VPC-native (using alias IP).
  5. Set Automatically create secondary ranges to false.
  6. Select the Network and Node subnet you want to use with your cluster.
  7. Select the Pod secondary range and Services secondary range to use with your cluster.

Creating a subnetwork automatically

The following procedures create a new alias IP cluster with an automatically-generated subnetwork.

gcloud

To create a new cluster that uses Alias IPs, run the following command:

gcloud container clusters create --enable-ip-alias --create-subnetwork name=my-cluster-subnet

In this command, the new cluster is automatically configured with IP ranges and a subnetwork. You can either provide a name for the subnetwork (in this example, name=my-cluster-subnet) or provide an empty string ("") to have a name automatically generated.

To configure the cluster yourself, run the following command:

gcloud container clusters create [CLUSTER_NAME] --enable-ip-alias \
--create-subnetwork="" [--cluster-ipv4-cidr [RANGE] --services-ipv4-cidr [RANGE] ]

In this command:

  • [CLUSTER_NAME] is the name you choose for the cluster.
  • --enable-ip-alias flag indicates that the cluster uses Alias IPs.
  • --create-subnetwork flag causes a subnetwork for the cluster to be automatically created.
  • The optional --cluster-ipv4-cidr flag indicates the size and location of the cluster's CIDR range. [RANGE] can be in the form of [IP]/[SIZE], such as 10.0.0.0/18, or simply /[SIZE], which causes the IP address to be automatically assigned. If this flag is omitted, a CIDR range is automatically assigned with default sizes.
  • The optional --services-ipv4-cidr flag indicates the size and location of the Service's CIDR range. The [RANGE] specifications are identical to --cluster-ipv4-cidr. The range cannot overlap with --cluster-ip4-cidr and vice-versa. If this flag is omitted, a CIDR range is automatically assigned with default sizes.

API

To create a cluster with Alias IPs, define a IPAllocationPolicy object in your cluster resource:

{
  "name": [CLUSTER_NAME],
  "description": [DESCRIPTION],
  ...
  “ipAllocationPolicy”: {
    "useIpAliases": true,
    "createSubnetwork": true,
    "subnetworkName": [SUBNET_NAME]
  },
  ...
}

"useIpAliases": true creates the cluster with Alias IPs. createSubnetwork automatically creates and provisions a subnetwork for the cluster. subnetworkName is optional; if left empty, a name is automatically chosen for the subnetwork.

Troubleshooting

This section provides guidance for resolving issues related to Alias IPs.

Not enough free IP space for Pod

Symptoms
  • Cluster is stuck in a provisioning state for an extended period of time
  • Cluster creation returns a Managed Instance Group (MIG) error
  • New nodes cannot be added to an existing cluster
Potential causes

Unallocated space in the Pods secondary range is not large enough for the nodes requested in the cluster. Refer to Considerations for cluster sizing for guidance in properly sizing IP ranges.

If this issue occurs during cluster creation, delete the cluster stuck in the provisioning state and create another with a secondary range that has sufficient space for the cluster. For example, if a user specifies a secondary CIDR Range smaller than /24 and assigns it to a cluster, then the cluster can become in stuck in the provisioning state.

If this issue occurs during node pool resize, you must make room for the new nodes by removing existing nodes.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Kubernetes Engine