This document describes a process to identify the Google Kubernetes Engine (GKE) CIDR block requirements. It then introduces a decision model that helps you identify which solution is recommended for any particular scenario. Finally, the document walks through various solutions, describes them briefly, and lists their corresponding advantages and disadvantages.
Identifying GKE CIDR block requirements
In this section, you determine the sizes of the CIDR blocks for your cluster. It's critical that you do this correctly because you cannot change or expand a CIDR block once you've created the cluster.
Determining the Node CIDR block size
Follow these steps to determine the Node CIDR block size:
Determine the maximum number of Nodes you need in your cluster over its lifetime.
If you can determine this number, which depends on the customer application, design, and growth forecasts, use it as your required number of Nodes. If you cannot determine the maximum number of Nodes that you need, use the quota limit of 5,000 Nodes as the maximum.
Determine the host bits needed to account for the required number of Nodes.
When you know the required Node count, you can use table 1 to calculate the host bits needed to generate a netmask. Find the number of required Nodes in the Nodes required column. For the next step, use the corresponding number in the Host bits for Nodes column.
Nodes required Host bits for Nodes 1-4 3 5-12 4 13–28 5 29–60 6 61–124 7 125–252 8 253–508 9 509–1020 10 1021–2044 11 2045–4092 12 4093–8188 13 Table 1. Bits needed for Node addresses.
Generate a CIDR block netmask, using the number from the Host bits for Node column that you determined in the previous step:
32 – host bits for Nodes = Node CIDR block netmask
The following figure shows this equation graphically.
For example, for a 5,000-Node cluster, table 1 shows that 13 host bits for Nodes are required in the Node netmask. To calculate the full netmask, substitute the number of host bits for Nodes in the equation, 32 – 13 = 19. The result is a netmask of
/19.
Determining the Pod CIDR block size
Follow these steps to determine the Pod CIDR block size:
Determine the maximum number of Pods per Node you need in your cluster over its lifetime.
If you can determine this number, which is application dependent, use it as your required number of Pods per Node. If you cannot determine the maximum number you need, use the quota limit of 110 Pods per Node as the maximum.
Determine the host bits needed for the required number of Pods.
When you know the required Pods per Node count, you can use table 2 to calculate the host bits needed to generate a netmask. Find the number of required Pods per Node in the Pods-per-Node count column. Use the corresponding number in the Host bits for Pods column for the next step.
Pods-per-Node count Host bits for Pods 1–8 4 9–16 5 17–32 6 33–64 7 65–110 8 Table 2. Bits needed for Pods per Node.
Generate a CIDR block netmask, using the number of host bits for Nodes that you determined in step two of the Determining the Node CIDR block size section and the number of host bits for Pods that you determined in the previous step:
32 – (host bits for Nodes + host bits for Pods) = Pod CIDR block netmask
The following figure shows this equation graphically.
For example, if you need 110 Pods per Node, you would need 8 host bits for the Pods per Node. You then take the host bits for Nodes (13) and substitute those numbers into the equation, 32 – (13 + 8) = 11. The result is a netmask of
/11.
Determining the cluster IP CIDR block size
Follow these steps to determine the CIDR block size of the cluster IP address:
Determine the maximum number of cluster IP addresses you need in your cluster over its lifetime.
If you can determine this number, which is application dependent, use it as your required number of cluster IP addresses. If you can't determine the maximum number you need, use the default
/20netmask. If you use the default netmask, you can skip the following step.Determine the netmask for the CIDR block of the cluster IP address.
When you know the maximum number of cluster IP addresses you need, you can use table 3 to find the netmask.
Number of cluster IP addresses Netmask 1–32 /2733–64 /2665–128 /25129–256 /24257–512 /23513–1,024 /221,025–2,048 /212,049–4,096 /204,097–8,192 /198,193–16,384 /1816,385–32,768 /1732,769–65,536 /16Table 3. Services' CIDR block netmask.
Understanding address demand
When you look at the CIDR block requirements derived from the preceding steps, notice that the Pod CIDR block places the largest demand for IP address space. Most of the time, the Node CIDR places the second largest demand, followed by the Services CIDR block.
Now that you've calculated the cluster IP address requirements, you need to review the available RFC 1918 IP address space and select a path forward.
Selecting the best solution
Figure 1 shows a decision tree that you can use to determine the best solution given your CIDR block requirements. The Reviewing the solutions section summarizes each solution.
Reviewing the solutions
This section describes each solution and when to use it, and summarizes any advantages, disadvantages, and other issues.
Assigning available address space
When to use:
- After reviewing the available IP address space and walking through figure 1,
you've determined that there's enough RFC 1918 address space available to
allocate for the cluster CIDR blocks. For example, although the
10.0.0.0/8address might be exhausted, there's enough space in the172.16.0.0/12or192.168.0.0/16address space to meet the CIDR block requirements.
Description:
- No other special configuration steps are required for this solution; allocate the address space and continue with the installation.
Advantages:
- Easiest of all the solutions.
- Requires no special configuration.
Disadvantages:
- Depletes available IP address space.
Other issues:
- To ensure that you are optimizing your IP address allocations, use the Flexible Pod CIDR feature.
Translating the GKE Pod CIDR block
When to use:
- After reviewing the available IP address space and walking through figure 1, you've determined that there is enough RFC 1918 address space available to allocate for the Node block, but not enough space for the Pod CIDR block. Therefore, you must translate the Pod (and possibly the Service) CIDR blocks.
Description:
To translate the Pod CIDR blocks in a GKE cluster, you implement the
ip-masquerade-agentfeature as shown in figure 2. This feature hides the Pod IP addresses behind the Node IP addresses. With this design, it's also possible to reuse CIDR blocks for the Service CIDR block.
Figure 2. NAT for a Pod CIDR block in a GKE cluster. (For a screen-readable PDF version, click the image.) This architecture has several components:
- New terminology to describe the CIDR block characteristics in relation to NAT.
- The
ip-masquerade-agentfeature. - Reused RFC 1918 CIDR blocks.
- Filtering at the on-premises connection.
Advantages:
- Alleviates IPv4 address allocation roadblocks for the Pod CIDR.
- Scales to largest supportable Pod CIDR block.
- Isolates the reused RFC 1918 CIDR blocks from the global routing table.
- Makes multi-cluster Istio meshes still possible.
- Makes network endpoint group configurations still possible.
Disadvantages:
- NAT introduces new issues—for example, address tethering and logging—that you need to address.
- Not all resources in the VPC can communicate with on-premises resources that are assigned the same CIDR range as the Pod CIDR range. This problem also exists for on-premises resources that share the Service CIDR block if the Service range is reused. The issue is that the reused CIDR blocks appear as routes local to the VPC and are therefore never routed outside the VPC.
Other issues:
- Be careful when you select which RFC 1918 CIDR blocks to reuse for the GKE CIDR blocks.
- Don't advertise the reused RFC 1918 CIDR blocks back to the on-premises network.
- Because reused CIDR blocks are in the VPC routing table, be careful when you use VPC peering or VPN connections.
Translating all the GKE CIDR blocks
When to use:
- After reviewing the available IP address space and walking through figure 1 you have determined that there is not enough RFC 1918 address space available to allocate to either the cluster Pod or Node CIDR blocks. Therefore, you must translate all the CIDR blocks.
Description:
To translate all the CIDR blocks in a GKE cluster, you implement the architecture shown in figure 3.
Figure 3. NAT for all CIDR blocks in a GKE cluster. (For a screen-readable PDF version, click the image.) This architecture has several components:
- New terminology to describe the CIDR block characteristics in relation to NAT.
- Reused RFC 1918 CIDR blocks.
- A host VPC that has a shared network.
- A Service VPC that is called the isolated VPC that contains the GKE cluster.
- A Compute Engine instance called the isolated-VPC gateway that performs NAT and has a network interface in both the isolated VPC and the host VPC.
- The concept of an internal load balancer CIDR block.
- Static routes in both VPCs that direct traffic appropriately.
- Filtering at the on-premises connection.
Advantages:
- Alleviates IPv4 address allocation roadblocks.
- Scales to the largest supportable cluster.
- Allows cookie-cutter replication for multiple clusters.
- Offers the best isolation of the reused RFC 1918 CIDR blocks from the global routing table.
- Makes network-endpoint group configurations still possible.
Disadvantages:
- Introduces more complexity than previous solutions.
- NAT introduces new issues—for example, address tethering and logging—that you need to address.
- Multicluster Istio meshes are not possible in some deployments.
Other issues:
- Take care when you select which RFC 1918 CIDR blocks to reuse for the GKE CIDR blocks.
- Don't advertise the reused RFC 1918 CIDR blocks back to the on-premises network.
What's next
- For a better understanding of the underlying technology in this document, read the following normative references: