This page shows you how to implement persistent IP addresses on Google Kubernetes Engine (GKE) Pods. You can take control of your GKE Pod networking with custom persistent IP address mapping. To learn more about persistent IP addresses, their use cases, and benefits, see About persistent IP addresses for GKE Pod.
Requirements
- GKE version 1.29 or later.
- Choose between reserving Google-provided IP addresses (Google-provided IP addresses) or bringing your own IP addresses (BYOIP).
- Configure your application running within the Pods to recognize and use assigned persistent IP addresses.
- Persistent IP addresses for GKE Pods require GKE Dataplane V2 and multi-networking enabled clusters.
Limitations
- The default network does not support Persistent IP addresses. You must enable multi-networking during cluster creation.
- Configure your applications to use assigned persistent IP addresses. GKE doesn't automatically add the IP address configuration to the Pod's network interfaces.
- You can associate each persistent IP address with a single Pod at a time.
When you have multiple Pods available, GKE typically sends
traffic to the newest Pod. However, GKE only does this if the
newest Pod is healthy, meaning the Pod has the
Ready
condition status asTrue
by default. You can configure this behavior and change it using thereactionMode
setting onGKEIPRoute
. - GKE only supports IPv4 addresses as persistent IP addresses.
- GKE only supports Layer 3 or Device type multi-networks for persistent IP addresses.
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task,
install and then
initialize the
gcloud CLI. If you previously installed the gcloud CLI, get the latest
version by running
gcloud components update
.
Read about IP addresses.
Review the requirements and limitations.
Pricing
The following Network Function Optimizer (NFO) features are supported only on clusters that are in Projects enabled with GKE Enterprise:
- Multi-network support for Pods
- Persistent IP addresses support for Pods (Preview)
- Multi-network network policies (Preview)
- Service Steering support for Pods (Preview)
To understand the charges that apply for enabling Google Kubernetes Engine (GKE) Enterprise edition, see GKE Enterprise Pricing.
Implement persistent IP addresses for GKE Pods
Persistent IP addresses in GKE offer a way to give your Pods a stable network identity, even if the Pods themselves are updated or moved around.
This section summarizes the workflow to implement persistent IP addresses for GKE Pods:
- Create a cluster: Create a cluster that has the Gateway API, GKE Dataplane V2, and multi-networking enabled.
- Reserve an IP address: Decide whether you need an external (publicly accessible) or internal (Google Cloud only) IP address and reserve it. Choose the same region as your GKE cluster.
- Create a Gateway: Configure the Kubernetes Gateway object that holds your
reserved persistent IP addresses and lets you create rules (
GKEIPRoutes
) about which Pods in your cluster can use those persistent IP addresses. - Create or identify workloads with additional networks for persistent IP addresses: Prepare Pods to use persistent IP addresses by enabling multiple network interfaces and defining the network where the persistent IP addresses reside.
- Create the GKEIPRoute object for selected workloads: Configure
GKEIPRoute
to assign the persistent IP address to a specific Pod. You can use labels to target the right Pod(s) and can optionally configure how the routing reacts to Pod changes. - Configure your application awareness: Configure your application within the Pod to actively use the persistent IP address.
- Monitoring: Keep track of the status of your Gateway and
GKEIPRoute
objects to ensure everything is working as expected.
To implement persistent IP address for GKE Pods, perform the following steps:
Step 1: Create a GKE Dataplane V2 and multi networking enabled GKE cluster
To enable the advanced network routing and IP address management capabilities necessary for implementing persistent IP addresses on GKE Pods, you must create a GKE Dataplane V2 and multi-networking enabled cluster.
Step 2: Reserve a persistent IP addresses
To establish reliable network identities for your Pods and set up persistent IP addresses, you must first acquire persistent IP addresses. You can choose between reserving Google-provided IP addresses or using your own (BYOIP).
Step 2a: Reserve Google-provided IP addresses
To reserve external IP addresses, run the following command:
gcloud compute addresses create ADDRESS_NAME \
--region=REGION
Replace the following:
ADDRESS_NAME
: the name that you want to associate with this address.REGION
: the region where you want to reserve this address. This region should be the same region as the Pod that you want to attach the IP address to.Note: You must specify a region when reserving an IP address because forwarding rules, which handle traffic routing for persistent IP addresses, are regional. Your IP address and GKE cluster must be in the same region for routing to work correctly.
To reserve internal IP addresses, run the following command:
gcloud compute addresses create ADDRESS_NAME \
--region REGION
--subnet SUBNETWORK \
--addresses IP_ADDRESS
Replace the following:
ADDRESS_NAME
: the names of one or more addresses that you want to reserve. In case of multiple addresses, specify all the addresses as a list, separated by spaces. For example, example-address-1 example-address-2 example-address-3REGION
: the region for this request.SUBNETWORK
: the subnet for this internal IPv4 address.
To ensure traffic is routed correctly within your private network, internal IP addresses must belong to a designated subnet.
For more information on external and internal IP addresses, see Reserve a static external IP address and Reserve a static internal IP address.
Step 2b: Bring your own IP addresses (BYOIP)
You can bring your own IP addresses (BYOIP), instead of relying on Google-provided IP addresses. BYOIP is helpful if you need specific IP addresses for your applications or are moving existing systems into Google Cloud. To use BYOIP, Google validates that you own the IP address range, and after the IP addresses are imported to Google Cloud, you can assign them as the persistent IP addresses for GKE Pods. For more information, see Use bring your own IP addresses.
Step 3: Create Gateway objects
Gateway objects hold the IP addresses and define which Pods are eligible to use them. To control how persistent IP addresses are assigned to your GKE Pods, you'll use Gateway objects.
- Create a Kubernetes Gateway object of the appropriate class:
gke-persistent-regional-external-managed
for external (public) IP addresses.gke-persistent-regional-internal-managed
for internal (Google Cloud-only) IP addresses.
- Within the Gateway's addresses section, list the persistent IP addresses (Google-provided or BYOIP) that this Gateway manages.
Use the
Listeners
section to determine which Pods (and their associatedGKEIPRoute
objects) can potentially use the Gateway's IP addresses.Listeners
act as a filter based on the GKEIPRoute's namespace where aGKEIPRoute
object exists.You can choose from the following Kubernetes namespace selection options:
- All Namespaces: Any
GKEIPRoute
in the cluster. - Selector:
GKEIPRoute
in GKEIPRoute's namespaces matching a specific label. - Same Namespace: Only
GKEIPRoutes
within the same GKEIPRoute's namespace as the Gateway.
- All Namespaces: Any
The following example provides cluster-wide access to external persistent IP addresses, allowing any Pod to potentially use them.
Save the following sample manifest as allowed-pod-ips.yaml
:
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
namespace: default
name: allowed-pod-ips
spec:
gatewayClassName: gke-persistent-regional-external-managed
listeners:
- name: default
port: 443
protocol: none
allowedRoutes:
namespaces:
from: All
addresses:
- value: "34.123.10.1/32"
type: "gke.networking.io/cidr"
- value: "34.123.10.2/32"
type: "gke.networking.io/cidr"
where:
- addresses: lists all the IP addresses whose permissions are managed by the specific Gateway.
- listeners: used to identify the namespaces from which
GKEIPRoute
objects can reference this Gateway.
Apply the manifest to the cluster:
kubectl apply -f allowed-pod-ips.yaml
Step 4: Create or identify workloads with additional networks for persistent IP addresses
Set up multi-network Pods and create network objects denoting networks that the persistent IP address belongs to.
Step 5: Create the GKEIPRoute
object for selected workloads
To assign a persistent IP address to a selected Pod, create a GKEIPRoute
object.
Save the following sample manifest as my-ip-route.yaml
:
kind: GKEIPRoute
apiVersion: networking.gke.io/v1
metadata:
namespace: default
name: my-ip-route
spec:
parentRefs:
- name: allowed-pod-ips
namespace: default
addresses:
- value: "34.123.10.1/32"
type: "gke.networking.io/cidr"
network: blue-network
reactionMode: ReadyCondition
podSelector: # Only one pod is selected.
matchLabels:
component: proxy
where:
- parentRefs: points to the Gateway from which persistent IP addresses are used. This field is immutable.
- addresses: lists all the persistent IP addresses that are routed to the
Pod identified with
podSelector
. This field is mutable. For IPv4, only /32 addresses are supported. - podSelector: specifies labels that identify the Pod where the persistent
IP addresses are routed to. This field is mutable and applies to the same
namespace in which
GKEIPRoute
is placed. If you select multiple Pods, two additional factors are taken under consideration: the Pod creation time (GKE picks the newest) and setting ofreactionMode
field. - reactionMode: specifies how this feature behaves when a specific Pod
(selected by the
podSelector
) gets created or deleted. This field is optional and defaults toReadyCondition
. TheReadyCondition
field is immutable. You can set thereactionMode
to control how the feature acts when Pods are created, deleted, or updated. - network: points to the network interface on a Pod to which persistent IP addresses are routed. This field is immutable.
Apply the manifest to the cluster:
kubectl apply -f my-ip-route.yaml
Assign persistent IP addresses to StatefulSet Pods
To assign a persistent IP address to a specific Pod within a StatefulSet, use the Pod's predictable hostname and Kubernetes' automatic labeling, as shown in the following example:
Save the following sample manifest as my-pod-ips.yaml
:
kind: GKEIPRoute
apiVersion: networking.gke.io/v1
metadata:
namespace: proxy-ss-ns
name: my-pod-ips
spec:
parentRefs:
- name: allowed-pod-ips
namespace: default
addresses:
- value: "34.123.10.1/32"
type: "gke.networking.io/cidr"
- value: "34.123.10.2/32"
type: "gke.networking.io/cidr"
network: blue-network
reactionMode: ReadyCondition
podSelector:
matchLabels:
statefulset.kubernetes.io/pod-name: proxy-ss-1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: proxy-ss-ns
name: proxy-ss
spec:
selector:
matchLabels:
component: proxy
serviceName: "proxy"
replicas: 3
template:
metadata:
annotations:
networking.gke.io/default-interface: 'eth0'
networking.gke.io/interfaces: '[{"interfaceName":"eth0","network":"default"}, {"interfaceName":"eth1","network":"blue-network"}]'
labels:
component: proxy
spec:
containers:
- name: hello-app
image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
Apply the manifest to the cluster (ensure you have a Network named "blue-network"):
kubectl apply -f my-pod-ips.yaml
Assign persistent IP addresses to Deployment Pods
To assign a persistent IP address to the newest Pod in a Deployment, apply a
GKEIPRoute
with the following configuration:
Save the following sample manifest as my-pod-ips.yaml
:
kind: GKEIPRoute
apiVersion: networking.gke.io/v1
metadata:
namespace: proxy-deploy-ns
name: my-pod-ips
spec:
parentRefs:
- name: allowed-pod-ips
namespace: default
addresses:
- value: "34.123.10.1/32"
type: "gke.networking.io/cidr"
- value: "34.123.10.2/32"
type: "gke.networking.io/cidr"
network: blue-network
reactionMode: ReadyCondition
podSelector:
matchLabels:
component: proxy
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: proxy-deploy-ns
name: proxy-deploy
spec:
selector:
matchLabels:
component: proxy
replicas: 4 # Latest Pod is used
template:
metadata:
annotations:
networking.gke.io/default-interface: 'eth0'
networking.gke.io/interfaces: '[{"interfaceName":"eth0","network":"default"}, {"interfaceName":"eth1","network":"blue-network"}]'
labels:
component: proxy
spec:
containers:
- name: hello-app
image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
Apply the manifest to the cluster (ensure you have a Network named "blue-network"):
kubectl apply -f my-ip-route.yaml
Step 6: Use persistent IP addresses inside the Pod
Assigning a persistent IP address to a GKE Pod using a GKEIPRoute
doesn't automatically make the IP addresses usable by your application. The
persistent IP addresses are handled at a network routing level, but your Pod's
default configuration won't be aware of it. You must configure your application's
configuration to recognize and use the address within the Pod. To achieve this,
your Pod will require privilege permissions.
To configure your application, consider the following options:
- net.ipv4.ip_nonlocal_bind: Modify a system setting to allow your application to use IP addresses not directly assigned to its interface.
- ip address add: Use this command within your application's logic to manually add the persistent IP address to an interface.
- Raw sockets: For even more control, your application could directly interact with the network stack (advanced).
- Userspace IP address stack: In specialized cases, a separate application might run within the Pod to manage the IP address (very advanced).
Customize persistent IP address behavior during Pod changes
This section describes how the persistent IP address for GKE Pod
behaves when the targeted Pod is created or deleted. The GKE
controller monitors your Pods and the GKEIPRoute
configuration. When it detects
an update is happening, it automatically reassigns the persistent IP address to
a suitable Pod according to your chosen reactionMode
.
Understand how the persistent IP address feature automatically handles Pod changes and the configuration options available to you:
- Decide between
ReadyCondition
or
Exists
within the
reactionMode
field of your
GKEIPRoute
configuration. Consider your application's needs regarding how quickly the IP address is assigned versus strict readiness requirements. - If you are using
ReadyCondition
to ensure readiness, make sure that your Pods have correctly implemented Kubernetes readiness probes. Otherwise, the persistent IP address might not function as intended. - We recommend that you monitor the status of your Pods and the
Conditions
field of theGKEIPRoute
object to ensure the system is working correctly. Thetrue
status of theReady
condition indicates that the system is working correctly.
Troubleshoot communication with persistent IP addresses for Pods
This section shows you how to resolve issues related to persistent IP addresses for Pods.
NoPodsFound
when there are no matching Pods found
Symptom
The GKEIPRoute
object specifies a podSelector
(a set of labels) to identify
which Pods are associated with the persistent IP address. The
NoPodsFound
status indicates that there are no Pods within the targeted
GKEIPRoute's
namespaces that have the matching labels.
Potential causes
- Incorrect labels: The Pod you intend to use the persistent IP address with might have the wrong labels, or no labels at all.
- No Pods exist: If the
reactionMode == Exists
, check if the Pod is assigned to a node by checking thepod.Spec.nodeName
field. There might not be any Pods running in theGKEIPRoute's
namespace that match the selector. - Pod not Ready: If
reactionMode == ReadyCondition
, check if the Pod status isREADY
. Even if a matching Pod exists, if it's not in aReady
state, it can't serve traffic and so it isn't selected.
Resolution
- Check Your Labels: Double-check that the labels in your
GKEIPRoute's
podSelector
match the labels you've applied to the intended Pod. - Verify Pod Existence: Make sure a Pod with the correct labels actually
exists in the
GKEIPRoute's
namespaces specified by your Gateway'sListeners
. If thereactionMode == Exists
, check if the Pod is assigned to a node by checking thepod.Spec.nodeName
field Confirm Pod Readiness: If
reactionMode == ReadyCondition
, check if the Pod status isREADY
. Ensure the Pod is in theReady
state using the following command:kubectl get pods -n <namespace>
Pods in other states (for example, "Pending", "Error") aren't selected.
Configure your Pods to respond on the assigned persistent IP address.
Mutated
when a matching Pod was found and persistent IP address programming is in progress
Symptom
The GKEIPRoute
status shows "Mutated", indicating that the persistent IP address
configuration for a matching Pod is in progress.
Potential Cause:
You can expect the "Mutated" status during configuration as the system is setting up the GKE datapath and Google Cloud resources for the persistent IP address.
Resolution:
- Wait and retry: In most cases, the configuration process completes
automatically within a short time. Check the status after waiting. It
will change to
Ready
when successful. - Investigate further (if necessary): If the "Mutated" status persists for
an extended period, this might indicate a configuration error. Examine the
other status conditions on your
GKEIPRoute
:- Accepted: Indicates if your
GKEIPRoute
setup is valid. - DPV2Ready: Indicates if the datapath on the node is programmed correctly.
- GCPReady: Indicates if the Google Cloud resources are set up as expected.
- Accepted: Indicates if your
Look for error messages within these conditions to help troubleshoot the issue.