This page describes how to configure bundled load balancing for Anthos clusters on bare metal. Anthos clusters on bare metal deploys Layer 4 load balancers that run on either a dedicated pool of worker nodes or on the same nodes as the control plane.
See Overview of load balancers for examples of load-balancing topologies available in Anthos clusters on bare metal.
- All load balancer nodes must be in the same Layer 2 subnet.
- All VIPs must be in the load balancer nodes subnet and be routable through the subnet's gateway.
- The gateway of the load balancer subnet must listen to gratuitous ARP messages and forward ARP packets to the load balancer nodes.
cluster.spec.loadBalancer section of the cluster configuration file
to configure bundled load balancing. For information about cluster
configuration files and examples of valid configurations. see one of the
This value must be
bundled to enable bundled load balancing.
This value specifies the destination port to be used for traffic sent to the Kubernetes control plane (the Kubernetes API servers).
This value specifies the destination IP address to be used for traffic sent to
the Kubernetes control plane (the Kubernetes API servers). This IP address must
be in the same Layer 2 subnet as the nodes running load balancers. Do not list
this address in the
address pools section of the
This value specifies the IP address to be used for Services behind the load balancer for ingress traffic. This field is not allowed in admin cluster configuration files. This address must be listed in the address pools section of the configuration.
This section of the configuration contains one or more address pools specified in this format:
- name: pool-name avoidBuggyIPs: boolean manualAssign: boolean addresses: - ip-range - ip-range
- name: The name of the address pool, pool-name, for your own organizational purposes.
- avoidBuggyIPs: (Optional)
true, the pool omits IP addresses ending in
.255. Some network hardware drops traffic to these special addresses. You can omit this field, its default value is
- manualAssign: (Optional).
true, addresses in this pool are not automatically assigned to Kubernetes Services. If
true, an IP address in this pool is used only when it is specified explicitly by a service. You can omit this field, its default value is false.
- addresses A list of one or more nonoverlapping IP address ranges.
ip-range can be specified in either CIDR notation (like
198.51.100.0/24) or range notation (like
198.51.100.0-198.51.100.10, with no spaces around the dash).
The IP address ranges in the
addresses list must not overlap and must be in
the same subnet as the nodes running load balancers.
This section of the configuration specifies a list of nodes to run load
balancers on. Load balancer nodes can run regular workloads by default; there is
no special taint on those nodes. The example below shows a node pool with two
nodes. The first node,
18.104.22.168, uses the
k8sIP field to specify the node's
IP address in the cluster. The
22.214.171.124 address is only used for SSH access.
nodePoolSpec: nodes: - address: 126.96.36.199 k8sIP: 10.0.0.32 - address: 10.0.0.33
All nodes in the load balancer node pool must be in the same Layer 2 subnet as the
load balancer VIPs configured in the
loadBalancer.addressPools section of the configuration file.
If a node has a
k8sIP configured, only that address needs to be in the same Layer 2
subnet as the other load balancer VIPs.
nodePoolSpec is not set, the bundled load balancers run on the control
plane nodes. We recommend you run load balancers on separate node pools if
Control plane load balancing
The control plane load balancer serves the control plane virtual IP address (VIP). Anthos clusters on bare metal runs Keepalived and HAProxy as Kubernetes static pods on the load-balancer nodes to announce the control plane VIP. Keepalived uses the Virtual Router Redundancy Protocol (VRRP) on the load balancer nodes for high availability.
Data plane load balancing
The data plane load balancer is for all Kubernetes Services of type
Anthos clusters on bare metal uses MetalLB
running in Layer 2 mode for data plane load balancing. Data plane load balancing can
only be configured through Anthos clusters on bare metal, do not modify MetalLB's ConfigMap
directly. You can use all MetalLB features including
IP address sharing across Services.
See the MetalLB documentation for feature information.
MetalLB runs a speaker Pod on each node using a daemonset, using memberlist for high availability. There is a MetalLB dedicated load balancer node for each Kubernetes Service, rather than one for the entire cluster. This way traffic is distributed across load balancer nodes if there are multiple Services.
The data plane load balancers can run on either the control plane nodes or on a subset of worker nodes. Bundling data plane load balancers on the control plane nodes increases utilization of the control plane nodes. However, bundling on the control plane nodes also increases the risk of overloading the control plane and increases the risk profile of confidential information on the control plane, such as SSH keys.
Preserving client source IP address
LoadBalancer Service created with the bundled Layer 2 load balancing
solution uses the default
Cluster setting for the external traffic policy.
spec.externalTrafficPolicy: Cluster, routes external traffic to
cluster-wide endpoints, but it also obscures the client source IP address.
The following sections describe how to configure your cluster to preserve the client source IP address.
Kubernetes does source network address translation (NAT) for
Services. To retain the client source IP addresses, set
Local. Kubernetes won't perform source
NAT anymore, but you must make sure there are pods running exactly on the node
IP you picked.
externalTrafficPolicy: Local in your
LoadBalancer Services, set
your application pods to run exactly on the load balancer nodes. Add the
nodeSelector to your application pods to make this change:
apiVersion: v1 kind: Pod ... spec: nodeSelector: baremetal.cluster.gke.io/lbnode: "true" ...
If your applications are HTTP services, you can achieve client IP visibility by configuring ingress components:
istio-ingressService for editing:
kubectl edit service -n gke-system istio-ingress
externalTrafficPolicy: Localto the
spec, save and exit the editor.
apiVersion: v1 kind: Service ... spec: ... externalTrafficPolicy: Local
istio-ingressDeployment for editing:
kubectl edit deployment -n gke-system istio-ingress
Add the following
nodeSelectorto the Deployment, save and exit the editor.
apiVersion: apps/v1 kind: Deployment ... spec: ... template: ... spec: ... nodeSelector: baremetal.cluster.gke.io/lbnode: "true" ...
Now, all of your services behind Ingress see a
X-Forwarded-For header with the
client IP, like the following example: