GKE On-Prem uses Kubernetes networking concepts like Service and Ingress. This document describes how GKE On-Prem networking is configured out of the box.
Cluster services operations and Island Mode
GKE On-Prem uses an Island Mode configuration in which Pods can directly talk to each other within a cluster, but cannot be reached from outside the cluster. This configuration forms an "island" within the network that is not connected to the external network. Clusters use BGP (via the Calico CNI plugin) to form a full node-to-node mesh across the cluster nodes, allowing Pod to reach other Pods within the cluster directly.
All egress traffic from the Pod to targets outside the cluster is NAT'd by the node IP. GKE On-Prem includes an L7 load balancer with an Envoy-based ingress controller that handles Ingress object rules for ClusterIP Services deployed within the cluster. The ingress controller itself is exposed as a NodePort Service in the cluster.
The ingress NodePort service can be reached through a L3/L4 F5 load balancer. The installation configures a virtual IP address (VIP) (with port 80 and 443) on the load balancer. The VIP points to the ports in the NodePort Service for the Ingress controller. This is how external clients can access services in the cluster.
User clusters can run Services of type LoadBalancer as long as a
loadBalancerIP field is configured in the Service's specification. In the
loadBalancerIP field, you need to provide the VIP that you want to use.
This will be configured on F5, pointing to the NodePorts of the Service.
As an alternative to using the F5 load balancer, you can
enable manual load balancing mode.
If you choose to use manual load balancing , you cannot run Services of
LoadBalancer. Instead, you can create Services of type
manually configure your load balancer to use them as backends. Also, you can
expose Services to outside clients by using an Ingress object.
Figure: GKE On-Prem networking.
- Node IP addresses
- DHCP or statically-assigned IP addresses for the nodes (alternatively called virtual machines or VMs). Must be routable within the data center. You can manually assign static IPs.
- Pod CIDR block
- Non-routable CIDR block for all Pods in the cluster. From this range, smaller /24 ranges are assigned per node. If you need an N node cluster, ensure this block is large enough to support N /24 blocks.
- Services CIDR block
- In Island Mode, similar to Pod CIDR block, so only used within the cluster. Any private CIDR block which does not overlap with the nodes, VIPs, or Pod CIDR block. You can share the same block among clusters. The size of the block determines the number of services. One Service IP is needed for the ingress service, and ten or more IPs for Kubernetes services like cluster DNS, etc.
- Services VIPs
- N number of routable IP addresses to be configured on F5 for L4 ingress when
you expose a Service. These VIPs are the same as the
loadBalancerIPvalues you set when you create Services of type LoadBalancer.
- Control plane VIP
- A routable IP address to configure on F5 load balancer for the Kubernetes API server.
- Ingress VIP
- A routable IP address to configure on the F5 load balancer for L7 ingress in conjunction with the Envoy proxies running on each node.
Networking configuration options
You have several options for configuring your clusters' networking:
- You can choose one global vCenter network, or use a different vCenter network for admin and user clusters.
- You can use a HTTP proxy, and you can choose which addresses you want to exclude from proxying.
- You can choose DHCP or static IP allocation.
- You can choose integrated or manual load balancing mode.
During installation, you specify your preferences in the GKE On-Prem configuration file.
vCenter network configurations
network field determines which vCenter network to use for the clusters:
- The global
vcenter.networkfield specifies one specific network.
admincluster.vcenter.networkoverrides the global field, and specifies the network to use for the admin cluster.
usercluster.vcenter.networkoverrides the global field, and specifies the network to use for the user cluster.
Example: Access a web application via URL
Suppose that you have a guestbook web application running in your cluster
as a Deployment named
frontend. You want to connect to the application using a
www.guestbook.com. You need some way of mapping the URL to the Deployment
running in your cluster. You can do this using a Kubernetes Ingress object.
To begin, you first create a wildcard DNS entry for
points to the cluster's existing ingress VIP:
*.guestbook.com A [INGRESS_VIP]
Next, you need to create a Service for the frontend Deployment. Running
expose creates a Service that logically groups the Deployment's Pods and
provides them with a common IP address within the cluster:
kubectl expose deployment frontend
This creates a Service of type ClusterIP, like this:
apiVersion: v1 kind: Service metadata: labels: app: guestbook name: frontend spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: app: guestbook type: ClusterIP
You need to map the URL,
www.guestbook.com, to the frontend Service you
just created. Applying the Ingress below creates that mapping:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: frontend labels: app: guestbook spec: rules: - host: www.guestbook.com http: paths: - backend: serviceName: frontend # name of the frontend Service servicePort: 80
Now, visiting www.guestbook.com opens the web application in your browser.
Here's how this works under the hood:
- Because you created that wildcard DNS entry, when you visit the URL, you're accessing the cluster's ingress VIP.
- The cluster looks for the correct Ingress object based on the hostname, which in this case is www.guestbook.com.
- The traffic is port forwarded to a frontend Pod.
Example: Access a web application via IP address
If your application is not a web application, or if you have networking constraints, you might prefer to create a VIP specifically for your service. You can do this using a Kubernetes Service of type LoadBalancer.
The Service below creates a VIP specifically for the
apiVersion: v1 kind: Service metadata: labels: app: guestbook name: frontend spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: app: guestbook type: LoadBalancer loadBalancerIP: [IP_ADDRESS]
After you apply this Service, you'd see the VIP in your F5 console, and in the console's Pools menu you'd see the IP addresses of the nodes. Visiting the IP address would load the application.