This page shows you how to tell Google Kubernetes Engine (GKE) to schedule your Pods together, separately, or in specific locations.
Workload separation lets you use taints and tolerations to tell GKE to separate Pods onto different nodes, place Pods on nodes that meet specific criteria, or to schedule specific workloads together. What you need to do to configure workload separation depends on your GKE cluster configuration. The following table describes the differences:
Workload separation configuration | |
---|---|
|
Add a toleration for a specific key:value pair to your Pod specification, and select that key:value pair using a nodeSelector. GKE creates nodes, applies the corresponding node taint, and schedules the Pod on the node. For instructions, refer to Separate workloads in Autopilot clusters on this page. |
Standard without node auto-provisioning |
For instructions, refer to Isolate your workloads in dedicated node pools. |
This guide uses an example scenario in which you have two workloads, a batch job and a web server, that you want to separate from each other.
When to use workload separation in GKE
Workload separation is useful when you have workloads that perform different roles and shouldn't run on the same underlying machines. Some example scenarios include the following:
- You have a batch coordinator workload that creates Jobs that you want to keep separate.
- You run a game server with a matchmaking workload that you want to separate from session Pods.
- You want to separate parts of your stack from each other, such as separating a server from a database.
- You want to separate some workloads for compliance or policy reasons.
Pricing
In Autopilot clusters, you're billed for the resources that your Pods request while running. For details, refer to Autopilot pricing. Pods that use workload separation have higher minimum resource requests enforced than regular Pods.
In Standard clusters, you're billed based on the hardware configuration and size of each node, regardless of whether Pods are running on the nodes. For details, refer to Standard pricing.
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI.
Ensure that you have a GKE cluster. To learn how to create a cluster, use one of the following:
Separate workloads in Autopilot clusters
To separate workloads from each other, add a toleration and a node selector to each workload specification that defines the node on which the workload should run. This method also works on Standard clusters that have node auto-provisioning enabled.
Save the following manifest as
web-server.yaml
:apiVersion: apps/v1 kind: Deployment metadata: name: web-server spec: replicas: 6 selector: matchLabels: pod: nginx-pod template: metadata: labels: pod: nginx-pod spec: tolerations: - key: group operator: Equal value: "servers" effect: NoSchedule nodeSelector: group: "servers" containers: - name: web-server image: nginx
This manifest includes the following fields:
spec.tolerations
: GKE can place the Pods on nodes that have thegroup=servers:NoSchedule
taint. GKE can't schedule Pods that don't have this toleration on those nodes.spec.nodeSelector
: GKE must place the Pods on nodes that have thegroup: servers
node label.
GKE adds the corresponding labels and taints to nodes that GKE automatically provisions to run these Pods.
Save the following manifest as
batch-job.yaml
:apiVersion: batch/v1 kind: Job metadata: name: batch-job spec: completions: 5 backoffLimit: 3 ttlSecondsAfterFinished: 120 template: metadata: labels: pod: pi-pod spec: restartPolicy: Never tolerations: - key: group operator: Equal value: "jobs" effect: NoSchedule nodeSelector: group: "jobs" containers: - name: pi image: perl command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
This manifest includes the following fields:
spec.tolerations
: GKE can place the Pods on nodes that have thegroup=jobs:NoSchedule
taint. GKE can't schedule Pods that don't have this toleration on those nodes.spec.nodeSelector
: GKE must place the Pods on nodes that have thegroup: jobs
node label.
GKE adds the corresponding labels and taints to nodes that GKE automatically provisions to run these Pods.
Deploy the workloads:
kubectl apply -f batch-job.yaml web-server.yaml
When you deploy the workloads, GKE does the following for each workload:
- GKE looks for existing nodes that have the corresponding node taint and node label specified in the manifest. If nodes exist and have available resources, GKE schedules the workload on the node.
- If GKE doesn't find an eligible existing node to schedule the workload, GKE creates a new node and applies the corresponding node taint and node label based on the manifest. GKE places the Pod on the new node.
The presence of the NoSchedule
effect in the node taint ensures that workloads
without a toleration don't get placed on the node.
Verify the workload separation
List your Pods to find the names of the nodes:
kubectl get pods --output=wide
The output is similar to the following:
NAME READY ... NODE
batch-job-28j9h 0/1 ... gk3-sandbox-autopilot-nap-1hzelof0-ed737889-2m59
batch-job-78rcn 0/1 ... gk3-sandbox-autopilot-nap-1hzelof0-ed737889-2m59
batch-job-gg4x2 0/1 ... gk3-sandbox-autopilot-nap-1hzelof0-ed737889-2m59
batch-job-qgsxh 0/1 ... gk3-sandbox-autopilot-nap-1hzelof0-ed737889-2m59
batch-job-v4ksf 0/1 ... gk3-sandbox-autopilot-nap-1hzelof0-ed737889-2m59
web-server-6bb8cd79b5-dw4ds 1/1 ... gk3-sandbox-autopilot-nap-1eurxgsq-f2f3c272-n6xm
web-server-6bb8cd79b5-g5ld6 1/1 ... gk3-sandbox-autopilot-nap-1eurxgsq-9f447e18-275z
web-server-6bb8cd79b5-jcdx5 1/1 ... gk3-sandbox-autopilot-nap-1eurxgsq-9f447e18-275z
web-server-6bb8cd79b5-pxdzw 1/1 ... gk3-sandbox-autopilot-nap-1eurxgsq-ccd22fd9-qtfq
web-server-6bb8cd79b5-s66rw 1/1 ... gk3-sandbox-autopilot-nap-1eurxgsq-ccd22fd9-qtfq
web-server-6bb8cd79b5-zq8hh 1/1 ... gk3-sandbox-autopilot-nap-1eurxgsq-f2f3c272-n6xm
This output shows that the batch-job
Pods and the web-server
Pods always run
on different nodes.
Limitations of workload separation with taints and tolerations
You can't use the following key prefixes for workload separation:
- GKE and Kubernetes-specific keys
- *cloud.google.com/
- *kubelet.kubernetes.io/
- *node.kubernetes.io/
You should use your own, unique keys for workload separation.
Separate workloads in Standard clusters without node auto-provisioning
Separating workloads in Standard clusters without node auto-provisioning requires that you manually create node pools with the appropriate node taints and node labels to accommodate your workloads. For instructions, refer to Isolate your workloads in dedicated node pools. Only use this approach if you have specific requirements that require you to manually manage your node pools.
What's next
- Enforce pre-defined security policies using the PodSecurity admission controller
- Run a full-stack workload at scale