Isolate your workloads in dedicated node pools


This page shows you how to reduce the risk of privilege escalation attacks in your cluster by telling Google Kubernetes Engine (GKE) to schedule your workloads on a separate, dedicated node pool away from privileged GKE-managed workloads.

Overview

GKE clusters use privileged GKE-managed workloads to enable specific cluster functionality and features, such as metrics gathering. These workloads are given special permissions to run correctly in the cluster.

Workloads that you deploy to your nodes might have the potential to be compromised by a malicious entity. Running these workloads alongside privileged GKE-managed workloads means that an attacker who breaks out of a compromised container can use the credentials of the privileged workload on the node to escalate privileges in your cluster.

Preventing container breakouts

Your primary defense should be your applications. GKE has multiple features that you can use to harden your clusters and Pods. In most cases, we strongly recommend using GKE Sandbox to isolate your workloads. GKE Sandbox is based on the gVisor open source project, and implements the Linux kernel API in the userspace. Each Pod runs on a dedicated kernel that sandboxes applications to prevent access to privileged system calls in the host kernel. Workloads running in GKE Sandbox are automatically scheduled on separate nodes, isolated from other workloads.

You should also follow the recommendations in Harden your cluster's security.

Avoiding privilege escalation attacks

If you can't use GKE Sandbox, and you want an extra layer of isolation in addition to other hardening measures, you can use node taints and node affinity to schedule your workloads on a dedicated node pool. A node taint tells GKE to avoid scheduling workloads without a corresponding toleration (such as GKE-managed workloads) on those nodes. The node affinity on your own workloads tells GKE to schedule your Pods on the dedicated nodes.

Limitations of node isolation

  • Attackers can still initiate Denial-of-Service (DoS) attacks from the compromised node.
  • Compromised nodes can still read many resources, including all Pods and namespaces in the cluster.
  • Compromised nodes can access Secrets and credentials used by every Pod running on that node.
  • Using a separate node pool to isolate your workloads can impact your cost efficiency, autoscaling, and resource utilization.
  • Compromised nodes can still bypass egress network policies.
  • Some GKE-managed workloads must run on every node in your cluster, and are configured to tolerate all taints.
  • If you deploy DaemonSets that have elevated permissions and can tolerate any taint, those Pods may be a pathway for privilege escalation from a compromised node.

How node isolation works

To implement node isolation for your workloads, you must do the following:

  1. Taint and label a node pool for your workloads.
  2. Update your workloads with the corresponding toleration and node affinity rule.

This guide assumes that you start with one node pool in your cluster. Using node affinity in addition to node taints isn't mandatory, but we recommend it because you benefit from greater control over scheduling.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Google Cloud CLI.
  • Set up default Google Cloud CLI settings for your project by using one of the following methods:
    • Use gcloud init, if you want to be walked through setting project defaults.
    • Use gcloud config, to individually set your project ID, zone, and region.

    gcloud init

    1. Run gcloud init and follow the directions:

      gcloud init

      If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

      gcloud init --console-only
    2. Follow the instructions to authorize the gcloud CLI to use your Google Cloud account.
    3. Create a new configuration or select an existing one.
    4. Choose a Google Cloud project.
    5. Choose a default Compute Engine zone.
    6. Choose a default Compute Engine region.

    gcloud config

    1. Set your default project ID:
      gcloud config set project PROJECT_ID
    2. Set your default Compute Engine region (for example, us-central1):
      gcloud config set compute/region COMPUTE_REGION
    3. Set your default Compute Engine zone (for example, us-central1-c):
      gcloud config set compute/zone COMPUTE_ZONE
    4. Update gcloud to the latest version:
      gcloud components update

    By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

  • Choose a specific name for the node taint and the node label that you want to use for the dedicated node pools. For this example, we use workloadType=untrusted.

Taint and label a node pool for your workloads

Create a new node pool for your workloads and apply a node taint and a node label. When you apply a taint or a label at the node pool level, any new nodes, such as those created by autoscaling, will automatically get the specified taints and labels.

You can also add node taints and node labels to existing node pools. If you use the NoExecute effect, GKE evicts any Pods running on those nodes that don't have a toleration for the new taint.

To add a taint and a label to a new node pool, run the following command:

gcloud container node-pools create POOL_NAME \
    --cluster CLUSTER_NAME \
    --node-taints KEY=VALUE:EFFECT \
    --node-labels NODE_LABELS

Replace the following:

  • POOL_NAME: the name of the new node pool for your workloads.
  • CLUSTER_NAME: the name of your GKE cluster.
  • KEY=VALUE: a key value pair associated with a scheduling EFFECT. For example, workloadType=untrusted.
  • EFFECT: one of the following effect values: NoSchedule, PreferNoSchedule, or NoExecute. NoExecute provides a better eviction guarantee than NoSchedule.

Add a toleration and a node affinity rule to your workloads

After you taint the dedicated node pool, no workloads can schedule on it unless they have a toleration corresponding to the taint you added. Add the toleration to the specification for your workloads to let those Pods schedule on your tainted node pool.

If you labelled the dedicated node pool, you can also add a node affinity rule to tell GKE to only schedule your workloads on that node pool.

The following example adds a toleration for the workloadType=untrusted:NoExecute taint and a node affinity rule for the workloadType=untrusted node label.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: my-app
  namespace: default
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      tolerations:
      - key: workloadType
        operator: Equal
        value: untrusted
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: workloadType
                operator: In
                values:
                - "untrusted" 
      containers:
      - name: sleep
        image: ubuntu
        command: ["/bin/sleep", "inf"]

When you update your Deployment with kubectl apply, GKE recreates the affected Pods. The node affinity rule forces the Pods onto the dedicated node pool that you created. The toleration allows only those Pods to be placed on the nodes.

Verify that the separation works

To verify that the scheduling works correctly, run the following command and check whether your workloads are on the dedicated node pool:

kubectl get pods -o=wide

Recommendations and best practices

After setting up node isolation, we recommend that you do the following:

  • Restrict specific node pools to GKE-managed workloads only by adding the components.gke.io/gke-managed-components taint. Adding this taint prevents your own Pods from scheduling on those nodes, improving the isolation.
  • When creating new node pools, prevent most GKE-managed workloads from running on those nodes by adding your own taint to those node pools.
  • Whenever you deploy new workloads to your cluster, such as when installing third-party tooling, audit the permissions that the Pods require. When possible, avoid deploying workloads that use elevated permissions to shared nodes.

What's next