Local SSDs

This page provides an overview of local SSD support in Kubernetes, and how to use local SSDs with Kubernetes Engine.

Overview

Local SSDs provide high-performance, ephemeral storage to every node in the cluster. Local SSDs provide higher throughput and lower latency than standard disks. Local caching and processing are suitable workloads for local SSDs.

You can create node pools with local SSDs within your cluster's machine type limits and your project's quotas.

Restrictions

Ensure that you are aware of the following restrictions as you use local SSDs:

  • Because local SSDs are physically attached to the node's host virtual machine instance, any data stored in them only exists on that node. As the data stored on the disks is local, ensure that your application is resilient to having this data being unavailable.
  • Data stored on local SSDs are ephemeral. A Pod that writes to a local SSD might lose access to the data stored on the disk if the Pod is rescheduled away from that node. Additionally, terminating, upgrading, or repairing a node causes the data to be erased.
  • You cannot add local SSDs to an existing node pool.

Creating a cluster with local SSDs

You can create a cluster with local SSDs using Google Cloud Platform Console or the gcloud command-line tool.

Console

You can create a cluster or node pool with local SSDs from the Kubernetes Engine menu in GCP Console.

To create a cluster in which the default pool uses local SSD disks, perform the following steps:

  1. Visit the Kubernetes Engine menu in GCP Console.

    Visit the Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure your cluster as desired. Then, from the Local SSD disks (per node) field, enter the desired number of SSDs as an absolute number.
  4. Click Create.

To create a node pool with local SSD disks in an existing cluster:

  1. Visit the Kubernetes Engine menu in GCP Console.

    Visit the Kubernetes Engine menu

  2. Select the desired cluster.

  3. Click Edit.
  4. From the Node pools menu, click Add node pool.
  5. Configure the node pool as desired. Then, from the Local SSD disks (per node) field, enter the desired number of SSDs as an absolute number.
  6. Click Save.

gcloud

To create a cluster or node pool with local SSDs using gcloud, specify the --local-ssd count flag.

To create a cluster in which the default pool uses local SSD disks, run the following command:

gcloud container clusters create [CLUSTER_NAME] --local-ssd-count [NUMBER_OF_DISKS]

where [NUMBER_OF_DISKS] is the desired number of disks as an absolute number.

To create a node pool with local SSD disks in an existing cluster:

gcloud beta container node-pools create [POOL_NAME] --cluster [CLUSTER_NAME] \
--local-ssd-count [NUMBER_OF_DISKS]

The --local-ssd-count specifies the number of local SSDs to be created per node. The maximum number varies by the machine type and region (see Local SSDs). When nodes are created, the local SSDs are automatically formatted and mounted on the host OS at the subdirectory /mnt/disks/, with each local SSD mounted at a "ssd#" directory.

Using local SSDs

The following section explains how to use local SSDs with Kubernetes Engine.

Local SSDs can be accessed through one of two methods:

  • Hostpath volumes are recommended for:

    • Workloads using DaemonSets.
    • Workloads that use dedicated node pools. All the local SSDs have to be accessed at the same path across all instances of a DaemonSet.
  • Local PersistentVolumes are available in beta beginning with Kubernetes 1.10 and are recommended for:

    • Workloads using StatefulSets and volumeClaimTemplates.
    • Workloads that share node pools. Each local SSD can be reserved through a PersistentVolumeClaim (PVC), and specific host paths are not encoded directly in the Pod spec.
    • Pods that require data gravity to the same local SSD. A Pod is always scheduled to the same node as its local PersistentVolume.

Example using hostPath volumes

For example, if one creates a node pool with three local SSDs, the host OS mounts the disks at /mnt/disks/ssd0, /mnt/disks/ssd1 and /mnt/disks/ssd2. Your Kubernetes containers access the disks using the hostPath parameter in defined in your object's configuration file.

Here is an example Pod configuration file which references a local SSD, /mnt/disks/ssd0:

apiVersion: v1
kind: Pod
metadata:
  name: "test-ssd"
spec:
  containers:
  - name: "shell"
    image: "ubuntu:14.04"
    command: ["/bin/sh", "-c"]
    args: ["echo 'hello world' > /test-ssd/test.txt && sleep 1 && cat /test-ssd/test.txt"]
    volumeMounts:
    - mountPath: "/test-ssd/"
      name: "test-ssd"
  volumes:
  - name: "test-ssd"
    hostPath:
      path: "/mnt/disks/ssd0"
  nodeSelector:
    cloud.google.com/gke-local-ssd: "true"

Example using local PersistentVolumes

Beginning with Kubernetes version 1.10, local SSDs can be specified as PersistentVolumes.

Currently, you can create PersistentVolumes from local SSDs by manually creating a PersistentVolume, or by running the local volume static provisioner.

Manually creating the PersistentVolume

You can manually create a PersistentVolume for each local SSD on each node in your cluster.

To reference a local SSD on a specific node, you must specify the nodeAffinity field in a PersistentVolume object. For example, the following is a PersistentVolume specification for a local SSD mounted at /mnt/disks/ssd0 on node gke-test-cluster-default-pool-926ddf80-f166:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: "example-local-pv"
spec:
  capacity:
    storage: 375Gi
  accessModes:
  - "ReadWriteOnce"
  persistentVolumeReclaimPolicy: "Retain"
  storageClassName: "local-storage"
  local:
    path: "/mnt/disks/ssd0"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: "kubernetes.io/hostname"
          operator: "In"
          values: "gke-test-cluster-default-pool-926ddf80-f166"

Running the local volume static provisioner

You can create PersistentVolumes for local SSDs automatically using the local volume static provisioner. The provisioner is a DaemonSet which manages the local SSDs on each node, creates and deletes the PersistentVolumes for them, and cleans up the data on the local SSD when the PersistentVolume is released.

To run the local volume static provisioner:

  1. Download the provisioner_generated_gce_ssd_count.yaml specification from the external-storage repo, and modify the specification's namespace fields as needed.

    The specification includes:

    • ServiceAccount for the provisioner
    • ClusterRole and ClusterRoleBindings for permissions to:
      • Create and Delete PersistentVolume objects
      • Get Node objects
    • ConfigMap with provisioner settings for Kubernetes Engine
    • DaemonSet for running the provisioner
  2. Link your IAM account to the Kubernetes cluster-admin role. This is required to get ClusterRole creation permissions.

  3. Deploy the provisioner:

    kubectl apply -f provisioner_generated_gce_ssd_count.yaml
    

Once the provisioner is running successfully, it will create a PersistentVolume object for each local SSD in the cluster.

Enabling delayed volume binding

For improved scheduling, it is recommended to also create a StorageClass with volumeBindingMode: WaitForFirstConsumer. This will delay PersistentVolumeClaim (PVC) binding until Pod scheduling, so that a local SSD is chosen from an appropriate node that can run the Pod. This enhanced scheduling behavior means that Pod CPU and memory requests, node affinity, Pod affinity and anti-affinity, and multiple PVC requests are considered along with which nodes have available local SSDs.

local_class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: "local-scsi"
provisioner: "kubernetes.io/no-provisioner"
volumeBindingMode: "WaitForFirstConsumer"

To create a StorageClass with delayed binding, run the following command:

kubectl apply -f local_class.yaml

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Kubernetes Engine