Local solid-state drives (SSDs) are fixed-size SSD drives, which can be mounted to a single Compute Engine VM. You can use local SSDs on GKE to get highly performant storage that is not persistent (ephemeral) that is attached to every node in your cluster. Local SSDs also provide higher throughput and lower latency than standard disks.
If you are using GKE with Standard clusters, you can attach local SSDs to nodes when creating node pools. This improves the performance of ephemeral storage for your workloads that use emptyDir volume. You can create node pools with local SSDs within your cluster's machine type limits and your project's quotas.
A local SSD is attached to only one node, and nodes themselves are ephemeral. A workload can be scheduled onto a different node at any point in time.
For more information on the benefits and limitations of local SSDs, such as performance and the allowed number of SSDs per machine type, see Local SSDs in the Compute Engine documentation.
Why choose Local SSDs
Local SSDs are a good choice if your workloads have any of the following requirements:
- You want to run applications that download and process data, such as AI or machine learning, analytics, batch processing, local caching, and in-memory databases.
- Your applications have specialized storage needs and you want raw block access on high-performance ephemeral storage.
- You want to run specialized data applications and want to operate local SSDs like a node level cache for your Pods. You can use this approach to drive better performance for in-memory database applications like Aerospike or Redis.
Usage pattern for local SSDs
To use local SSDs in your clusters, follow these general steps:
- Provision a node pool with local SSD attached: To set up a GKE node pool in your cluster with local SSD attached, call a Google Cloud CLI command. The command then calls a local SSD API provided by GKE to create a GKE node pool with local SSD attached and configured with specific local SSD characteristics, such as the type (NVMe, SCSI), file format (emptyDir, raw block), and data sharing. To learn more about options for provisioning local SSDs, see local SSDs APIs.
- Access data from local SSDs: To use the data from local SSDs, you can use a Kubernetes construct such as emptyDir, hostpath volume, or local persistent volume. To learn more about these options, see Access local SSDs.
Local SSD APIs on GKE
The following table summarizes the recommended APIs that GKE provides for provisioning local SSDs on clusters. Use the gcloud CLI to call these APIs.
To see examples of how to create clusters and node pools with local SSD, see Provision ephemeral storage with local SSDs
| Local SSD API | gcloud CLI command | GKE availability | Local SSD profile |
|---|---|---|---|
| Ephemeral Storage Local SSD API | gcloud container clusters create |
v1.25.3-gke.1800 or later |
Storage technology: NVMe Data shared across pods: No Data lifecycle: Pod Size and need for RAID configuration: Up to 9 TiB. GKE automatically configures RAID under the hood. Format: File System (Kubernetes emptyDir) Kubernetes scheduler integration: Fully integrated by default. Kubernetes scheduler will ensure space on the node before placement and scale nodes if needed. |
| Local NVMe SSD Block API | gcloud container clusters create |
v1.25.3-gke.1800 or later |
Storage technology: NVMe Data shared across pods: Yes, via hostPath or Local PVs. Data lifecycle: Node Size and need for RAID configuration: Up to 9 TiB. You need to manually configure RAID for larger sizes. Format: Raw block Kubernetes scheduler integration: No, by default. You need to ensure capacity on nodes and handle noisy neighbors. If you opt-in to local PV, scheduling is integrated. |
Support for existing local SSD APIs
The following table summarizes these existing local SSD APIs and their recommended substitutes:
| Existing local SSD API | gcloud CLI command | Local SSD profile | Recommended GA version of the local SSD API |
|---|---|---|---|
| Local SSD Count API | gcloud container clusters create |
Storage technology: SCSI Data shared across pods: Yes, via hostPath or Local PVs Data lifecycle: Node Size and need for RAID configuration: 375 GiB. You need to manually configure RAID for larger sizes. Format: File System (ext-4) Kubernetes scheduler integration: No by default. You need to ensure capacity on nodes and handle noisy neighbors. If you opt-in to local PV, scheduling is integrated. |
gcloud container clusters create |
| Ephemeral Storage API (beta) | gcloud beta container clusters create |
Storage technology: NVMe Data shared across pods: No Data lifecycle: Pod Size and need for RAID configuration: Up to 9 TiB. GKE automatically configures RAID under the hood. Format: File System (Kubernetes emptyDir) Kubernetes scheduler integration: Fully integrated by default. Kubernetes scheduler will ensure space on nodes before placement and scale nodes if needed. |
gcloud container clusters create |
| Local SSD Volumes API (alpha) | gcloud alpha container clusters create |
Storage technology: NVMe or SCSI Data shared across pods: No Data lifecycle: Node Size and need for RAID configuration: 375 GiB. You need to manually configure RAID for larger sizes. Format: File system (ext-4) or raw block Kubernetes scheduler integration: No by default, You need to ensure capacity on nodes and handle noisy neighbors. |
gcloud container clusters create |
Local SSD access
You can access local SSDs with one of the following methods.
emptyDir volume
In GKE version v1.25.3-gke.1800 and later, you can use ephemeral storage as
an emptyDir volume
backed by local SSDs, via the --ephemeral-storage-local-ssd option. We recommend
this approach for most cases, including applications that need high-performance
ephemeral scratch space.
GKE lets you configure a node pool to mount node ephemeral storage on local SSD with an NVMe interface.
To learn more, see this example.
hostPath volume
We recommend this approach if you have workloads that use DaemonSets, or workloads that use dedicated node pools. All the local SSDs have to be accessed at the same path across all instances of a DaemonSet.
To learn more, see this example and the open-source Kubernetes Volumes documentation.
Local persistent volume
A local persistent volume represents a local disk that is attached to a single node. Local persistent volumes allow you to share local SSDs across Pods. Because the local disk is a local SSD, the data is still ephemeral.
We recommend this approach if any of the following run on your cluster:
- Workloads using StatefulSets and volumeClaimTemplates.
- Workloads that share node pools. Each local SSD can be reserved through a PersistentVolumeClaim, and specific host paths are not encoded directly in the Pod spec.
- Pods that require data gravity to the same local SSD. A Pod is always scheduled to the same node as its local PersistentVolume.
To learn more, see this example and the open-source Kubernetes Volumes documentation.
Block access
Block devices enable random access to data in fixed-size blocks. Some specialized applications require direct access to a block device because, for example, the file system layer introduces unneeded overhead.
Common scenarios for using block devices include:
- Databases where data is organized directly on the underlying storage.
- Software which itself implements some kind of storage service (software defined storage systems).
In GKE version v1.25.3-gke.1800 and later, you can create clusters and node pools with raw block local NVMe SSDs attached, by using the--local-nvme-ssd-blockoption. You can then customize block storage by formatting a filesystem of choice (such as ZFS or HDFS) and configuring RAID for storage size. This option is suitable if you need additional control to run workloads with specific requirements for block device local SSD access.
You can also use the block access approach with local SSDs if you want a unified data cache to share data across Pods, and the data is tied to a node lifecycle. To do so, install a DaemonSet with a RAID configuration, format a filesystem, and use a hostPath volume or local PV to share data across Pods.
Restrictions
- Pods using emptyDir transparently use local SSD; however, this is true for all Pods on all nodes in that node pool. GKE does not support having, in the same node pool, some Pods that use local SSD emptyDir volumes backed by local SSD and other Pods that use emptyDir volumes backed by a node boot disk. If you have workloads that use emptyDir volumes backed by a node boot disk, schedule the workloads on a different node pool.
- Autopilot clusters and node auto-provisioning is not supported for local SSDs.