GKE persistent volumes & provisioning

Standard

This page provides an overview of Persistent Volumes (PVs), Persistent Volume Claims (PVCs), and Storage Classes in Google Kubernetes Engine (GKE). It focuses on storage backed by Compute Engine persistent disks.

You learn how to effectively create, manage, and troubleshoot persistent storage within your GKE clusters to help ensure data security, high availability, and optimal performance.

This page is for Storage specialists who create and allocate storage and configure and manage data security, protection, and access and permissions. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE user roles and tasks.

PersistentVolumes

PersistentVolume resources are used to manage durable storage in a cluster. In GKE, a PersistentVolume is typically backed by a persistent disk.

You can use other storage solutions like NFS instead. Filestore is a NFS solution on Google Cloud. To learn how to set up a Filestore instance as an NFS PV solution for your GKE clusters, see Access Filestore instances with the Filestore CSI driver in the Filestore documentation.

The PersistentVolume lifecycle is managed by Kubernetes. A PersistentVolume can be dynamically provisioned; you don't have to manually create and delete the backing storage.

PersistentVolume resources are cluster resources that exist independently of Pods. This means that the disk and data represented by a PersistentVolume continue to exist as the cluster changes and as Pods are deleted and recreated. PersistentVolume resources can be provisioned dynamically through PersistentVolumeClaims, or they can be explicitly created by a cluster administrator.

Caution: Using the [fsGroup] setting with large PersistentVolumes can cause mounts to fail. For more information, see Troubleshooting Volume mount failures.

To learn more about PersistentVolume resources, refer to the Kubernetes Persistent Volumes documentation and the Persistent Volumes API reference.

PersistentVolumeClaims

A PersistentVolumeClaim is a request for and claim to a PersistentVolume resource. PersistentVolumeClaim objects request a specific size, access mode, and StorageClass for the PersistentVolume. If a PersistentVolume that satisfies the request exists or can be provisioned, the PersistentVolumeClaim is bound to that PersistentVolume.

Pods use claims as volumes. The cluster inspects the claim to find the bound volume and mounts that volume for the Pod.

Portability is another advantage of using PersistentVolumes and PersistentVolumeClaims. You can use the same Pod specification across different clusters and environments because PersistentVolume is an interface to the actual backing storage.

StorageClasses

Volume implementations such as Compute Engine persistent disk Container Storage Interface (CSI) Driver are configured through StorageClass resources.

GKE creates a default StorageClass for you which uses the balanced persistent disk type (ext4). The default StorageClass is used when a PersistentVolumeClaim doesn't specify a StorageClassName. You can replace the provided default StorageClass with your own. For instructions, see Change the default StorageClass.

You can create your own StorageClass resources to describe different classes of storage. For example, classes might map to quality-of-service levels, or to backup policies. This concept is sometimes called "profiles" in other storage systems.

If you are using a cluster with Windows node pools, you must create a StorageClass and specify a StorageClassName in the PersistentVolumeClaim because the default fstype (ext4) is not supported with Windows. If you are using a Compute Engine persistent disk, you must use NTFS as the file storage type.

When defining a StorageClass, you must list a provisioner. On GKE, we recommend that you use one of the following provisioners:

• Compute Engine persistent disk CSI
• Kubernetes Filestore CSI

Note: Support for provisioners other than those listed is not covered by Cloud Customer Care, except for the open source SMB CSI Driver for Kubernetes, which is provided on a best effort basis. Third party support for other provisioners might also be available.

Dynamically provision PersistentVolumes

Most of the time, you don't need to directly configure PersistentVolume objects or create Compute Engine persistent disks. Instead, you can create a PersistentVolumeClaim and Kubernetes automatically provisions a persistent disk for you.

The following manifest describes a request for a disk with 30 gibibytes (GiB) of storage whose access mode allows it to be mounted as read-write by a single node. It also creates a Pod that consumes the PersistentVolumeClaim as a volume.

# pvc-pod-demo.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
  storageClassName: standard-rwo
---
kind: Pod
apiVersion: v1
metadata:
  name: pod-demo
spec:
  volumes:
    - name: pvc-demo-vol
      persistentVolumeClaim:
       claimName: pvc-demo
  containers:
    - name: pod-demo
      image: nginx
      resources:
        limits:
          cpu: 10m
          memory: 80Mi
        requests:
          cpu: 10m
          memory: 80Mi
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: pvc-demo-vol

When you create this PersistentVolumeClaim object with kubectl apply -f pvc-pod-demo.yaml, Kubernetes dynamically creates a corresponding PersistentVolume object.

Because the storage class standard-rwo uses volume binding mode WaitForFirstConsumer, the PersistentVolume won't be created until a Pod is scheduled to consume the volume.

The following example shows the PersistentVolume created.

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: pd.csi.storage.gke.io
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/pd-csi-storage-gke-io
  name: pvc-c9a44c07-cffa-4cd8-b92b-15bac9a9b984
  uid: d52af557-edf5-4f96-8e89-42a3008209e6
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 30Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: pvc-demo
    namespace: default
    uid: c9a44c07-cffa-4cd8-b92b-15bac9a9b984
  csi:
    driver: pd.csi.storage.gke.io
    csi.storage.k8s.io/fstype: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1660085000920-8081-pd.csi.storage.gke.io
    volumeHandle: projects/xxx/zones/us-central1-c/disks/pvc-c9a44c07-cffa-4cd8-b92b-15bac9a9b984
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.gke.io/zone
          operator: In
          values:
          - us-central1-c
  persistentVolumeReclaimPolicy: Delete
  storageClassName: standard-rwo
  volumeMode: Filesystem
status:
  phase: Bound

Assuming that you haven't replaced the storage class standard-rwo, this PersistentVolume is backed by a new, empty Compute Engine persistent disk.

Note: Modifying the topology.kubernetes.io/zone label during the lifetime of a PersistentVolume can result in mount failures.

Delete persistent storage

During dynamic provisioning, you can control whether your persistent storage is deleted when a PersistentVolumeClaim (PVC) is deleted. To do this, you set the reclaimPolicy field in your StorageClass configuration to either Delete or Retain.

• reclaimPolicy: Delete: This is the default policy. When the PVC is deleted, the PersistentVolume (PV) object and the underlying disk (such as a Compute Engine persistent disk) are automatically deleted. Use this policy for temporary workloads or for compliance requirements where data must be cleaned up after a job is complete.

• reclaimPolicy: Retain: If you set this policy, the PV and the underlying disk are not deleted when you delete the PVC. The PV status changes to Released. The data is preserved, but the volume cannot be used by a new PVC until an administrator manually reclaims it.

Manually managing retained volumes

When a PersistentVolume has a Released status, you must manually decide what to do with the underlying data and disk.

• To permanently delete the storage: You must delete both the Kubernetes PersistentVolume resource and the underlying Compute Engine disk.

  1. Delete the PersistentVolume: bash kubectl delete pv <your-pv-name>
  2. Find the name of the underlying disk by describing the YAML configuration file for the PV and looking for the volumeHandle or a similar field.
  3. Delete the Compute Engine persistent disk: bash gcloud compute disks delete <your-disk-name> --zone=<your-zone>

• To reuse the volume: Reclaiming a volume with a Released status for use by another PVC is an advanced task that involves manually cleaning the disk data and editing the PersistentVolume object to make it Available again. For more information about the detailed procedure, see the official Kubernetes documentation on Reclaiming a PersistentVolume.

To prevent or mitigate data loss, we also recommend enabling Backup for GKE and scheduling regular backups of your GKE cluster.

Troubleshooting unexpected disk deletion

If you observe that your underlying persistent disks are being deleted when you don't expect them to be, the most common cause is a reclaimPolicy field that's set to Delete in the StorageClass configuration used by your PersistentVolumeClaim.

To diagnose this issue, complete the following steps:

1. Check the StorageClass of your PVC: bash kubectl get pvc <your-pvc-name> -o jsonpath='{.spec.storageClassName}'

2. Inspect the reclaimPolicy field: bash kubectl get sc <your-storageclass-name> -o yaml

3. Align the policy with your intent:

   • If the policy is Delete and your application requires data to persist, you must update your application to use a StorageClass configuration that sets the policy to Retain.
   • If the policy is Delete and your application is ephemeral or subject to compliance rules requiring data cleanup, then the configuration is correct and working as intended.

For more information, see the Kubernetes documentation about Reclaiming.

Manage persistent storage during cluster deletion

When a GKE cluster is deleted, GKE retains PersistentVolume resources with either the Delete or Retain reclaim policy.

To remove PersistentVolume resources when you delete a cluster, you can manually delete the namespace of PersistentVolumeClaim resources, which will trigger deletion for PersistentVolume objects with the Delete policy. Alternatively, you can delete individual PersistentVolumeClaim resources. However, GKE doesn't wait before these deletion activities are complete before starting to delete the cluster. So, if you delete a namespace then immediately delete the cluster, PersistentVolume resources with Delete policies might not be deleted.

After cluster deletion, you can view remaining PersistentVolume resources in the Google Cloud console.

To view any unused resources, such as unused PersistentVolume resources, you can view recommendations for idle resources.

Access modes

PersistentVolume resources support the following access modes:

• ReadWriteOnce: The volume can be mounted as read-write by a single node.
• ReadOnlyMany: The volume can be mounted read-only by many nodes.
• ReadWriteMany: The volume can be mounted as read-write by many nodes. PersistentVolume resources that are backed by Compute Engine persistent disks don't support this access mode.
• ReadWriteOncePod: The volume can be mounted as read-write by a single Pod only.

Using Compute Engine persistent disks as ReadOnlyMany

ReadWriteOnce is the most common use case for persistent disks and works as the default access mode for most applications. Compute Engine persistent disks also support ReadOnlyMany mode so that many applications or many replicas of the same application can consume the same disk at the same time. An example use case is serving static content across multiple replicas.

Note: You can't attach persistent disks in write mode on multiple nodes at the same time. See Deployments versus StatefulSets.

For instructions, refer to Use persistent disks with multiple readers.

Use pre-existing persistent disks as PersistentVolumes

Dynamically provisioned PersistentVolume resources are empty when they are created. If you have an existing Compute Engine persistent disk populated with data, you can introduce it to your cluster by manually creating a corresponding PersistentVolume resource. The persistent disk must be in the same zone as the cluster nodes.

Refer to this example of how to create a Persistent Volume backed by a pre-existing persistent disk.

Note: Mounting a persistent disk from a different Google Cloud project to a Pod in your current Google Cloud project is not supported.

Deployments versus StatefulSets

You can use a PersistentVolumeClaim or VolumeClaim templates in higher level controllers such as Deployments or StatefulSets respectively.

Deployments are designed for stateless applications so all replicas of a Deployment share the same PersistentVolumeClaim. Since the replica Pods created are identical to each other, only volumes with the ReadWriteMany mode can work in this setting.

Even Deployments with one replica using ReadWriteOnce volume are not recommended. This is because the default Deployment strategy creates a second Pod before bringing down the first Pod on a recreate. The Deployment may fail in deadlock as the second Pod can't start because the ReadWriteOnce volume is already in use, and the first Pod won't be removed because the second Pod has not yet started. Instead, use a StatefulSet with ReadWriteOnce volumes.

StatefulSets are the recommended method of deploying stateful applications that require a unique volume per replica. By using StatefulSets with PersistentVolumeClaim templates, you can have applications that can scale up automatically with unique PersistentVolumesClaims associated to each replica Pod.

Regional persistent disks

Regional persistent disks are multi-zonal resources that replicate data between two zones in the same region, and can be used similarly to zonal persistent disks. In the event of a zonal outage or if cluster nodes in one zone become unschedulable, Kubernetes can failover workloads using the volume to the other zone. You can use regional persistent disks to build highly available solutions for stateful workloads on GKE. You must ensure that both the primary and failover zones are configured with enough resource capacity to run the workload.

Regional SSD persistent disks are an option for applications such as databases that require both high availability and high performance. For more details, see Block storage performance comparison.

As with zonal persistent disks, regional persistent disks can be dynamically provisioned as needed or manually provisioned in advance by the cluster administrator. To learn how to add regional persistent disks, see Provisioning regional persistent disks.

Zones in persistent disks

Zonal persistent disks are zonal resources and regional persistent disks are multi-zonal resources. When you add persistent storage to your cluster, unless a zone is specified, GKE assigns the disk to a single zone. GKE chooses the zone at random. Once a persistent disk is provisioned, any Pods referencing the disk are scheduled to the same zone as the persistent disk. However, Pods or Deployments don't inherently recognize the zone of pre-existing persistent disks. To ensure Pods are scheduled correctly with pre-existing persistent disks, use zonal placement methods like nodeAffinity in your Pod or Deployment specifications to target the appropriate zones.

Volume binding mode WaitForFirstConsumer

If you dynamically provision a persistent disk in your cluster, we recommend you set the WaitForFirstConsumer volume binding mode on your StorageClass. This setting instructs Kubernetes to provision a persistent disk in the same zone that the Pod gets scheduled to. It respects Pod scheduling constraints such as anti-affinity and node selectors. Anti-affinity on zones allows StatefulSet Pods to be spread across zones along with the corresponding disks.

Following is an example StorageClass for provisioning zonal persistent disks that sets WaitForFirstConsumer:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: slow
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-balanced
  csi.storage.k8s.io/fstype: ext4
volumeBindingMode: WaitForFirstConsumer

For an example using regional persistent disks, see Provisioning regional persistent disks.

What's next

• Learn about StatefulSets, the recommended method of deploying stateful applications.
• Learn how to deploy a stateful application using a StatefulSet.
• Learn to use persistent disks in a cluster.
• Learn how to create a disk that can be read from multiple nodes.
• Learn how to create persistent disks backed by SSDs.
• Learn how to provision regional persistent disks.