In Google Kubernetes Engine (GKE), you can use the Kubernetes volume snapshot feature for persistent volumes in your GKE clusters.
Volume snapshots let you create a copy of your volume at a specific point in time. You can use this copy to bring a volume back to a prior state or to provision a new volume.
From GKE version 1.17 or later, you can provision and attach volume snapshots with the following components:
Requirements
To use volume snapshots on GKE, you must meet the following requirements:
Use a CSI driver that supports snapshots. The in-tree persistent disk driver does not support snapshots.
- The Compute Engine persistent disk CSI driver is installed by default on new Linux clusters running GKE version 1.18.10-gke.2100 or later, or version 1.19.3-gke.2100 or later. You can also enable the Compute Engine persistent disk CSI driver on an existing cluster.
- You can also automatically deploy the Filestore CSI driver. For a list of all CSI drivers that support snapshots, see the "Other features" column in Drivers in the Kubernetes documentation.
Use control plane versions 1.17 or later. To use the Compute Engine persistent disk CSI Driver in a VolumeSnapshot, use GKE versions 1.17.6-gke.4 or later. To use the Filestore CSI Driver in a VolumeSnapshot, use GKE versions 1.21 or later.
Have an existing
PersistentVolumeClaim
to use for a snapshot. ThePersistentVolume
you use for a snapshot source must be managed by a CSI driver. You can verify that you're using a CSI driver by checking that thePersistentVolume
spec has acsi
section withdriver: pd.csi.storage.gke.io
orfilestore.csi.storage.gke.io
. If the PersistentVolume is dynamically provisioned by the CSI driver as described in the following sections, it's managed by the CSI driver.
Limitations
All restrictions for creating a disk snapshot on Compute Engine also apply to GKE.
Best practices
Be sure to follow best practices for Compute Engine disk snapshots when using volume snapshots on GKE.
Before you begin
Before you start, make sure you have performed the following tasks:
- Enable the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI.
Creating and using a volume snapshot
The examples in this document show you how to do the following tasks:
- Create a
PersistentVolumeClaim
and Deployment. - Add a file to the PersistentVolume that the Deployment uses.
- Create a
VolumeSnapshotClass
to configure the snapshot. - Create a volume snapshot of the PersistentVolume.
- Delete the test file.
- Restore the
PersistentVolume
to the snapshot you created. - Verify that the restoration worked.
To use a volume snapshot, you must complete the following steps:
- Create a
VolumeSnapshotClass
object to specify the CSI driver and deletion policy for your snapshot. - Create a
VolumeSnapshot
object to request a snapshot of an existing PersistentVolumeClaim. - Reference the
VolumeSnapshot
in aPersistentVolumeClaim
to restore a volume to that snapshot or create a new volume using the snapshot.
Create a PersistentVolumeClaim and a Deployment
To create the
PersistentVolumeClaim
object, save the following manifest asmy-pvc.yaml
:apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-pvc spec: storageClassName: standard-rwo accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
For
spec.storageClassName
, you can specify any storage class that uses a supported CSI driver. This example uses thestandard-rwo
storage class installed by default with the Compute Engine persistent disk CSI driver. For more information, refer to Using the Compute Engine persistent disk CSI driver.Apply the manifest:
kubectl apply -f my-pvc.yaml
To create a Deployment, save the following manifest as
my-deployment.yaml
:apiVersion: apps/v1 kind: Deployment metadata: name: hello-app spec: selector: matchLabels: app: hello-app template: metadata: labels: app: hello-app spec: containers: - name: hello-app image: google/cloud-sdk:slim args: [ "sleep", "3600" ] volumeMounts: - name: sdk-volume mountPath: /usr/share/hello/ volumes: - name: sdk-volume persistentVolumeClaim: claimName: my-pvc
Apply the manifest:
kubectl apply -f my-deployment.yaml
Check the status of the Deployment:
kubectl get deployment hello-app
It might take some time for the Deployment to become ready. You can run the preceding command until you see an output similar to the following:
NAME READY UP-TO-DATE AVAILABLE AGE hello-app 1/1 1 1 2m55s
Add a test file to the volume
List the Pods in the Deployment:
kubectl get pods -l app=hello-app
The output is similar to the following:
NAME READY STATUS RESTARTS AGE hello-app-6d7b457c7d-vl4jr 1/1 Running 0 2m56s
Create a test file in a Pod:
kubectl exec POD_NAME \ -- sh -c 'echo "Hello World!" > /usr/share/hello/hello.txt'
Replace
POD_NAME
with the name of the Pod.Verify that the file exists:
kubectl exec POD_NAME \ -- sh -c 'cat /usr/share/hello/hello.txt'
The output is similar to the following:
Hello World!
Create a VolumeSnapshotClass
object
Create a VolumeSnapshotClass
object to specify the CSI driver and
deletionPolicy
for your volume snapshot. You can reference
VolumeSnapshotClass
objects when you create VolumeSnapshot
objects.
Save the following manifest as
volumesnapshotclass.yaml
.v1
Use the
v1
API version for clusters running versions 1.21 or later.apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: my-snapshotclass driver: pd.csi.storage.gke.io deletionPolicy: Delete
v1beta1
Use the
v1beta1
API version for clusters running versions 1.21 or earlier.apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshotClass metadata: name: my-snapshotclass driver: pd.csi.storage.gke.io deletionPolicy: Delete
In this example:
The
driver
field is used by the CSI driver to provision the snapshot. In this example,pd.csi.storage.gke.io
uses the Compute Engine persistent disk CSI Driver.The
deletionPolicy
field tells GKE what to do with theVolumeSnapshotContent
object and the underlying snapshot when the boundVolumeSnapshot
object is deleted. SpecifyDelete
to delete theVolumeSnapshotContent
object and the underlying snapshot. SpecifyRetain
if you want to keep theVolumeSnapshotContent
and the underlying snapshot.
To use a custom storage location, add a
storage-locations
parameter to the snapshot class. To use this parameter, your clusters must use version 1.21 or later.apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: my-snapshotclass parameters: storage-locations: us-east2 driver: pd.csi.storage.gke.io deletionPolicy: Delete
Apply the manifest:
kubectl apply -f volumesnapshotclass.yaml
Create a VolumeSnapshot
A VolumeSnapshot
object is a request for a snapshot of an existing
PersistentVolumeClaim
object. When you create a VolumeSnapshot
object,
GKE automatically creates and binds it with a VolumeSnapshotContent
object, which is a resource in your cluster like a PersistentVolume
object.
Save the following manifest as
volumesnapshot.yaml
.v1
Use the
v1
API version for clusters running versions 1.21 or later.apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: my-snapshot spec: volumeSnapshotClassName: my-snapshotclass source: persistentVolumeClaimName: my-pvc
v1beta1
Use the
v1beta1
API version for clusters running versions 1.21 or earlier.apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: my-snapshot spec: volumeSnapshotClassName: my-snapshotclass source: persistentVolumeClaimName: my-pvc
Apply the manifest:
kubectl apply -f volumesnapshot.yaml
After you create a volume snapshot, GKE creates a corresponding
VolumeSnapshotContent
object in the cluster. This object stores the snapshot and bindings ofVolumeSnapshot
objects. You do not interact withVolumeSnapshotContents
objects directly.Confirm that GKE created the
VolumeSnapshotContents
object:kubectl get volumesnapshotcontents
The output is similar to the following:
NAME AGE snapcontent-cee5fb1f-5427-11ea-a53c-42010a1000da 55s
After the volume snapshot content is created, the CSI driver you specified in
the VolumeSnapshotClass
creates a snapshot on the corresponding storage
system. After GKE creates a snapshot on the storage system and
binds it to a VolumeSnapshot
object on the cluster, the snapshot is ready to
use. You can check the status by running the following command:
kubectl get volumesnapshot \
-o custom-columns='NAME:.metadata.name,READY:.status.readyToUse'
If the snapshot is ready to use, the output is similar to the following:
NAME READY
my-snapshot true
Delete the test file
Delete the test file that you created:
kubectl exec POD_NAME \ -- sh -c 'rm /usr/share/hello/hello.txt'
Verify that the file no longer exists:
kubectl exec POD_NAME \ -- sh -c 'cat /usr/share/hello/hello.txt'
The output is similar to the following:
cat: /usr/share/hello/hello.txt: No such file or directory
Restore the volume snapshot
You can reference a VolumeSnapshot
in a PersistentVolumeClaim
to provision
a new volume with data from an existing volume or restore a volume to a
state that you captured in the snapshot.
To reference a VolumeSnapshot
in a PersistentVolumeClaim
, add the dataSource
field to your PersistentVolumeClaim
.
In this example, you reference the VolumeSnapshot
that you created in a new
PersistentVolumeClaim
and update the Deployment to use the new claim.
Save the following manifest as
pvc-restore.yaml
:apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-restore spec: dataSource: name: my-snapshot kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io storageClassName: standard-rwo accessModes: - ReadWriteOnce resources: requests: storage: 1Gi
Apply the manifest:
kubectl apply -f pvc-restore.yaml
Update the
my-deployment.yaml
file to use the newPersistentVolumeClaim
:... volumes: - name: my-volume persistentVolumeClaim: claimName: pvc-restore
Apply the updated manifest:
kubectl apply -f my-deployment.yaml
Check that the snapshot restored successfully
Get the name of the new Pod that GKE creates for the updated Deployment:
kubectl get pods -l app=hello-app
Verify that the test file exists:
kubectl exec NEW_POD_NAME \
-- sh -c 'cat /usr/share/hello/hello.txt'
Replace NEW_POD_NAME
with the name of the new Pod that
GKE created.
The output is similar to the following:
Hello World!
Clean up
To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.
Delete the
VolumeSnapshot
:kubectl delete volumesnapshot my-snapshot
Delete the
VolumeSnapshotClass
:kubectl delete volumesnapshotclass my-snapshotclass
Delete the Deployment:
kubectl delete deployments hello-app
Delete the
PersistentVolumeClaim
objects:kubectl delete pvc my-pvc pvc-restore
What's next
- Read the Kubernetes Volume Snapshot documentation.
- Learn about volume expansion.
- Learn how to manually install a CSI driver.