Scale your storage performance with Hyperdisk

The Compute Engine Persistent Disk CSI driver is the primary way for you to access Hyperdisk storage with GKE clusters.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

Requirements

To use Hyperdisk volumes in GKE, your clusters must meet the following requirements:

Create a Hyperdisk volume for GKE

This section provides an overview of creating a Hyperdisk volume backed by the Compute Engine CSI driver in GKE.

Create a StorageClass

The following Persistent Disk storage Type fields are provided by the Compute Engine Persistent Disk CSI driver to support Hyperdisk Throughput and Hyperdisk Extreme:

  • hyperdisk-throughput
  • hyperdisk-extreme

To create a new StorageClass with the throughput or IOPS level you want, use pd.csi.storage.gke.io in the provisioner field, and specify one of the Hyperdisk storage types.

Hyperdisk Throughput

Provision throughput using the provisionedThroughputOnCreate parameter when defining the StorageClass. See Plan the performance level for your Hyperdisk volume for guidance on allowable values for throughput. These values should be expressed in MB/s using the "Mi" qualifier; for example, if your required throughput is 50 MB/s, specify "50Mi" when creating the StorageClass.

  1. Save the following manifest in a file named hdt-example-class.yaml:

      apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: throughput-storage
  provisioner: pd.csi.storage.gke.io
  volumeBindingMode: WaitForFirstConsumer
  allowVolumeExpansion: true
  parameters:
    type: hyperdisk-throughput
    provisioned-throughput-on-create: "50Mi"

  2. Create the StorageClass:

      kubectl create -f hdt-example-class.yaml

Hyperdisk Extreme

Provision IOPS using the provisioned-iops-on-create parameter when defining the StorageClass. See Plan the performance level for your Hyperdisk volume for guidance on allowable values for IOPS. The value should be expressed without any qualifiers; for example, if your required IOPS is 1,000 I/O per second, specify "1000" when creating the StorageClass.

  1. Save the following manifest in a file named hdx-example-class.yaml:

      apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: extreme-storage
  provisioner: pd.csi.storage.gke.io
  volumeBindingMode: WaitForFirstConsumer
  allowVolumeExpansion: true
  parameters:
    type: hyperdisk-extreme
    provisioned-iops-on-create: '1000'

  2. Create the StorageClass:

      kubectl create -f hdx-example-class.yaml

To find the name of the StorageClasses available in your cluster, run the following command:

kubectl get sc

Create a PersistentVolumeClaim

You can create a PersistentVolumeClaim that references the Compute Engine Persistent Disk CSI driver's StorageClass.

Hyperdisk Throughput

In this example, you specify the targeted storage capacity of the Hyperdisk Throughput volume as 2 TiB.

  1. Save the following PersistentVolumeClaim manifest in a file named pvc-example.yaml:

      kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: podpvc
  spec:
    accessModes:
    - ReadWriteOnce
    storageClassName: throughput-storage
    resources:
      requests:
        storage: 2Ti

  2. Apply the PersistentVolumeClaim that references the StorageClass you created from the earlier example:

      kubectl apply -f pvc-example.yaml

Hyperdisk Extreme

In this example, you specify the minimum storage capacity of the Hyperdisk Extreme volume as 64 GiB.

  1. Save the following PersistentVolumeClaim manifest in a file named pvc-example.yaml:

      kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: podpvc
  spec:
    accessModes:
    - ReadWriteOnce
    storageClassName: extreme-storage
    resources:
      requests:
        storage: 64Gi

  2. Apply the PersistentVolumeClaim that references the StorageClass you created from the earlier example:

      kubectl apply -f pvc-example.yaml

Create a Deployment to consume the Hyperdisk volume

When using Pods with PersistentVolumes, we recommend that you use a workload controller (such as a Deployment or StatefulSet).

  1. The following example creates a manifest that configures a Pod for deploying a Nginx webserver using the PersistentVolumeClaim created in the previous section. Save the following example manifest as hyperdisk-example-deployment.yaml:

      apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: web-server-deployment
    labels:
      app: nginx
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: nginx
    template:
      metadata:
        labels:
          app: nginx
      spec:
        containers:
        - name: nginx
          image: nginx
          volumeMounts:
          - mountPath: /var/lib/www/html
            name: mypvc
        volumes:
        - name: mypvc
          persistentVolumeClaim:
            claimName: podpvc
            readOnly: false

  2. To create a Deployment based on the hyperdisk-example-deployment.yaml manifest file, run the following command:

      kubectl apply -f hyperdisk-example-deployment.yaml

  3. Confirm the Deployment was successfully created:

      kubectl get deployment

    It might take a few minutes for Hyperdisk instances to complete provisioning. When the deployment completes provisioning, it reports a READY status.

  4. You can check the progress by monitoring your PersistentVolumeClaim status by running the following command:

      kubectl get pvc

Provision a Hyperdisk volume from a snapshot

To create a new Hyperdisk volume from an existing Persistent Disk snapshot, use the Google Cloud console, the Google Cloud CLI, or the Compute Engine API. To learn how to create a Persistent Disk snapshot, see Creating and using volume snapshots.

Console

  1. Go to the Disks page in the Google Cloud console.

    Go to Disks

  2. Click Create Disk.

  3. Under Disk Type, choose one of the following for disk type:

    • Hyperdisk Extreme
    • Hyperdisk Throughput

  4. Under Disk source type, click Snapshot.

  5. Select the name of the snapshot to restore.

  6. Select the size of the new disk, in GiB. This number must be equal to or larger than the original source disk for the snapshot.

  7. Set the Provisioned throughput or Provisioned IOPS you want for the disk, if different from the default values.

  8. Click Create to create the Hyperdisk volume.

gcloud

Run the gcloud compute disks create command to create the Hyperdisk volume from a snapshot.

Hyperdisk Throughput

gcloud compute disks create DISK_NAME \
    --size=SIZE \
    --source-snapshot=SNAPSHOT_NAME \
    --provisioned-throughput=TRHROUGHPUT_LIMIT \
    --type=hyperdisk-throughput

Replace the following:

  • DISK_NAME: the name of the new disk.
  • SIZE: the size, in gibibytes (GiB or GB) or tebibytes (TiB or TB), of the new disk. Acceptable sizes range, in 1 GiB increments, from 2 TiB to 32 TiB inclusive.
  • SNAPSHOT_NAME: the name of the snapshot being restored.
  • THROUGHPUT_LIMIT: Optional: For Hyperdisk Throughput disks, this is an integer that represents the throughput, measured in MB/s, that the disk can handle. The throughput must be specified within the following limits:
    • Minimum value: 10 MB/s per TiB of capacity.
    • Maximum value: The lesser of 600 MB/s per volume or 90 MB/s per TiB of capacity.

If you don't specify a value for throughput, a default value is used, which is 90 MB/s per TiB of disk capacity, but not more than the maximum supported throughput per disk.

Hyperdisk Extreme

gcloud compute disks create DISK_NAME \
    --size=SIZE \
    --source-snapshot=SNAPSHOT_NAME \
    --provisioned-iops=IOPS_LIMIT \
    --type=hyperdisk-iops

Replace the following:

  • DISK_NAME: the name of the new disk.
  • SIZE: the size, in gibibytes (GiB or GB) or tebibytes (TiB or TB), of the new disk. Acceptable sizes range, in 1 GiB increments, from 64 GiB to 64 TiB, inclusive.
  • SNAPSHOT_NAME: the name of the snapshot being restored.
  • IOPS_LIMIT: Optional: For Hyperdisk Extreme disks, this is the number of I/O operations per second that the disk can handle. The IOPS must be specified within the following limits:
    • Minimum value: Two IOPS per GiB of disk capacity.
    • Maximum value: The lesser of 350,000 or 1,000 times the capacity measured in GiB.

If you don't specify a value for IOPS, a default value is used, which is the lesser of 350,000 or 100 times the disk capacity measured in GiB. For example, you can provision IOPs as follows:

  • For a 64 GiB disk, you can provision IOPS in the range of 128 to 64000. The default IOPS is 6400.
  • For a 64,000 GiB disk, you can provision IOPS in the range of 128,000 to 350,000. The default IOPS is 350,000.

Create a snapshot for a Hyperdisk volume

To create a snapshot from a Hyperdisk volume, follow the same steps as creating a snapshot for a Persistent Disk volume:

Update the provisioned throughput of an existing Hyperdisk Throughput volume

This section covers how to modify provisioned performance for Hyperdisk Throughput or IOPS volumes.

Hyperdisk Throughput

To update the provisioned throughput level of your Hyperdisk Throughput volume, follow the Google Cloud console, gcloud CLI, or Compute Engine API instructions in Changing the provisioned performance for a Hyperdisk volume.

You can change the provisioned throughput level (up to once every 4 hours) for a Hyperdisk Throughput volume after volume creation. New throughput levels might take up to 15 minutes to take effect. During the performance change, any performance SLA and SLO are not in effect. You can change the throughput level of an existing volume at any time, regardless of whether the disk is attached to a running instance or not.

The new throughput level you specify must adhere to the supported values for Hyperdisk volumes.

Hyperdisk IOPS

To update the provisioned IOPS level of your Hyperdisk Extreme volume, follow the Google Cloud console, gcloud CLI, or Compute Engine API instructions in Changing the provisioned performance for a Hyperdisk volume.

You can change the provisioned IOPS level (up to once every 4 hours) for a Hyperdisk IOPS volume after volume creation. New IOPS levels might take up to 15 minutes to take effect. During the performance change, any performance SLA and SLO are not in effect. You can change the IOPS level of an existing volume at any time, regardless of whether the disk is attached to a running instance or not.

The new IOPS level you specify must adhere to the supported values for Hyperdisk volumes.

To update the provisioned IOPS level for a Hyperdisk Extreme volume, you must identify the name of the Persistent Disk backing your PersistentVolumeClaim and PersistentVolume resources:

  1. Go to the Object browser in the Google Cloud console.

    Go to Object Browser

  2. Find the entry for your PersistentVolumeClaim object.

  3. Click the Volume link .

  4. Open the YAML tab of the associated PersistentVolume. Scroll down to find the CSI volumeHandle value.

  5. Note the last element of this handle (it should have a value like "pvc-XXXXX"). This is the name of your PersistentVolumeClaim. You should also take note of the project and zone.

Monitor throughput or IOPS on a Hyperdisk volume

To monitor the provisioned performance of your Hyperdisk volume, see Analyze provisioned IOPS and throughput in the Compute Engine documentation.

Troubleshooting

This section provides troubleshooting guidance to resolve issues with Hyperdisk volumes on GKE.

Cannot change performance or capacity: ratio out of range

The following error occurs when you attempt to change the provisioned performance level or capacity, but the performance level or capacity that you picked is outside of the range that is acceptable for the volume:

  • Requested provisioned throughput cannot be higher than <value>.
  • Requested provisioned throughput cannot be lower than <value>.
  • Requested provisioned throughput is too high for the requested disk size.
  • Requested provisioned throughput is too low for the requested disk size.
  • Requested disk size is too high for current provisioned throughput.

The throughput provisioned for Hyperdisk Throughput volumes must meet the following requirements:

  • At least 10 MB/s per TiB of capacity, and no more than 90 MB/s per TiB of capacity.
  • At most 600 MB/s per volume.

To resolve this issue, correct the requested throughput or capacity to be within the allowable range and reissue the command.

Cannot change performance: rate limited

The following error occurs when you attempt to change the provisioned performance level, but the performance level has already been changed within the last 4 hours:

Cannot update provisioned throughput due to being rate limited.
Cannot update provisioned iops due to being rate limited.

Hyperdisk Throughput and IOPS volumes can have their provisioned performance updated once every 4 hours. To resolve this issue, wait for the cool-down timer for the volume to elapse, and then reissue the command.

What's next