Optimizing Persistent Disk and Local SSD Performance

Persistent disks are the most common storage options due to their price, performance, and predictability. However, you can create instances with local SSDs for even greater performance and lower latency, but without the data redundancy and durability that you get from persistent disks. When you configure a storage option for applications that run on your instances, use the following processes:

  • Determine how much space you need.
  • Determine what performance characteristics your applications require.
  • Configure your instances to optimize storage performance.

This document discusses block storage options that you can attach to Compute Engine instances. To see a complete list of storage options on Google Cloud Platform, read Choosing a storage option.

Block storage performance comparison

Consider your storage size and performance requirements to help you determine the correct disk type and size for your instances. Performance requirements for a given application are typically separated into two distinct IO patterns.

  • Small reads and writes
  • Large reads and writes

For small reads and writes, the limiting factor is random input/output operations per second IOPS.

For large reads and writes, the limiting factor is throughput.

Standard persistent disks SSD persistent disks Local SSD (SCSI) Local SSD (NVMe)
Maximum sustained IOPS
Read IOPS per GB 0.75 30 266.7 453.3
Write IOPS per GB 1.5 30 186.7 240
Read IOPS per instance 3,000 15,000 - 40,000* 400,000 680,000
Write IOPS per instance 15,000 15,000 - 30,000* 280,000 360,000
Maximum sustained throughput (MB/s)
Read throughput per GB 0.12 0.48 1.04 1.77
Write throughput per GB 0.12 0.48 0.73 0.94
Read throughput per instance 180 240 - 800* 1,560 2,650
Write throughput per instance 120 240 - 400* 1,090 1,400

* SSD persistent disks can achieve greater IOPS and throughput performance on instances with greater numbers of vCPUs. Read SSD persistent disk performance limits for details.

Comparing persistent disk to a physical hard drive

When you specify the size of your persistent disks, consider how these disks compare to traditional physical hard drives. The following tables compare standard persistent disks and SSD persistent disks to the typical performance that you would expect from a 7200 RPM SATA drive, which typically achieves 75 IOPS or 120 MB/s.

IO Type IO Pattern Size required to match a 7200 RPM SATA drive
Standard persistent disk SSD persistent disk
Small random reads 75 small random reads 100 GB 3 GB
Small random writes 75 small random writes 50 GB 3 GB
Streaming large reads 120 MB/s streaming reads 1000 GB 250 GB
Streaming large writes 120 MB/s streaming writes 1000 GB 250 GB

Size, Price, and Performance Summary

While you have several inputs to consider when you select a volume type and size for your application, one factor you do not need to consider is the price of using your volume. Persistent Disk has no per-IO costs, so there is no need to estimate monthly I/O to calculate budget for what you will spend on disks.

Consider only the relative costs of standard persistent disks compared to SSD persistent disks. Standard persistent disks are priced at $0.040 per GB and SSD persistent disks are priced at $0.170 per GB. However, performance caps increase with the size of the volume, so look at the price per IOPS for IOPS oriented workloads.

Standard persistent disks are approximately $0.053 per random read IOPS and $0.0266 per random write IOPS. SSD persistent disks are $0.0057 per random read IOPS and $0.0057 per random write IOPS. The price per IOPS for SSD persistent disks is true up to the point where they reach the IOPS limits of the instance or the vCPU count for that instance.

SSD persistent disks reach their limit of 40,000 random read IOPS at 1334 GB and 30,000 random write IOPS at 1000 GB. In contrast, standard persistent disks reach their limit of 3,000 random read IOPS at 4 TB and 15,000 random write IOPS at 10 TB.

Standard persistent disk

Standard persistent disk performance scales linearly up to the VM performance limits. The vCPU count for your instance does not limit the performance of standard persistent disks.

A vCPU count of 2 or more for your instance does not limit the performance of standard persistent disks. A vCPU count of 1 for your instance will experience a reduced write limit because it is constrained by the network egress limits which is proportional to the vCPU count.

Standard persistent disk IOPS and throughput performance increases linearly with the size of the disk until it reaches the following per-instance limits:

  • Read throughput: Up to 180 MB/s at a 1.5 TB disk size.
  • Write throughput: Up to 120 MB/s at a 1 TB disk size.
  • Read IOPS: Up to 3,000 IOPS at a 4 TB disk size.
  • Write IOPS: Up to 15,000 IOPS at a 10 TB disk size.

To gain persistent disk performance benefits on your existing instances, resize your persistent disks to increase IOPS and throughput per persistent disk.

Volume Size (GB) Monthly Price Sustained Random IOPS Sustained Throughput (MB/s)
Read Write Read Write
10 $0.40 * * * *
32 $1.28 24 48 3 3
64 $2.56 48 96 7 7
128 $4.00 96 192 15 15
256 $10.24 192 384 30 30
512 $20.48 384 768 61 61
1000 $40.00 750 1500 120 120
1500 $80.00 1500 3000 180 120
2048 $81.92 1536 3072 180 120
4000 $160.00 3000 6000 180 120
8192 $327.68 3000 12288 180 120
10000 $400.00 3000 15000 180 120
16384 $655.36 3000 15000 180 120
32768 $1,310.72 3000 15000 180 120
65536 $2,621.44 3000 15000 180 120

* Use this volume size only for boot volumes. IO bursting will be relied upon for any meaningful tasks.

SSD persistent disk

SSD persistent disk performance scales linearly until it reaches either the limits of the volume or the limits of each Compute Engine instance. In general, instances with greater numbers of vCPUs can achieve higher throughput and IOPS limits.

Unlike standard persistent disks, SSD persistent disks can achieve better IOPS performance on instances that have greater numbers of vCPUs.

Instance vCPU count Sustained Random IOPS Sustained Throughput (MB/s)
Read Write Read Write
15 or fewer vCPUs 15,000 15,000 240 240
16 to 31 vCPUs 25,000 25,000 480 240
32+ vCPUs 40,000 30,000 800 400

To improve SSD persistent disk performance on your existing instances, change the machine type of the instance to increase the per-vm limits and resize your persistent disks to increase IOPS and throughput per persistent disk.

Volume Size (GB) Monthly Price Sustained Random IOPS Sustained Throughput (MB/s)
Reads Writes Reads Writes
10 $1.70 300 300 4.8 4.8
32 $5.44 960 960 15 15
64 $10.88 1920 1920 30 30
128 $21.76 3840 3840 61 61
256 $43.52 7680 7680 122 122
500 $85.00 15000 15000 240 240
834 $141.78 25000 25000 400 400
1000 $170.00 30000 30000 480 400
1334 $226.78 40000 30000 640 400
1667 $283.39 40000 30000 800 400
2048 $348.16 40000 30000 800 400
4096 $696.32 40000 30000 800 400
8192 $1,392.64 40000 30000 800 400
16384 $5,570.56 40000 30000 800 400
32768 $5,570.56 40000 30000 800 400
65536 $11,141.12 40000 30000 800 400

Simultaneous reads and writes

For standard persistent disks, simultaneous reads and writes share the same performance limits. As your instance uses more read throughput or IOPS, it will be able to perform fewer writes. Instances that use more write throughput will be able to make fewer reads.

SSD persistent disks are capable of achieving their maximum throughput limits for both reads and writes simultaneously. For IOPS, however, SSD persistent disks cannot reach their maximum read and write limits simultaneously. To achieve maximum throughput limits during simultaneous reads and writes, optimize your IO size so that the volume can meet its throughput limits without reaching an IOPS bottleneck.

Instance IOPS limits for simultaneous reads and writes:

Standard persistent disk SSD persistent disk (8 vCPUs) SSD persistent disk (32+ vCPUs)
Read Write Read Write Read Write
3000 IOPS 0 IOPS 15000 IOPS 0 IOPS 40000 IOPS 0 IOPS
2250 IOPS 3750 IOPS 11250 IOPS 3750 IOPS 30000 IOPS 3700 IOPS
1500 IOPS 7500 IOPS 7500 IOPS 7500 IOPS 20000 IOPS 15000 IOPS
750 IOPS 11250 IOPS 3750 IOPS 11250 IOPS 10000 IOPS 22500 IOPS
0 IOPS 15000 IOPS 0 IOPS 15000 IOPS 0 IOPS 30000 IOPS

Instance throughput limits for simultaneous reads and writes:

Standard persistent disk SSD persistent disk (8 vCPUs) SSD persistent disk (32+ vCPUs)
Read Write Read Write Read Write
180 MB/s 0 MB/s 240 MB/s 240 MB/s 800 MB/s 400 MB/s
135 MB/s 30 MB/s
90 MB/s 60 MB/s
45 MB/s 90 MB/s
0 MB/s 120 MB/s

Network egress caps on write throughput

Each persistent disk write operation contributes to your virtual machine instance's cumulative network egress traffic. This means that persistent disk write operations are capped by your instance's network egress cap.

To calculate the maximum persistent disk write traffic that a virtual machine instance can issue, subtract an instance’s other network egress traffic from its 2 Gbit/s/vCPU network cap. The remaining throughput represents the throughput available to you for persistent disk write traffic.

Because persistent disk storage has 3.3x data redundancy, each write has to be written 3.3 times. This means that a single write operation counts as 3.3 I/O operations.

The following figures are the persistent disk I/O caps per virtual machine instance, based on the network egress caps for the virtual machine. These figures are based on an instance that has no additional IP traffic.

Standard persistent disk Solid-state persistent disks
Number of vCPUs Standard persistent disk write limit (MB/s) Standard volume size needed to reach limit (GB) SSD persistent disk write limit (MB/s) SSD persistent disk size needed to reach limit (GB)
1 78 650 78 163
2 120 1000 156 326
4 120 1000 240 500
8 120 1000 240 500
16 120 1000 240 500
32+ 120 1000 480 1000

To derive the figures above, divide the network egress cap – 2 Gbit/s, which is equivalent to 256 MB/s – by the data redundancy multiplier (3.3):

Number of max write I/O for one vCPU = 256 / 3.3 = ~78 MB/s of I/O issued by your standard persistent disk

Using the standard persistent disk write throughput/GB figure provided in the performance chart presented earlier, you can now derive an appropriate disk size as well:

Desired disk size = 78 / 0.12 = ~650 GB

Optimizing persistent disk and local SSD performance

You can optimize persistent disks and local SSDs to handle your data more efficiently.

Optimizing persistent disks

Persistent disks can give you the performance described in the disk type chart, but the virtual machine must drive sufficient usage to reach the performance caps. After you size your persistent disk volumes appropriately for your performance needs, your application and operating system might need some tuning.

In this section, we describe a few key elements that can be tuned for better performance and follow with discussion of how to apply some of them to specific types of workloads.

Disable lazy initialization and enable DISCARD commands

Persistent disks support DISCARD or TRIM commands, which allow operating systems to inform the disks when blocks are no longer in use. DISCARD support allows the operating system to mark disk blocks as no longer needed, without incurring the cost of zeroing out the blocks.

On most Linux operating systems, you enable DISCARD when you mount a persistent disk to your instance. Windows 2012 R2 instances enable DISCARD by default when you mount a persistent disk. Windows 2008 R2 does not support DISCARD.

Enabling DISCARD can boost general runtime performance, and it can also speed up the performance of your disk when it is first mounted. Formatting an entire disk volume can be time consuming. As such, so-called "lazy formatting" is a common practice. The downside of lazy formatting is that the cost is often then paid the first time the volume is mounted. By disabling lazy initialization and enabling DISCARD commands, you can get fast format and mount.

  • Disable lazy initialization and enable DISCARD during format by passing the following parameters to mkfs.ext4:

    -E lazy_itable_init=0,lazy_journal_init=0,discard
    

    The lazy_journal_init=0 parameter does not work on instances with CentOS 6 or RHEL 6 images. For those instances, format persistent disks without that parameter.

    -E lazy_itable_init=0,discard
    
  • Enable DISCARD commands on mount, pass the following flag to the mount command:

    -o discard
    

IO queue depth

Many applications have setting that influence their IO queue depth to tune performance. Higher queue depths increase IOPS, but can also increase latency. Lower queue depths decrease per-IO latency, but sometimes at the expense of IOPS.

Readahead cache

To improve IO performance, operating systems employ techniques such as readahead where more of a file than was requested is read into memory with the assumption that subsequent reads are likely to need that data. Higher readahead increases throughput, but at the expense of memory and IOPs. Lower readahead increases IOPS, but at the expense of throughput.

On linux systems, you can get and set the readahead value with the blockdev command:

$ sudo blockdev --getra /dev/
$ sudo blockdev --setra  /dev/

The readahead value is <desired_readahead_bytes> / 512 bytes.

For example, if you desire a 8 MB readahead, 8 MB is 8388608 bytes (8 * 1024 * 1024).

8388608 bytes / 512 bytes = 16384

And you would set:

$ sudo blockdev --setra 16384 /dev/

Free CPUs

Reading and writing to Persistent Disk requires CPU cycles from your virtual machine. To achieve very high, consistent IOPS levels requires having CPUs free to process IO.

IOPS-oriented workloads

Databases, whether SQL or NoSQL, have usage patterns of random access to data. The following are suggested for IOPS-oriented workloads:

  • Lower readahead values are typically suggested in best practices documents for MongoDB, Apache Cassandra, and other database applications

  • IO queue depth values of 1 per each 400-800 IOPS, up to a limit of 64 on large volumes

  • One free CPU for every 2000 random read IOPS and 1 free CPU for every 2500 random write IOPS

Throughput-oriented workloads

Streaming operations, such as a Hadoop job, benefit from fast sequential reads. As such, larger block sizes can increase streaming performance. The default block size on volumes is 4K. For throughput-oriented workloads, values of 256KB or above are recommended.

Optimizing SSD persistent disk performance

The performance by disk type chart describes the expected, achievable performance limits for solid-state persistent disks. To optimize your applications and virtual machine instances to achieve these speeds, use the following best practices:

  • Make sure your application is issuing enough I/O

    If your application is issuing less IOPS than the limit described in the chart above, you won't reach that level of IOPS. For example, on a 500 GB disk, the expected IOPS limit is 15,000 IOPS. However, if you issue less than that, or if you issue I/O operations that are larger than 16 KB, you won't achieve 15,000 IOPS.

  • Make sure to issue I/O with enough parallelism

    Use a high-enough queue depth that you are leveraging the parallelism of the operating system. If you issue 1000 IOPS but do so in a synchronous manner with a queue depth of 1, you will achieve far less IOPS than the limit described in the chart. At a minimum, your application should have a queue depth of at least 1 per every 400-800 IOPS.

  • Make sure there is enough available CPU on the virtual machine instance issuing the I/O

    If your virtual machine instance is starved for CPU, your application won't be able to manage the IOPS described above. As a rule of thumb, you should have one available CPU for every 2000-2500 IOPS of expected traffic.

  • Make sure your application is optimized for a reasonable temporal data locality on large disks

    If your application accesses data distributed across different parts of a disk over short period of time (hundreds of GB per vCPU), you won't achieve optimal IOPS. For best performance, optimize for temporal data locality, weighing factors like the fragmentation of the disk and the randomness of accessed parts of the disk.

  • Make sure the I/O scheduler in the operating system is configured to meet your specific needs

    On Linux-based systems, you can set the I/O scheduler to noop to achieve the highest number of IOPS on SSD-backed devices.

Optimizing Local SSDs

The performance by disk type chart describes the expected, achievable performance limits for local SSD devices. To optimize your applications and virtual machine instances to achieve these speeds, use the following best practices:

Use the Linux Guest Environment optimizations for Local SSD

By default, most Compute Engine-provided Linux images will automatically run an optimization script that configures the instance for peak local SSD performance. The script enables certain Queue sysfs files settings that enhance the overall performance of your machine and masks interrupt requests (IRQs) to specific virtual CPUs (vCPUs). This script only optimizes performance for Compute Engine local SSD devices.

Ubuntu, SLES, and older images might not be configured to include this performance optimization. If you are using any of these images, or an image older than v20141218, you can install the Linux Guest Environment to enable these optimizations instead.

Select the best image for NVMe or SCSI interfaces

Local SSDs operate best with either the NVMe or SCSI interface type depending on the image that you use on the boot disk for your instance. Choose an interface for your local SSD devices that works best with your boot disk image. If your instances connect to local SSDs using SCSI interfaces, you can enable multi-queue SCSI on the guest operating system to achieve optimal performance over the SCSI interface.

Enable multi-queue SCSI on instances with custom images and local SSDs

Some public images support multi-queue SCSI. If you require multi-queue SCSI capability on custom images that you import to your project, you must enable it yourself. Your imported Linux images can use multi-queue SCSI only if they include kernel version 3.19 or later.

To enable multi-queue SCSI on a custom image, import the image with the VIRTIO_SCSI_MULTIQUEUE guest operating system feature enabled and add an entry to your GRUB config:

CentOS

  1. Import your custom image using the API and include a guestOsFeatures item with a type value of VIRTIO_SCSI_MULTIQUEUE.

  2. Create an instance using your custom image and attach one or more local SSDs.

  3. Connect to your instance through SSH.

  4. Check the value of the /sys/module/scsi_mod/parameters/use_blk_mq file

    $ cat /sys/module/scsi_mod/parameters/use_blk_mq
    

    If the value of this file is Y, then multi-queue SCSI is already enabled on your imported image. If the value of the file is N, include scsi_mod.use_blk_mq=Y in the GRUB_CMDLINE_LINUX entry in your GRUB config file and restart the system.

    1. Open the /etc/default/grub GRUB config file in a text editor.

      $ sudo vi /boot/grub2/grub
      

    2. Add scsi_mod.use_blk_mq=Y to the GRUB_CMDLINE_LINUX entry.

      GRUB_CMDLINE_LINUX=" vconsole.keymap=us console=ttyS0,38400n8 vconsole.font=latarcyrheb-sun16 scsi_mod.use_blk_mq=Y"
      
    3. Save the config file.

    4. Run the grub2-mkconfig command to regenerate the GRUB file and complete the configuration.

      $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
      

    5. Reboot the instance.

      $ sudo reboot
      

Ubuntu

  1. Import your custom image using the API and include a guestOsFeatures item with a type value of VIRTIO_SCSI_MULTIQUEUE.

  2. Create an instance using your custom image and attach one or more local SSDs using the SCSI interface.

  3. Connect to your instance through SSH.

  4. Check the value of the /sys/module/scsi_mod/parameters/use_blk_mq file.

    $ cat /sys/module/scsi_mod/parameters/use_blk_mq
    

    If the value of this file is Y, then multi-queue SCSI is already enabled on your imported image. If the value of the file is N, include scsi_mod.use_blk_mq=Y in the GRUB_CMDLINE_LINUX entry in your GRUB config file and restart the system.

    1. Open the /etc/default/grub.d/50-cloudimg-settings.cfg GRUB config file in a text editor.

      $ sudo vi /etc/default/grub.d/50-cloudimg-settings.cfg
      

    2. Add scsi_mod.use_blk_mq=Y to the GRUB_CMDLINE_LINUX entry.

      GRUB_CMDLINE_LINUX="scsi_mod.use_blk_mq=Y"
      
    3. Save the config file.

    4. Run the update-grub command to regenerate the GRUB file and complete the configuration.

      $ sudo update-grub
      

    5. Reboot the instance.

      $ sudo reboot
      

Disable write cache flushing

Filesystems, databases, and other applications use cache flushing to ensure that data is committed to durable storage at various checkpoints. For most storage devices, this default makes sense. However, write cache flushes are fairly slow on local SSDs. You can increase the write performance for some applications by disabling automatic flush commands in those applications or by disabling flush options at the file system level.

Local SSDs always flush cached writes within two seconds regardless of the flush commands that you set for your filesystems and applications, so temporary hardware failures can cause you to lose only two seconds of cached writes at most. Permanent hardware failures can still cause loss of all data on the device whether the data is flushed or not, so you should still backup critical data to persistent disks or Cloud Storage buckets.

To disable write cache flushing on ext4 file systems, include the nobarrier in your mount options or in your /etc/fstab entries. For example:

$ sudo mount -o discard,defaults,nobarrier /dev/[LOCAL_SSD_ID] /mnt/disks/[MNT_DIR]

where: [LOCAL_SSD_ID] is the device ID for the local SSD that you want to mount.

Benchmarking local SSD performance

The local SSD performance figures provided in the Performance section were achieved using specific settings on the local SSD instance. If your instance is having trouble reaching these performance limits and you have already configured the instance using the recommended local SSD settings, you can compare your performance limits against the published limits by replicating the settings used by the Compute Engine team.

  1. Create a local SSD instance that has four or eight vCPUs for each device, depending on your workload. For example, if you had four local SSD devices attached to an instance, you should use a 16-vCPU machine type.

  2. Run the following script on your machine, which replicates the settings used to achieve these speeds. Note that the --bs parameter defines the block size, which affects the results for different types of read and write operations.

    # install dependencies
    sudo apt-get -y update
    sudo apt-get install -y build-essential git libtool gettext autoconf \
    libgconf2-dev libncurses5-dev python-dev fio
    
    # blkdiscard
    git clone git://git.kernel.org/pub/scm/utils/util-linux/util-linux.git
    cd util-linux/
    ./autogen.sh
    ./configure --disable-libblkid
    make
    sudo mv blkdiscard /usr/bin/
    sudo blkdiscard /dev/disk/by-id/google-local-ssd-0
    
    # full write pass - measures write bandwidth with 1M blocksize
    sudo fio --name=writefile --size=100G --filesize=100G \
    --filename=/dev/disk/by-id/google-local-ssd-0 --bs=1M --nrfiles=1 \
    --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 \
    --iodepth=200 --ioengine=libaio
    
    # rand read - measures max read IOPs with 4k blocks
    sudo fio --time_based --name=benchmark --size=100G --runtime=30 \
    --filename=/dev/disk/by-id/google-local-ssd-0 --ioengine=libaio --randrepeat=0 \
    --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
    --numjobs=4 --rw=randread --blocksize=4k --group_reporting
    
    # rand write - measures max write IOPs with 4k blocks
    sudo fio --time_based --name=benchmark --size=100G --runtime=30 \
    --filename=/dev/disk/by-id/google-local-ssd-0 --ioengine=libaio --randrepeat=0 \
    --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \
    --numjobs=4 --rw=randwrite --blocksize=4k --group_reporting
    

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Compute Engine Documentation