Persistent disks are the most common storage options due to their price, performance, and predictability. However, you can create instances with local SSDs for even greater performance and lower latency, but without the data redundancy and durability that you get from persistent disks. When you configure a storage option for applications that run on your instances, use the following processes:
- Determine how much space you need.
- Determine what performance characteristics your applications require.
- Configure your instances to optimize storage performance.
This document discusses block storage options that you can attach to Compute Engine instances. To see a complete list of storage options on Google Cloud Platform, read Choosing a storage option.
Block storage performance comparison
Consider your storage size and performance requirements to help you determine the correct disk type and size for your instances. Performance requirements for a given application are typically separated into two distinct IO patterns.
- Small reads and writes
- Large reads and writes
For small reads and writes, the limiting factor is random input/output operations per second IOPS.
For large reads and writes, the limiting factor is throughput.
|Standard persistent disks||SSD persistent disks||Local SSD (SCSI)||Local SSD (NVMe)|
|Maximum sustained IOPS|
|Read IOPS per GB||0.75||30||266.7||453.3|
|Write IOPS per GB||1.5||30||186.7||240|
|Read IOPS per instance||3,000||15,000 - 40,000*||400,000||680,000|
|Write IOPS per instance||15,000||15,000 - 30,000*||280,000||360,000|
|Maximum sustained throughput (MB/s)|
|Read throughput per GB||0.12||0.48||1.04||1.77|
|Write throughput per GB||0.12||0.48||0.73||0.94|
|Read throughput per instance||180||240 - 800*||1,560||2,650|
|Write throughput per instance||120||240 - 400*||1,090||1,400|
* SSD persistent disks can achieve greater IOPS and throughput performance on instances with greater numbers of vCPUs. Read SSD persistent disk performance limits for details.
Comparing persistent disk to a physical hard drive
When you specify the size of your persistent disks, consider how these disks compare to traditional physical hard drives. The following tables compare standard persistent disks and SSD persistent disks to the typical performance that you would expect from a 7200 RPM SATA drive, which typically achieves 75 IOPS or 120 MB/s.
|IO Type||IO Pattern||Size required to match a 7200 RPM SATA drive|
|Standard persistent disk||SSD persistent disk|
|Small random reads||75 small random reads||100 GB||3 GB|
|Small random writes||75 small random writes||50 GB||3 GB|
|Streaming large reads||120 MB/s streaming reads||1000 GB||250 GB|
|Streaming large writes||120 MB/s streaming writes||1000 GB||250 GB|
Size, Price, and Performance Summary
While you have several inputs to consider when you select a volume type and size for your application, one factor you do not need to consider is the price of using your volume. Persistent Disk has no per-IO costs, so there is no need to estimate monthly I/O to calculate budget for what you will spend on disks.
Consider only the relative costs of standard persistent disks compared to SSD persistent disks. Standard persistent disks are priced at $0.040 per GB and SSD persistent disks are priced at $0.170 per GB. However, performance caps increase with the size of the volume, so look at the price per IOPS for IOPS oriented workloads.
Standard persistent disks are approximately $0.133 per random read IOPS and $0.0266 per random write IOPS. SSD persistent disks are $0.0057 per random read IOPS and $0.0057 per random write IOPS. The price per IOPS for SSD persistent disks is true up to the point where they reach the IOPS limits of the instance or the vCPU count for that instance.
SSD persistent disks reach their limit of 40,000 random read IOPS at 1334 GB and 30,000 random write IOPS at 1000 GB. In contrast, standard persistent disks reach their limit of 3,000 random read IOPS at 4 TB and 15,000 random write IOPS at 10 TB.
Standard persistent disk
Standard persistent disk performance scales linearly up to the volume performance limits. The vCPU count for your instance does not limit the performance of standard persistent disks.
Standard persistent disk IOPS and throughput performance increases linearly with the size of the disk until it reaches the following per-instance limits:
- Read throughput: Up to 180 MB/s at a 1.5 TB disk size.
- Write throughput: Up to 120 MB/s at a 1 TB disk size.
- Read IOPS: Up to 3,000 IOPS at a 4 TB disk size.
- Write IOPS: Up to 15,000 IOPS at a 10 TB disk size.
To gain persistent disk performance benefits on your existing instances, resize your persistent disks to increase IOPS and throughput per persistent disk.
|Volume Size (GB)||Monthly Price||Sustained Random IOPS||Sustained Throughput (MB/s)|
* Use this volume size only for boot volumes. IO bursting will be relied upon for any meaningful tasks.
SSD persistent disk
SSD persistent disk performance scales linearly until it reaches either the limits of the volume or the limits of each Compute Engine instance. In general, instances with greater numbers of vCPUs can achieve higher throughput and IOPS limits.
Unlike standard persistent disks, SSD persistent disks can achieve better IOPS performance on instances that have greater numbers of vCPUs.
|Instance vCPU count||Sustained Random IOPS||Sustained Throughput (MB/s)|
|15 or fewer vCPUs||15,000||15,000||240||240|
|16 to 31 vCPUs||25,000||25,000||480||240|
To improve SSD persistent disk performance on your existing instances, change the machine type of the instance to increase the per-vm limits and resize your persistent disks to increase IOPS and throughput per persistent disk.
|Volume Size (GB)||Monthly Price||Sustained Random IOPS||Sustained Throughput (MB/s)|
Simultaneous reads and writes
For standard persistent disks, simultaneous reads and writes share the same performance limits. As your instance uses more read throughput or IOPS, it will be able to perform fewer writes. Instances that use more write throughput will be able to make fewer reads.
SSD persistent disks are capable of achieving their maximum throughput limits for both reads and writes simultaneously. For IOPS, however, SSD persistent disks cannot reach their maximum read and write limits simultaneously. To achieve maximum throughput limits during simultaneous reads and writes, optimize your IO size so that the volume can meet its throughput limits without reaching an IOPS bottleneck.
Instance IOPS limits for simultaneous reads and writes:
|Standard persistent disk||SSD persistent disk (8 vCPUs)||SSD persistent disk (32+ vCPUs)|
|3000 IOPS||0 IOPS||15000 IOPS||0 IOPS||40000 IOPS||0 IOPS|
|2250 IOPS||3750 IOPS||11250 IOPS||3750 IOPS||30000 IOPS||3700 IOPS|
|1500 IOPS||7500 IOPS||7500 IOPS||7500 IOPS||20000 IOPS||15000 IOPS|
|750 IOPS||11250 IOPS||3750 IOPS||11250 IOPS||10000 IOPS||22500 IOPS|
|0 IOPS||15000 IOPS||0 IOPS||15000 IOPS||0 IOPS||30000 IOPS|
Instance throughput limits for simultaneous reads and writes:
|Standard persistent disk||SSD persistent disk (8 vCPUs)||SSD persistent disk (32+ vCPUs)|
|180 MB/s||0 MB/s||240 MB/s||240 MB/s||800 MB/s||400 MB/s|
|135 MB/s||30 MB/s|
|90 MB/s||60 MB/s|
|45 MB/s||90 MB/s|
|0 MB/s||120 MB/s|
Network egress caps on write throughput
Each persistent disk write operation contributes to your virtual machine instance's cumulative network egress traffic. This means that persistent disk write operations are capped by your instance's network egress cap.
To calculate the maximum persistent disk write traffic that a virtual machine instance can issue, subtract an instance’s other network egress traffic from its 2 Gbit/s/vCPU network cap. The remaining throughput represents the throughput available to you for persistent disk write traffic.
Because persistent disk storage has 3.3x data redundancy, each write has to be written 3.3 times. This means that a single write operation counts as 3.3 I/O operations.
The following figures are the persistent disk I/O caps per virtual machine instance, based on the network egress caps for the virtual machine. These figures are based on an instance that has no additional IP traffic.
|Standard persistent disk||Solid-state persistent disks|
|Number of vCPUs||Standard persistent disk write limit (MB/s)||Standard volume size needed to reach limit (GB)||SSD persistent disk write limit (MB/s)||SSD persistent disk size needed to reach limit (GB)|
To derive the figures above, divide the network egress cap – 2 Gbit/s, which is equivalent to 256 MB/s – by the data redundancy multiplier (3.3):
Number of max write I/O for one vCPU = 256 / 3.3 = ~78 MB/s of I/O issued by your standard persistent disk
Using the standard persistent disk write throughput/GB figure provided in the performance chart presented earlier, you can now derive an appropriate disk size as well:
Desired disk size = 78 / 0.12 = ~650 GB
Optimizing persistent disk and local SSD performance
You can optimize persistent disks and local SSDs to handle your data more efficiently.
Optimizing persistent disks
Persistent disks can give you the performance described in the disk type chart, but the virtual machine must drive sufficient usage to reach the performance caps. After you size your persistent disk volumes appropriately for your performance needs, your application and operating system might need some tuning.
In this section, we describe a few key elements that can be tuned for better performance and follow with discussion of how to apply some of them to specific types of workloads.
Disable lazy initialization and enable DISCARD commands
Persistent disks support DISCARD (or TRIM commands, which allow operating systems to inform the disks when blocks are no longer in use. DISCARD support allows the operating system to mark disk blocks as no longer needed, without incurring the cost of zeroing out the blocks.
On most Linux operating systems, you enable DISCARD when you mount a persistent disk to your instance. Windows 2012 R2 instances enable DISCARD by default when you mount a persistent disk. Windows 2008 R2 does not support DISCARD.
Enabling DISCARD can boost general runtime performance, and it can also speed up the performance of your disk when it is first mounted. Formatting an entire disk volume can be time consuming. As such, so-called "lazy formatting" is a common practice. The downside of lazy formatting is that the cost is often then paid the first time the volume is mounted. By disabling lazy initialization and enabling DISCARD commands, you can get fast format and mount.
Disable lazy initialization and enable DISCARD during format by passing the following parameters to mkfs.ext4:
Enable DISCARD commands on mount, pass the following flag to the mount command:
IO queue depth
Many applications have setting that influence their IO queue depth to tune performance. Higher queue depths increase IOPS, but can also increase latency. Lower queue depths decrease per-IO latency, but sometimes at the expense of IOPS.
To improve IO performance, operating systems employ techniques such as readahead where more of a file than was requested is read into memory with the assumption that subsequent reads are likely to need that data. Higher readahead increases throughput, but at the expense of memory and IOPs. Lower readahead increases IOPS, but at the expense of throughput.
On linux systems, you can get and set the readahead value with the blockdev command:
$ sudo blockdev --getra /dev/
$ sudo blockdev --setra
The readahead value is
<desired_readahead_bytes> / 512 bytes.
For example, if you desire a 8 MB readahead, 8 MB is 8388608 bytes (8 * 1024 * 1024).
8388608 bytes / 512 bytes = 16384
And you would set:
$ sudo blockdev --setra 16384 /dev/
Reading and writing to Persistent Disk requires CPU cycles from your virtual machine. To achieve very high, consistent IOPS levels requires having CPUs free to process IO.
Databases, whether SQL or NoSQL, have usage patterns of random access to data. The following are suggested for IOPS-oriented workloads:
IO queue depth values of 1 per each 400-800 IOPS, up to a limit of 64 on large volumes
One free CPU for every 2000 random read IOPS and 1 free CPU for every 2500 random write IOPS
Streaming operations, such as a Hadoop job, benefit from fast sequential reads. As such, larger block sizes can increase streaming performance. The default block size on volumes is 4K. For throughput-oriented workloads, values of 256KB or above are recommended.
Optimizing SSD persistent disk performance
The performance by disk type chart describes the expected, achievable performance limits for solid-state persistent disks. To optimize your applications and virtual machine instances to achieve these speeds, use the following best practices:
Make sure your application is issuing enough I/O
If your application is issuing less IOPS than the limit described in the chart above, you won't reach that level of IOPS. For example, on a 500 GB disk, the expected IOPS limit is 15,000 IOPS. However, if you issue less than that, or if you issue I/O operations that are larger than 16 KB, you won't achieve 15,000 IOPS.
Make sure to issue I/O with enough parallelism
Use a high-enough queue depth that you are leveraging the parallelism of the operating system. If you issue 1000 IOPS but do so in a synchronous manner with a queue depth of 1, you will achieve far less IOPS than the limit described in the chart. At a minimum, your application should have a queue depth of at least 1 per every 400-800 IOPS.
Make sure there is enough available CPU on the virtual machine instance issuing the I/O
If your virtual machine instance is starved for CPU, your application won't be able to manage the IOPS described above. As a rule of thumb, you should have one available CPU for every 2000-2500 IOPS of expected traffic.
Make sure your application is optimized for a reasonable temporal data locality on large disks
If your application accesses data distributed across different parts of a disk over short period of time (hundreds of GB per vCPU), you won't achieve optimal IOPS. For best performance, optimize for temporal data locality, weighing factors like the fragmentation of the disk and the randomness of accessed parts of the disk.
Make sure the I/O scheduler in the operating system is configured to meet your specific needs
On Linux-based systems, you can set the I/O scheduler to
noopto achieve the highest number of IOPS on SSD-backed devices.
Optimizing Local SSDs
The performance by disk type chart describes the expected, achievable performance limits for local SSD devices. To optimize your applications and virtual machine instances to achieve these speeds, use the following best practices:
Use the Linux Guest Environment optimizations for Local SSD
By default, most Compute Engine-provided Linux images will automatically run an optimization script that configures the instance for peak local SSD performance. The script enables certain Queue sysfs files settings that enhance the overall performance of your machine and masks interrupt requests (IRQs) to specific virtual CPUs (vCPUs). This script only optimizes performance for Compute Engine local SSD devices.
Ubuntu, SLES, and older images might not be configured to include this performance optimization. If you are using any of these images, or an image older than v20141218, you can install the Linux Guest Environment to enable these optimizations instead.
Select the best image for NVMe or SCSI interfaces
Local SSDs operate best with either the NVMe or SCSI interface type depending on the image that you use on the boot disk for your instance. Choose an interface for your local SSD devices that works best with your boot disk image.
Disable write cache flushing
Filesystems, databases, and other applications use cache flushing to ensure that data is committed to durable storage at various checkpoints. For most storage devices, this default makes sense. However, write cache flushes are fairly slow on local SSDs. You can increase the write performance for some applications by disabling automatic flush commands in those applications or by disabling flush options at the file system level.
Local SSDs always flush cached writes within two seconds regardless of the flush commands that you set for your filesystems and applications, so temporary hardware failures can cause you to lose only two seconds of cached writes at most. Permanent hardware failures can still cause loss of all data on the device whether the data is flushed or not, so you should still backup critical data to persistent disks or Cloud Storage buckets.
To disable write cache flushing on
ext4 file systems, include the
in your mount options or in your
/etc/fstab entries. For example:
$ sudo mount -o discard,defaults,nobarrier /dev/[LOCAL_SSD_ID] /mnt/disks/[MNT_DIR]
[LOCAL_SSD_ID] is the device ID for the local SSD that you want to
Benchmarking local SSD performance
The local SSD performance figures provided in the Performance section were achieved using specific settings on the local SSD instance. If your instance is having trouble reaching these performance limits and you have already configured the instance using the recommended local SSD settings, you can compare your performance limits against the published limits by replicating the settings used by the Compute Engine team.
Create a local SSD instance that has four or eight vCPUs for each device, depending on your workload. For example, if you had four local SSD devices attached to an instance, you should use a 16-vCPU machine type.
Run the following script on your machine, which replicates the settings used to achieve these speeds. Note that the
--bsparameter defines the block size, which affects the results for different types of read and write operations.
# install dependencies sudo apt-get -y update sudo apt-get install -y build-essential git libtool gettext autoconf \ libgconf2-dev libncurses5-dev python-dev fio # blkdiscard git clone git://git.kernel.org/pub/scm/utils/util-linux/util-linux.git cd util-linux/ ./autogen.sh ./configure --disable-libblkid make sudo mv blkdiscard /usr/bin/ sudo blkdiscard /dev/disk/by-id/google-local-ssd-0 # full write pass - measures write bandwidth with 1M blocksize sudo fio --name=writefile --size=100G --filesize=100G \ --filename=/dev/disk/by-id/google-local-ssd-0 --bs=1M --nrfiles=1 \ --direct=1 --sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1 \ --iodepth=200 --ioengine=libaio # rand read - measures max read IOPs with 4k blocks sudo fio --time_based --name=benchmark --size=100G --runtime=30 \ --filename=/dev/disk/by-id/google-local-ssd-0 --ioengine=libaio --randrepeat=0 \ --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \ --numjobs=4 --rw=randread --blocksize=4k --group_reporting # rand write - measures max write IOPs with 4k blocks sudo fio --time_based --name=benchmark --size=100G --runtime=30 \ --filename=/dev/disk/by-id/google-local-ssd-0 --ioengine=libaio --randrepeat=0 \ --iodepth=128 --direct=1 --invalidate=1 --verify=0 --verify_fatal=0 \ --numjobs=4 --rw=randwrite --blocksize=4k --group_reporting