Google Cloud Platform

Compute Engine Disks: Price, Performance, and Persistence

Introduction

Google Compute Engine endeavors to provide nimble, reliable, and inexpensive compute power and storage for you to build your services. General Availability of Google Compute Engine brings a single, unified block storage offering with Persistent Disks. Persistent Disks provide straight-forward, consistent and reliable storage at a consistent and reliable price, removing the need for a separate local Scratch Disk offering.

This paper provides an overview of Persistent Disks on Compute Engine, and highlights key information for existing customers to navigate the transition.

Persistent Disk on Compute Engine

Compute Engine Persistent Disk provide network-attached block storage, much like a high speed and highly reliable Storage Area Network (SAN), to Compute Engine instances.

The core features of Google Persistent Disk are:

  • Resilience
  • Performance Consistency
  • Encryption
  • Checksums

In addition, by providing storage as a service, Persistent Disks offer manageability features that make your storage more nimble.

Persistent Disks can be

  • Detached from instances and reattached to new instances
  • Copied quickly to globally accessible snapshots
  • Attached to multiple nodes in read-only mode

Core Features

Resilience

Persistent Disks have built-in redundancy to protect your data against equipment failure and to remain available through datacenter maintenance events. Your instances, free of local Scratch Disks, can be moved by Google Live Migration to newer hardware without your intervention. This allows Google datacenters to be maintained at the highest level; software and hardware can be continually updated to ensure excellent performance and reliability for your cloud-based services.

Performance Consistency

Persistent Disk stripes data across many physical volumes within a zone to improve performance consistency. As a result, volumes of the same size in the same zone generally perform at the same level at a given time. Volume performance can vary over time as the aggregate IO load of the zone rises and falls, but by striping across many physical volumes, that variability is generally reduced.

Encryption

Persistent Disk data is always encrypted when it is outside of your virtual machine instance. When the instance sends a write to disk, the virtual machine monitor transparently encrypts it. When your instance reads data from disk, the virtual machine monitor transparently decrypts it. At rest and in flight, your Persistent Disk data is encrypted.

Checksums

Checksums are calculated for all Persistent Disk IOs so that we can ensure that what you read is what you wrote. Persistent Disk guarantees that data is written uncorrupted. If one of the replicas gets lost or corrupted later, the data is recovered from another replica. Only if all replicas become corrupt or lost will a read fail; Persistent Disk will never return corrupted data.

Persistent Disk manageability features

Persistent Disks live independently of the Compute Engine instance(s) they are attached to. This allows for Persistent Disks to be managed in ways not available to Scratch Disks. Scratch Disk space is tied to the life of an instance; if the instance is terminated for any reason, all Scratch Disk data is lost.

Can be detached from instances

Need to take down a Compute Engine instance to resize it or to replace or upgrade the running software? You won't lose the associated data; you won't need to rebuild the disk. Take the instance down and then bring up a new one with the disk attached.

For this type of maintenance, having fast persistent storage instead of local scratch means:

  • Less downtime
  • Faster upgrades
  • Simplified management
  • Lower overall network traffic and disk IO

Can be copied quickly to globally accessible snapshots

Persistent Disks can be backed up on a regular basis using Snapshots. Persistent Disk redundancy is within a zone, while snapshots are automatically replicated across multiple zones. If one zone is completely lost in a disaster, you can quickly recover by creating new Persistent Disks in another zone from snapshots you have created. Also if an end user deletes critical data, you can create a new Persistent Disk from a previous snapshot to recover it.

Repeated snapshots of the same Persistent Disk are incremental, meaning that only changes since the previous snapshot are stored. This allows for faster backups and reduces snapshot storage charges.

Can be attached to multiple nodes in read-only mode

Persistent Disks can be attached to multiple nodes in read-only mode (when not attached to any instance in read-write mode). You can distribute static content across multiple Compute Engine instances without incurring the cost of replicating the storage.

Persistent Disk ease of use

There are complexities and costs typically associated with block storage that Compute Engine Persistent Disks do away with.

No need to trade off price and performance for reliability and functionality

Compute Engine Persistent Disk prices are comparable to what was previously charged for Scratch. Persistent Disk performance in most cases will be better than Scratch. You don't need to be bound to the restrictions of a local disk to get higher performance.

No need to search for the best performing disks

Persistent Disks give consistent performance. There is no need to spin up a bunch of extra disks, test them for performance, and select only the best. There are no outliers to discard.

No need to stripe multiple disks

You have no need to aggregate multiple disks in a RAID array to get improved performance and reliability. Google already stripes your data across multiple disks, so you are getting the performance of parallel I/O and the reliability of replicated blocks.

No need to pre-warm volumes

When you add a volume, that volume is available to run at full speed. There is no need to pre-warm the volume; doing so needlessly consumes resources.

Selecting the right Persistent Disk

When you purchase Persistent Disk storage, you generally need to consider only how much space and what performance you need, and then buy the disk volume that gives you both. For some applications, you may also need to consider virtual machine limitations, discussed further below.

Persistent Disks are priced at $0.04 per GB per month[1] with no separate I/O costs. There is no need to estimate monthly usage to calculate budget for what you will spend on disks.

Persistent Disk performance increases with the size of the volume until it reaches the maximum performance available for a single virtual machine instance. The performance caps on reads and writes can be outlined with two distinct IO patterns, "small" and "large". For small reads and writes, the limiting factor is input/output operations per second (IOPs). For large reads and writes, the limiting factor is bandwidth.

The following table illustrates the performance limits for a 1 TB Persistent Disk volume for each of the IO patterns.

IO Pattern

Limit For 1 TB Volume

Small Random Reads

300 IOPs

Small Random Writes

1500 IOPs

Large Reads

120 MB/s

Large Writes

90 MB/s

The performance increases linearly with the size of the disk. Thus if you require 600 small read IOPs, then you would purchase at least a 2 TB volume.

Performance tip:

Many applications allow setting their IO queue depth to tune performance. Higher queue depths increase IOPs, but can also increase latency. Lower queue depths decrease per-IO latency, but sometimes at the expense of IOPs.

As a general guideline, an IO depth of 4-8 will provide the balance of IOPs and latency most applications need. Applications with very high IOPs needs, on large volumes, can benefit from IO depths as high as 64.

Each virtual machine instance has its own disk performance limits as illustrated in the following table.

IO Pattern

VM Limits

Small Reads

1 core: 1300 IOPs
2 core: 1500 IOPs
4 core: 2000 IOPs
8 core: 2000 IOPs

Small Writes

1 core: 1800 IOPs
2 core: 1800 IOPs
4 core: 2400 IOPs
8 core: 2400 IOPs

The IOPs limits are the smaller of what the volume can support and what the VM can support. If you have picked a volume size due to its IO capability, make sure to attach it to a VM that can handle that amount of IO.

VM throughput maximums also scale with volume size up to a VM limit. The VM limit for Persistent Disk bandwidth is 180 MB/s for reads and 120 MB/s for writes. Generally speaking, larger VMs will achieve higher bandwidth.

Examples

The following set of examples demonstrates how to select a Persistent Disk size based on performance requirements.

Example 1

Suppose you have a database installation (small writes) that requires a maximum random write rate of 300 IOPs:

So you would purchase a Persistent Disk of at least 200 GB.

Example 2

Suppose you have a database installation (small writes) that requires a maximum random write rate of 1800 IOPs:

So you would purchase a Persistent Disk of at least 1200 GB.

Example 3

Suppose you have a database installation (small writes) that requires a maximum random write rate of 4800 IOPs:

You would need Persistent Disks totalling at least 3200 GB. Note that the maximum performance for a single Compute Engine instance is either 1800 IOPs (1 or 2 cores) or 2400 IOPs (4 or 8 cores). You could then shard your data over:

  • three 1 or 2 core instances, each with a Persistent Disk of at least 1033 GB, or
  • two 4 or 8 core instances, each with a Persistent Disk of at least 1600 GB

Limits (size)

Persistent Disks can be up to 10 TB.

Changes for Existing Persistent Disk Users

Benefits

If you are an existing Persistent Disk user, you can now have better disk performance and more predictable monthly costs. You may need to use larger Persistent Disk volumes to achieve current performance, but note that with the decrease in Persistent Disk pricing, you will likely see a reduction in costs[2]. Your costs will also now be more predictable as there are no IO charges. See the Appendix for a detailed walkthrough to ensure your Persistent Disks are correctly sized.

What do you need to do?

Existing Persistent Disk volumes will continue to be unchanged for six months after General Availability (December 3rd, 2013). During that time, you should migrate your data to a new Persistent Disk volume. Six months after GA, old volumes will be automatically converted to use the new performance characteristics. If you do not migrate to a larger volume you may experience a performance drop. Instructions for migrating your data can be found here.

Changes for Existing Scratch Disk Users

Benefits

Most existing Scratch Disk users will get better performance with Persistent Disks, along with all of the benefits mentioned above. In addition, you are freed from the old Scratch Disk limitations.

Old Scratch Disk sizes were restricted by the size of the virtual machine instance. You were limited to allocating 420 GB, 870 GB, 1770 GB, or 3540 GB disks. You had to buy specific disk sizes, and if you needed larger disks you needed to buy a larger virtual machine. With Persistent Disks, you can attach volumes of any size to any size VM.

Compared to the old Scratch Disk offering, the cost per GB is slightly higher, but for most cases, the ability to create the right amount of space needed and select the right machine instance type makes this a price decrease.

What do you need to do?

Your existing virtual machines with Scratch Disks for boot volumes will continue to run until March 1, 2014, at which time they will be terminated. In addition, if you have a project created prior to General Availability (December 3rd, 2013), you will be able to continue to create new instances with Scratch Disks.

To migrate Scratch Disk data to a new Persistent Disk, follow the instructions below.

What about Scratch boot disk performance?

Boot volumes are typically small (10 GB by default) and would therefore have too low expected performance for boot or package installation. To address this specific concern, Google has implemented a burst capability for boot volumes that enables short bursts of IO. This applies to booting, package installation, and any other sporadic short term IO activity.

Conclusion

Google Compute Engine Persistent Disks provide a network block storage offering that is so high performing, consistent in implementation, and inexpensively priced that we believe it allows local Scratch Disks (and their limitations) to become a thing of the past.

Use Google Compute Engine Persistent Disks for their:

  • Performance (high and consistent)
  • Price (low and predictable)
  • Safety (redundancy, encryption, and checksum verification)
  • Management (simplicity and flexibility)

Google will continually improve Persistent Disk features and performance to satisfy ever more demanding workloads and customer requirements.

References and Resources

For more information on Google Compute Engine Persistent Disks see:

Appendix A: New volume sizing to transition from old Persistent Disk

Existing customers, we want to make sure your Persistent Disks are sized correctly after the transition for both your storage and disk performance needs. We also want to make sure that you are not paying more. Remember that performance is now tied to the size of your Persistent Disk, so you may need to increase the size of your Persistent Disk to get the same performance you currently get.

Let's take a quick look at what has changed:

  • Price
    • Dropped per-GB charges from $0.10 to $0.04 per GB per month
    • Eliminated per-IO charges from $0.10 per 1 million to $0.00
  • Performance
    • Previously fixed maximums at:
      • 250 read IOPs
      • 600 write IOPs
      • 120 MB/s read throughput
      • 120 MB/s write throughput
    • Now scales with volume size at:
      • 300 read IOPs / 1 TB
      • 1500 write IOPs / 1 TB
      • 120 MB/s read throughput / 1 TB
      • 90 MB/s write throughput / 1 TB

Happy with old Persistent Disk performance

As a reference point, we will walk through the case where old Persistent Disk performance was exactly what you needed and you are not looking to either raise or lower those caps.

As there are two major categories of performance limits (IOPs and throughput), we will look at the two scenarios separately.

Scenario 1: IOPs-oriented workloads

For IOPs-oriented workloads, we calculate the disk size that gives you the same performance as the old Persistent Disk. Note that most applications do not need even this level of performance under peak loads.

Random reads:

Random writes:

Thus if your application requires bursting up to 250 random read IOPs, then to have the same performance with new Persistent Disk as with old Persistent Disk, you need a volume of at least 833 GB.

Now compare cost

A new 833 GB Persistent Disk costs $33.32 per month (833 GB x $0.04/GB; for reference, an old Persistent Disk of size 833 GB would have cost $83.30 per month plus IO charges).

For $33.32 per month, what size disk could you have had with old Persistent Disk? This is a difficult comparison because you would have also paid IO costs with old Persistent disk. For simplicity of comparison, we will assume there was no IO cost previously, and only include the $0.10/GB monthly price.

An old Persistent Disk of size 333 GB without IO charges cost $33.30 per month.

Now take a look at the table below to determine what you need to do and to ensure your Persistent Disk costs have not gone up.

If your Persistent Disk is

Then

Larger than 833 GB

Do nothing and you will get the same or better performance and your cost will drop at least 60%.

Between 333 GB and 833 GB

Increase the size of your Persistent Disk to 833 GB and still save money (up to 60% on fixed charges, even more on IO charges).

If your Persistent Disk is less than 333 GB, then we need to refine the calculation by factoring in your old IO costs in order to verify that you are saving money when you increase the size of your volume to 833 GB.

See the instructions below on Calculating old Persistent Disk per-IO costs. When you have that value, then calculate:

and

If you are not saving money with new Persistent Disks, contact your Google sales representative.

Scenario 2: Throughput-oriented workloads

For throughput-oriented workloads, we calculate the disk size that gives you the same performance as the old Persistent Disk. Note that most applications do not need even this level of performance under peak loads.

Large reads:

Large writes:

Thus if your application requires peak write throughput of 120 MB/s, then to have the same performance with new Persistent Disk as with old Persistent Disk, you need a volume of at least 1333 GB.

Now compare cost

A new 1333 GB Persistent Disk costs $53.32 per month (1333 GB x $0.04/GB; for reference, an old Persistent Disk of size 1333 GB would have cost $133.30 per month plus IO charges).

For $53.32 per month, what size disk could you have had with old Persistent Disk? This is a difficult comparison because you would have also paid IO costs with old Persistent disk. For simplicity of comparison, we will assume there was no IO cost previously, and only include the $0.10/GB monthly price.

An old Persistent Disk of size 533 GB without IO charges would cost $53.30 per month.

Now take a look at the table below to determine what you need to do and to ensure your Persistent Disk costs have not gone up.

If your Persistent Disk is

Then

Larger than 1333 GB

Do nothing and you will get the same or better performance and your cost will drop at least 60%.

Between 533 GB and 1333 GB

Increase the size of your Persistent Disk to 1333 GB and still save money (up to 60% on fixed charges, even more on IO charges).

If your Persistent Disk is less than 533 GB, then we need to refine the calculation by factoring in your old IO costs in order to verify that you are saving money when you increase the size of your volume to 1333 GB.

See the instructions below on Calculating old Persistent Disk per-IO costs. When you have that value, then calculate:

and

If you are not saving money with new Persistent Disks, contact your Google sales representative.

Need something other than old Persistent Disk performance limits

The new Persistent Disk feature allows you to choose your IO limits by selecting the size of your disk. The preceding section walked through the changes necessary to mimic the old Persistent Disk static performance limits.

If your service requires lower limits than the old Persistent Disk or you now want to take advantage of higher performance limits, review the sizing procedure above. Your new price will be:

Calculating old Persistent Disk IO costs

For some amount of time that provides a representative sample for your application (1 hour, 1 day, or 1 week), monitor your Persistent Disk with a tool such as iostat. To install iostat on Google images, run:

$ sudo apt-get install systat

Tools such as iostat work with the system device name. You can find the device name by listing the Persistent Disk in /dev/disk/by-id. The name of the Persistent Disk in this directory will be prefixed with "google-". For example suppose your Persistent Disk is named "prod-db1-pd", then:

$ basename $(readlink /dev/disk/by-id/google-prod-db1-pd)
sdb

Now run iostat for your desired duration. In this example, we run it for 60 seconds:

$ iostat -d $(basename $(readlink /dev/disk/by-id/google-prod-db1-pd)) 60
Linux 3.3.8-gcg-201309241443  (prod-db1-pd)  11/07/13  _x86_64_  (1 CPU)

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdb               5.11         0.03        21.88       1840    1435752
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdb            1315.85         0.00      5266.60          0     316628

Take the "tps" value from the last line of output (this represents operations per second) and multiply it by the average number of seconds in a month, 2,629,800[3]:

With the total operations in a month, calculate the cost:

Thus 1315.85 operations per second sustained would have cost $346.04 per month with old Persistent Disk IO charges.

Appendix B: New volume sizing to transition from Scratch Disk

Existing Scratch Disk customers, we want to make sure your Persistent Disks are sized correctly after the transition for both your storage and disk performance needs.

Recall that the new Persistent Disk per-GB price is higher than Scratch ($0.04 vs. $0.029), but you don't need to allocate disk you don't need. That fixed price of Scratch Disks was part of the per-hour instance price. This price will be represented in calculations below as the "Instance Savings" of moving from the "Local Disk" (-d) instances to the "Diskless" instance types.

Happy with Scratch Disk performance

The maximum performance for random read and write IOPs for the new Persistent Disk is significantly higher than for Scratch. The maximum performance for total throughput is very similar to that for Scratch.

As a reference point, we will walk through the case where Scratch Disk performance was exactly what you needed and you are not looking to either raise or lower those caps.

As there are two major categories of performance limits (IOPs and throughput), we will look at the two scenarios separately.

Scenario 1: IOPs-oriented workloads

The table below provides a recommended minimum Persistent Disk size for each of the n1-standard instance types. The recommended size is intended to achieve at least the same random read and random write performance as Scratch. The table also contains, the monthly Persistent Disk cost for the recommended size and the monthly fixed cost for Scratch Disk.

The Scratch Disk cost is calculated from the instance pricing difference between the instance type ("Diskless") and its "-d" ("Local Disk") version.

Instance Type

Minimum Persistent Disk

Persistent Disk Cost

Scratch Disk Cost

n1-standard-1

60 GB

$2.40

$12.42

n1-standard-2

125 GB

$5.00

$25.57

n1-standard-4

250 GB

$10.00

$50.40

n1-standard-8

500 GB

$20.00

$100.80

Scenario 2: Throughput-oriented workloads

The table below provides a recommended minimum Persistent Disk size for each of the n1-standard instance types. The recommended size is intended to achieve at least the same throughput performance as Scratch. The table also contains, the monthly Persistent Disk cost for the recommended size and the monthly fixed cost for Scratch Disk.

The Scratch Disk cost is calculated from the instance pricing difference between the instance type ("Diskless") and its "-d" ("Local Disk") version.

Instance Type

Minimum Persistent Disk

Persistent Disk Cost

Scratch Disk Cost

n1-standard-1

278 GB

$11.12

$12.42

n1-standard-2

656 GB

$26.24

$25.57

n1-standard-4

1333 GB

$53.32

$50.40

To achieve a similar maximum throughput to the n1-standard-8 with Scratch Disk, your workload must be sharded across multiple virtual machine instances.

If you have any questions about Persistent Disk sizing and cost in transitioning from Scratch Disks, contact your Google sales representative.

Appendix C: Moving data from existing data disks

As a best practice it is always recommended that one use a data disk separate from a boot disk. It is also a best practice to have procedures to recreate a boot disk on instance failure and procedures to recreate data from durable storage.

The instructions here are provided for copying an old Scratch Disk or an old Persistent Disk being used as a data disk. Should you already have procedures for restoring the data from backup or other durable storage, you may choose to do so.

Copying Data

Whether moving data from a Scratch Disk or an old Persistent Disk, the steps are the same:

  1. Create a new Persistent Disk of the right size
  2. Attach the new Persistent Disk to an existing instance
  3. Format and mount the new Persistent Disk
  4. Stop any services that might be writing to the source disk
  5. Remount the source disk read only
  6. Copy the data from the old disk to the new disk
  7. Unmount both disks
  8. Mount the new Persistent Disk in place of the old disk

All of the steps can be performed on the virtual machine instance. If service accounts are not enabled on the virtual machine instance, gcutil will prompt you to complete an authentication workflow in order to complete steps 2 and 3. Alternatively steps 2 and 3 may be completed on any machine that has already performed gcutil authentication for your project:

  1. Set environment variables
    NEWDISK_SIZE_GB=size
    NEWDISK_NAME=newdisk
    NEWDISK_MOUNT=/mnt/newdata
    NEWDISK_ZONE=zone
    
    # For a Scratch Disk, olddisk should be something like "ephemeral-disk-0"
    # For a Persistent Disk, olddisk should be the name of the Persistent Disk
    OLDDISK_NAME=olddisk
    OLDDISK_MOUNT=/mnt/data
    
  2. Create a new Persistent Disk
    gcutil adddisk ${NEWDISK_NAME} \
      --zone=${NEWDISK_ZONE} --size_gb=${NEWDISK_SIZE_GB}
    
  3. Attach the Persistent Disk
    gcutil attachdisk $(hostname) --disk=${NEWDISK_NAME}
    
  4. Format and mount the new Persistent Disk
    sudo mkdir ${NEWDISK_MOUNT}
    
    NEWDISK_DEVICE=$(basename $(readlink /dev/disk/by-id/google-${NEWDISK_NAME}))
    sudo /usr/share/google/safe_format_and_mount -m "mkfs.ext4 -F" \
      /dev/${NEWDISK_DEVICE} ${NEWDISK_MOUNT}
    
  5. Stop any services that may be writing to the old disk
  6. Remount the old disk in read-only mode (if possible)
    OLDDISK_DEVICE=$(basename $(readlink /dev/disk/by-id/google-${OLDDISK_NAME}))
    sudo mount -o remount,ro /dev/${OLDDISK_DEVICE}
  7. Copy the data from the old disk to the new Persistent Disk
    sudo cp -rax ${OLDDISK_MOUNT}/* ${NEWDISK_MOUNT}
    
  8. Unmount the old and new disks
    sudo umount ${OLDDISK_MOUNT}
    sudo umount ${NEWDISK_MOUNT}
  9. Mount the new disk at the old mount point
    sudo mount /dev/${NEWDISK_DEVICE} ${OLDDISK_MOUNT}
    
  10. Remove the temporary new disk mount point
    sudo rmdir ${NEWDISK_MOUNT}
    



[1] As of December 3rd, 2013.

[2] Prices may be higher for existing small Persistent Disks that have sporadic bursty IO needs.

[3] (365.25 days/year / 12 months/year) * 24 hours/day * 60 minutes/hour * 60 seconds/minute

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.