Compute Engine offers several storage options for your VMs. Each of the following storage options has unique price and performance characteristics:
- Persistent Disk volumes provide high-performance and redundant network
storage. Each Persistent Disk volume is
striped
across hundreds of physical disks.
- By default, VMs use zonal Persistent Disk, and store your
data on volumes located within a single zone, such as
us-west1-c
. - You can also create regional Persistent Disk volumes, which synchronously replicate data between disks located in two zones and provide protection if a zone becomes unavailable.
- By default, VMs use zonal Persistent Disk, and store your
data on volumes located within a single zone, such as
- Google Cloud Hyperdisk volumes offer the fastest redundant network storage for Compute Engine, with configurable performance and volumes that can be dynamically resized.
- Local SSD disks are physical drives attached directly to the same server as your VM. They can offer better performance, but are ephemeral.
- Cloud Storage buckets provide affordable object storage.
- You can also use Filestore with your VMs for high performance file storage.
Each storage option has unique price and performance characteristics. For cost comparisons, see disk pricing. If you are not sure which option to use, the most common solution is to add a Persistent Disk volume to your VM.
Introduction
By default, each Compute Engine VM has a single boot disk that contains the operating system. The boot disk data is typically stored on a Persistent Disk volume. When your applications require additional storage space, you can provision one or more of the following storage volumes to your VM.
To learn more about each storage option, review the following table:
Balanced Persistent Disk |
SSD Persistent Disk |
Standard Persistent Disk |
Extreme Persistent Disk |
Hyperdisk Balanced (Preview) | Hyperdisk Extreme | Hyperdisk Throughput | Local SSDs | Cloud Storage buckets | |
---|---|---|---|---|---|---|---|---|---|
Storage type | Cost-effective and reliable block storage | Fast and reliable block storage | Efficient and reliable block storage | Highest performance Persistent Disk block storage option with customizable IOPS | High performance for demanding workloads with a friendly budget | Fastest block storage option with customizable IOPS | Cost-effective and throughput-oriented block storage with customizable throughput | High performance local block storage | Affordable object storage |
Minimum capacity per disk | Zonal: 10 GiB Regional: 10 GiB |
Zonal: 10 GiB Regional: 10 GiB |
Zonal: 10 GiB Regional: 200 GiB |
500 GiB | 4 GiB | 64 GiB | 2 TiB | 375 GiB | n/a |
Maximum capacity per disk | 64 TiB | 64 TiB | 64 TiB | 64 TiB | 64 TiB | 64 TiB | 32 TiB | 375 GiB | n/a |
Capacity increment | 1 GiB | 1 GiB | 1 GiB | 1 GiB | 1 GiB | 1 GiB | 1 GiB | Depends on the machine type† | n/a |
Maximum capacity per VM | 257 TiB* | 257 TiB* | 257 TiB* | 257 TiB* | 512 TiB* | 512 TiB* | 512 TiB* | 9 TiB | Almost infinite |
Scope of access | Zone | Zone | Zone | Zone | Zone | Zone | Zone | Instance | Global |
Data redundancy | Zonal and multi-zonal | Zonal and multi-zonal | Zonal and multi-zonal | Zonal | Zonal | Zonal | Zonal | None | Regional, dual-regional or multi-regional |
Encryption at rest | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
Custom encryption keys | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes |
How-to | Add an extreme Persistent Disk | Add a Hyperdisk | Add a Local SSD | Connect a bucket |
In addition to the storage options that Google Cloud provides, you can deploy alternative storage solutions on your VMs.
- Create a file server or distributed file system on Compute Engine to use as a network file system with NFSv3 and SMB3 capabilities.
- Mount a RAM disk within the VM memory to create a block storage volume with high throughput and low latency.
Block storage resources have different performance characteristics. Consider your storage size and performance requirements when determining the correct block storage type for your VMs.
For information about performance limits for each disk type, see:
Persistent Disk volumes created in multi-writer mode have specific IOPS and throughput limits. For details, see performance of Persistent Disk in multi-writer mode.
Persistent Disk
Persistent Disk volumes are durable network storage devices that your virtual machine (VM) instances can access like physical disks in a desktop or a server. The data on each Persistent Disk volume is distributed across several physical disks. Compute Engine manages the physical disks and the data distribution for you to ensure redundancy and optimal performance.
Persistent Disk volumes are located independently from your VM, so you can detach or move Persistent Disk volumes to keep your data even after you delete your VMs. Persistent Disk performance scales automatically with size, so you can resize your existing Persistent Disk volumes or add more Persistent Disk volumes to a VM to meet your performance and storage space requirements.
Persistent Disk types
When you configure a persistent disk, you can select one of the following disk types:
- Balanced persistent disks (
pd-balanced
)- An alternative to performance (pd-ssd) persistent disks
- Balance of performance and cost. For most VM shapes, except very large ones, these disks have the same maximum IOPS as SSD persistent disks and lower IOPS per GB. This disk type offers performance levels suitable for most general-purpose applications at a price point between that of standard and performance (pd-ssd) persistent disks.
- Backed by solid-state drives (SSD).
- Performance (SSD) persistent disks (
pd-ssd
)- Suitable for enterprise applications and high-performance databases that require lower latency and more IOPS than standard persistent disks provide.
- Designed for single-digit millisecond latencies; the observed latency is application specific.
- Backed by solid-state drives (SSD).
- Standard persistent disks (
pd-standard
)- Suitable for large data processing workloads that primarily use sequential I/Os.
- Backed by standard hard disk drives (HDD).
- Extreme persistent disks (
pd-extreme
)- Offer consistently high performance for both random access workloads and bulk throughput.
- Designed for high-end database workloads.
- Allow you to provision the target IOPS.
- Backed by solid-state drives (SSD).
- Available with a limited number of machine types.
If you create a disk in the Google Cloud console, the default disk type is
pd-balanced
. If you create a disk using the gcloud CLI or the
Compute Engine API, the default disk type is pd-standard
.
For information about machine type support, refer to the following:
Durability of Persistent Disk
Disk durability represents the probability of data loss, by design, for a typical disk in a typical year, using a set of assumptions about hardware failures, the likelihood of catastrophic events, isolation practices and engineering processes in Google data centers, and the internal encodings used by each disk type. Persistent Disk data loss events are extremely rare and have historically been the result of coordinated hardware failures, software bugs, or a combination of the two. Google also takes many steps to mitigate the industry-wide risk of silent data corruption. Human error by a Google Cloud customer, such as when a customer accidentally deletes a disk, is outside the scope of Persistent Disk durability.
There is a very small risk of data loss occurring with a regional persistent disk due to its internal data encodings and replication. Regional persistent disks provide twice as many replicas as zonal Persistent Disk, with their replicas distributed between two zones in the same region, so they provide high availability and can be used for disaster recovery if an entire data center is lost and cannot be recovered (although that has never happened). The additional replicas in a second zone can be accessed immediately if a primary zone becomes unavailable during a long outage.
Note that durability is in the aggregate for each disk type, and does not represent a financially-backed service level agreement (SLA).
The table below shows durability for each disk type's design. 99.999% durability means that with 1,000 disks, you would likely go a hundred years without losing a single one.
Zonal standard Persistent Disk | Zonal balanced Persistent Disk | Zonal SSD Persistent Disk | Zonal extreme Persistent Disk | Regional standard Persistent Disk | Regional balanced Persistent Disk | Regional SSD Persistent Disk |
---|---|---|---|---|---|---|
Better than 99.99% | Better than 99.999% | Better than 99.999% | Better than 99.9999% | Better than 99.999% | Better than 99.9999% | Better than 99.9999% |
Zonal Persistent Disk
Ease of use
Compute Engine handles most disk management tasks for you so that you do not need to deal with partitioning, redundant disk arrays, or subvolume management. Generally, you don't need to create larger logical volumes, but you can extend your secondary attached Persistent Disk capacity to 257 TiB per VM and apply these practices to your Persistent Disk volumes if you want. You can save time and get the best performance if you format your Persistent Disk volumes with a single file system and no partition tables.
If you need to separate your data into multiple unique volumes, create additional disks rather than dividing your existing disks into multiple partitions.
When you require additional space on your Persistent Disk volumes, resize your disks rather than repartitioning and formatting.
Performance
Persistent Disk performance is predictable and scales linearly with provisioned capacity until the limits for an VM's provisioned vCPUs are reached. For more information about performance scaling limits and optimization, see Configure disks to meet performance requirements.
Standard Persistent Disk volumes are efficient and economical for handling sequential read/write operations, but they aren't optimized to handle high rates of random input/output operations per second (IOPS). If your apps require high rates of random IOPS, use SSD or extreme Persistent Disk. SSD Persistent Disk is designed for single-digit millisecond latencies. Observed latency is application specific.
Compute Engine optimizes performance and scaling on Persistent Disk volumes automatically. You don't need to stripe multiple disks together or pre-warm disks to get the best performance. When you need more disk space or better performance, resize your disks and possibly add more vCPUs to add more storage space, throughput, and IOPS. Persistent Disk performance is based on the total Persistent Disk capacity attached to a VM and the number of vCPUs that the VM has.
For boot devices, you can reduce costs by using a standard Persistent Disk. Small, 10 GiB Persistent Disk volumes can work for basic boot and package management use cases. However, to ensure consistent performance for more general use of the boot device, use a balanced Persistent Disk as your boot disk.
Each Persistent Disk write operation contributes to the cumulative network egress traffic for your VM. This means that Persistent Disk write operations are capped by the network egress cap for your VM.
Reliability
Persistent Disk has built-in redundancy to protect your data against equipment failure and to ensure data availability through datacenter maintenance events. Checksums are calculated for all Persistent Disk operations, so we can ensure that what you read is what you wrote.
Additionally, you can create snapshots of Persistent Disk to protect against data loss due to user error. Snapshots are incremental, and take only minutes to create even if you snapshot disks that are attached to running VMs.
Multi-writer mode
You can attach an SSD Persistent Disk in multi-writer mode to up to two N2 VMs simultaneously so that both VMs can read and write to the disk.
Persistent Disk in multi-writer mode provides a shared block storage capability and presents an infrastructural foundation for building highly-available shared file systems and databases. These specialized file systems and databases should be designed to work with shared block storage and handle cache coherence between VMs by using tools such as SCSI Persistent Reservations.
However, Persistent Disk with multi-writer mode should generally not be used directly and you should be aware that many file systems such as EXT4, XFS, and NTFS are not designed to be used with shared block storage. For more information about the best practices when sharing Persistent Disk between VMs, see Best practices.
If you require a fully managed file storage, you can mount a Filestore file share on your Compute Engine VMs.
To enable multi-writer mode for new Persistent Disk volumes, create a new
Persistent Disk and specify the --multi-writer
flag in the gcloud CLI
or the multiWriter
property in the Compute Engine API. For more information, see
Share Persistent Disk volumes between VMs.
Persistent Disk encryption
Compute Engine automatically encrypts your data before it travels outside of your VM to the Persistent Disk storage space. Each Persistent Disk remains encrypted either with system-defined keys or with customer-supplied keys. Google distributes Persistent Disk data across multiple physical disks in a manner that users do not control.
When you delete a Persistent Disk volume, Google discards the cipher keys, rendering the data irretrievable. This process is irreversible.
If you want to control the encryption keys that are used to encrypt your data, create your disks with your own encryption keys.
Restrictions
You cannot attach a Persistent Disk volume to an VM in another project.
You can attach a balanced Persistent Disk to a maximum of 10 VMs in read-only mode.
For custom machine types or predefined machine types with a minimum of 1 vCPU, you can attach up to 128 Persistent Disk volumes.
Each Persistent Disk volume can be up to 64 TiB in size, so there is no need to manage arrays of disks to create large logical volumes. Each VM can attach only a limited amount of total Persistent Disk space and a limited number of individual Persistent Disk volumes. Predefined machine types and custom machine types have the same Persistent Disk limits.
Most VMs can have up to 128 Persistent Disk volumes and up to 257 TiB of total disk space attached. Total disk space for a VM includes the size of the boot disk.
Shared-core machine types are limited to 16 Persistent Disk volumes and 3 TiB of total Persistent Disk space.
Creating logical volumes larger than 64 TiB might require special consideration. For more information about larger logical volume performance see logical volume size.
Regional Persistent Disk
Regional Persistent Disk volumes have storage qualities that are similar to zonal Persistent Disk. However, regional Persistent Disk volumes provide durable storage and replication of data between two zones in the same region.
If you are designing robust systems or high availability services on Compute Engine, use regional Persistent Disk combined with other best practices such as backing up your data using snapshots. Regional Persistent Disk volumes are also designed to work with regional managed instance groups.
In the unlikely event of a zonal outage, you can usually failover your workload
running on regional Persistent Disk to another zone by using the
--force-attach
flag. The --force-attach
flag lets you attach the regional Persistent Disk to
a standby VM even if the disk can't be detached from the original VM
due to its unavailability. To learn more, see Regional Persistent Disk
failover. You can't force attach a zonal
Persistent Disk to a VM.
Performance
Regional Persistent Disk volumes are designed for workloads that require a lower Recovery Point Objective (RPO) and Recovery Time Objective (RTO) compared to using Persistent Disk snapshots.
Regional Persistent Disk are an option when write performance is less critical than data redundancy across multiple zones.
Like zonal Persistent Disk, regional Persistent Disk can achieve greater IOPS and throughput performance on VMs with a greater number of vCPUs. For more information about this and other limitations, see Configure disks to meet performance requirements.
When you need more disk space or better performance, you can resize your regional disks to add more storage space, throughput, and IOPS.
Reliability
Compute Engine replicates data of your regional Persistent Disk to the zones you selected when you created your disks. The data of each replica is spread across multiple physical machines within the zone to ensure redundancy.
Similar to zonal Persistent Disk, you can create snapshots of Persistent Disk to protect against data loss due to user error. Snapshots are incremental, and take only minutes to create even if you snapshot disks that are attached to running VMs.
Restrictions
Google Cloud Hyperdisk
Google Cloud Hyperdisk is Google's next generation block storage. By offloading and dynamically scaling out storage processing, it decouples storage performance from the VM type and size. Hyperdisk offers substantially higher performance, flexibility and efficiency.
Hyperdisk Balanced (Preview)
Hyperdisk Balanced for Compute Engine is a good fit for a wide range of use cases such as line of business (LOB) applications, web applications, and medium-tier databases not requiring the performance of Hyperdisk Extreme.
Hyperdisk Balanced volumes let you dynamically tune the capacity, IOPS, and throughput for your workloads.
Hyperdisk Extreme
Hyperdisk Extreme offers the fastest block storage available. It is suitable for high-end workloads that need the highest throughput and IOPS.
Hyperdisk Extreme volumes let you dynamically tune the capacity and IOPS for your workloads.
Hyperdisk Throughput
Hyperdisk Throughput is a good fit for scale-out analytics including Hadoop and Kafka, data drives for cost sensitive apps, and cold storage.
Hyperdisk Throughput volumes let you dynamically adjust to a change in workload requirements. You can change the provisioned throughput level without downtime or interruption to your workloads.
Hyperdisk volumes are created and managed like Persistent Disk, with the additional ability to set the provisioned IOPS or throughput level and change that value at any time. There is no direct migration path from Persistent Disk to Hyperdisk. Instead, you can create a snapshot and restore the snapshot to a new Hyperdisk volume.
For more information about Hyperdisk, see About Hyperdisk.
Hyperdisk encryption
Compute Engine automatically encrypts your data upon writing to a Hyperdisk volume.
Data persistence on Hyperdisk
Disk durability represents the probability of data loss, by design, for a typical disk in a typical year. Durability is calculated using a set of assumptions about hardware failures, such as:
- The likelihood of catastrophic events
- Isolation practices
- Engineering processes in Google data centers
- The internal encodings used by each disk type
Hyperdisk Balanced (Preview) offers greater than 99.999% durability. Hyperdisk Extreme offers greater than 99.9999% durability. Hyperdisk Throughput offers greater than 99.99% durability.
Local SSDs
Local SSDs are physically attached to the server that hosts your VM. Local SSDs have higher throughput and lower latency than standard Persistent Disk or SSD Persistent Disk. The data that you store on a Local SSD persists only until the VM is stopped or deleted. You can attach multiple Local SSD disks to your VM, depending on the number of vCPUs.
The size of each Local SSD disk is fixed at 375 GiB. For more storage, add multiple Local SSD disks to your VM when you create the VM. The maximum number of Local SSD disks you can attach to a VM depends on the machine type and the number of vCPUs in use.
Properties of Local SSD disks
Review the following sections to understand the behavior and characteristics of Local SSD disks.
Data persistence on Local SSDs
Review Local SSD data persistence to learn what events preserve your Local SSD data and what events can cause your Local SSD data to be unrecoverable.
General limitations
- You can create a VM with a maximum of 32 Local SSD partitions
for 12 TiB of Local SSD disk space using the
c3d-standard-360-lssd
machine type. - You can create a VM with a maximum of 32 Local SSD partitions
for 12 TiB of Local SSD disk space using the
c3-standard-176-lssd
machine type. - You can create a VM with a maximum of 16 or 24 Local SSD partitions for 6 TB or 9 TB of Local SSD space, respectively, using N1, N2, and N2D machine types.
- For C2, C2D, A2, M1, and M3 machine types, you can create a VM with a maximum of 8 Local SSD partitions, for a total of 3 TB Local SSD space.
- To reach the maximum IOPS limits, use a VM with 32 or more vCPUs.
- VMs with shared-core machine types can't attach Local SSD partitions.
- You can't attach Local SSDs to E2, Tau T2D, Tau T2A, H3, and M2 machine types.
- You can't use customer-supplied encryption keys with Local SSD disks. Compute Engine automatically encrypts your data when it is written to Local SSD storage space.
Local SSDs and machine types
You can attach Local SSDs to most machine types available on Compute Engine, as shown in the Machine series comparison table. However, there are constraints around how many Local SSDs you can attach based on each machine type. For more information, see Choosing a valid number of Local SSDs.
Performance
Local SSDs are designed to offer very high IOPS and low latency. Unlike Persistent Disk, you must manage the striping on Local SSD disks yourself.
Local SSD performance depends on several factors. For more information, see Local SSD performance and Optimizing Local SSD performance.
Cloud Storage buckets
Cloud Storage buckets are the most flexible, scalable, and durable storage option for your VMs. If your apps don't require the lower latency of Persistent Disks and Local SSDs, you can store your data in a Cloud Storage bucket.
Connect your VM to a Cloud Storage bucket when latency and throughput aren't a priority and when you must share data easily between multiple VMs or zones.
Properties of Cloud Storage buckets
Review the following sections to understand the behavior and characteristics of Cloud Storage buckets.
Performance
The performance of Cloud Storage buckets depends on the storage class that you select and the location of the bucket relative to your VM.
Using the Cloud Storage Standard storage class in the same location as your VM gives performance that is comparable to Persistent Disk but with higher latency and less consistent throughput characteristics. Using the Standard storage class in a dual-region stores your data redundantly across two regions. For optimal performance when using a dual-region, your VMs should be located in one of the regions that is part of the dual-region.
Nearline storage, Coldline storage, and Archive storage classes are primarily for long-term data archival. Unlike the Standard storage class, these classes have minimum storage durations and incur data retrieval costs. Consequently, they are best for long-term storage of data that is accessed infrequently.
Reliability
All Cloud Storage buckets have built-in redundancy to protect your data against equipment failure and to ensure data availability through datacenter maintenance events. Checksums are calculated for all Cloud Storage operations to help ensure that what you read is what you wrote.
Flexibility
Unlike Persistent Disk, Cloud Storage buckets aren't restricted to the zone where your VM is located. Additionally, you can read and write data to a bucket from multiple VMs simultaneously. For example, you can configure VMs in multiple zones to read and write data in the same bucket rather than replicate the data to Persistent Disk in multiple zones.
Cloud Storage encryption
Compute Engine automatically encrypts your data before it travels outside of your VM to Cloud Storage buckets. You don't need to encrypt files on your VMs before you write them to a bucket.
Just like Persistent Disk volumes, you can encrypt buckets with your own encryption keys.
Writing and reading data from Cloud Storage buckets
Write and read files from Cloud Storage buckets by using the
gcloud storage
command-line tool
or a Cloud Storage client library.
gcloud storage
By default, the gcloud storage
command-line tool is installed on most VMs
that use public images.
If your VM doesn't have the gcloud storage
command-line tool, you can
install it.
Connect to your Linux VMs or Connect to your Windows VMs using SSH or another connection method.
- In the Google Cloud console, go to the VM instances page.
-
In the list of virtual machine instances, click SSH in the row of
the instance that you want to connect to.
If you have never used
gcloud storage
on this VM before, use the gcloud CLI to set up credentials.gcloud init
Alternatively, if your VM is configured to use a service account with a Cloud Storage scope, you can skip this step.
Use the
gcloud storage
tool to create a bucket, write data to buckets, and read data from those buckets. To write or read data from a specific bucket, you must have access to the bucket. You can read data from any bucket that is publicly accessible.Optionally, you can also stream data to Cloud Storage.
Client library
If you configured your VM to use a service account with a Cloud Storage scope, you can use the Cloud Storage API to write and read data from Cloud Storage buckets.
- In the Google Cloud console, go to the VM instances page.
-
In the list of virtual machine instances, click SSH in the row of
the instance that you want to connect to.
Install and configure a client library for your preferred language.
If necessary, follow the insert code samples to create a Cloud Storage bucket on the VM.
Follow the insert code samples to write data and read data, and include code in your app that writes or reads a file from a Cloud Storage bucket.
What's next
- Add a Persistent Disk volume to your VM.
- Add a regional Persistent Disk to your VM.
- Create a VM with Local SSDs.
- Create a file server or distributed file system.
- Review the quotas for disks.
- Mount a RAM disk on your VM.