Share persistent disks between VMs

Stay organized with collections Save and categorize content based on your preferences.

You can attach an SSD persistent disk in multi-writer mode to up to two N2 virtual machine (VM) instances simultaneously so that both VMs can read and write to the disk. To enable multi-writer mode for new persistent disks, create a new persistent disk and specify the --multi-writer flag in the gcloud CLI or the multiWriter property in the Compute Engine API.

Persistent disks in multi-writer mode provide a shared block storage capability and present an infrastructural foundation for building distributed storage systems and similar highly available services. When using persistent disks in multi-writer mode, use a scale-out storage software system that has the ability to coordinate access to persistent disk devices across multiple VMs. Examples of these storage systems include Lustre and IBM Spectrum Scale. Most single VM file systems such as EXT4, XFS, and NTFS are not designed to be used with shared block storage. For more information, see Best practices in this document. If you require a fully managed file storage, you can mount a Filestore file share on your Compute Engine VMs.

Persistent disks in multi-writer mode support a subset of SCSI-3 Persistent Reservations (SCSI PR) commands. High-availability applications can use these commands for I/O fencing and failover configurations.

The following SCSI PRcommands are supported:

  • IN {REPORT CAPABILITIES, READ FULL STATUS, READ RESERVATION, READ KEYS}
  • OUT {REGISTER, REGISTER AND IGNORE EXISTING KEY, RESERVE, PREEMPT, CLEAR, RELEASE}

Restrictions

  • Available only for SSD type persistent disks.
  • You can create a persistent disk in multi-writer mode in any zone, but you can only attach that disk to VMs in the following locations:
    • australia-southeast1
    • europe-west1
    • us-central1 (us-central1-a and us-central1-c zones only)
    • us-east1 (us-east1-d zone only)
    • us-west1 (us-west1-b and us-west1-c zones only)
  • Attached VMs must have an N2 machine type.
  • Minimum size: 10 GB
  • Maximum attached instances: 2
  • Multi-writer mode persistent disks do not support persistent disk metrics.
  • Disks in multi-writer mode cannot change to read-only mode.
  • You cannot use disk images or snapshots to create persistent disks in multi-writer mode.
  • You cannot create snapshots or images from persistent disks in multi-writer mode.
  • Lower IOPS limits. See disk performance for details.
  • You cannot resize a multi-writer persistent disk.
  • When creating a VM using the Google Cloud CLI, you cannot create a multi-writer persistent disk using the --create-disk flag.

Best practices

  • I/O fencing using SCSI PR commands results in a crash consistent state of persistent disk data. Some file systems do not have crash consistency and therefore might become corrupt if you use SCSI PR commands.
  • Many file systems such as EXT4, XFS, and NTFS are not designed to be used with shared block storage and do not have mechanisms to synchronize or perform operations that originate from multiple VM instances.
  • Before you use persistent disks in multi-writer mode, ensure that you understand your file system and how it can be safely used with shared block storage and simultaneous access from multiple VMs.

Performance

Persistent disks created in multi-writer mode have specific IOPS and throughput limits.

Zonal SSD persistent disk multi-writer mode
Maximum sustained IOPS
Read IOPS per GB 30
Write IOPS per GB 30
Read IOPS per instance 15,000–100,000*
Write IOPS per instance 15,000–100,000*
Maximum sustained throughput (MB/s)
Read throughput per GB 0.48
Write throughput per GB 0.48
Read throughput per instance 240–1,200*
Write throughput per instance 240–1,200*
* Persistent disk IOPS and throughput performance depends on disk size, instance vCPU count, and I/O block size, among other factors.
Attaching a multi-writer disk to multiple virtual machine instances does not affect aggregate performance or cost. Each machine gets a share of the per-disk performance limit.

To learn how to share persistent disks between multiple VMs, see Share persistent disks between VMs.

Share a zonal persistent disk between VM instances

Share a disk in read-only mode between multiple VMs

You can attach a non-boot persistent disk to more than one VM in read-only mode, which lets you share static data between multiple VMs. Sharing static data between multiple VMs from one persistent disk is less expensive than replicating your data to unique disks for individual instances.

If you need to share dynamic storage space between multiple VMs, you can use one of the following options:

Console

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. In the list of instances in your project, click the name of the instance where you want to attach the disk. The VM instance details page opens.

  3. On the instance details page, click Edit.

  4. In the Additional disks section, click one of the following:

    1. Add a disk to add a disk in read-only mode to the instance.
    2. Attach existing disk to select an existing disk and attach it in read-only mode to your instance.

  5. Specify other options for your disk.

  6. Click Done to apply the changes.

  7. Click Save to apply your changes to the instance.

  8. Connect to the instance and mount the disk.

  9. Repeat this process to add the disk to other instances in read-only mode.

gcloud

In the gcloud CLI, use the compute instances attach-disk command and specify the --mode flag with the ro option.

gcloud compute instances attach-disk INSTANCE_NAME \
  --disk DISK_NAME \
  --mode ro

Replace the following:

  • INSTANCE_NAME: the name of the instance where you want to attach the zonal persistent disk
  • DISK_NAME: the name of the disk that you want to attach

After you attach the disk, connect to the instance and mount the disk.

Repeat this command for each instance where you want to add this disk in read-only mode.

API

In the API, construct a POST request to the compute.instances.attachDisk method method. In the request body, specify the mode parameter as READ_ONLY.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/INSTANCE_NAME/attachDisk

{
 "source": "zones/ZONE/disks/DISK_NAME",
 "mode": "READ_ONLY"
}

Replace the following:

  • INSTANCE_NAME: the name of the instance where you want to attach the zonal persistent disk
  • PROJECT_ID: your project ID
  • ZONE: the zone where your disk is located
  • DISK_NAME: the name of the disk that you are attaching

After you attach the disk, connect to the instance and mount the disk.

Repeat this request for each instance where you want to add this disk in read-only mode.

Share an SSD persistent disk in multi-writer mode between VM instances

You can share an SSD persistent disk in multi-writer mode between N2 instances in the same zone. See Persistent disk multi-writer mode for details about how this mode works. You can create and attach multi-writer persistent disks using the following process:

gcloud

Create and attach a zonal persistent disk by using the gcloud CLI:

  1. Use the gcloud beta compute disks create command command to create a zonal persistent disk. Include the --multi-writer flag to indicate that the disk must be sharable between the instances in multi-writer mode.

    gcloud beta compute disks create DISK_NAME \
   --size DISK_SIZE \
   --type pd-ssd \
   --multi-writer

    Replace the following:

    • DISK_NAME: the name of the new disk
    • DISK_SIZE: the size, in GB, of the new disk Acceptable sizes range from 1 GB to 65,536 GB for SSD persistent disks, or 200 GB to 65,536 GB for standard persistent disks in multi-writer mode.

  2. After you create the disk, attach it to any running or stopped instance with an N2 machine type. Use the gcloud compute instances attach-disk command:

    gcloud compute instances attach-disk INSTANCE_NAME \
   --disk DISK_NAME

    Replace the following:

    • INSTANCE_NAME: the name of the N2 instance where you are adding the new zonal persistent disk
    • DISK_NAME: the name of the new disk that you are attaching to the instance

  3. Repeat the gcloud compute instances attach-disk command but replace INSTANCE_NAME` with the name of your second instance.

After you create and attach a new disk to an instance, format and mount the disk using a shared-disk file system. Most file systems are not capable of using shared storage. Confirm that your file system supports these capabilities before you use it with multi-writer persistent disk. You cannot mount the disk to multiple VMs using the same process you would normally use to mount the disk to a single VM.

API

Use the Compute Engine API to create and attach an SSD persistent disk to N2 instances in multi-writer mode.

  1. In the API, construct a POST request to create a zonal persistent disk using the disks.insert method. Include the name, sizeGb, and type properties. To create this new disk as an empty and unformatted non-boot disk, do not specify a source image or a source snapshot for this disk. Include the multiWriter property with a value of True to indicate that the disk must be sharable between the instances in multi-writer mode.

    POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/disks

{
"name": "DISK_NAME",
"sizeGb": "DISK_SIZE",
"type": "zones/ZONE/diskTypes/pd-ssd",
"multiWriter": "True"
}

    Replace the following:

    • PROJECT_ID: your project ID
    • ZONE: the zone where your instance and new disk are located
    • DISK_NAME: the name of the new disk
    • DISK_SIZE: the size, in GB, of the new disk Acceptable sizes range from 1 GB to 65,536 GB for SSD persistent disks, or 200 GB to 65,536 GB for standard persistent disks in multi-writer mode.

  2. Construct a POST request to the compute.instances.attachDisk method, and include the URL to the zonal persistent disk that you just created:

    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/INSTANCE_NAME/attachDisk

{
"source": "/compute/v1/projects/PROJECT_ID/zones/ZONE/disks/DISK_NAME"
}

    Replace the following:

    • PROJECT_ID: your project ID
    • ZONE: the zone where your instance and new disk are located
    • INSTANCE_NAME: the name of the instance where you are adding the new persistent disk.
    • DISK_NAME: the name of the new disk

  3. Repeat the disks.insert command, but specify the second instance instead.

After you create and attach a new disk to an instance, format and mount the disk using a shared-disk file system. Most file systems are not capable of using shared storage. Confirm that your file system supports these capabilities before you use it with multi-writer persistent disk.

What's next