Regional persistent disk failover

Regional persistent disks provide synchronous replication of data between two zones in a region. Regional persistent disks can be a good building block to use when you implement high availability (HA) services in Compute Engine. Regional persistent disks are also designed to work with regional managed instance groups.

Failure scenarios

With regional persistent disks, when the device is in replicating mode, data is automatically replicated to two zones in a region. A write is acknowledged back to a VM when it is durably persisted in both replicas.

If replication to one zone fails or is very slow for a while, the disk switches to unreplicated mode. In this mode, write is acknowledged after it is durably persisted in one replica.

If and when Compute Engine detects that replication can be resumed, data previously written since the device entered unreplicated mode is synced to both zones and the disk returns to replicating mode. The transition is fully automated and transparent to the customer.

RPO and RTO are undefined while a device is in unreplicated mode. To minimize data and/or availability loss in the event of a failure of a disk operating in unreplicated mode, we recommend that you back up your regional persistent disks regularly using snapshots. You can recover a disk by restoring the snapshot.

Zonal failures

A regional persistent disk is replicated in two zones:

One replica is located in the same zone as the VM instance to which it is attached (the primary zone).
The other replica is located in an alternate zone in the same region (the secondary zone).

In the event that the primary zone fails, you can fail over your regional persistent disk to a VM instance in another zone by using the --force-attach flag with the attach-disk command.

In this scenario, you might not be able to detach a disk from the instance because the instance can't be reached to perform the detach operation. Force-attach lets you attach a regional persistent disk to a VM instance even if that disk is currently attached to another instance.

After you complete the force-attach operation, Compute Engine prevents the original VM from writing to the disk. Using force-attach lets you safely regain access to your data and recover your service. You also have the option to manually shut down the instance after you perform the force-attach step.

In the event that the secondary zone fails, the unhealthy replica comes back into sync with the healthy replica automatically when the secondary zone recovers.

Initial state	Failure	New state	Action
Two healthy zones	The primary zone fails	The healthy replica has all disk data. The newly unhealthy replica is not guaranteed to have all disk data.	Force attach the disk to a VM in the healthy zone.
Two healthy zones	The secondary zone fails	The unhealthy replica is not guaranteed to have all disk data until the zone recovers.	No action needed. The unhealthy replica is brough back into sync when the zone recovers.
One healthy zone One unhealthy zone	The healthy zone fails	Both replicas are in unhealthy zones and cannot serve traffic. The disk is unavailable. If the zonal outage is temporary, no data is lost. If the zonal outage is permanent, data that was written only to the healthy replica is permanently lost.	We do not recommend force attaching because the disk cannot serve traffic. You cannot create a snapshot of the disk until the zone recovers. As a best practice, back up the regional persistent disks regularly using snapshots.

Application and VM failures

In the event of outages caused by VM misconfiguration, an unsuccessful OS upgrade, or other application failures, you can force-attach your regional persistent disk to a VM instance in the same zone.

Failure category and (probability)	Failure types	Action
Application failure (High)	Application unresponsive Application admin actions (for example, upgrade) Human error (for example, misconfiguration of parameters such as SSL certificate or ACLs.	Application control plane can trigger failover based on health check thresholds.
VM failure (Medium)	Infrastructure/hardware failure VM unresponsive due to CPU contention, intermediate network interruption	VMs are usually autohealed. The application control plane can trigger failover based on health check thresholds.
Application corruption (Low-Medium)	Application data corruption (for example, due to application bugs or an unsuccessful OS upgrade)	Application recovery: Try application-specific recovery tools if available. For example, MySQL database page corruption. Restore from logical replication archive. For example, a read replica or logical log archive such as PostgreSQL continuous archiving.

Failover your regional persistent disk using `force-attach`

To force attach an existing disk to a VM, perform the following steps:

Console

Go to the VM instances page.

Go to VM instances
Select your project.
Click the name of the VM you want to change.
On the details page, click Edit.
In the Additional disks section, click Attach additional disk.
Select the regional persistent disk from the drop-down list.
To force attach the disk, select the Force-attach disk checkbox.
Click Done, and then click Save.

You can perform the same steps to force-attach a disk to the original VM after the failure is resolved.

gcloud

In the gcloud CLI, use the instances attach-disk command to attach the replica disk to a VM instance. Include the --disk-scope flag and set it to regional.

gcloud compute instances attach-disk INSTANCE_NAME  \
    --disk DISK_NAME --disk-scope regional \
    --force-attach

Replace the following:

INSTANCE_NAME: the name of the new VM instance in the region
DISK_NAME: the name of the disk

After you force-attach the disk, mount the file systems on the disk, if necessary. The instance can use the force-attached disk to continue read and write operations.

API

Construct a POST request to the compute.instances.attachDisk method method, and include the URL to the persistent disk that you just created. To attach the disk to the new VM instance, the forceAttach=true query parameter is required, even though the primary instance still has the disk.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/INSTANCE_NAME/attachDisk?forceAttach=true

{
 "source": "projects/PROJECT_ID/regions/REGION/disks/DISK_NAME"
}