Regional Persistent Disk is a storage option that enables you to implement high availability (HA) services in Compute Engine. Regional Persistent Disk synchronously replicates data between two zones in the same region and ensures HA for disk data for up to one zonal failure.
This document explains how you can use regional Persistent Disk to build HA services.
Building high availability services with regional Persistent Disk
This section explains how you can build HA services with regional Persistent Disk.
Design considerations
Before you start designing a HA service, understand the characteristics of the application, the file system, and the operating system. These characteristics are the basis for the design and can rule out various approaches. For example, if an application does not support application-level replication, some corresponding design options are not applicable.
Similarly, if the application, the file system, or the operating system are not crash tolerant, then using regional persistent disks or even zonal persistent disk snapshots might not be an option. Crash tolerance is defined as the ability to recover from an abrupt termination without losing or corrupting data that was already committed to a persistent disk prior to the crash.
Consider the following when designing for high availability:
- The effect of using regional persistent disks or other solutions on the application and disk write performance.
- The service recovery time objective. Understand how quickly your service must recover from a zonal outage and the SLA requirements.
- The cost to build a resilient and reliable service architecture.
In terms of cost, the following options are for synchronous and asynchronous application replication:
Use two instances of the database and VM. In this case the following items determine the total cost:
- VM instance costs
- Persistent disk costs
- Costs of maintaining application replication
Use a single VM with regional persistent disks. To achieve high availability with a regional persistent disk, use the same VM instance and persistent disk components, but also include a regional persistent disk. Regional persistent disks are double the cost per byte compared to zonal persistent disks because they are replicated in two zones.
However, using regional persistent disks might reduce your maintenance cost because the data is automatically replicated to two replicas without the requirement of maintaining application replication.
Do not start the second VM until failover is required. You can reduce host costs even more by starting the back-up VM only on demand during failover rather than maintaining the VM as a hot standby.
Compare cost, performance, and resiliency
The following table highlights the trade-offs in cost, performance, and resiliency for the different service architectures.
| HA service architecture |
Zonal persistent disk snapshots |
Application level synchronous |
Application level asynchronous |
HA solution using regional persistent disk |
|---|---|---|---|---|
| Protects against application, VM, zone failure1 | ||||
| Mitigation against application corruption (Example: application crash-intolerance) | 2 | 2 | ||
| Cost | $ |
$$
|
$$
|
$1.5x - $$
|
| Application performance |
|
|
|
|
| Suited for application with low RPO requirement (Very low tolerance for data loss) |
|
|
|
|
| Storage recovery time from disaster4 |
|
|
|
|
1 Using regional persistent disks or snapshots is not sufficient to protect from and mitigate against failures and corruptions. Your application, file system, and possibly other software components must be crash consistent or use some sort of quiescing.
2 The replication of some applications does provide mitigation against some application corruptions. For example, MySQL primary application corruption doesn't cause its replica VM instances to become corrupted as well. Review your application's documentation for details.
3 Data loss means unrecoverable loss of data committed to persistent storage. Any non-committed data is still lost.
4 Failover performance doesn't include file system check and application recovery and load after failover.
Building HA database services using regional persistent disks
This section covers high level concepts for building HA solutions for stateful database services (MySQL, Postgres, etc.) using regional persistent disks and Compute Engine.
If there are broad outages in Google Cloud, for example, if a whole region becomes unavailable, your application might become unavailable. Depending on your needs, consider cross-regional replication techniques for even higher availability.
Database HA configurations typically have at least two VM instances. Preferably these VM instances are part of one or more managed instance groups:
- A primary VM instance in the primary zone
- A standby VM instance in a secondary zone
A primary VM instance has at least two persistent disks: a boot disk, and a regional persistent disk. The regional persistent disk contains database data and any other mutable data that should be preserved to another zone in case of an outage.
A standby VM instance requires a separate boot disk to be able to recover from configuration-related outages, which could result from an operating system upgrade, for example. You cannot force attach a boot disk to another VM during a failover.
The primary and standby VM instances are configured to use a load balancer with the traffic directed to the primary VM based on health check signals. This configuration is also known as a hot standby. The disaster recovery scenario for data outlines other failover configurations, which might be more appropriate for your scenario.
Challenges with database replication
The following table lists some common challenges with setting up and managing application synchronous or semi-synchronous replication (like MySQL) and how they compare to block replication with regional persistent disks.
| Challenges | Application synchronous or semi-synchronous replication |
Block replication with regional persistent disks |
|---|---|---|
| Maintaining stable replication between primary and failover replica. | There are a number of things that could go wrong and cause a VM instance
to fall out of HA mode:
|
Storage failures are handled by regional persistent disks. This happens
transparently to the application except for a possible fluctuation in the
disk's performance. There must be user-defined health checks to reveal any application or VM issues and trigger failover. |
| The end-to-end failover time is longer than desired. | The time taken for the failover operation doesn't have an upper bound. Waiting for all transactions to be replayed (step 2 above) could take an arbitrarily long time, depending on the schema and the load on the database. | Regional persistent disks provide synchronous replication, so the
failover time is bounded by the sum of the following latencies:
|
| Split-brain | To avoid split-brain, both approaches require provisions to ensure that there is only one primary at a time. | |
Health checks
The health checks used by the load balancer are implemented by the health check agent. The health check agent serves two purposes:
- The health check agent resides within the primary and secondary VMs to monitor the VM instances and communicate with the load balancer to direct traffic. This works best when configured with instance groups.
- The health check agent syncs with the application-specific regional control plane and makes failover decisions based on control plane behavior. The control plane must be in a zone that differs from the VM instance whose health it is monitoring.
The health check agent itself must be fault tolerant. For example, notice that,
in the image that follows, the control plane is separated from the primary VM
instance, which resides in zone us-central1-a, while the standby VM resides
in zone us-central1-f.
What's next
- Learn how to create and manage regional Persistent Disk volumes.
- Learn how to build scalable and resilient web applications on Google Cloud.
- Review the disaster recovery planning guide.