This page describes how to set up a high availability configuration for AlloyDB Omni. This page only covers creating a new AlloyDB Omni instance in a high availability configuration. It doesn't cover converting existing instances to high availability.
Before you begin
Read the High availability and data resilience.
If you don't already have one, create a Google Cloud project.
Ensure that billing is enabled for your project.
Open Cloud Shell in the Google Cloud console.
In the Google Cloud console, clone the following source repository.
git clone https://github.com/GoogleCloudPlatform/cloud-solutions.git
Installation
In this guide, we deploy a three node Patroni cluster with AlloyDB Omni and a three node cluster etcd as the configuration store. In the front of the cluster, we use HAProxy in a managed instance group for the floating IP address so that the failover is transparent to clients.
The initial configuration for this setup is shown in the following diagram:
Figure 1. Configuration where HAProxy manages the connection between clients and the primary node.
If an outage occurs, the configuration changes to the following diagram:
Figure 2. Updated configuration where the failed primary node is replaced by the standby node.
If the number of clients that connect to the database becomes an issue and you have performance issues due to the high number of simultaneous database connections, we recommend that you add application-side connection pooling. If you can't do that, then you can add database-side connection pooling using a tool like PgBouncer.
Deployment
In Cloud Shell and after cloning the Cloud Solutions repository, navigate to the terraform directory.
cd cloud-solutions/projects/alloydbomni-ha-patroni-etcd/terraform
Create and edit a
terraform.tfvars
file. In the file, set values for the following variables.project_id = "PROJECT_ID" region = "REGION" zones = "ZONES" node_count = 3 cluster_name = "CLUSTER_NAME" replication_user_password = "REPLICATION_USER_PASSWORD" postgres_super_user_password = "PG_SUPER_USER_PASSWORD"
Descriptions for each variable can be found on GitHub in the variables configuration file.
Run the Terraform script to create all the resources.
terraform init && terraform apply
This script creates and configures the following:
Three nodes for your etcd cluster
Three nodes for your Patroni cluster
One node for HAProxy
Configuring Patroni to be synchronous
To make Patroni use only synchronous replication in your three nodes cluster,
add configuration items like synchronous_mode
, synchronous_node_count
,
synchronous_commit
, and synchronous_standby_names
in the bootstrap section
in your Patroni configuration files. The Patroni configuration is defined in the
startup script template
and in the /alloydb/config/patroni.yml
file on the Patroni nodes. To use
synchronous replication, your Patroni bootstrap configuration should look as
follows:
bootstrap:
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
synchronous_mode: true
synchronous_node_count: 2
postgresql:
use_pg_rewind: true
use_slots: true
parameters:
hot_standby: "on"
wal_keep_segments: 20
max_wal_senders: 8
max_replication_slots: 8
synchronous_commit: remote_apply
synchronous_standby_names: '*'
When synchronous_mode
is turned on, Patroni uses synchronous replication
between its primary and the other replicas. The parameter
synchronous_node_count
is used by Patroni to manage the number of synchronous
standby replicas. Patroni manages precise number of synchronous standby
replicas based on parameter synchronous_node_count
and adjusts the state in
the configuration store and in the synchronous_standby_names
as members join
and leave. For more information about synchronous replication, see the
Replication modes
section in Patroni's documentation.