Manage high availability in Kubernetes

This page shows you how to enable and test high availability (HA) on your Kubernetes-based AlloyDB Omni database cluster. Performing the tasks documented here requires basic knowledge about applying Kubernetes manifest files and using the kubectl command-line tool.

Overview

You can enable HA in your database cluster by directing the AlloyDB Omni Kubernetes Operator to create a single standby replica of your primary database instance. The AlloyDB Omni Operator configures your database cluster to continuously update the data on this replica, matching all changes to data on your primary instance.

If your primary instance becomes unavailable for more than 90 seconds, then the AlloyDB Omni Operator automatically fails over from the primary database instance to the standby instance.

You can configure a standby replica to be used as a read-only instance. For more information, see Use a standby as a read-only instance

The AlloyDB Omni Operator supports both automatic and manual failover. Automatic failover is enabled by default.

Failover results in the following sequence of events:

The AlloyDB Omni Operator takes the primary database instance offline.
The AlloyDB Omni Operator turns the standby replica into the new primary database instance.
The AlloyDB Omni Operator deletes the previous primary database instance.
The AlloyDB Omni Operator creates a new standby replica.

Enable HA

Before you enable HA on your database cluster, ensure that your Kubernetes cluster has the following:

Storage for two complete copies of your data
Compute resources for two database instances running in parallel

To enable HA, follow these steps:

Modify the database cluster's manifest to include an availability section under its spec section. This section defines the number of standbys that you want to add by setting the numberOfStandbys parameter.
```
spec:
  availability:
    numberOfStandbys: NUMBER_OF_STANDBYS
```
Replace NUMBER_OF_STANDBYS with the number of standbys you want to add. The maximum value is 5. If you're setting up HA and are unsure about the number of standbys you need, then start by setting the value to either 1 or 2.
Re-apply the manifest.

Disable HA

To disable HA, follow these steps:

Set numberOfStandbys to 0 in the cluster's manifest:
```
spec:
  availability:
    numberOfStandbys: 0
```
Re-apply the manifest.

Disable automatic failover

Automatic failovers are enabled by default on database clusters.

To disable a failover, follow these steps:

Set enableAutoFailover to false in the cluster's manifest:
```
spec:
  availability:
    enableAutoFailover: false
```
Re-apply the manifest.

Trigger a manual failover

To trigger a manual failover, create and apply a manifest for a new failover resource:

apiVersion: alloydbomni.dbadmin.goog/v1
kind: Failover
metadata:
  name: FAILOVER_NAME
  namespace: NAMESPACE
spec:
  dbclusterRef: DB_CLUSTER_NAME

Replace the following:

FAILOVER_NAME: a name for this resource—for example, failover-1.
NAMESPACE: the namespace for this failover resource, which must match the namespace of the database cluster that it applies to.
DB_CLUSTER_NAME: the name of the database cluster to fail over.

To monitor the failover, run the following command:

kubectl get failover FAILOVER_NAME -o jsonpath={.status.state} -n NAMESPACE

Replace the following:

FAILOVER_NAME: the name that you assigned the failover resource when you created it.
NAMESPACE: the namespace of the database cluster.

The command returns Success after the new primary database instance is ready for use. To monitor the status of the new standby instance, see the next section.

Use standby replica as a read-only instance

To use a standby replica as a read-only instance, complete the following steps:

Modify the database cluster's manifest to set the enableStandbyAsReadReplica parameter to true.
```
spec:
  availability:
    enableStandbyAsReadReplica: true
```
Re-apply the manifest.

Verify that the read-only endpoint is reported in the status field of the DBCluster object:

kubectl describe dbcluster -n NAMESPACE DB_CLUSTER_NAME

The following example response shows the endpoint of the read-only instance:

  Status:
  [...]
  Primary: 
    [...]
    Endpoints:
      Name: Read-Write
      Value: 10.128.0.81:5432
      Name: Read-Only
      Value: 10.128.0.82:5432

Verify HA on a database cluster

To view the current HA status of a database cluster, check the HAReady condition of that cluster's status. If this value has a status set to True, then HA is set up and working on the database cluster.

To check this value on the command line, run the following command:

kubectl get dbcluster.alloydbomni.dbadmin.goog DB_CLUSTER_NAME -o jsonpath={.status.conditions[?(@.type == \'HAReady\')]} -n NAMESPACE

Replace the following:

DB_CLUSTER_NAME: the name of the database cluster.
NAMESPACE: the namespace of the database cluster.