Manual failover

This page gives an overview of manual failover for Cloud Memorystore for Redis. To learn how to perform a failover, see Initiating a manual failover.

What is a manual failover?

A standard tier Cloud Memorystore for Redis instance uses a replica node to back up the master node. A normal failover occurs when the master node becomes unhealthy, causing the replica to be designated as the new master. A manual failover differs from a normal failover because you initiate it yourself. For more information on how Cloud Memorystore for Redis replication works, see High availability.

Why initiate a manual failover?

Initiating a manual failover allows you to test how your application responds to a failover. This knowledge can ensure a smoother failover process if an unexpected failover occurs later on.

Optional data protection mode

The two available data protection modes are:

  • limited-data-loss mode (default).
    • Manual failover always runs in limited-data-loss mode, unless you change the mode.
  • force-data-loss mode.

To change the data protection mode, use one of the following commands:

gcloud redis instances failover INSTANCE_NAME --data-protection-mode=force-data-loss

or

gcloud redis instances failover INSTANCE_NAME --data-protection-mode=limited-data-loss

How data protection modes work

If you want to test how your application will behave in a real failover scenario, you can use the force-data-loss mode because it most accurately represents the conditions of a failover in disaster recovery.

Any failover from the master to the replica risks some data loss. The limited-data-loss mode keeps that data loss to a minimum by verifying that the difference in synchronization between your master and replica is below 30MB before initiating the failover.

The force-data-loss mode overrides this check on master-replica synchronization. If you use the force-data-loss mode when the replica synchronization is more than 30 MB behind the master, you could lose 30 MB of data or more.

Bytes pending replication metric

The bytes pending replication metric tells you how many remaining bytes the replica needs to copy before the master is fully backed up. You can access this metric in the Google Cloud Platform Console on the instance details page. To view the instance details page, click the instance id in your project's instances list page.

Alternatively, access the Stackdriver metrics explorer for your project, and search for the redis.googlapis.com/replication/offset_diff metric.

When to run a manual failover

Manual failovers using the default limited-data-loss protection mode only succeed if the bytes pending replication metric is less than 30MB. If you want to run a manual failover with bytes pending replication higher than 30MB, use the force-data-loss protection mode.

If you are trying to preserve as much data as possible, temporarily stop your application from writing to the Redis instance, and wait to run your manual failover until the bytes pending replication metric is as low as you deem acceptable.

Potential issues blocking a manual failover

  • Running a manual failover on a Basic Tier instance does not work because Basic Tier instances do not have replicas.

  • If your Redis instance is unhealthy, then the manual failover operation is blocked.

  • If your instance has incomplete operations pending, such as scaling or updating, the manual failover operation is blocked. You must wait until your instance is in the READY state to run a manual failover.

Client application connection

When your master node fails over to the replica, existing connections to Cloud Memorystore for Redis are dropped. However, on reconnect, your application is automatically redirected to the new master node using the same connection string or IP address.

Verifying a manual failover

You can verify the success of a manual failover operation with the Google Cloud Platform Console, Stackdriver, or gcloud.

GCP Console verification

Before you start a manual failover, go to the Cloud Memorystore for Redis instances list page, and click the name of your instance.

Then, under Instance Properties, view which zones your master and replica are in. Make a note of the zones. Check this page again when you complete your manual failover to confirm that the master node and replica node switched zones.

Stackdriver verification

Go to your project's Stackdriver Metrics Explorer page and search for the Node Role metric. To view the Node Role chart below, filter by your instance_id.

image

The Stackdriver chart represents the master and replica nodes with two lines. When a node's line has a value of zero on the chart, it is the replica node. When a node's line has a value of one on the chart, it is the master node. The chart represents a failover by showing how the lines switch from one to zero, and zero to one, respectively.

gcloud verification

Before you initiate a manual failover, use the following command to check which zone your master node is in:

gcloud redis instances describe [INSTANCE_ID] --region=[REGION]

Your master node is in the zone labeled currentLocationId. Make a note of the zone.

After you complete a manual failover, you can confirm that your master node switched to a new zone by running the gcloud redis instances describe command again and checking that the currentLocationId changed zones.

Additionally, the locationId label tells you the zone in which you originally provisioned your master node. The alternativeLocationId label tells you the zone in which system originally provisioned your replica node. Each time a failover occurs the master and replica switch between these two zones. However, the zones associated with locationId and alternativeLocationId do not change.

Kunde den här sidan hjälpa dig? Berätta:

Skicka feedback om ...

Google Cloud Memorystore for Redis