Configuring an Instance for High Availability

MySQL   |  PostgreSQL

This page describes how to configure a Second Generation instance for high availability.

The easiest way to configure an instance for high availability is to do it when you create the instance. You can also enable high availability on an existing instance.

For more information about the high availability configuration, see Overview of the High Availability Configuration.

Creating a new instance configured for high availability

When you create a Second Generation instance, you can configure it for high availability; Cloud SQL creates the failover replica at the same time that it creates the master.

To create an instance configured for high availability:

Console

1. Go to the Cloud SQL Instances page in the Google Cloud Platform Console.

   Go to the Cloud SQL Instances page

2. Create a Second Generation instance, selecting the High availability checkbox under the Enable auto backups and high availability configuration option section, and leaving automated backups and binary logging enabled.

gcloud

1. Create the master instance and its failover replica:

   gcloud sql instances create [MASTER_INSTANCE_NAME] \
                                    --backup-start-time [BACKUP_WINDOW_START_TIME] \
                                    --failover-replica-name [FAILOVER_REPLICA_NAME] \
                                    --tier [MACHINE_TYPE] --enable-bin-log
   

   The backup window start time is in the format HH:MM.

2. Configure the root user on the master:

   gcloud sql users set-password root % \
                         --instance [MASTER_INSTANCE_NAME] --password [PASSWORD]
   

For a complete list of available parameters, see the gcloud sql instances create reference page.

cURL

1. Create the master instance and its failover replica:

   ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
   curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
        --header 'Content-Type: application/json' \
        --data '{"name":"[MASTER_INSTANCE_NAME]", "region":"[REGION]",
                 "settings": {"tier":"[MACHINE_TYPE]",
                 "backupConfiguration": {"binaryLogEnabled": true, "enabled": true}},
                 "failoverReplica": {"name": "[FAILOVER_REPLICA_NAME]"}}' \
        -X POST \
        https://www.googleapis.com/sql/v1beta4/projects/[PROJECT-ID]/instances
   

2. When the master finishes initializing, update the root password on the master:

   curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
        --header 'Content-Type: application/json' \
        --data '{"name": "root", "host": "%", "password": "[ROOT_PASSWORD]"}' \
        'https://www.googleapis.com/sql/v1beta4/projects/[PROJECT-ID]/instances/[INSTANCE_NAME]/users?host=%25&name=root'
   

   This change is automatically propagated to the replica.

For the complete list of parameters for this request, see the instances:insert page.

For more details about creating an instance, see Creating Instances.

Configuring an existing instance for high availability

Console

The process for configuring an existing Second Generation instance for high availability consists of these steps:

• Start failover replica creation.
• If needed, enable backups and binary logging on the instance.
• Finish failover replica creation.
• Optionally, test failover.

The Google Cloud Platform Console helps you through most of this process.

To configure an existing instance for high availability:

1. Go to the Cloud SQL Instances page in the Google Cloud Platform Console.

   Go to the Cloud SQL Instances page

2. Find the Second Generation instance you want to configure for high availability in the instance list and click Add.

   This opens the Create failover replica page. If your instance is ready for replica creation, you are shown the fields that you can edit for the replica, and you can skip to step 6. Otherwise, continue with the next step.

3. If the instance does not have binary logs or backups enabled, enable them now by selecting both checkboxes and clicking Save and restart.

   Note: Enabling binary logging on an existing instance causes the instance to be restarted. Existing database connections are lost and must be reopened.

   You should see the Create failover replica page.

4. Review the current settings of the replica, and update any settings as needed.

   You can change the name of the replica, add an authorized network, and change the zone of the replica. The MySQL flags should be identical on the master and the replica.

5. Click Create to create the replica.

   You are returned to the instance page for the master while the replica initializes.

gcloud

1. Check the status of the master instance:

   gcloud sql instances describe [MASTER_INSTANCE_NAME]

   If the databaseReplicationEnabled property is true, the instance is a replica; you cannot create a failover replica for a replica.

2. If the enabled property under backupConfiguration is false, enable backups for the master instance now:

   gcloud sql instances patch [MASTER_INSTANCE_NAME] --backup-start-time [HH:MM]

   The backup-start-time parameter is specified in 24-hour time, in the UTC±00 time zone, and specifies the start of a 4-hour backup window. Backups can start any time during the backup window.

3. If the binaryLogEnabled property is false, enable binary logs:

   gcloud sql instances patch --enable-bin-log [MASTER_INSTANCE_NAME]

   Enabling binary logs causes the instance to be restarted.

4. Create the replica:

   gcloud sql instances create [FAILOVER_REPLICA_NAME] \
   --master-instance-name [MASTER_INSTANCE_NAME] --tier [MACHINE_TYPE] --replica-type FAILOVER
   

   Machine type can be any value usable for Second Generation instances.

cURL

1. Check the status of the master instance:

   ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
   curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
        -X GET \
        https://www.googleapis.com/sql/v1beta4/projects/[PROJECT-ID]/instances/[MASTER_INSTANCE_NAME]?fields=settings
   

   The output will contain the backupConfiguration, for example:

   "backupConfiguration": {
       "kind": "sql#backupConfiguration",
       "startTime": "12:00",
       "enabled": true,
       "binaryLogEnabled": true
   }
   

2. If either enabled or binaryLogEnabled are false, use the patch method of the instances resource to enable them both. Specify the properties of the backup configuration you want to keep as is and those you want to change.

   To enable backups, set enabled to true and the startTime to a value which is the start of the backup window. To enable binary logging, set binaryLogEnabled to true.

   curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
        --header 'Content-Type: application/json' \
        --data '{"settings" : {"backupConfiguration" : {"startTime": "[HH:MM]", "enabled": true, "binaryLogEnabled": true}}}' \
        -X PATCH \
        https://www.googleapis.com/sql/v1beta4/projects/[PROJECT-ID]/instances/[MASTER_INSTANCE_NAME]
   

3. Use the insert method of the instances resource to create the read replica.

   curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
        --header 'Content-Type: application/json' \
        --data '{"name": "[REPLICA_NAME]", "masterInstanceName": "[MASTER_INSTANCE_NAME]",
                 "region": "[MASTER_REGION]", "databaseVersion": "[MASTER_DATABASE_VERSION]",
                 "replicaConfiguration": {"failoverTarget": true},
                 "settings": {"tier":"[MACHINE_TYPE]"}}' \
        -X POST \
        https://www.googleapis.com/sql/v1beta4/projects/[PROJECT-ID]/instances
   

   The machine type can be any Second Generation machine type.

Initiating failover

Testing failover is optional, but is recommended so that you can see how your application responds in the event of a zone failure. Connections to the master instance are reset to make sure there no writes can be made to the master during the failover process.

Note: This feature helps you test your application's response to an automatic failover. Make sure your instance has fully recovered from any previous failover before you trigger another failover.

Console

1. Go to the Cloud SQL Instances page in the Google Cloud Platform Console.

   Go to the Cloud SQL Instances page

2. Click the instance name to open its Instance details page.
3. Click Failover in the button bar.

   The instance fails over to its failover replica.

gcloud

Initiate the failover:

gcloud sql instances failover [MASTER_INSTANCE_NAME]

cURL

1. Describe the master instance to get the value of the `settingsVersion` field.

   ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
   curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
        -X GET \
        https://www.googleapis.com/sql/v1beta4/projects/[PROJECT-ID]/instances/[MASTER_INSTANCE_NAME]
   

2. Initiate the failover:

   curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
        --header 'Content-Type: application/json' \
        --data '{"failoverContext":{"settingsVersion":"[SETTINGS_VERSION]"}}' \
             -X POST \
        https://www.googleapis.com/sql/v1beta4/projects/[PROJECT-ID]/instances/[MASTER_INSTANCE_NAME]/failover
             

Creating an alert for replication lag

The time required for a failover operation depends on the amount of replication lag when failover is initiated. You can use Stackdriver Monitoring in the Google Cloud Platform Console to alert you when replication lag exceeds a threshold.

For more information about replication lag, search for "seconds_behind_master" on the SHOW SLAVE STATUS Syntax page in the MySQL documentation.

For more information about Stackdriver Monitoring, see the Stackdriver Monitoring documentation.

Note: Newly created Cloud SQL instances can take some time to appear in the Stackdriver Monitoring interface. If you do not see an instance that was recently created, check back again in an hour or so.

Setting an alert for a specific failover replica

1. Click the Stackdriver logo in the upper-left corner to return to the main Stackdriver Monitoring page.
2. Click Create Alerting Policy.
3. Enter a descriptive name for your alert.
4. Select Metric Threshold and click Next to open the Target tab.
5. For Resource Type, select Cloud SQL.
6. For Applies To, select Single, then select your failover replica.
7. Click Next to open the Configuration tab.

8. Enter the following values:

   Field	Value
   If Metric	Seconds Behind Master
   Condition	above
   Thresholds	120 (or whatever value is appropriate for your environment)
   For	3 minutes

9. Select your desired Notification methods, and click Save Condition.

Setting an alert for a group of failover replicas

If you plan to create multiple failover replicas in the same project, it could be easier to set the alert for the entire group, rather than on each failover replica individually. To create a group for monitoring, use a suffix for the failover replicas' names, such as "-failover". Make sure you do not use this suffix for any other types of Cloud SQL instances.

Create your failover replica group:

Go to the Cloud SQL Instances page in the Google Cloud Platform Console.

Go to the Cloud SQL Instances page

1. From the left-hand menu, select Monitoring to open the Stackdriver Monitoring page, and log in.
2. From the upper menu bar, select Groups > Create....
3. Enter a descriptive name for your group.
4. Leave Name and Contains selected, and enter your name substring.

5. Click Save Group.

   The summary page for the new group opens.

6. Confirm that the expected instances are included in the group.

Note: If you open the Groups Overview page, where all existing groups are listed, your group shows 0 instances. This is because Stackdriver uses the term "instances" to mean VMs and VM clusters. Cloud SQL instances are shown under Resources.

Create the alert for the failover replica group:

1. Click the Stackdriver logo in the upper-left corner to return to the main Stackdriver Monitoring page.
2. Click Create Alerting Policy.
3. Enter a descriptive name for your group alert.
4. Select Metric Threshold and click Next to open the Target tab.
5. For Resource Type, select Cloud SQL.
6. For Applies To, select Group, then select your failover group.
7. Leave Any Member Violates selected and click Next to open the Configuration tab.

8. Enter the following values:

   Field	Value
   If Metric	Seconds Behind Master
   Condition	above
   Thresholds	120 (or whatever value is appropriate for your environment)
   For	3 minutes

9. Select your desired Notification methods, and click Save Condition.

What's next

• Learn more about how the high availability configuration works.
• Test how your application responds to lost connections by restarting your instance.
• Learn more about managing your database connections.
• Create read replicas for your instance.
• Learn more about Stackdriver Monitoring.