HA cluster configuration guide for SAP HANA on RHEL

This guide shows you how to deploy and configure a Red Hat Enterprise Linux (RHEL) high-availability (HA) cluster for an SAP HANA 1.0 SPS 12 or later scale-up system on Google Cloud.

This guide includes the steps for:

  • Configuring Internal TCP/UDP Load Balancing to reroute traffic in the event of a failure
  • Configuring a Pacemaker cluster on RHEL to manage the SAP systems and other resources during a failover

This guide also includes steps for configuring SAP HANA system replication, but refer to the SAP documentation for the definitive instructions.

To deploy a SAP HANA system without a Linux high-availability cluster or standby hosts, use the SAP HANA deployment guide.

To configure an HA cluster for SAP HANA on SUSE Linux Enterprise Server (SLES), see the HA cluster configuration guide for SAP HANA scale-up on SLES.

This guide is intended for advanced SAP HANA users who are familiar with Linux high-availability configurations for SAP HANA.

The system that this guide deploys

Following this guide, you will deploy two SAP HANA instances and set up an HA cluster on RHEL. You deploy each SAP HANA instance on a Compute Engine VM in a different zone within the same region. A high-availability installation of SAP NetWeaver is not covered in this guide.

Overview of a high-availability Linux cluster for a single-node SAP HANA scaleup system

The deployed cluster includes the following functions and features:

  • Two host VMs, each with an instance of SAP HANA
  • Synchronous SAP HANA system replication.
  • The Pacemaker high-availability cluster resource manager.
  • A STONITH fencing mechanism.
  • Automatic restart of the failed instance as the new secondary instance.

This guide has you use the Cloud Deployment Manager templates that are provided by Google Cloud to deploy the Compute Engine virtual machines (VMs) and the SAP HANA instances, which ensures that the VMs and the base SAP HANA systems meet SAP supportability requirements and conform to current best practices.

SAP HANA Studio is used in this guide to test SAP HANA system replication. You can use SAP HANA Cockpit instead, if you prefer. For information about installing SAP HANA Studio, see:

Prerequisites

Before you create the SAP HANA high availability cluster, make sure that the following prerequisites are met:

  • You or your organization has a Google Cloud account and you have created a project for the SAP HANA deployment. For information about creating Google Cloud accounts and projects, see Setting up your Google account in the SAP HANA Deployment Guide.
  • The SAP HANA installation media is stored in a Cloud Storage bucket that is available in your deployment project and region. For information about how to upload SAP HANA installation media to a Cloud Storage bucket, see Downloading SAP HANA in the SAP HANA Deployment Guide.
  • If you are using VPC internal DNS, the value of the VmDnsSetting variable in your project metadata must be either GlobalOnly or ZonalPreferred to enable the resolution of the node names across zones. The default setting of VmDnsSetting is ZonalOnly. For more information, see:

Creating a network

For security purposes, create a new network. You can control who has access by adding firewall rules or by using another access control method.

If your project has a default VPC network, don't use it. Instead, create your own VPC network so that the only firewall rules in effect are those that you create explicitly.

During deployment, VM instances typically require access to the internet to download Google's monitoring agent. If you are using one of the SAP-certified Linux images that are available from Google Cloud, the VM instance also requires access to the internet in order to register the license and to access OS vendor repositories. A configuration with a NAT gateway and with VM network tags supports this access, even if the target VMs do not have external IPs.

To set up networking:

  1. Go to Cloud Shell.

    Go to Cloud Shell

  2. To create a new network in the custom subnetworks mode, run:

    gcloud compute networks create [YOUR_NETWORK_NAME] --subnet-mode custom

    where [YOUR_NETWORK_NAME] is the name of the new network. The network name can contain only lowercase characters, digits, and the dash character (-).

    Specify --subnet-mode custom to avoid using the default auto mode, which automatically creates a subnet in each Compute Engine region. For more information, see Subnet creation mode.

  3. Create a subnetwork, and specify the region and IP range:

    gcloud compute networks subnets create [YOUR_SUBNETWORK_NAME] \
            --network [YOUR_NETWORK_NAME] --region [YOUR_REGION] --range [YOUR_RANGE]

    where:

    • [YOUR_SUBNETWORK_NAME] is the new subnetwork.
    • [YOUR_NETWORK_NAME] is the name of the network you created in the previous step.
    • [REGION] is the region where you want the subnetwork.
    • [YOUR_RANGE] is the IP address range, specified in CIDR format, such as 10.1.0.0/24. If you plan to add more than one subnetwork, assign non-overlapping CIDR IP ranges for each subnetwork in the network. Note that each subnetwork and its internal IP ranges are mapped to a single region.
  4. Optionally, repeat the previous step and add additional subnetworks.

Setting up a NAT gateway

If you intend to create one or more VMs that will not have public IP addresses, you must create a NAT gateway so that your VMs can access the Internet to download Google's monitoring agent.

If you intend to assign an external public IP address to your VM, you can skip this step.

To create a NAT gateway:

  1. Create a VM to act as the NAT gateway in the subnet you just created:

    gcloud compute instances create [YOUR_VM_NAME] --can-ip-forward \
            --zone [YOUR_ZONE]  --image-family [YOUR_IMAGE_FAMILY] \
            --image-project [YOUR_IMAGE_PROJECT] \
            --machine-type=[YOUR_MACHINE_TYPE] --subnet [YOUR_SUBNETWORK_NAME] \
            --metadata startup-script="sysctl -w net.ipv4.ip_forward=1; iptables \
            -t nat -A POSTROUTING -o eth0 -j MASQUERADE" --tags [YOUR_VM_TAG]

    where:

    • [YOUR_VM_NAME] is the name of the VM you are creating that want to use for the NAT gateway.
    • [YOUR_ZONE] is the zone where you want the VM.
    • [YOUR_IMAGE_FAMILY] and [YOUR_IMAGE_PROJECT] specify the image you want to use for the NAT gateway.
    • [YOUR_MACHINE_TYPE] is any supported machine type. If you expect high network traffic, choose a machine type with that has at least eight virtual CPUs.
    • [YOUR_SUBNETWORK_NAME] is the name of the subnetwork where you want the VM.
    • [YOUR_VM_TAG] is a tag that is applied to the VM you are creating. If you use this VM as a bastion host, this tag is used to apply the related firewall rule only to this VM.
  2. Create a route that is tagged so that traffic passes through the NAT VM instead of the default Internet gateway:

    gcloud compute routes create [YOUR_ROUTE_NAME] \
            --network [YOUR_NETWORK_NAME] --destination-range 0.0.0.0/0 \
            --next-hop-instance [YOUR_VM_NAME] --next-hop-instance-zone \
            [YOUR_ZONE] --tags [YOUR_TAG_NAME] --priority 800

    where:

    • [YOUR_ROUTE_NAME] is the name of the route you are creating.
    • [YOUR_NETWORK_NAME] is the network you created.
    • [YOUR_VM_NAME] is the VM you are using for your NAT gateway.
    • [YOUR_ZONE] is the zone where the VM is located.
    • [YOUR_TAG_NAME] is the tag on the route that directs traffic through the NAT VM.
  3. If you also want to use the NAT gateway VM as a bastion host, run the following command. This command creates a firewall rule that allows inbound SSH access to this instance from the Internet:

    gcloud compute firewall-rules create allow-ssh --network [YOUR_NETWORK_NAME] --allow tcp:22 --source-ranges 0.0.0.0/0 --target-tags "[YOUR_VM_TAG]"

    where:

    • [YOUR_NETWORK_NAME] is the network you created.
    • [YOUR_VM_TAG] is the tag you specified when you created the NAT gateway VM. This tag is used so this firewall rule applies only to the VM that hosts the NAT gateway, and not to all VMs in the network.

Adding firewall rules

By default, an implied firewall rule blocks incoming connections from outside your Virtual Private Cloud (VPC) network. To allow incoming connections, set up a firewall rule for your VM. After an incoming connection is established with a VM, traffic is permitted in both directions over that connection.

You can create a firewall rule to allow external access to specified ports, or to restrict access between VMs on the same network. If the default VPC network type is used, some additional default rules also apply, such as the default-allow-internal rule, which allows connectivity between VMs on the same network on all ports.

Depending on the IT policy that is applicable to your environment, you might need to isolate or otherwise restrict connectivity to your database host, which you can do by creating firewall rules.

Depending on your scenario, you can create firewall rules to allow access for:

  • The default SAP ports that are listed in TCP/IP of All SAP Products.
  • Connections from your computer or your corporate network environment to your Compute Engine VM instance. If you are unsure of what IP address to use, talk to your company's network administrator.
  • Communication between VMs when, for example, your database server and application server are running on different VMs. To enable communication between VMs, you must create a firewall rule to allow traffic that originates from the subnetwork.

To create a firewall rule:

Console

  1. In the Cloud Console, go to the Firewall rules page.

    OPEN FIREWALL RULES

  2. At the top of the page, click Create firewall rule.

    • In the Network field, select the network where your VM is located.
    • In the Targets field, specify the resources on Google Cloud that this rule applies to. For example, specify All instances in the network. Or to to limit the rule to specific instances on Google Cloud, enter tags in Specified target tags.
    • In the Source filter field, select one of the following:
      • IP ranges to allow incoming traffic from specific IP addresses. Specify the range of IP addresses in the Source IP ranges field.
      • Subnets to allow incoming traffic from a particular subnetwork. Specify the subnetwork name in the following Subnets field. You can use this option to allow access between the VMs in a 3-tier or scaleout configuration.
    • In the Protocols and ports section, select Specified protocols and ports and enter tcp:[PORT_NUMBER].
  3. Click Create to create your firewall rule.

gcloud

Create a firewall rule by using the following command:

$ gcloud compute firewall-rules create firewall-name
--direction=INGRESS --priority=1000 \
--network=network-name --action=ALLOW --rules=protocol:port \
--source-ranges ip-range --target-tags=network-tags

Deploying the VMs and SAP HANA

Before you begin configuring the HA cluster, you define and deploy the VM instances and SAP HANA systems that will serve as the primary and secondary nodes in your HA cluster.

To define and deploy the systems, you use the same Cloud Deployment Manager template that you use to deploy a SAP HANA system in the SAP HANA deployment guide.

However, to deploy two systems instead of one, you need to add the definition for the second system to the configuration file by copying and pasting the definition of the first system. After you create the second definition, you need to change the resource and instance names in the second definition. To protect against a zonal failure, specify a different zone in the same region. All other property values in the two definitions stay the same.

After the SAP HANA systems have deployed successfully, you define and configure the HA cluster.

The following instructions use the Cloud Shell, but are generally applicable to the Cloud SDK.

  1. Confirm that your current quotas for resources such as persistent disks and CPUs are sufficient for the SAP HANA systems you are about to install. If your quotas are insufficient, deployment fails. For the SAP HANA quota requirements, see Pricing and quota considerations for SAP HANA.

    Go to the quotas page

  2. Open the Cloud Shell or, if you installed the Cloud SDK on your local workstation, open a terminal.

    Go to the Cloud Shell

  3. Download the template.yaml configuration file template for the SAP HANA high-availability cluster to your working directory by entering the following command in the Cloud Shell or Cloud SDK:

    wget https://storage.googleapis.com/sapdeploy/dm-templates/sap_hana/template.yaml
  4. Optionally, rename the template.yaml file to identify the configuration it defines.

  5. Open the template.yaml file in the Cloud Shell code editor or, if you are using the Cloud SDK, the text editor of your choice.

    To open the Cloud Shell code editor, click the pencil icon in the upper right corner of the Cloud Shell terminal window.

  6. In the template.yaml file, complete the definition of the first VM and SAP HANA system. Specify the property values by replacing the brackets and their contents with the values for your installation. The properties are described in the following table.

    To create the VM instances without installing SAP HANA, delete or comment out all of the lines that begin with sap_hana_.

    Property Data type Description
    instanceName String The name of the VM instance currently being defined. Specify different names in the primary and secondary VM definitions. Names must be specified in lowercase letters, numbers, or hyphens.
    instanceType String The type of Compute Engine virtual machine that you need to run SAP HANA on. If you need a custom VM type, specify a predefined VM type with a number of vCPUs that is closest to the number you need while still being larger. After deployment is complete, modify the number of vCPUs and the amount of memory..
    zone String The Google Cloud zone in which to deploy the VM instance that your are defining. Specify different zones in the same region for the primary and secondary VM definitions. The zones must be in the same region that you selected for your subnet.
    subnetwork String The name of the subnetwork you created in a previous step. If you are deploying to a shared VPC, specify this value as [SHAREDVPC_PROJECT]/[SUBNETWORK]. For example, myproject/network1.
    linuxImage String The name of the Linux operating-system image or image family that you are using with SAP HANA. To specify an image family, add the prefix family/ to the family name. For example, family/rhel-7-6-sap-ha. To specify a specific image, specify only the image name. For the list of available images and families, see the Images page in Cloud Console.
    linuxImageProject String The Google Cloud project that contains the image you are going to use. This project might be your own project or a Google Cloud image project, such as rhel-sap-cloud. For more information about GCP image projects, see the Images page in the Compute Engine documentation.
    sap_hana_deployment_bucket String The name of the GCP storage bucket in your project that contains the SAP HANA installation and revision files that you uploaded in a previous step. Any upgrade revision files in the bucket are applied to SAP HANA during the deployment process.
    sap_hana_sid String The SAP HANA system ID (SID). The ID must consist of three alphanumeric characters and begin with a letter. All letters must be uppercase.
    sap_hana_instance_number Integer The instance number, 0 to 99, of the SAP HANA system. The default is 0.
    sap_hana_sidadm_password String The password for the operating system (OS) administrator. Passwords must be at least eight characters and include at least one uppercase letter, one lowercase letter, and one number.
    sap_hana_system_password String The password for the database superuser. Passwords must be at least 8 characters and include at least one uppercase letter, one lowercase letter, and one number.
    sap_hana_sidadm_uid Integer The default value for the sidadm user ID is 900 to avoid user-created groups conflicting with SAP HANA. You can change this to a different value if you need to.
    sap_hana_sapsys_gid Integer The default group ID for sapsys is 79. By specifying a value above you can override this value to your requirements.
    sap_hana_scaleout_nodes Integer Specify 0. These instructions are for scale-up SAP HANA systems only.
    networkTag String A network tag that represents your VM instance for firewall or routing purposes. If you specify `publicIP: No` and do not specify a network tag, be sure to provide another means of access to the internet.
    publicIP Boolean Optional. Determines whether a public IP address is added to your VM instance. The default is Yes.
    serviceAccount String Optional. Specifies a service account to be used by the host VMs and by the programs that run on the host VMs. Specify the member email account of the service account. For example, svc-acct-name@project-id.iam.gserviceaccount.com. By default, the Compute Engine default service account is used. For more information, see Identity and access management for SAP programs on Google Cloud.
  7. Create the definition of the second VM and SAP HANA system by copying the definition of the first and pasting the copy after the first definition. See the example following these steps.

  8. In the definition of the second system, specify different values for the following properties than you specified in the first definition:

    • name
    • instanceName
    • zone
  9. Create the instances:

    gcloud deployment-manager deployments create deployment-name --config template-name.yaml

    The above command invokes the Deployment Manager, which deploys the VMs, downloads the SAP HANA software from your storage bucket, and installs SAP HANA, all according to the specifications in your template.yaml file.

    Deployment processing consists of two stages. In the first stage, Deployment Manager writes its status to the console. In the second stage, the deployment scripts write their status to Cloud Logging.

Example of a complete template.yaml configuration file

The following example shows a completed template.yaml configuration file that deploys two VM instances with a SAP HANA system installed.

The file contains the definitions of two resources to deploy: sap_hana_primary and sap_hana_secondary. Each resource definition contains the definitions for a VM and a SAP HANA instance.

The sap_hana_secondary resource definition was created by copying and pasting the first definition, and then modifying the values of name, instanceName, and zone properties. All other property values in the two resource definitions are the same.

The properties networkTag, serviceAccount, sap_hana_sidadm_uid, and sap_hana_sapsys_gid are from the Advanced Options section of the configuration file template. The properties sap_hana_sidadm_uid and sap_hana_sapsys_gid are included to show their default values, which are used because the properties are commented out.

imports:
‐ path: https://storage.googleapis.com/sapdeploy/dm-templates/sap_hana/sap_hana.py

resources:
‐ name: sap_hana_primary
  type: https://storage.googleapis.com/sapdeploy/dm-templates/sap_hana/sap_hana.py
  properties:
    instanceName: hana-ha-vm-1
    instanceType: n2-highmem-32
    zone: us-central1-a
    subnetwork: example-subnet-us-central1
    linuxImage: family/rhel-7-6-sap-ha
    linuxImageProject: rhel-sap-cloud
    sap_hana_deployment_bucket: hana2-sp4-rev46
    sap_hana_sid: HA1
    sap_hana_instance_number: 22
    sap_hana_sidadm_password: Google123
    sap_hana_system_password: Google123
    sap_hana_scaleout_nodes: 0
    networkTag: cluster-ntwk-tag
    serviceAccount: limited-roles@example-project-123456.iam.gserviceaccount.com
    # sap_hana_sidadm_uid: 900
    # sap_hana_sapsys_gid: 79

‐ name: sap_hana_secondary
  type: https://storage.googleapis.com/sapdeploy/dm-templates/sap_hana/sap_hana.py
  properties:
    instanceName: hana-ha-vm-2
    instanceType: n2-highmem-32
    zone: us-central1-c
    subnetwork: example-subnet-us-central1
    linuxImage: family/rhel-7-6-sap-ha
    linuxImageProject: rhel-sap-cloud
    sap_hana_deployment_bucket: hana2-sp4-rev46
    sap_hana_sid: HA1
    sap_hana_instance_number: 22
    sap_hana_sidadm_password: Google123
    sap_hana_system_password: Google123
    sap_hana_scaleout_nodes: 0
    networkTag: cluster-ntwk-tag
    serviceAccount: limited-roles@example-project-123456.iam.gserviceaccount.com
    # sap_hana_sidadm_uid: 900
    # sap_hana_sapsys_gid: 79

Create firewall rules that allow access to the host VMs

If you haven't done so already, create firewall rules that allow access to each host VM from the following sources:

  • For configuration purposes, your local workstation, a bastion host, or a jump server
  • For access between the cluster nodes, the other host VMs in the HA cluster

When you create VPC firewall rules, you specify the network tags that you defined in the template.yaml configuration file to designate your host VMs as the target for the rule.

To verify deployment, define a rule to allow SSH connections on port 22 from a bastion host or your local workstation.

For access between the cluster nodes, add a firewall rule that allows all connection types on any port from other VMs in the same subnetwork.

Make sure that the firewall rules for verifying deployment and for intra-cluster communication are created before proceeding to the next section. For instructions, see Adding firewall rules.

Verifying the deployment of the VMs and SAP HANA

Before you begin configuring the HA cluster, verify that the VMs and SAP HANA were deployed correctly by checking the logs, the OS directory mapping, and the SAP HANA installation.

Checking the logs

  1. Open Cloud Logging to check for errors and monitor the progress of the installation.

    Go to Cloud Logging

  2. On the Resources tab, select Global as your logging resource.

    • If "INSTANCE DEPLOYMENT COMPLETE" is displayed, Deployment Manager processing is complete and you can proceed to the next step.
    • If you see a quota error:

      1. On the IAM & Admin Quotas page, increase any of your quotas that do not meet the SAP HANA requirements that are listed in the SAP HANA Planning Guide.
      2. On the Deployment Manager Deployments page, delete the deployment to clean up the VMs and persistent disks from the failed installation.
      3. Rerun the Deployment Manager.

      Logging display.

Checking the configuration of the VMs and SAP HANA

  1. After the SAP HANA system deploys without errors, connect to each VM by using SSH. From the Compute Engine VM instances page, you can click the SSH button for each VM instance, or you can use your preferred SSH method.

    SSH button on Compute Engine VM instances page.

  2. Change to the root user.

    $ sudo su -
  3. At the command prompt, enter df -h. On each VM, ensure that you see the /hana directories, such as /hana/data.

    Filesystem                        Size  Used Avail Use% Mounted on
    /dev/sda2                          30G  4.0G   26G  14% /
    devtmpfs                          126G     0  126G   0% /dev
    tmpfs                             126G     0  126G   0% /dev/shm
    tmpfs                             126G   17M  126G   1% /run
    tmpfs                             126G     0  126G   0% /sys/fs/cgroup
    /dev/sda1                         200M  9.7M  191M   5% /boot/efi
    /dev/mapper/vg_hana-shared        251G   49G  203G  20% /hana/shared
    /dev/mapper/vg_hana-sap            32G  240M   32G   1% /usr/sap
    /dev/mapper/vg_hana-data          426G  7.0G  419G   2% /hana/data
    /dev/mapper/vg_hana-log           125G  4.2G  121G   4% /hana/log
    /dev/mapper/vg_hanabackup-backup  512G   33M  512G   1% /hanabackup
    tmpfs                              26G     0   26G   0% /run/user/900
    tmpfs                              26G     0   26G   0% /run/user/899
    tmpfs                              26G     0   26G   0% /run/user/1000
  4. Change to the SAP admin user by replacing sid in the following command with the system ID that you specified in the configuration file template.

    # su - sidadm
  5. Ensure that the SAP HANA services, such as hdbnameserver, hdbindexserver, and others, are running on the instance by entering the following command:

    > HDB info

Disable SAP HANA autostart

For each SAP HANA instance in the cluster, make sure that SAP HANA autostart is disabled. For failovers, Pacemaker manages the starting and stopping of the SAP HANA instances in a cluster.

  1. On each host as sidadm, stop SAP HANA:

    > HDB stop
  2. On each host, open the SAP HANA profile by using an editor, such as vi:

    vi /usr/sap/SID/SYS/profile/SID_HDBinst_num_host_name
  3. Set the Autostart property to 0:

    Autostart=0
  4. Save the profile.

  5. On each host as sidadm, start SAP HANA:

    > HDB start

Optional: Configure SSH keys on the primary and secondary VMs

The SAP HANA secure store (SSFS) keys need to be synchronized between the hosts in the HA cluster. To simplify the synchronization, and to allow files like backups to be copied between the hosts in the HA cluster, these instructions authorize direct SSH connections between the two hosts.

Your organization is likely to have guidelines that govern internal network communications. If necessary, after deployment is complete you can remove the metadata from the VMs and the keys from the authorized_keys directory.

If setting up direct SSH connections does not comply with your organization's guidelines, you can synchronize the SSFS keys and transfer files by using other methods, such as:

To enable SSH connections between the primary and secondary instances, follow these steps.

  1. On the primary host VM:

    1. SSH into the VM.

    2. Generate an SSH key for the user that needs the host-to-host SSH connection. The user is typically you.

      $ ssh-keygen
    3. At the prompts, accept the defaults by pressing enter.

    4. Update the primary VM's metadata with information about the SSH key for the secondary VM.

      $ gcloud compute instances add-metadata secondary-host-name \
           --metadata "ssh-keys=$(whoami):$(cat ~/.ssh/id_rsa.pub)" \
           --zone secondary-zone
    5. Authorize the primary VM to itself

      $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  2. On the secondary host VM:

    1. SSH into the VM.

    2. Generate an SSH key for the user that needs the host-to-host SSH connection.

      $ ssh-keygen
    3. Update the secondary VM's metadata with information about the SSH key for the primary VM.

      $ gcloud compute instances add-metadata primary-host-name \
            --metadata "ssh-keys=$(whoami):$(cat ~/.ssh/id_rsa.pub)" \
            --zone primary-zone
    4. Authorize the secondary VM to itself

      $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    5. Confirm that the SSH keys are set up properly by opening an SSH connection from the secondary system to the primary system.

      $ ssh primary-host-name
  3. On the primary host VM, confirm the connection by opening an SSH connection to the secondary host VM:

    $ ssh secondary-host-name

Back up the databases

Create backups of your databases to initiate database logging for SAP HANA system replication and create a recovery point.

If you have multiple tenant databases in an MDC configuration, back up each tenant database.

The Deployment Manager template uses /hanabackup/data/SID as the default backup directory.

To create backups of new SAP HANA databases:

  1. On the primary host, switch to sidadm. Depending on your OS image, the command might be different.

    sudo -i -u sidadm
  2. Create database backups:

    • For a SAP HANA single-database-container system:

      > hdbsql -t -u system -p system-password -i inst-num \
        "backup data using file ('full')"

      The following example shows a successful response from a new SAP HANA system:

      0 rows affected (overall time 18.416058 sec; server time 18.414209 sec)
    • For a SAP HANA multi-database-container system (MDC), create a backup of the system database as well as any tenant databases:

      > hdbsql -t -d SYSTEMDB -u system -p system-password -i inst_num \
        "backup data using file ('full')"
      > hdbsql -t -d SID -u system -p system-password -i inst-num \
        "backup data using file ('full')"

    The following example shows a successful response from a new SAP HANA system:

    0 rows affected (overall time 16.590498 sec; server time 16.588806 sec)
  3. Confirm that the logging mode is set to normal:

    > hdbsql -u system -p system-password -i inst_num \
      "select value from "SYS"."M_INIFILE_CONTENTS" where key='log_mode'"

    You should see:

    VALUE
    "normal"

Enable SAP HANA system replication

As a part of enabling SAP HANA system replication, you need to copy the data and key files for the SAP HANA secure stores on the file system (SSFS) from the primary host to the secondary host. The method that this procedure uses to copy the files is just one possible method that you can use.

  1. On the primary host as sidadm, enable system replication:

    > hdbnsutil -sr_enable --name=primary-host-name
  2. On the secondary host as sidadm, stop SAP HANA:

    > HDB stop
  3. On the primary host, using the same user account that you used to set up SSH between the host VMs, copy the key files to the secondary host. For convenience, the following commands also define an environment variable for your user account ID:

    $ sudo cp /usr/sap/SID/SYS/global/security/rsecssfs ~/rsecssfs -r
    $ myid=$(whoami)
    $ sudo chown ${myid} -R /home/"${myid}"/rsecssfs
    $ scp -r rsecssfs $(whoami)@secondary-host-name:rsecssfs
    $ rm -r /home/"${myid}"/rsecssfs
    
  4. On secondary host, as the same user as the preceding step:

    1. Replace the existing key files in the rsecssfs directories with the files from the primary host and set the file permissions to limit access:

      $ SAPSID=SID
      $ sudo rm /usr/sap/"${SAPSID}"/SYS/global/security/rsecssfs/data/SSFS_"${SAPSID}".DAT
      $ sudo rm /usr/sap/"${SAPSID}"/SYS/global/security/rsecssfs/key/SSFS_"${SAPSID}".KEY
      $ myid=$(whoami)
      $ sudo cp /home/"${myid}"/rsecssfs/data/SSFS_"${SAPSID}".DAT \
        /usr/sap/"${SAPSID}"/SYS/global/security/rsecssfs/data/SSFS_"${SAPSID}".DAT
      $ sudo cp /home/"${myid}"/rsecssfs/key/SSFS_"${SAPSID}".KEY \
        /usr/sap/"${SAPSID}"/SYS/global/security/rsecssfs/key/SSFS_"${SAPSID}".KEY
      $ sudo chown "${SAPSID,,}"adm:sapsys \
        /usr/sap/"${SAPSID}"/SYS/global/security/rsecssfs/data/SSFS_"${SAPSID}".DAT
      $ sudo chown "${SAPSID,,}"adm:sapsys \
        /usr/sap/"${SAPSID}"/SYS/global/security/rsecssfs/key/SSFS_"${SAPSID}".KEY
      $ sudo chmod 644 \
        /usr/sap/"${SAPSID}"/SYS/global/security/rsecssfs/data/SSFS_"${SAPSID}".DAT
      $ sudo chmod 640 \
        /usr/sap/"${SAPSID}"/SYS/global/security/rsecssfs/key/SSFS_"${SAPSID}".KEY
    2. Cleanup the files in your home directory.

      $ rm -r /home/"${myid}"/rsecssfs
    3. As sidadm, register the secondary SAP HANA system with SAP HANA system replication:

      > hdbnsutil -sr_register --remoteHost=primary-host-name --remoteInstance=inst_num \
      --replicationMode=syncmem --operationMode=logreplay --name=secondary-host-name
    4. As sidadm, start SAP HANA:

      > HDB start

Validating system replication

On the primary host as sidadm, confirm that SAP HANA system replication is active by running the following python script:

$ python $DIR_INSTANCE/exe/python_support/systemReplicationStatus.py

If replication is set up properly, among other indicators, the following values are displayed for the xsengine, nameserver, and indexserver services:

  • The Secondary Active Status is YES
  • The Replication Status is ACTIVE

Also, the overall system replication status shows ACTIVE.

SAP HANA 1.0 SPS 12 only: Create a linux user for the monitoring agent

You need to register resource agents as an SAP HANA database user so they can run queries on the system replication status. The resource agents need CATALOG READ and MONITOR ADMIN privileges.

To register the resource agents as a database user:

  1. On the primary host:

    1. As sidadm, create user rhelhasync:

      > hdbsql -i inst_num -u system -p system-password \
        "create user rhelhasync password \"monitoring-user-password\""
      > hdbsql -i inst_num -u system -p system-password \
        "grant CATALOG READ to rhelhasync"
      > hdbsql -i inst_num -u system -p system-password \
        "grant MONITOR ADMIN to rhelhasync"
      > hdbsql -i inst_num -u system -p system-password \
        "ALTER USER rhelhasync DISABLE PASSWORD LIFETIME"
    2. As root, store the credential so it is accessible by the root user:

      # /usr/sap/SID/HDBinst_num/exe/hdbuserstore \
        SET SAPHANASIDSR localhost:3inst_num15 rhelhasync \
        "monitoring-user-password"
    3. As root, confirm that the credentials were stored successfully:

      # /usr/sap/SID/HDBinst_num/exe/hdbuserstore list

      You should see output similar to the following example:

      ha1adm@hana-ha-vm-1:/usr/sap/HA1/HDB22> /usr/sap/HA1/HDB22/exe/hdbuserstore list
      DATA FILE       : /usr/sap/HA1/home/.hdb/hana-ha-vm-1/SSFS_HDB.DAT
      KEY FILE        : /usr/sap/HA1/home/.hdb/hana-ha-vm-1/SSFS_HDB.KEY
      
      KEY SAPHANAHA1SR
       ENV : localhost:32215
       USER: rhelhasync
    4. As root, confirm that root can connect to the database by using the stored key:

      # /usr/sap/SID/HDBinst_num/exe/hdbsql -U SAPHANASIDSR -i inst_num \
        "select distinct REPLICATION_STATUS from SYS.M_SERVICE_REPLICATION"
  2. On the secondary host:

    1. As root, store the credential so it is accessible by the root user:

      # /usr/sap/SID/HDBinst_num/exe/hdbuserstore \
        SET SAPHANASIDSR localhost:3inst_num15 rhelhasync \
          "monitoring-user-password"
    2. As root, confirm that the credentials were stored successfully:

      # /usr/sap/SID/HDBinst_num/exe/hdbuserstore list

      You should see output similar to the following example:

      ha1adm@hana-ha-vm-2:/usr/sap/HA1/HDB22> /usr/sap/HA1/HDB22/exe/hdbuserstore list
      DATA FILE       : /usr/sap/HA1/home/.hdb/hana-ha-vm-2/SSFS_HDB.DAT
      KEY FILE        : /usr/sap/HA1/home/.hdb/hana-ha-vm-2/SSFS_HDB.KEY
      KEY SAPHANAHA1SR
       ENV : localhost:32215
       USER: rhelhasync

If you get error messages or are prompted to change the password, use either the hdbsql command or SAP HANA Studio to confirm that the password for the resource agent user is not configured to be changed on first login or that it has not expired.

Configure the Cloud Load Balancing failover support

The Internal TCP/UDP Load Balancing service with failover support routes traffic to the active host in an SAP HANA cluster based on a health check service. This offers protection in an Active/Passive configuration and can be extended to support an Active/Active (Read-enabled secondary) configuration.

Reserve an IP address for the virtual IP

The virtual IP (VIP) address , which is sometimes referred to as a floating IP address, follows the active SAP HANA system. The load balancer routes traffic that is sent to the VIP to the VM that is currently hosting the active SAP HANA system.

  1. Open Cloud Shell:

    Go to Cloud Shell

  2. Reserve an IP address for the virtual IP. This is the IP address that applications use to access SAP HANA. If you omit the --addresses flag, an IP address in the specified subnet is chosen for you:

    $ gcloud compute addresses create vip-name \
      --region cluster-region --subnet cluster-subnet \
      --addresses vip-address

    For more information about reserving a static IP, see Reserving a static internal IP address.

  3. Confirm IP address reservation:

    $ gcloud compute addresses describe vip-name \
      --region cluster-region

    You should see output similar to the following example:

    address: 10.0.0.19
    addressType: INTERNAL
    creationTimestamp: '2020-05-20T14:19:03.109-07:00'
    description: ''
    id: '8961491304398200872'
    kind: compute#address
    name: vip-for-hana-ha
    networkTier: PREMIUM
    purpose: GCE_ENDPOINT
    region: https://www.googleapis.com/compute/v1/projects/example-project-123456/regions/us-central1
    selfLink: https://www.googleapis.com/compute/v1/projects/example-project-123456/regions/us-central1/addresses/vip-for-hana-ha
    status: RESERVED
    subnetwork: https://www.googleapis.com/compute/v1/projects/example-project-123456/regions/us-central1/subnetworks/example-subnet-us-central1

Create instance groups for your host VMs

  1. In Cloud Shell, create two unmanaged instance groups and assign the primary host VM to one and the secondary host VM to the other:

    $ gcloud compute instance-groups unmanaged create primary-ig-name \
      --zone=primary-zone
    $ gcloud compute instance-groups unmanaged add-instances primary-ig-name \
      --zone=primary-zone \
      --instances=primary-host-name
    $ gcloud compute instance-groups unmanaged create secondary-ig-name \
      --zone=secondary-zone
    $ gcloud compute instance-groups unmanaged add-instances secondary-ig-name \
      --zone=secondary-zone \
      --instances=secondary-host-name
    
  2. Confirm the creation of the instance groups:

    $ gcloud compute instance-groups unmanaged list

    You should see output similar to the following example:

    NAME          ZONE           NETWORK          NETWORK_PROJECT        MANAGED  INSTANCES
    hana-ha-ig-1  us-central1-a  example-network  example-project-123456 No       1
    hana-ha-ig-2  us-central1-c  example-network  example-project-123456 No       1

Create a Compute Engine health check

  1. In Cloud Shell, create the health check. For the port used by the health check, choose a port that is in the private range, 49152-65535, to avoid clashing with other services. The check-interval and timeout values are slightly longer than the defaults so as to increase failover tolerance during Compute Engine live migration events. You can adjust the values, if necessary:

    $ gcloud compute health-checks create tcp health-check-name --port=healthcheck-port-num \
      --proxy-header=NONE --check-interval=10 --timeout=10 --unhealthy-threshold=2 \
      --healthy-threshold=2
  2. Confirm the creation of the health check:

    $ gcloud compute health-checks describe health-check-name

    You should see output similar to the following example:

    checkIntervalSec: 10
    creationTimestamp: '2020-05-20T21:03:06.924-07:00'
    healthyThreshold: 2
    id: '4963070308818371477'
    kind: compute#healthCheck
    name: hana-health-check
    selfLink: https://www.googleapis.com/compute/v1/projects/example-project-123456/global/healthChecks/hana-health-check
    tcpHealthCheck:
     port: 60000
     portSpecification: USE_FIXED_PORT
     proxyHeader: NONE
    timeoutSec: 10
    type: TCP
    unhealthyThreshold: 2

Create a firewall rule for the health checks

Define a firewall rule for a port in the private range that allows access to your host VMs from the IP ranges that are used by Compute Engine health checks, 35.191.0.0/16 and 130.211.0.0/22. For more information, see Creating firewall rules for health checks.

  1. If you don't already have one, create a firewall rule to allow the health checks:

    $ gcloud compute firewall-rules create  rule-name \
      --network network-name \
      --action ALLOW \
      --direction INGRESS \
      --source-ranges 35.191.0.0/16,130.211.0.0/22 \
      --target-tags network-tags \
      --rules tcp:hlth-chk-port-num

    For example:

    gcloud compute firewall-rules create  fw-allow-health-checks \
    --network example-network \
    --action ALLOW \
    --direction INGRESS \
    --source-ranges 35.191.0.0/16,130.211.0.0/22 \
    --target-tags cluster-ntwk-tag \
    --rules tcp:60000

Configure the load balancer and failover group

  1. Create the load balancer backend service:

    $ gcloud compute backend-services create backend-service-name \
      --load-balancing-scheme internal \
      --health-checks health-check-name \
      --no-connection-drain-on-failover \
      --drop-traffic-if-unhealthy \
      --failover-ratio 1.0 \
      --region cluster-region \
      --global-health-checks
  2. Add the primary instance group to the backend service:

    $ gcloud compute backend-services add-backend backend-service-name \
      --instance-group primary-ig-name \
      --instance-group-zone primary-zone \
      --region cluster-region
  3. Add the secondary, failover instance group to the backend service:

    $ gcloud compute backend-services add-backend backend-service-name \
      --instance-group secondary-ig-name \
      --instance-group-zone secondary-zone \
      --failover \
      --region cluster-region
  4. Create a forwarding rule. For the IP address, specify the IP address that you reserved for the VIP:

    $ gcloud compute forwarding-rules create rule-name \
      --load-balancing-scheme internal \
      --address vip-address \
      --subnet cluster-subnet \
      --region cluster-region \
      --backend-service backend-service-name \
      --ports ALL

Test the load balancer configuration

Even though your backend instance groups won't register as healthy until later, you can test the load balancer configuration by setting up a listener to respond to the health checks. After setting up a listener, if the load balancer is configured correctly, the status of the backend instance groups changes to healthy.

The following sections present different methods that you can use to test the configuration.

Testing the load balancer with the socat utility

You can use the socat utility to temporarily listen on the health check port.

  1. On both host VMs, install the socat utility:

    $ sudo yum install -y socat

  2. Start a socat process to listen for 60 seconds on the health check port:

    $ sudo timeout 60s socat - TCP-LISTEN:hlth-chk-port-num,fork

  3. In Cloud Shell, after waiting a few seconds for the health check to detect the listener, check the health of your backend instance groups:

    $ gcloud compute backend-services get-health backend-service-name \
      --region cluster-region

    You should see output similar to the following:

    ---
    backend: https://www.googleapis.com/compute/v1/projects/example-project-123456/zones/us-central1-a/instanceGroups/hana-ha-ig-1
    status:
     healthStatus:
     ‐ healthState: HEALTHY
       instance: https://www.googleapis.com/compute/v1/projects/example-project-123456/zones/us-central1-a/instances/hana-ha-vm-1
       ipAddress: 10.0.0.35
       port: 80
     kind: compute#backendServiceGroupHealth
    ---
    backend: https://www.googleapis.com/compute/v1/projects/example-project-123456/zones/us-central1-c/instanceGroups/hana-ha-ig-2
    status:
     healthStatus:
     ‐ healthState: HEALTHY
       instance: https://www.googleapis.com/compute/v1/projects/example-project-123456/zones/us-central1-c/instances/hana-ha-vm-2
       ipAddress: 10.0.0.34
       port: 80
     kind: compute#backendServiceGroupHealth

Testing the load balancer using port 22

If port 22 is open for SSH connections on your host VMs, you can temporarily edit the health checker to use port 22, which has a listener that can respond to the health checker.

To temporarily use port 22, follow these steps:

  1. Click your health check in the console:

    Go to Health checks page

  2. Click Edit.

  3. In the Port field, change the port number to 22.

  4. Click Save and wait a minute or two.

  5. In Cloud Shell, check the health of your backend instance groups:

    $ gcloud compute backend-services get-health backend-service-name \
      --region cluster-region

    You should see output similar to the following:

    ---
    backend: https://www.googleapis.com/compute/v1/projects/example-project-123456/zones/us-central1-a/instanceGroups/hana-ha-ig-1
    status:
     healthStatus:
     ‐ healthState: HEALTHY
       instance: https://www.googleapis.com/compute/v1/projects/example-project-123456/zones/us-central1-a/instances/hana-ha-vm-1
       ipAddress: 10.0.0.35
       port: 80
     kind: compute#backendServiceGroupHealth
    ---
    backend: https://www.googleapis.com/compute/v1/projects/example-project-123456/zones/us-central1-c/instanceGroups/hana-ha-ig-2
    status:
     healthStatus:
     ‐ healthState: HEALTHY
       instance: https://www.googleapis.com/compute/v1/projects/example-project-123456/zones/us-central1-c/instances/hana-ha-vm-2
       ipAddress: 10.0.0.34
       port: 80
     kind: compute#backendServiceGroupHealth
  6. When you are done, change the health check port number back to the original port number.

Set up Pacemaker

The following procedure configures the Red Hat implementation of a Pacemaker cluster on Compute Engine VMs for SAP HANA.

The procedure is based on Red Hat documentation for configuring high-availability clusters, including (a Red Hat subscription is required):

Install the cluster agents on both nodes

Complete the following steps on both nodes.

  1. As root, install the Pacemaker components:

    # yum -y install pcs pacemaker fence-agents-gce resource-agents-gcp resource-agents-sap-hana
    # yum update -y

    If you are using a Google-provided RHEL-for-SAP image, these packages are already installed, but some updates might be required.

  2. Set the password for the hacluster user, which is installed as part of the packages:

    # passwd hacluster
  3. Specify a password for hacluster at the prompts.

  4. In the RHEL images provided by Google Cloud, the OS firewall service is active by default. Configure the firewall service to allow high-availability traffic:

    # firewall-cmd --permanent --add-service=high-availability
    # firewall-cmd --reload
  5. Start the pcs service and configure it to start at boot time:

    # systemctl start pcsd.service
    # systemctl enable pcsd.service
  6. Check the status of the pcs service:

    # systemctl status pcsd.service

    You should see output similar to the following:

    ● pcsd.service - PCS GUI and remote configuration interface
      Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled; vendor preset: disabled)
      Active: active (running) since Sat 2020-06-13 21:17:05 UTC; 25s ago
        Docs: man:pcsd(8)
              man:pcs(8)
    Main PID: 31627 (pcsd)
      CGroup: /system.slice/pcsd.service
              └─31627 /usr/bin/ruby /usr/lib/pcsd/pcsd
    Jun 13 21:17:03 hana-ha-vm-1 systemd[1]: Starting PCS GUI and remote configuration interface...
    Jun 13 21:17:05 hana-ha-vm-1 systemd[1]: Started PCS GUI and remote configuration interface.
  7. In the /etc/hosts file, add the full host name and the internal IP addresses of both hosts in the cluster. For example:

    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    10.0.0.40 hana-ha-vm-1.us-central1-a.c.example-project-123456.internal hana-ha-vm-1  # Added by Google
    10.0.0.41 hana-ha-vm-2.us-central1-c.c.example-project-123456.internal hana-ha-vm-2
    169.254.169.254 metadata.google.internal  # Added by Google

    For more information from Red Hat about setting up the /etc/hosts file on RHEL cluster nodes, see https://access.redhat.com/solutions/81123.

Create the cluster

  1. As root on either node, authorize the hacluster user:

    # pcs cluster auth primary-host-name secondary-host-name
  2. At the prompts, enter the hacluster user name and the password that you set for the hacluster user.

  3. Create the cluster and set it to start automatically:

    # pcs cluster setup --name cluster-name primary-host-name secondary-host-name
    # pcs cluster enable --all
    # pcs cluster start --all

Set up fencing

RHEL images that are provided by Google Cloud include a fence_gce fencing agent that is specific to Google Cloud. You use fence_gce to create fence devices for each host VM.

To see all of the options that are available with the fence_gce fencing agent, issue fence_gce -h.

  1. On the primary host as root:

    1. Create a fencing device for each host VM:

      # pcs stonith create primary-fence-name fence_gce \
        port=primary-host-name \
        zone=primary-host-zone \
        project=project-id
      # pcs stonith create secondary-fence-name fence_gce \
        port=secondary-host-name \
        zone=secondary-host-zone \
        project=project-id
    2. Constrain each fence device to the other host VM:

      # pcs constraint location primary-fence-name \
        prefers secondary-host-name=INFINITY
      # pcs constraint location secondary-fence-name \
        prefers primary-host-name=INFINITY
  2. On the primary host as root, test the secondary fence device:

    1. Shut down the secondary host VM:

      # fence_gce -o off -n secondary-host-name --zone=secondary-host-zone

      If the command is successful, you lose connectivity to the secondary host VM and it appears stopped on the VM instances page in the Cloud Console. You might need to refresh the page.

    2. Restart the secondary host VM:

      # fence_gce -o on -n secondary-host-name --zone=secondary-host-zone
  3. On the secondary host as root, test the primary fence device by repeating the preceding steps using the values for the primary host in the commands.

  4. On either host as root, check the status of the cluster:

    # pcs status

    The fence resources appear in the resources section of the cluster status, similar to the following example:

    [root@hana-ha-vm-2 ~]# pcs status
    Cluster name: hana-ha-cluster
    Stack: corosync
    Current DC: hana-ha-vm-1 (version 1.1.19-8.el7_6.5-c3c624ea3d) - partition with quorum
    Last updated: Mon Jun 15 17:19:07 2020
    Last change: Mon Jun 15 17:18:33 2020 by root via cibadmin on hana-ha-vm-1
    
    2 nodes configured
    2 resources configured
    
    Online: [ hana-ha-vm-1 hana-ha-vm-2 ]
    
    Full list of resources:
    
     STONITH-hana-ha-vm-1   (stonith:fence_gce):    Started hana-ha-vm-2
     STONITH-hana-ha-vm-2   (stonith:fence_gce):    Started hana-ha-vm-1
    
    Daemon Status:
      corosync: active/enabled
      pacemaker: active/enabled
      pcsd: active/enabled

SAP HANA 2.0 and later: enable the SAP HANA HA/DR provider hook

The SAP HANA HA/DR provider hook allows SAP HANA to send out notifications for certain events and improves failure detection. Red Hat highly recommends its use.

The following steps enable the SAP HANA HA/DR provider hook.

  1. As root on either host, stop the cluster:

    # pcs cluster stop --all
  2. On both hosts:

    1. As sidadm, stop SAP HANA:

      > HDB stop
    2. As root, install the provided SAP HANA script:

      # mkdir -p /hana/shared/myHooks
      # cp /usr/share/SAPHanaSR/srHook/SAPHanaSR.py /hana/shared/myHooks
      # chown -R sidadm:sapsys /hana/shared/myHooks
    3. As root, open the global.ini file for editing:

      # vi /hana/shared/SID/global/hdb/custom/config/global.ini
    4. Add the following definitions to the global.ini file:

      [ha_dr_provider_SAPHanaSR]
      provider = SAPHanaSR
      path = /hana/shared/myHooks
      execution_order = 1
      
      [trace]
      ha_dr_saphanasr = info
    5. As root, create a sudo configuration file to allow the hook script to update the node attributes when the srConnectionChanged()hook is called:

      # vi /etc/sudoers.d/20-saphana
    6. In the /etc/sudoers.d/20-saphana file, add the following text:

      1 Cmnd_Alias SOK   = /usr/sbin/crm_attribute -n hana_SID_glob_srHook -v SOK -t crm_config -s SAPHanaSR
      2 Cmnd_Alias SFAIL = /usr/sbin/crm_attribute -n hana_SID_glob_srHook -v SFAIL -t crm_config -s SAPHanaSR
      3 sidadm ALL=(ALL) NOPASSWD: SOK, SFAIL
    7. As sidadm, start SAP HANA:

      > HDB start
    8. As sidadm, test the status reported by the hook script:

      > cdtrace
      > awk '/ha_dr_SAPHanaSR.*crm_attribute/ { printf "%s %s %s %s\n",$2,$3,$5,$16 }' nameserver_*

Set the cluster defaults

Set up migration thresholds and stickiness to determine the number of failovers to attempt before failure and to set the system to try restarting on the current host first. This only needs to be set on one node to apply to the cluster.

  1. As root on either host, start the cluster:

    # pcs cluster start --all #start the cluster
  2. Set the resource defaults:

    # pcs resource defaults resource-stickiness=1000
    $ pcs resource defaults migration-threshold=5000

    The property resource-stickiness controls how likely a service is to stay running where it is. Higher values make the service more sticky. A value of 1000 means that the service is very sticky.

    The property migration-threshold specifies the number of failures that must occur before a service fails over to another host. A value of 5000 is high enough to prevent failover for shorter-lived error situations.

    You can check the resource defaults by entering pcs resource defaults.

  3. If the startup-fencing and stonith-enabled properties are not already set totrue, set them to true now:

    # pcs property set startup-fencing="true"
    # pcs property set stonith-enabled="true"

    You can check your proprerty settings with pcs property list.

Create the SAPHanaTopology resource

The SAPHanaTopology resource gets the status and configuration of HANA System Replication on the nodes. It also checks the SAP host agent.

  1. As root on either host, create the SAPHanaTopology resource:

    # pcs resource create topology_resource_name SAPHanaTopology SID=SID \
       InstanceNumber=inst_num \
       op start timeout=600 \
       op stop timeout=300 \
       op monitor interval=10 timeout=600 \
       clone clone-max=2 clone-node-max=1 interleave=true
  2. After the resource is created, check the configuration. Append -clone to the resource name to include the clone set information in the response:

    # pcs resource show topology_resource_name-clone

    You should see output similar to the following:

    Clone: SAPHanaTopology_HA1_22-clone
    Meta Attrs: clone-max=2 clone-node-max=1 interleave=true
    Resource: SAPHanaTopology_HA1_22 (class=ocf provider=heartbeat type=SAPHanaTopology)
     Attributes: InstanceNumber=22 SID=HA1
     Operations: methods interval=0s timeout=5 (SAPHanaTopology_HA1_22-methods-interval-0s)
                 monitor interval=10 timeout=600 (SAPHanaTopology_HA1_22-monitor-interval-10)
                 reload interval=0s timeout=5 (SAPHanaTopology_HA1_22-reload-interval-0s)
                 start interval=0s timeout=600 (SAPHanaTopology_HA1_22-start-interval-0s)
                 stop interval=0s timeout=300 (SAPHanaTopology_HA1_22-stop-interval-0s)

You can also check the cluster attributes by using the crm_mon -A1 command.

Create the SAPHana resource

The SAPHana resource agent manages the databases that are configured for SAP HANA system replication.

The following parameters in the SAPHana resource definition are optional:

  • AUTOMATED_REGISTER, which, when set to true, automatically registers the former primary as secondary when the DUPLICATE_PRIMARY_TIMEOUT expires after a takeover. The default is false.
  • DUPLICATE_PRIMARY_TIMEOUT, which sets the time difference in seconds between two primary timestamps if a dual-primary situation occurs. The default is 7200.
  • PREFER_SITE_TAKEOVER, which determines if local restarts are tried before failover is initiated. The default is false.

For additional information about these parameters see, Installing and Configuring a Red Hat Enterprise Linux 7.6 (and later) High-Availability Cluster on Google Cloud. A Red Hat subscription is required.

  1. As roote on either host, create the SAP HANA resource:

    # pcs resource create sap_hana_resource_name SAPHana SID=SID \
        InstanceNumber=inst_num \
        PREFER_SITE_TAKEOVER=true DUPLICATE_PRIMARY_TIMEOUT=7200 AUTOMATED_REGISTER=true \
        op start timeout=3600 \
        op stop timeout=3600 \
        op monitor interval=61 role="Slave" timeout=700 \
        op monitor interval=59 role="Master" timeout=700 \
        op promote timeout=3600 \
        op demote timeout=3600 \
        master meta notify=true clone-max=2 clone-node-max=1 interleave=true
  2. Check the resulting resource attibutes:

    # pcs resource show sap_hana_resource_name

    You should see output similar to the following example:

     Resource: SAPHana_HA1_22 (class=ocf provider=heartbeat type=SAPHana)
      Attributes: AUTOMATED_REGISTER=true DUPLICATE_PRIMARY_TIMEOUT=7200 InstanceNumber=22 PREFER_SITE_TAKEOVER=true SID=HA1
      Meta Attrs: clone-max=2 clone-node-max=1 interleave=true notify=true
      Operations: demote interval=0s timeout=3600 (SAPHana_HA1_22-demote-interval-0s)
                  methods interval=0s timeout=5 (SAPHana_HA1_22-methods-interval-0s)
                  monitor interval=61 role=Slave timeout=700 (SAPHana_HA1_22-monitor-interval-61)
                  monitor interval=59 role=Master timeout=700 (SAPHana_HA1_22-monitor-interval-59)
                  promote interval=0s timeout=3600 (SAPHana_HA1_22-promote-interval-0s)
                  reload interval=0s timeout=5 (SAPHana_HA1_22-reload-interval-0s)
                  start interval=0s timeout=3600 (SAPHana_HA1_22-start-interval-0s)
                  stop interval=0s timeout=3600 (SAPHana_HA1_22-stop-interval-0s)
  3. After the resource is started, check the node attributes to see the current state of the SAP HANA databases on the nodes:

    # crm_mon -A1

    You should see output similar to the following:

    Stack: corosync
    Current DC: hana-ha-vm-2 (version 1.1.19-8.el7_6.5-c3c624ea3d) - partition with quorum
    Last updated: Tue Jun 16 20:07:51 2020
    Last change: Tue Jun 16 20:07:26 2020 by root via crm_attribute on hana-ha-vm-1
    
    2 nodes configured
    6 resources configured
    
    Online: [ hana-ha-vm-1 hana-ha-vm-2 ]
    
    Active resources:
    
    STONITH-hana-ha-vm-1   (stonith:fence_gce):    Started hana-ha-vm-2
    STONITH-hana-ha-vm-2   (stonith:fence_gce):    Started hana-ha-vm-1
    Clone Set: SAPHanaTopology_HA1_22-clone [SAPHanaTopology_HA1_22]
        Started: [ hana-ha-vm-1 hana-ha-vm-2 ]
    Master/Slave Set: SAPHana_HA1_22-master [SAPHana_HA1_22]
        Masters: [ hana-ha-vm-1 ]
        Slaves: [ hana-ha-vm-2 ]
    
    Node Attributes:
    * Node hana-ha-vm-1:
       + hana_ha1_clone_state              : PROMOTED
       + hana_ha1_op_mode                  : logreplay
       + hana_ha1_remoteHost               : hana-ha-vm-2
       + hana_ha1_roles                    : 4:P:master1:master:worker:master
       + hana_ha1_site                     : hana-ha-vm-1
       + hana_ha1_srmode                   : syncmem
       + hana_ha1_sync_state               : PRIM
       + hana_ha1_version                  : 1.00.122.27.1568902538
       + hana_ha1_vhost                    : hana-ha-vm-1
       + lpa_ha1_lpt                       : 1592338046
       + master-SAPHana_HA1_22             : 150
    * Node hana-ha-vm-2:
       + hana_ha1_clone_state              : DEMOTED
       + hana_ha1_op_mode                  : logreplay
       + hana_ha1_remoteHost               : hana-ha-vm-1
       + hana_ha1_roles                    : 4:S:master1:master:worker:master
       + hana_ha1_site                     : hana-ha-vm-2
       + hana_ha1_srmode                   : syncmem
       + hana_ha1_sync_state               : SOK
       + hana_ha1_version                  : 1.00.122.27.1568902538
       + hana_ha1_vhost                    : hana-ha-vm-2
       + lpa_ha1_lpt                       : 30
       + master-SAPHana_HA1_22             : 100

Create a virtual IP address resource

You need to create a cluster resource for the VIP. The VIP resource is localized to the primary operating system and is not routable by other hosts. The load balancer routes traffic that is sent to the VIP to the backend host based on the health check.

As root on either host:

# pcs resource create resource_name \
  IPaddr2 ip="vip-address" nic=eth0 cidr_netmask=32

The vip-address value is the same IP address that you reserved earlier and specified in the forwarding rule for the front-end of your load balancer. Change the network interface as appropriate for your configuration.

Create the constraints

You create constraints to define which services need to start first, and which services need to run together on the same host. For example, the IP address must be on the same host as the primary HANA instance.

  1. Define the start order constraint:

    # pcs constraint order topology_resource_name-clone \
      then sap_hana_resource_name-master symmetrical=false

    The specification of symmetrical=false means that the constraint applies to startup only and not to shutdown.

    However, because you set interleave=true for these resources in a previous step, the processes can start in parallel. In other words, you can start SAPHana on any node as soon as SAPHanaTopology is running.

  2. Define the colocation constraint:

    # pcs constraint colocation add vip_resource_name \
      with master sap_hana_resource_name-master 2000

    Using a score of 2000 rather than the default INFINITY allows the cluster to take down the IPaddr2 resource in the event that no Master is promoted in the SAPHana resource, so that tools like SAP Management Console or SAP LaMa can use the address to query the status of the SAP instance.

  3. Check the constraints:

    # pcs constraint

    You should see output similar to the following:

    Location Constraints:
     Resource: STONITH-hana-ha-vm-1
       Enabled on: hana-ha-vm-2 (score:INFINITY)
     Resource: STONITH-hana-ha-vm-2
       Enabled on: hana-ha-vm-1 (score:INFINITY)
    Ordering Constraints:
     start SAPHanaTopology_HA1_22-clone then start SAPHana_HA1_22-master (kind:Mandatory) (non-symmetrical)
    Colocation Constraints:
     rsc_vip_HA1_22 with SAPHana_HA1_22-master (score:2000) (rsc-role:Started) (with-rsc-role:Master)
    Ticket Constraints:

Install listeners and create a health check resource

To configure a health check resource, you need to install the listeners first.

Install a listener

The load balancer uses a listener on the health-check port of each host to determine where the primary instance of the SAP HANA cluster is running.

  1. On each host as root, install a simple TCP listener. These instructions install and use HAProxy as the listener.

    # yum install haproxy
  2. Open the configuration file haproxy.cfg for editing:

    # vi /etc/haproxy/haproxy.cfg
    1. In the defaults section of the haproxy.cfg, change the mode to tcp.

    2. After the defaults section, create a new section by adding:

      #---------------------------------------------------------------------
      # Health check listener port for SAP HANA HA cluster
      #---------------------------------------------------------------------
      listen healthcheck
        bind *:healthcheck-port-num

      The bind port is the same port that you used when you created the health check.

      When you are done, your updates should look similar to the following example:

      #---------------------------------------------------------------------
      # common defaults that all the 'listen' and 'backend' sections will
      # use if not designated in their block
      #---------------------------------------------------------------------
      defaults
        mode                    tcp
        log                     global
        option                  httplog
        option                  dontlognull
        option http-server-close
        option forwardfor       except 127.0.0.0/8
        option                  redispatch
        retries                 3
        timeout http-request    10s
        timeout queue           1m
        timeout connect         10s
        timeout client          1m
        timeout server          1m
        timeout http-keep-alive 10s
        timeout check           10s
        maxconn                 3000
      
      #---------------------------------------------------------------------
      # Set up health check listener for SAP HANA HA cluster
      #---------------------------------------------------------------------
      listen healthcheck
       bind *:60000
  3. On each host as root, start the service to confirm it is correctly configured:

    # systemctl start haproxy.service
  4. On the Load balancer page in the Cloud Console, click your load balancer entry:

    Load balancing page

    In the Backend section on the Load balancer details page, if the HAProxy service is active on both hosts, you see 1/1 in the Healthy column of each instance group entry.

    Screen capture shows `1/1` in the Healthy column of both instance groups,
indicating that they are both healthy.

  5. On each host, stop the HAProxy service:

    # systemctl stop haproxy.service

    After you stop the HAProxy service on each host, 0/1 displays in the Healthy column of each instance group.

    Screen capture shows `0/1` in the Healthy column of each
instance group, indicating that there is no active listener.

    Later, when the health check is configured, the cluster restarts the listener on the master node.

Create the healthcheck resource

  1. On either host as root, create a healthcheck resource for the HAProxy service:

    # pcs resource create healthcheck_resource_name service:haproxy
  2. Confirm that the health check service is active on the same host as your master SAP HANA instance and VIP resource:

    # pcs status

    If the health check resource is not on the primary host, move it with the following command:

    # pcs resource move healthcheck_resource_name target_host_name
    # pcs resource clear healthcheck_resource_name

    The command pcs resource clear leaves the resource at its new location but removes the unwanted location constraint that the pcs resource move command created.

    In the status, the resources section should look similar to the following example:

    Full list of resources:
    
    STONITH-hana-ha-vm-1   (stonith:fence_gce):    Started hana-ha-vm-2
    STONITH-hana-ha-vm-2   (stonith:fence_gce):    Started hana-ha-vm-1
    Clone Set: SAPHanaTopology_HA1_22-clone [SAPHanaTopology_HA1_22]
        Started: [ hana-ha-vm-1 hana-ha-vm-2 ]
    Master/Slave Set: SAPHana_HA1_22-master [SAPHana_HA1_22]
        Masters: [ hana-ha-vm-1 ]
        Slaves: [ hana-ha-vm-2 ]
    rsc_vip_HA1_22 (ocf::heartbeat:IPaddr2):       Started hana-ha-vm-1
    rsc_healthcheck_HA1    (service:haproxy):      Started hana-ha-vm-2
  3. Group the VIP and health check resources together:

    # pcs resource group add rsc-group-name healthcheck_resource_name vip_resource_name

    In the cluster status, the resources section should look similar to the following example:

    Full list of resources:
    
    STONITH-hana-ha-vm-1   (stonith:fence_gce):    Started hana-ha-vm-2
    STONITH-hana-ha-vm-2   (stonith:fence_gce):    Started hana-ha-vm-1
    Clone Set: SAPHanaTopology_HA1_22-clone [SAPHanaTopology_HA1_22]
        Started: [ hana-ha-vm-1 hana-ha-vm-2 ]
    Master/Slave Set: SAPHana_HA1_22-master [SAPHana_HA1_22]
        Masters: [ hana-ha-vm-1 ]
        Slaves: [ hana-ha-vm-2 ]
    Resource Group: g-primary
        rsc_healthcheck_HA1        (service:haproxy):      Started hana-ha-vm-1
        rsc_vip_HA1_22     (ocf::heartbeat:IPaddr2):       Started hana-ha-vm-1
  4. Create a constraint that locates the new group on the same node as the master SAP HANA instance.

    # pcs constraint colocation add rsc-group-name with master sap_hana_resource_name-master 2000

    Your final constraints should look similar to the following example:

    # pcs constraint
    Location Constraints:
     Resource: STONITH-hana-ha-vm-1
       Enabled on: hana-ha-vm-2 (score:INFINITY)
     Resource: STONITH-hana-ha-vm-2
       Enabled on: hana-ha-vm-1 (score:INFINITY)
    Ordering Constraints:
     start SAPHanaTopology_HA1_22-clone then start SAPHana_HA1_22-master (kind:Mandatory) (non-symmetrical)
    Colocation Constraints:
     rsc_vip_HA1_22 with SAPHana_HA1_22-master (score:2000) (rsc-role:Started) (with-rsc-role:Master)
     g-primary with SAPHana_HA1_22-master (score:2000) (rsc-role:Started) (with-rsc-role:Master)
    Ticket Constraints:

Test failover

Test your cluster by simulating a failure on the primary host. Use a test system or run the test on your production system before you release the system for use.

Backup the system before the test.

You can simulate a failure in a variety of ways, including:

  • HDB stop
  • HDB kill
  • shutdown -r (on the master node)
  • ip link set eth0 down
  • echo c > /proc/sysrq-trigger

These instructions use ip link set eth0 down to take the network interface offline, because it validates both failover as well as fencing.

  1. On the master host, as root, take the network interface offline:

    # ip link set eth0 down
  2. Follow the progress of the failover in Logging:

    Go to Logging

    The following example shows the log entries for a successful failover:

    Screenshot of the Logging logs for a failover

  3. Reconnect to either host using SSH and change to the root user.

  4. Enter pcs status to confirm that the primary host is now active on the VM that used to contain the secondary host. Automatic restart is enabled in the cluster, so the stopped host will restart and assume the role of secondary host, as shown in the following example.

    Cluster name: hana-ha-cluster
    Stack: corosync
    Current DC: hana-ha-vm-2 (version 1.1.19-8.el7_6.5-c3c624ea3d) - partition with quorum
    Last updated: Wed Jun 17 01:04:36 2020
    Last change: Wed Jun 17 01:03:58 2020 by root via crm_attribute on hana-ha-vm-2
    
    2 nodes configured
    8 resources configured
    
    Online: [ hana-ha-vm-1 hana-ha-vm-2 ]
    
    Full list of resources:
    
    STONITH-hana-ha-vm-1   (stonith:fence_gce):    Started hana-ha-vm-2
    STONITH-hana-ha-vm-2   (stonith:fence_gce):    Started hana-ha-vm-1
    Clone Set: SAPHanaTopology_HA1_22-clone [SAPHanaTopology_HA1_22]
        Started: [ hana-ha-vm-1 hana-ha-vm-2 ]
    Master/Slave Set: SAPHana_HA1_22-master [SAPHana_HA1_22]
        Masters: [ hana-ha-vm-2 ]
        Slaves: [ hana-ha-vm-1 ]
    Resource Group: g-primary
        rsc_healthcheck_HA1        (service:haproxy):      Started hana-ha-vm-2
        rsc_vip_HA1_22     (ocf::heartbeat:IPaddr2):       Started hana-ha-vm-2
    
    Daemon Status:
     corosync: active/enabled
     pacemaker: active/enabled
     pcsd: active/enabled

Troubleshooting

You can find the logs in the following directories:

  • /var/log/pacemaker.log
  • /var/log/cluster/corosync.log

Valid cluster status:

# pcs status
Cluster name: hana-ha-cluster
Stack: corosync
Current DC: hana-ha-vm-1 (version 1.1.19-8.el7_6.5-c3c624ea3d) - partition with quorum
Last updated: Wed Jun 17 00:34:16 2020
Last change: Wed Jun 17 00:34:09 2020 by root via crm_attribute on hana-ha-vm-1

2 nodes configured
8 resources configured

Online: [ hana-ha-vm-1 hana-ha-vm-2 ]

Full list of resources:

 STONITH-hana-ha-vm-1   (stonith:fence_gce):    Started hana-ha-vm-2
 STONITH-hana-ha-vm-2   (stonith:fence_gce):    Started hana-ha-vm-1
 Clone Set: SAPHanaTopology_HA1_22-clone [SAPHanaTopology_HA1_22]
     Started: [ hana-ha-vm-1 hana-ha-vm-2 ]
 Master/Slave Set: SAPHana_HA1_22-master [SAPHana_HA1_22]
     Masters: [ hana-ha-vm-1 ]
     Slaves: [ hana-ha-vm-2 ]
 Resource Group: g-primary
     rsc_healthcheck_HA1        (service:haproxy):      Started hana-ha-vm-1
     rsc_vip_HA1_22     (ocf::heartbeat:IPaddr2):       Started hana-ha-vm-1

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Completing the NAT gateway installation

If you created a NAT gateway, complete the following steps.

  1. Add tags to all instances:

    export NETWORK_NAME="[YOUR_NETWORK_NAME]"
    export TAG="[YOUR_TAG_TEXT]"
    gcloud compute instances add-tags "[PRIMARY_VM_NAME]" --tags="$TAG" --zone=[PRIMARY_VM_ZONE]
    gcloud compute instances add-tags "[SECONDARY_VM_NAME]" --tags="$TAG" --zone=[SECONDARY_VM_ZONE]
  2. Delete external IPs:

    gcloud compute instances delete-access-config "[PRIMARY_VM_NAME]" --access-config-name "external-nat" --zone=[PRIMARY_VM_ZONE]
    gcloud compute instances delete-access-config "[SECONDARY_VM_NAME]" --access-config-name "external-nat" --zone=[SECONDARY_VM_ZONE]

Connecting to SAP HANA

If the host VMs don't have an external IP address for SAP HANA, you can only connect to the SAP HANA instances through the bastion instance using SSH or through the Windows server through SAP HANA Studio.

  • To connect to SAP HANA through the bastion instance, connect to the bastion host, and then to the SAP HANA instance(s) by using an SSH client of your choice.

  • To connect to the SAP HANA database through SAP HANA Studio, use a remote desktop client to connect to the Windows Server instance. After connection, manually install SAP HANA Studio and access your SAP HANA database.

Performing post-deployment tasks

Before using your SAP HANA instance, we recommend that you configure and backup your new SAP HANA database.

For more information:

What's next

See the following resource for more information: