Internal Load Balancing using HAProxy on Google Compute Engine

This solution walks through the setup of an internal load balancer that uses HAProxy on a dedicated Compute Engine instance to distribute HTTP requests across a set of three Compute Engine instances running the Apache web server.

A load balancer accepts incoming network traffic and distributes it among a group of application servers. For example, the Google Compute Engine Load Balancing service distributes incoming traffic from the Internet to a group of Compute Engine instances. The following diagram shows how Compute Engine Network Load Balancing works.

Diagram describing Compute Engine Network Load Balancing

An internal load balancer distributes network traffic on a private network that is not exposed to the Internet. Internal load balancing is useful not only for intranet applications where all traffic remains on a private network, but also for complex web applications where a frontend sends requests to backend servers by using a private network.

You can use Compute Engine to create your own internal load balancer by using a software load balancer. One way to implement such a load balancer is by using HAProxy, an open source software load balancer, on a Compute Engine instance. The following diagram offers a high-level description of how to use HAProxy for internal load balancing on Google Cloud Platform.

High-level diagram of HAProxy as an internal load balancer

The solid green arrows in the diagram represent traffic on a private network. The HAProxy server acts as an internal load balancer because it is not exposed to the Internet. The HAProxy server receives requests on the private network and distributes them across a group of backend servers that are accessible only through internal IP addresses.

This solution creates a total of five Compute Engine instances: one for HAProxy, three backend Apache servers, and one micro instance for testing. The following diagram shows how the five instances interact.

Detailed diagram of HAProxy as an internal load balancer

Reliability considerations

Running HAProxy on its own instance does introduce a single point of failure so we'll use the Compute Engine Instance Group Manager API, currently in beta release, to keep the load balancer running. The Instance Group Manager monitors the status of the HAProxy's Compute Engine instance and recreates the instance if it stops running.

The fact that the HAProxy server's underlying instance is running, however, does not guarantee that the HAProxy load balancer service is running. To ensure that the HAProxy service restarts in the event that it fails, we'll use systemd, a system management utility that is part of the Debian 8 default installation.

Objectives

  • Set up three backend Apache servers
  • Set up a managed instance group of one Compute Engine instance running HAProxy
  • Set up a Compute Engine micro instance with an external IP address for testing
  • Configure HAProxy to load balance traffic to the backend Apache servers

Prerequisites

  1. Create a new Google Cloud Platform Console project to make cleanup easier.

  2. Use the Google Cloud Platform Console to enable the Compute Engine and Instance Group Manager APIs.

  3. Install the Google Cloud SDK.

  4. Configure your workspace to make commands less verbose. Substitute your project's values for project-id and my-zone in the following commands.

    me@local:~$ gcloud config set project project-id
    me@local:~$ gcloud config set compute/zone my-zone
    

Create a group of three backend Apache servers

You can use an internal load balancer to distribute traffic to a wide variety of computing resources, such as database servers, key-value caches, and web servers. In this solution, we use a group of three web servers running the Apache web server. In a subsequent solution, we'll demonstrate how to use an internal load balancer with an autoscaled group of backend servers.

Create a custom image that contains Apache Server

Follow these steps to create a custom image for the backend servers.

  1. Create a Compute Engine instance to use as the base for the custom image.

    me@local:~$ gcloud compute instances create apache-base --image debian-8
    
  2. Sign into the instance and install Apache2.

    me@local:~$ gcloud compute ssh apache-base
    apache-base:~$ sudo apt-get update && sudo apt-get install -y apache2
    apache-base:~$ exit
    
  3. Terminate the instance but keep the boot disk.

    me@local:~$ gcloud compute instances delete apache-base --keep-disks boot
    
  4. Create the custom image using the source disk that you just created.

    me@local:~$ gcloud compute images create apache-base-image --source-disk apache-base
    

Create three backend Apache servers based on the custom image

Run the following command to create three backend servers that are all running Apache server.

me@local:~$ gcloud compute instances create apache1 apache2 apache3 --image \
apache-base-image --no-address --metadata startup-script='#! /bin/bash
HOSTNAME=$(hostname)
sudo cat << EOF | sudo tee /var/www/html/index.html
This is $HOSTNAME
EOF'

The --no-address option generates instances that do not have external IP addresses. The --metadata option includes a startup script that edits the home page of each Apache server so that it prints the name of the instance. This makes it easier to check whether the load balancer is distributing requests to all three backend servers.

Compute Engine returns the names of the servers and their internal IP addresses. Note the internal IP addresses for each of the three backend servers. You'll need them for the HAProxy configuration file.

Create a micro instance for testing

You cannot use your development machine to test the load balancer and backend servers because they do not have external IP addresses. Follow these steps to create a micro instance that has an external IP address so that you can test the backend servers and the load balancer.

  1. Create the test instance with --machine-type set to f1-micro.

    me@local:~$ gcloud compute instances create test-instance --machine-type f1-micro
    
  2. Establish an SSH connection to the test-instance instance:

    gcloud compute ssh test-instance

  3. Use curl to check the backend web servers.

    me@test-instance:~$ curl apache1
    me@test-instance:~$ curl apache2
    me@test-instance:~$ curl apache3
    me@test-instance:~$ exit
    

Create an internal load balancer running HAProxy

In this solution, the HAProxy service runs on a dedicated Compute Engine instance. To reduce the risk of having a single point of failure, we can take measures to protect the health of both the Compute Engine instance and the HAProxy service itself.

To ensure that the underlying Compute Engine instance keeps running, we'll use a managed instance group. If the HAProxy instance is in a managed instance group, Instance Group Manager checks the status of the server and recreates the instance if it stops running. Instance Group Manager recreates the image even if you use the gcloud compute instances command—or the Compute Engine API—to stop or terminate the instance. However, if you use the gcloud compute instance-groups managed delete-instances commmand—or the Instance Group Manager API—to delete the instance, the instance remains deleted.

You can create a managed instance group in three steps. First, create a custom image. Second, use the custom image to create an instance template. Third, use the instance template to create a managed instance group.

To ensure that the HAProxy service keeps running, we'll make sure that the unit configuration file for the HAProxy service is set to always restart. If properly configured, systemd immediately restarts a service if the process is killed, exits, or times out.

Follow these steps to create the HAProxy custom image, instance template, and managed instance group.

Create the base image for the HAProxy instance

  1. Use the debian-8 image to create a Compute Engine instance.

    me@local:~$ gcloud compute instances create haproxy-base --image debian-8
    
  2. Sign into the instance and install HAProxy.

    me@local:~$ gcloud compute ssh haproxy-base
    haproxy-base:~$ sudo apt-get update && sudo apt-get -y install haproxy
    
  3. Configure HAProxy by appending the list of backend servers to the configuration file. Replace internal-ip with the matching internal IP addresses that you noted in a previous step.

    haproxy-base:~$ echo -e "\n\n# Listen for incoming traffic
    listen apache-lb *:80
        mode http
        balance roundrobin
        option httpclose
        option forwardfor
        server apache1 internal-ip:80 check
        server apache2 internal-ip:80 check
        server apache3 internal-ip:80 check" | sudo tee -a /etc/haproxy/haproxy.cfg
    

If you forgot to substitute your actual IP addresses for the three occurrences of internal-ip before running the command, use a text editor to replace them before you proceed.

Confirm that the HAProxy service is set to always restart.

  1. Use a text editor to view a file named haproxy-check.sh.

    haproxy-base:~$ sudo nano /lib/systemd/system/haproxy.service
    
  2. Confirm that Restart=always is listed under the [Service] section. Add Restart=always if it is not listed.

    [Service]
    Environment=CONFIG=/etc/haproxy/haproxy.cfg
    ...
    Restart=always
    
  3. Exit the file, saving any changes, and then exit the shell.

    haproxy-base:~$ exit
    

Create the custom image

Use the same procedure you used to create the Apache server custom image to create a custom image for the HAProxy server.

  1. Terminate the instance but keep the boot disk.

    me@local:~$ gcloud compute instances delete haproxy-base --keep-disks boot
    
  2. Create the custom image using the source disk that you just created.

    me@local:~$ gcloud compute images create haproxy-base-image --source-disk haproxy-base
    

Create an HAProxy instance template based on the custom image

Use the --no-address option to prevent the assignment of an external IP address.

 me@local:~$ gcloud compute instance-templates create haproxy-template \
 --image haproxy-base-image --no-address

Create a managed instance group named haproxy

The --base-instance-name option allows you to set the prefix for the instance name that Instance Group Manager assigns to the instance it creates. For this example, use haproxy-server as the prefix. The instance name will be haproxy-server-xxxx where xxxx is a series of letters or numbers.

me@local:~$ gcloud compute instance-groups managed create haproxy \
--base-instance-name haproxy-server --size 1 --template haproxy-template

Get the name of the HAProxy server and test the load balancer

After you create the managed instance group, Instance Group Manager creates your instance for you. We'll use the gcloud compute instances command to find the name of the new instance.

  1. Get the name of the HAProxy instance and SSH into the instance.

    me@local:~$ gcloud compute instances list | grep -oE "haproxy-server-[a-z0-9]{4}"
    me@local:~$ gcloud compute ssh test-instance
    
  2. Use curl to test the proxy server. Replace haproxy-server-xxxx with the name of your proxy server. As you repeat this command, the result should return values from each of the three web servers.

    me@test-instance:~$ curl haproxy-server-xxxx
    
  3. Exit the test-instance shell.

    me@test-instance:~$ exit
    

Test the resiliency of the load balancer

We can introduce two failure scenarios to test the resiliency of the load balancer. First, we can stop the HAProxy's Compute Engine instance to test whether the Instance Group Manager recreates the instance and whether the startup script properly restarts the HAProxy service. Second, we can sign in to the HAProxy server and manually stop the HAProxy service to test whether systemd detects the failure and restarts the service.

Test the HAProxy instance

Follow these steps to test the resiliency of the HAProxy managed instance group.

  1. Use the gcloud compute instances command to stop the HAProxy instance. Remember to replace haproxy-server-xxxx with the name of your proxy server.

    me@local:~$ gcloud compute instances stop haproxy-server-xxxx
    
  2. Wait a few minutes and then check to see if the instance is running. Instance Group Manager should recreate the instance.

    me@local:~$ gcloud compute instances list
    
  3. After the instance is running again, you can sign on to the test-instance to verify that the load balancer is functioning. Sign out of the test-instance when you are done testing.

Test the HAProxy service

Testing the HAProxy service requires a few extra steps because you cannot connect directly to the HAProxy instance from your development machine. You can, however, connect to the test-instance and then use instance to instance ssh to access the haproxy-server-xxxx instance.

  1. Set up instance to instance ssh.

    me@local:~$ eval `ssh-agent`
    me@local:~$ ssh-add ~/.ssh/google\_compute\_engine
    
  2. Connect to the test-instance using the ssh-flag option.

    me@local:~$ gcloud compute ssh test-instance --ssh-flag="-A"
    
  3. Use ssh on the test-instance to connect to the haproxy-server-xxxx instance (replace haproxy-server-xxxx with the name of your haproxy-server instance).

    me@test-instance:~$ ssh haproxy-server-xxxx
    
  4. Check the status of the haproxy server. The command output should include the word running, a timestamp of the most recent service startup, and how long ago that was.

    me@haproxy-server-xxxx:~$ sudo service haproxy status | grep Active
    
  5. Kill the HAProxy server process to test whether the HAProxy service restarts.

    me@haproxy-server-xxxx:~$ sudo pkill haproxy
    
  6. Check the status again to see whether the HAProxy service restarted. The output should again report that the service is running, but the timestamp of the last service startup should be more recent.

    me@haproxy-server-xxxx:~$ sudo service haproxy status | grep Active
    
  7. Exit the shells for both instances.

    me@haproxy-server-xxxx:~$ exit
    me@test-instance:~$ exit
    

Cleaning up

After you've finished the HAProxy tutorial, you can clean up the resources you created on Google Cloud Platform so you won't be billed for them in the future. The following sections describe how to delete or turn off these resources.

Deleting the project

The easiest way to eliminate billing is to delete the project you created for the tutorial.

To delete the project:

  1. In the Cloud Platform Console, go to the Projects page.

    Go to the Projects page

  2. In the project list, select the project you want to delete and click Delete project. After selecting the checkbox next to the project name, click
      Delete project
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Deleting instances

To delete a Compute Engine instance:

  1. In the Cloud Platform Console, go to the VM Instances page.

    Go to the VM Instances page

  2. Click the checkbox next to your test-instance instance.
  3. Click the Delete button at the top of the page to delete the instance.

Deleting disks

To delete a Compute Engine disk:

  1. In the Cloud Platform Console, go to the Disks page.

    Go to the Disks page

  2. Click the checkbox next to the disk you want to delete.
  3. Click the Delete button at the top of the page to delete the disk.

Next Steps

You've now seen how to create an internal load balancing solution by using HAProxy on a Compute Engine instance that runs on a private network. You've also seen how to make the load balancer more resilient by using the Instance Group Manager API and systemd to monitor and repair the health of the load balancer.

You can extend or modify this solution to support different backend applications such as database servers or key-value stores. You can also make the backend server pool scalable by using the Compute Engine Autoscaler feature.

For information about how to load test your internal load balancer, see Distributed Load Testing Using Kubernetes.

Read about other load balancing solutions available on Google Cloud Platform:

Send feedback about...