Building High-throughput VPNs

This tutorial shows how to create secure, high-throughput VPNs and test their speed.

Secure communication between Google Cloud and other clouds or on-premises systems is a common, critical need. Fortunately, Google Cloud makes it easy for you to create secure Internet Protocol security (IPsec) virtual private networks (VPNs) to achieve this goal. If a single tunnel does not provide necessary throughput, Google Cloud can smoothly distribute traffic across multiple tunnels to provide additional bandwidth.


Create VPNs

  • Create and configure a Google Cloud project suitable for VPNs.
  • Create a Virtual Private Cloud (VPC) named cloud to simulate your Google Cloud network, and a VPC named on-prem (on-premises) to simulate an external network.
  • Create VPN gateways, forwarding rules, and addresses for the cloud VPC.
  • Form a tunnel for the new VPN, and route traffic through it.
  • Repeat the VPN creation process for the on-prem VPC, creating a second VPN.

Test VPNs

  • Create a virtual machine (VM) using Compute Engine for throughput load testing.
  • Test throughput speed of a single VPN using iperf.
  • Route traffic through both VPNs simultaneously, and test increased throughput speed using iperf.


This tutorial uses billable components of Google Cloud Platform, including:

The estimated price to run this tutorial, assuming you use every resource for an entire day, is approximately $10.01 based on this pricing calculator.

Before you begin

  1. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.

    Go to the project selector page

  2. Make sure that billing is enabled for your Cloud project. Learn how to confirm that billing is enabled for your project.

  3. Enable the Google Compute Engine API.

    Enable the API

  4. Review and familiarize yourself with how to create a VPN using Google Cloud; also review the VPC Network Overview.

Creating the cloud VPC

In this section, you:

  • Create a VPC to simulate your cloud production network.
  • Allow common types of traffic to flow through the VPC.
  • Create a subnet for deploying hosts.

  1. Go to Cloud Shell.


  2. In Cloud Shell, create a custom VPC named cloud associated with your Google Cloud project. This VPC allows you to use non-default IP addressing, but does not include any default firewall rules:

    gcloud compute networks create cloud --subnet-mode custom
  3. Enable SSH and icmp, because you'll need a secure shell to communicate with VMs during load testing:

    gcloud compute firewall-rules create cloud-fw --network cloud --allow tcp:22,icmp
  4. Create a subnet within this VPC and specify a region and IP range. In this solution, you'll be using and the us-east1 region:

    gcloud compute networks subnets create cloud-east --network cloud \
        --range --region us-east1

Creating the on-prem VPC

In this section, you create a simulation of your on-prem VPC, or any network you want to connect to cloud. In practice, you'll already have resources here, but for the purpose of creating tunnels and validating configurations, follow these steps:

  1. Using the gcloud command-line tool, create a new custom subnet VPC associated with your project. Name the new VPC on-prem:

    gcloud compute networks create on-prem --subnet-mode custom
  2. Enable SSH and icmp for hosts in the on-prem VPC, because you'll need a secure shell to communicate with VMs during load testing:

    gcloud compute firewall-rules create on-prem-fw --network on-prem --allow tcp:22,icmp
  3. Specify the subnet prefix for the region. In this example, you assign to the us-central1 region:

    gcloud compute networks subnets create on-prem-central \
        --network on-prem --range --region us-central1

Creating VPN gateways

Each environment requires VPN gateways for secure external communication. Follow these steps to create the initial gateways for your cloud and on-prem VPCs:

  1. Create a VPN gateway named on-prem-gw1 in the on-prem VPC and us-central1 region:

    gcloud compute target-vpn-gateways create on-prem-gw1 --network on-prem --region us-central1
  2. Create a VPN gateway named cloud-gw1 in the cloud VPC and us-east1 region:

    gcloud compute target-vpn-gateways create cloud-gw1 --network cloud --region us-east1

Creating a route-based VPN tunnel between local and Google Cloud networks

Now that you've created the VPN gateways, they each need a static, external IP address so that systems outside the VPC can communicate with them. The following steps show to create IP addresses and routes on the cloud and on-prem VPCs:

  1. Allocate the IP for the cloud-gw1 VPN gateway:

    gcloud compute addresses create cloud-gw1 --region us-east1
  2. Allocate the IP for the on-prem-gw1 VPN gateway:

    gcloud compute addresses create on-prem-gw1 --region us-central1
  3. Store the addresses created in the preceding steps so you won't have to look them up in later commands.

    1. First, for the cloud-gw1 gateway:

      cloud_gw1_ip=$(gcloud compute addresses describe cloud-gw1 \
          --region us-east1 --format='value(address)')
    2. Second, for the on-prem-gw1 gateway:

      on_prem_gw_ip=$(gcloud compute addresses describe on-prem-gw1 \
          --region us-central1 --format='value(address)')
  4. Create forwarding rules for IPsec on the cloud VPC. You'll need to create forwarding rules in both directions.

    1. Forward the Encapsulating Security Payload (ESP) protocol from cloud-gw1:

      gcloud compute forwarding-rules create cloud-1-fr-esp --ip-protocol ESP \
          --address $cloud_gw1_ip --target-vpn-gateway cloud-gw1 --region us-east1
    2. Forward UDP:500 traffic from cloud-gw1:

      gcloud compute forwarding-rules create cloud-1-fr-udp500 --ip-protocol UDP \
          --ports 500 --address $cloud_gw1_ip --target-vpn-gateway cloud-gw1 --region us-east1
    3. Forward UDP:4500 traffic from cloud-gw1:

      gcloud compute forwarding-rules create cloud-fr-1-udp4500 --ip-protocol UDP \
          --ports 4500 --address $cloud_gw1_ip --target-vpn-gateway cloud-gw1 --region us-east1
  5. In the same manner, create forwarding rules for the IPsec tunnel on the on-prem VPC. This step allows the IPsec tunnel to exit your firewalls, and as before must be done for the following protocols:

    1. Forward the ESP protocol from on-prem-gw1:

      gcloud compute forwarding-rules create on-prem-fr-esp --ip-protocol ESP \
          --address $on_prem_gw_ip --target-vpn-gateway on-prem-gw1 --region us-central1
    2. Forward UDP:500 traffic, used in establishing the IPsec tunnel from on-prem-gw1:

      gcloud compute forwarding-rules create on-prem-fr-udp500 --ip-protocol UDP --ports 500 \
          --address $on_prem_gw_ip --target-vpn-gateway on-prem-gw1 --region us-central1
    3. Forward UDP:4500 traffic, which carries the encrypted traffic from on-prem-gw1:

      gcloud compute forwarding-rules create on-prem-fr-udp4500 --ip-protocol UDP --ports 4500 \
          --address $on_prem_gw_ip --target-vpn-gateway on-prem-gw1 --region us-central1
  6. Generate a secret, which you use in the next step, where you create and validate the tunnels on-prem-tunnel1 and cloud-tunnel1. For details about how to create and securely store secrets, see secret management.

  7. Using the secret you just generated, create a tunnel for the local network, on-prem-tunnel1, and for the cloud-based network, cloud-tunnel1. Each network must have a VPN gateway, and the secrets must match. In the following two commands, replace [MY_SECRET] with the secret you generated.

    1. Create the VPN tunnel from on-prem to cloud:

      gcloud compute vpn-tunnels create on-prem-tunnel1 --peer-address $cloud_gw1_ip \
          --target-vpn-gateway on-prem-gw1 --ike-version 2 --local-traffic-selector \
          --remote-traffic-selector --shared-secret=[MY_SECRET] --region us-central1
    2. Create the VPN tunnel from cloud to on-prem:

      gcloud compute vpn-tunnels create cloud-tunnel1 --peer-address $on_prem_gw_ip \
          --target-vpn-gateway cloud-gw1 --ike-version 2 --local-traffic-selector \
          --remote-traffic-selector --shared-secret=[MY_SECRET] --region us-east1
  8. Now that you've created the gateways and built the tunnels, you need to add routes from the previously created subnets through the two tunnels.

    1. Route traffic from the on-prem VPC to the cloud range into the tunnel:

      gcloud compute routes create on-prem-route1 --destination-range \
          --network on-prem --next-hop-vpn-tunnel on-prem-tunnel1 \
          --next-hop-vpn-tunnel-region us-central1
    2. Route traffic from the cloud VPC to the on-prem range into the tunnel:

      gcloud compute routes create cloud-route1 --destination-range \
          --network cloud --next-hop-vpn-tunnel cloud-tunnel1 --next-hop-vpn-tunnel-region us-east1

Testing throughput over VPN

At this point, you've established a secure path between the on-prem and cloud VPCs. To test throughput, you will use iperf, an open-source tool for network load testing. You'll need a VM in each environment, one to send traffic and the other to receive it.

Single VPN load testing

  1. Create a virtual machine for the cloud VPC. Here, the name is cloud-loadtest, and this example uses a Debian Linux image for the OS. If you have an existing project, feel free to omit this step and use existing resources. Note that bandwidth for a VM is 2 Gbps * vCPUs, so you'll want a 4 vCPU minimum.

    gcloud compute instances create "cloud-loadtest" --zone "us-east1-b" \
        --machine-type "n1-standard-4" --subnet "cloud-east" \
        --image-family "debian-9" --image-project "debian-cloud" --boot-disk-size "10" \
        --boot-disk-type "pd-standard" --boot-disk-device-name "cloud-loadtest"
  2. Create a virtual machine for the on-prem VPC. This example uses the same Debian image as in the cloud VPC. Omit this step if you have existing resources. Here, the name is on-prem-loadtest.

    gcloud compute instances create "on-prem-loadtest" --zone "us-central1-a" \
        --machine-type "n1-standard-4" --subnet "on-prem-central" \
        --image-family "debian-9" --image-project "debian-cloud" --boot-disk-size "10" \
        --boot-disk-type "pd-standard" --boot-disk-device-name "on-prem-loadtest"
  3. Open TCP port 5001 on the on-prem network's firewall to accept iperf requests:

    gcloud compute firewall-rules create on-prem-iperf-fw --network on-prem --allow tcp:5001
  4. Using SSH, connect to each VM and install a copy of iperf. To install iperf on Debian, use the following command line:

    sudo apt-get install iperf
  5. On the on-prem-loadtest VM, run this command:

    iperf -s -i 5 -p 5001

    You have created an iperf server, on the VM, that reports its status every 5 seconds.

  6. On the cloud-loadtest VM, run this command:

    iperf -c -P 20 -x C -p 5001

You have created an iperf client with twenty streams, which will report values after 10 seconds of testing:

iperf testing

Note the [SUM] throughput value; if the result is sufficient for your traffic, you can stop here. However, if you need additional throughput, you'll want to create and use additional tunnels.

Multiple VPN load testing

If the routes match, Google Cloud VPN gateways perform per-flow equal-cost multi-path routing (ECMP) to load balance across multiple tunnels and deliver higher aggregate throughput. Since load balancing is per-flow, a single flow gains no benefit, but multiple flows do.

  1. Create an additional cloud VPN gateway,cloud-gw2, in the same region as cloud-gw1:

    gcloud compute target-vpn-gateways create cloud-gw2 --network cloud --region us-east1
  2. Create the address for cloud-gw2:

    gcloud compute addresses create cloud-gw2 --region us-east1
  3. Store the address for cloud-gw2:

    cloud_gw2_ip=$(gcloud compute addresses describe cloud-gw2 \
        --region us-east1 --format='value(address)')
  4. Create forwarding rules on cloud-gw2:

    gcloud compute forwarding-rules create cloud-2-fr-esp --ip-protocol ESP \
        --address $cloud_gw2_ip --target-vpn-gateway cloud-gw2 --region us-east1
    gcloud compute forwarding-rules create cloud-2-fr-udp500 --ip-protocol UDP --ports 500 \
        --address $cloud_gw2_ip --target-vpn-gateway cloud-gw2 --region us-east1
    gcloud compute forwarding-rules create cloud-fr-2-udp4500 --ip-protocol UDP --ports 4500 \
        --address $cloud_gw2_ip --target-vpn-gateway cloud-gw2 --region us-east1
  5. Tunnel from on-prem to cloud-gw2, replacing [MY_SECRET] with the secret you generated:

    gcloud compute vpn-tunnels create on-prem-tunnel2 --peer-address $cloud_gw2_ip \
        --target-vpn-gateway on-prem-gw1 --ike-version 2 --local-traffic-selector \
        --remote-traffic-selector --shared-secret=[MY_SECRET] --region us-central1
  6. Tunnel from cloud-gw2 to on-prem, replacing [MY_SECRET] with the secret you generated:

    gcloud compute vpn-tunnels create cloud-tunnel2 --peer-address $on_prem_gw_ip \
        --target-vpn-gateway cloud-gw2 --ike-version 2 --local-traffic-selector \
        --remote-traffic-selector --shared-secret=[MY_SECRET] --region us-east1
  7. Route from on-prem to cloud-gw2:

    gcloud compute routes create on-prem-route2 --destination-range --network on-prem \
        --next-hop-vpn-tunnel on-prem-tunnel2 --next-hop-vpn-tunnel-region us-central1
  8. Route from cloud-gw2 to on-prem:

    gcloud compute routes create cloud-route2 --destination-range \
        --network cloud --next-hop-vpn-tunnel cloud-tunnel2 --next-hop-vpn-tunnel-region us-east1

You have created two tunnels, cloud-tunnel1 and cloud-tunnel2, with identical routes. Traffic will be automatically load balanced between them.

Retesting throughput over VPN

  • Use iperf to retest the speed of the network:

    iperf -c -P 20 -x C

Note how the throughput value is larger than in the single-tunnel test:

iperf retesting

Also note that the limit for equal-cost multi-path routing VPN tunnels isn't 2, nor do the tunnels need to terminate in the same regions.

For additional scale, you can continue this process indefinitely. Also, to improve reliability, you can source or terminate these tunnels in different regional subnets.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Deleting the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

  1. In the Cloud Console, go to the Manage resources page.

    Go to Manage resources

  2. In the project list, select the project that you want to delete, and then click Delete.
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

  • Review Cloud Router to enable Border Gateway Protocol (BGP) and increase fault tolerance.
  • Review Cloud Interconnect for other interconnection options.
  • Monitor the VPN gateways with Cloud Monitoring.
  • Try out other Google Cloud features for yourself. Have a look at our tutorials.